summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2025-01-31[feature] Add `instance-stats-randomize` config option (#3718)Libravatar tobi1
* [feature] Add `instance-stats-randomize` config option * don't use cache (overkill)
2025-01-24[feature] Process incoming Undo Announce properly (#3676)Libravatar tobi2
* [feature] Process incoming Undo Announce properly * test undo announce
2025-01-23[feature] Push notifications (#3587)Libravatar Vyr Cossont16
* Update push subscription API model to be Mastodon 4.0 compatible * Add webpush-go dependency # Conflicts: # go.sum * Single-row table for storing instance's VAPID key pair * Generate VAPID key pair during startup * Add VAPID public key to instance info API * Return VAPID public key when registering an app * Store Web Push subscriptions in DB * Add Web Push sender (similar to email sender) * Add no-op push senders to most processor tests * Test Web Push notifications from workers * Delete Web Push subscriptions when account is deleted * Implement push subscription API * Linter fixes * Update Swagger * Fix enum to int migration * Fix GetVAPIDKeyPair * Create web push subscriptions table with indexes * Log Web Push server error messages * Send instance URL as Web Push JWT subject * Accept any 2xx code as a success * Fix malformed VAPID sub claim * Use packed notification flags * Remove unused date columns * Add notification type for update notifications Not used yet * Make GetVAPIDKeyPair idempotent and remove PutVAPIDKeyPair * Post-rebase fixes * go mod tidy * Special-case 400 errors other than 408/429 Most client errors should remove the subscription. * Improve titles, trim body to reasonable length * Disallow cleartext HTTP for Web Push servers * Fix lint * Remove redundant index on unique column Also removes redundant unique and notnull tags on ID column since these are implied by pk * Make realsender.go more readable * Use Tobi's style for wrapping errors * Restore treating all 5xx codes as temporary problems * Always load target account settings * Stub `policy` and `standard` * webpush.Sender: take type converter as ctor param * Move webpush.MockSender and noopSender into testrig
2025-01-23[feature] Serve bot accounts over AP as Service instead of Person (#3672)Libravatar tobi2
* pepis * oopsie doopsie * bollocks
2025-01-08[bugfix] More permissive CSV parsing for perm subs, text parse fix (#3638)Libravatar tobi1
* [bugfix] More permissive CSV parsing for perm subs, text parse fix * wee * change the way dry works, slightly * me oh my, i'm just a little guy * we're just normal men
2025-01-08[feature] Fetch + create domain permissions from subscriptions nightly (#3635)Libravatar tobi21
* peepeepoopoo * test domain perm subs * swagger * envparsing * dries your wets * start on docs * finish up docs * copy paste errors * rename actions package * rename force -> skipCache * move obfuscate parse nearer to where err is checked * make higherPrios a simple slice * don't use receiver for permsFrom funcs * add more context to error logs * defer finished log * use switch for permType instead of if/else * thanks linter, love you <3 * validate csv headers before full read * use bufio scanner
2025-01-08[chore] replace statuses.updated_at column with statuses.edited_at (#3636)Libravatar kim4
* update statuses table to replace updated_at column with edited_at * code comment * better code comments, fix setting of status + edit + mention + poll database times * fix log to logf call * fix status.EditIDs not being carried over in dereferencer.encrichStatus() * move status.EditID setting into handleStatusEdit()
2025-01-05[feature] Create/update/remove domain permission subscriptions (#3623)Libravatar tobi2
* [feature] Create/update/remove domain permission subscriptions * lint * envparsing * remove errant fmt.Println * create drafts, subs, exclude, from snapshot models * name etag column correctly * remove count column * lint
2024-12-23[feature] add support for clients editing statuses and fetching status ↵Libravatar kim13
revision history (#3628) * start adding client support for making status edits and viewing history * modify 'freshest' freshness window to be 5s, add typeutils test for status -> api edits * only populate the status edits when specifically requested * start adding some simple processor status edit tests * add test editing status but adding a poll * test edits appropriately adding poll expiry handlers * finish adding status edit tests * store both new and old revision emojis in status * add code comment * ensure the requester's account is populated before status edits * add code comments for status edit tests * update status edit form swagger comments * remove unused function * fix status source test * add more code comments, move media description check back to media process in status create * fix tests, add necessary form struct tag
2024-12-05[feature] add support for receiving federated status edits (#3597)Libravatar kim8
* add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-04[feature] unending polls (#3592)Libravatar kim1
* adds support for unending polls to be created locally * remove unused argument
2024-12-02[feature] Add global instance CSS customization setting (#3352)Libravatar Victor Dyotte1
Allow instance admins to add custom CSS that will affect every page of their instance. This is done with a new CustomCSS instance setting that works pretty much exactly like the Users CustomCSS property. This custom CSS is then requested for every page load. User styles/themes take precedence over this CSS. Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-11-28send out poll votes as separate create activities given that no other AP ↵Libravatar kim1
servers support multiple objects in a single activity (#3582)
2024-11-28[chore] rename New___(string) int signature functions to Parse___(string) ↵Libravatar kim1
int (#3580) * rename New___(string) int {} signature functions to Parse___(string) int {} * remove test output
2024-11-26[bugfix] Allow unsetting filter expiration dates (#3560)Libravatar Vyr Cossont4
* Regression tests for #3497 (v1 and v2) * use Nullable type for v2 form.expires_in --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-11-25[bugfix] notification types missing from link header (#3571)Libravatar kim2
* ensure notification types get included in link header query for notifications * fix type query keys
2024-11-25[performance] convert enum strings to ints (#3558)Libravatar kim4
* convert statuses.visibility and notifications.notification_type columns from type string -> int for performance / space savings * fix test trying to compare string to int * fix instance count query using string literal instead of gtsmodel const type * ensure a default value is always set * also migrate the account settings and sin bin status tables * initialize maps outside loops and place into singular enum mapping creation func * use int16 for enum types * update sinbinstatus creation to be from a snapshot at initial creation * add snapshot of poll type at creation time
2024-11-21[feature] Add domain permission drafts and excludes (#3547)Libravatar tobi4
* [feature] Add domain permission drafts and excludes * fix typescript complaining * lint * make filenames more consistent * test own domain excluded
2024-11-11[performance] minimise log field allocations (#3529)Libravatar kim2
* when appending log field only do so by minimal amount * move slice utils to separate package to fix import cycle, add GrowJust() and AppendJust() functions * fix GrowJust() not returning slice of same length * improved xslices tests * make AppendJust() test check for slice contents, fix AppendJust() final copying behaviour * add a +1 with field growth to try minimise allocation for log 'msg' field
2024-10-21[feature] Add image descriptions for default avatar + header; don't allow ↵v0.17.1Libravatar tobi2
editing default desc (#3473)
2024-10-16[chore] Upgrade golangci-lint, ignore existing int overflow warnings (#3420)Libravatar Markus Unterwaditzer3
* [chore] Bump tooling versions, bump go -> v1.23.0 * undo silly change * sign * bump go version in go.mod * allow overflow in imaging * goreleaser deprecation notices * [chore] Upgrade golangci-lint, ignore existing int overflow warnings There is a new lint for unchecked int casts. Integer overflows are bad, but the old code that triggers this lint seems to be perfectly fine. Instead of disabling the lint entirely for new code as well, grandfather in existing code. * fix golangci-lint documentation link * revert unrelated changes * revert another unrelated change * get rid of remaining nolint:gosec * swagger updates * apply review feedback * fix wrong formatting specifier thing * fix the linter for real --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-10-12[bugfix] Account.last_status_at is a date, not datetime (#3419)Libravatar Markus Unterwaditzer2
* [bugfix] Account.last_status_at is a date, not datetime Fix #3418 * update swagger
2024-10-11[bugfix] Ensure `pending_approval` set on statuses + status faves (#3415)Libravatar tobi1
* [bugfix] Ensure pending_approval set on statuses + status faves * set PendingApproval on boosts * assume not pending approval
2024-10-09[bugfix/frontend] Don't show replies to hidden parents; return 404 if no ↵v0.17.0-rc3Libravatar tobi1
"main" thread (#3411)
2024-10-08[feature] Distribute + ingest Accepts to followers (#3404)Libravatar tobi1
2024-10-04[bugfix] Don't try to add nil filtered statuses to context (#3388)Libravatar tobi1
2024-09-27[bugfix] fix media limit reader check (#3363)Libravatar kim1
* return nicer errors for frontend when media / emoji upload limit reached * fix reader limit check * add code comment
2024-09-23[feature] Show info for pending replies, allow implicit accept of pending ↵Libravatar tobi7
replies (#3322) * [feature] Allow implicit accept of pending replies * update wording
2024-09-23[chore] local instance count query caching, improved status context endpoint ↵Libravatar kim4
logging, don't log ErrHideStatus when timelining (#3330) * ensure that errors checking status visibility / converting aren't dropped * add some more context to error messages * include calling function name in log entries * don't error on timelining hidden status * further code to ignore statusfilter.ErrHideStatus type errors * remove unused error type * add local instance status / domain / user counts * add checks for localhost * rename from InstanceCounts to LocalInstance * improved code comment
2024-09-20[chore] ensure consistent caller name fetching regardless of compiler ↵Libravatar kim2
inlining (#3323) * move logging levels into log package itself * ensure inconsistent inlining doesn't mess with log calling function name * remove unused global variable * fix log level
2024-09-18[feature] Allow users to submit `interaction_policy` on new statuses (#3314)Libravatar tobi1
* [feature] Parse `interaction_policy` on status submission * beep boop * swagger? i barely know er
2024-09-16[bugfix] Ensure id set on outgoing Reject + Accept (#3312)Libravatar tobi1
2024-09-16[bugfix] Be more lenient when parsing mastodown following.csv (#3311)Libravatar tobi1
* [bugfix] Be more lenient when parsing mastodown following.csv * use follow.Notify
2024-09-16[performance] cache more database calls, reduce required database calls ↵Libravatar kim10
overall (#3290) * improvements to caching for lists and relationship to accounts / follows * fix nil panic in AddToList() * ensure list related caches are correctly invalidated * ensure returned ID lists are ordered correctly * bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded) * remove zero checks in uncached key load functions (go-structr now handles this) * fix issues after rebase on upstream/main * update the expected return order of CSV exports (since list entries are now down by entry creation date) * rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests * use returning statements on delete to get cache invalidation info * fixes to recent database delete changes * fix broken list entries delete sql * remove unused db function * update remainder of delete functions to behave in similar way, some other small tweaks * fix delete user sql, allow returning on err no entries * uncomment + fix list database tests * update remaining list tests * update envparsing test * add comments to each specific key being invalidated * add more cache invalidation explanatory comments * whoops; actually delete poll votes from database in the DeletePollByID() func * remove added but-commented-out field * improved comment regarding paging being disabled * make cache invalidation comments match what's actually happening * fix up delete query comments to match what is happening * rename function to read a bit better * don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call) * update function name in test * move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic() * use the poll_id column in poll_votes for selecting votes in poll ID * fix function name
2024-09-16[bugfix/chore] Always set the status sensitive if media + content-warning ↵Libravatar kim1
present (#3308) * always set the status sensitive flag to true if it has a content-warning with media * whoops use attachment ids instead of mention ids ... :facepalm:
2024-09-10[feature] Process `Reject` of interaction via fedi API, put rejected ↵Libravatar tobi5
statuses in the "sin bin" 😈 (#3271) * [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin" * update test * move nil check back to `rejectStatusIRI`
2024-09-09[feature] Implement exclusive lists (#3280)Libravatar Vyr Cossont5
Fixes #2616
2024-09-09[feature/frontend] Add options to include Unlisted posts or hide all posts ↵Libravatar tobi3
(#3272) * [feature/frontend] Add options to include Unlisted posts or hide all posts * finish up * swagger * move invalidate call into bundb package, avoid invalidating if not necessary * rename show_web_statuses => web_visibility * don't use ptr for webvisibility * last bits
2024-08-24[chore/bugfix] Fix missing insertion of preapproved interaction requests (#3228)Libravatar tobi1
2024-08-24[feature] Interaction requests client api + settings panel (#3215)Libravatar tobi18
* [feature] Interaction requests client api + settings panel * test accept / reject * fmt * don't pin rejected interaction * use single db model for interaction accept, reject, and request * swaggor * env sharting * append errors * remove ErrNoEntries checks * change intReqID to reqID * rename "pend" to "request" * markIntsPending -> mark interactionsPending * use log instead of returning error when rejecting interaction * empty migration * jolly renaming * make interactionURI unique again * swag grr * remove unnecessary locks * invalidate as last step
2024-08-22[feature] Use `local_only` field, deprecate `federated` field (#3222)Libravatar tobi4
* [feature] Use `local_only` field, deprecate `federated` field * use `deprecated` comment for form.Federated * nolint
2024-08-13[security] harden account update logic (#3198)Libravatar kim1
* on account update, ensure that public key has not changed * change expected error message * also support the case of changing account keys when expired (not waiting for handshake) * tweak account update hardening logic, add tests for updating account with pubkey expired * add check for whether incoming data was via federator, accepting keys if so * use freshest window for federated account updates + comment about it
2024-08-11[bugfix] updated pinned counts on status delete (#3188)Libravatar kim3
* include pinned status when incrementing / decrementing status counts * remove the pinned increment on status creation * code comments * microoptimize decr
2024-08-08[performance] move thumbnail generation to go code where possible (#3183)Libravatar kim1
* wrap thumbnailing code to handle generation natively where possible * more code comments! * add even more code comments! * add code comments about blurhash generation * maintain image rotation if contained in exif data * move rotation before resizing * ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs * use linear instead of nearest-neighbour for resizing * work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation * add header to new file * use thumb extension when getting thumb mime type * update test models and tests with new media processing * add suggested code comments * add note about thumbnail filter count reducing memory usage
2024-08-03[bugfix] fix emoji recaching operations (#3167)Libravatar kim2
* add test for emoji update image * update emoji recache to set the instance account id * don't refresh emoji if only not cached. in that case literally just recache * code comment * rename + move a few things * add some more code comments, and rename some functions to make logic a bit clearer * remove unnecessary nil check (the value can be nil) * comment wording * remove test data output * handle the case of caching an emoji which has been refreshed then uncached * allow overwriting on testrig storage as we do now on regular storage * fix emoji category ID not getting updated --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-02[chore] move PopulateAccountStats() nil check often performed into function ↵Libravatar kim4
itself (#3158) * move PopulateAccountStats() nil check often performed into function itself * fix test to take in mind we don't repopulate account stats if not-nil
2024-08-02[feature] Allow import of following and blocks via CSV (#3150)Libravatar tobi1
* [feature] Import follows + blocks via settings panel * test import follows
2024-07-31[feature] Allow users to export data via the settings panel (#3140)Libravatar tobi1
* [feature] Allow users to export data via the settings panel * rename/move some stuff
2024-07-30[feature] persist worker queues to db (#3042)Libravatar kim2
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup * ensure the tasks are sorted by creation time before pushing them * add migration to insert WorkerTask{} into database, add test for worker task persistence * add test for recovering worker queues from database * quick tweak * whoops we ended up with double cleaner job scheduling * insert each task separately, because bun is throwing some reflection error?? * add specific checking of cancelled worker contexts * add http request signing to deliveries recovered from database * add test for outgoing public key ID being correctly set on delivery * replace select with Queue.PopCtx() * get rid of loop now we don't use it * remove field now we don't use it * ensure that signing func is set * header values weren't being copied over :facepalm: * use ptr for httpclient.Request in delivery * move worker queue filling to later in server init process * fix rebase issues * make logging less shouty * use slices.Delete() instead of copying / reslicing * have database return tasks in ascending order instead of sorting them * add a 1 minute timeout to persisting worker queues
2024-07-29[feature] Implement following hashtags (#3141)Libravatar Vyr Cossont10
* Implement followed tags API * Insert statuses with followed tags into home timelines * Test following and unfollowing tags * Correct Swagger path params * Trim conversation caches * Migration for followed_tags table * Followed tag caches and DB implementation * Lint and tests * Add missing tag info endpoint, reorganize tag API * Unwrap boosts when timelining based on tags * Apply visibility filters to tag followers * Address review comments