summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2025-04-26[performance] rewrite timelines to rely on new timeline cache type (#3941)Libravatar kim20
* start work rewriting timeline cache type * further work rewriting timeline caching * more work integration new timeline code * remove old code * add local timeline, fix up merge conflicts * remove old use of go-bytes * implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr * remove old timeline package, add local timeline cache * remove references to old timeline types that needed starting up in tests * start adding page validation * fix test-identified timeline cache package issues * fix up more tests, fix missing required changes, etc * add exclusion for test.out in gitignore * clarify some things better in code comments * tweak cache size limits * fix list timeline cache fetching * further list timeline fixes * linter, ssssssssshhhhhhhhhhhh please * fix linter hints * reslice the output if it's beyond length of 'lim' * remove old timeline initialization code, bump go-structr to v0.9.4 * continued from previous commit * improved code comments * don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts * finish writing more code comments * some variable renaming, for ease of following * change the way we update lo,hi paging values during timeline load * improved code comments for updated / returned lo , hi paging values * finish writing code comments for the StatusTimeline{} type itself * fill in more code comments * update go-structr version to latest with changed timeline unique indexing logic * have a local and public timeline *per user* * rewrite calls to public / local timeline calls * remove the zero length check, as lo, hi values might still be set * simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions * swap the lo, hi values :facepalm: * add (now) missing slice reverse of tag timeline statuses when paging ASC * remove local / public caches (is out of scope for this work), share more timeline code * remove unnecessary change * again, remove more unused code * remove unused function to appease the linter * move boost checking to prepare function * fix use of timeline.lastOrder, fix incorrect range functions used * remove comments for repeat code * remove the boost logic from prepare function * do a maximum of 5 loads, not 10 * add repeat boost filtering logic, update go-structr, general improvements * more code comments * add important note * fix timeline tests now that timelines are returned in page order * remove unused field * add StatusTimeline{} tests * add more status timeline tests * start adding preloading support * ensure repeat boosts are marked in preloaded entries * share a bunch of the database load code in timeline cache, don't clear timelines on relationship change * add logic to allow dynamic clear / preloading of timelines * comment-out unused functions, but leave in place as we might end-up using them * fix timeline preload state check * much improved status timeline code comments * more code comments, don't bother inserting statuses if timeline not preloaded * shift around some logic to make sure things aren't accidentally left set * finish writing code comments * remove trim-after-insert behaviour * fix-up some comments referring to old logic * remove unsetting of lo, hi * fix preload repeatBoost checking logic * don't return on status filter errors, these are usually transient * better concurrency safety in Clear() and Done() * fix test broken due to addition of preloader * fix repeatBoost logic that doesn't account for already-hidden repeatBoosts * ensure edit submodels are dropped on cache insertion * update code-comment to expand CAS accronym * use a plus1hULID() instead of 24h * remove unused functions * add note that public / local timeline requester can be nil * fix incorrect visibility filtering of tag timeline statuses * ensure we filter home timeline statuses on local only * some small re-orderings to confirm query params in correct places * fix the local only home timeline filter func
2025-04-25[chore] Move deps to code.superseriousbusiness.org (#4054)Libravatar tobi5
2025-04-22[bugfix] Use util.IsNil for checking DomainPermission (#4040)Libravatar tobi1
2025-04-14ensure 'none' gets included in serializable visibilities (#4007)Libravatar kim1
2025-04-11[bugfix] Fix setting bot on/off (#3986)Libravatar tobi2
* [bugfix] Fix setting bot on/off * read client messages in tests * test fix
2025-04-10[feature] Implement /oauth/revoke for token revocation (#3983)Libravatar tobi1
2025-04-07[feature] add TOTP two-factor authentication (2FA) (#3960)Libravatar tobi5
* [feature] add TOTP two-factor authentication (2FA) * use byteutil.S2B to avoid allocations when comparing + generating password hashes * don't bother with string conversion for consts * use io.ReadFull * use MustGenerateSecret for backup codes * rename util functions
2025-04-06[chore] Migrate accounts to new table, relax uniqueness constraint of actor ↵Libravatar tobi8
`url` and collections (#3928) * [chore] Migrate accounts to new table, relax uniqueness constraint of actor url and collections * fiddle with it! (that's what she said) * remove unused cache fields * sillyness * fix tiny whoopsie
2025-04-04[feature] Allow editing domain blocks/allows, fix comment import (#3967)Libravatar tobi4
* start implementing editing of existing domain permissions * [feature] Allow editing domain blocks/allows, fix comment import * [bugfix] Use "comment" via /api/v1/instance * fix the stuff
2025-03-31[feature] Use blurhashes in frontend, tidy up gallery view a bit (#3948)Libravatar tobi1
* [feature] Use blurhashes in frontend, tidy up gallery view a bit * weeeeeeeeeeeeeeeee * beep boop
2025-03-26[feature] Allow user to choose "gallery" style layout for web view of ↵Libravatar tobi3
profile (#3917) * [feature] Allow user to choose "gallery" style web layout * find a bug and squish it up and all day long you'll have good luck * just a sec * [performance] reindex public timeline + tinker with query a bit * fiddling * should be good now * last bit of finagling, i'm done now i prommy * panic normally
2025-03-17[feature] Application creation + management via API + settings panel (#3906)Libravatar tobi6
* [feature] Application creation + management via API + settings panel * fix docs links * add errnorows test * use known application as shorter * add comment about side effects
2025-03-07[feature] Parse content warning to HTML, serialize via client API as ↵Libravatar tobi14
plaintext (#3876) * [feature] Parse content warning as HTML, serialize via API to plaintext * tidy up some cruft * whoops * oops * i'm da joker baybee * clemency muy lorde * rename some of the text functions for clarity * jiggle the opts * fiddle de deee * hopefully the last test fix i ever have to do in my beautiful life
2025-03-06[bugfix] Store and expose status content type (#3870)Libravatar ewwwin6
* Add ContentType to internal models * Add ContentType to API models StatusSource and StatusEdit * Add helpers to convert between API/internal StatusContentType * Write status content type on create/edit * Add migration * Update API docs go run github.com/go-swagger/go-swagger/cmd/swagger generate spec --scan-models --exclude-deps --output docs/api/swagger.yaml * ensure ContentType is updated anywhere Text is * Update docs, take care of TODOs * Set ContentType in more places where Text is set * We don't actually use ContentType on the API status model * Update StatusSource test * Remove unused helper function I copied * Revert change to StatusContentType swagger annotation I'm going to include this in a follow-on PR instead. * Add test for updating content type in edits * Return a value from processContentType instead of modifying the existing status Fixes an issue that was caught by the test I just added - the recorded edit would be marked with the *new* content type instead of the old one, which is obviously bad * Add test for handling of statuses with no stored content type * repurpose an existing test status instead of adding a new one to avoid breaking other tests * Add test to ensure newly created statuses always have content type saved * Do include content type on status API model actually This is mostly important when deleting and redrafting. The comment on `apimodel.Status.Text` implies that it's not sent except in response to status deletion, but actually this doesn't seem to be the case; it also appears to be present in responses to creations and normal fetches and stuff. So I'm treating `ContentType` the same here. * Update new tests to check content type on API statuses * Check content type of API statuses in all tests where text is checked * update other api tests with status content type field * Add test ensuring text and content type are returned when deleting a status * Convert processContentType to free function and remove unused parameter * check for the correct value in the deletion test * Be explicit about this test status having an empty content type * Use omitempty consistently on API models * clean up the final diff a bit * one more swagger regen for the road * Handle nil statuses in processContentType * Don't pass processContentType the entire edit form, it doesn't need it * Move processContentType to common.go and use for creation as well * Remove unused parameters to ContentTypeToAPIContentType
2025-03-04[feature] Add token review / delete to backend + settings panel (#3845)Libravatar tobi1
2025-03-03[feature] Refactor tokens, allow multiple app redirect_uris (#3849)Libravatar tobi11
* [feature] Refactor tokens, allow multiple app redirect_uris * move + tweak handlers a bit * return error for unset oauth2.ClientStore funcs * wrap UpdateToken with cache * panic handling * cheeky little time optimization * unlock on error
2025-03-02[chore] migrate oauth2 -> codeberg (#3857)Libravatar tobi2
2025-03-02[chore] github.com/superseriousbusiness/activity -> ↵Libravatar tobi3
codeberg.org/superseriousbusiness/activity (#3853)
2025-03-01[feature] Implement CSV import for mutes (#3696)Libravatar Xavier Vello1
* Implement CSV import for mutes * update swagger.yaml * update documentation * add ImportTestSuite.TestImportMutes * fix comment typo
2025-02-26[feature] Enforce OAuth token scopes (#3835)Libravatar tobi11
* move tokenauth to apiutil * enforce scopes * docs * update test models, remove deprecated "follow" * file header * tests * tweak scope matcher * simplify... * fix tests * log user out of settings panel in case of oauth error
2025-02-20[bugfix] update fedi api to support multiple separate votes in same multiple ↵Libravatar kim2
choice poll (#3809)
2025-02-19[feature] Forward-compatibility with Approval objects (#3807)Libravatar tobi1
* vendor * [feature] Forward-compatibility with Approval objects * vendor the thing * fix leetle bug * lil syntax tweak for beloved kimb
2025-02-17adds more code comments and some small code formatting tweaks (#3799)Libravatar kim2
2025-02-13[bugfix] Return 404 when web context target status hidden (#3792)Libravatar tobi1
2025-02-12[feature] Implement backfilling statuses thru scheduled_at (#3685)Libravatar Vyr Cossont4
* Implement backfilling statuses thru scheduled_at * Forbid mentioning others in backfills * Update error messages & codes * Add new tests for backfilled statuses * Test that backfilling doesn't timeline or notify * Fix check for absence of notification * Test that backfills do not cause federation * Fix type of apimodel.StatusCreateRequest.ScheduledAt in tests * Add config file switch and min date check
2025-02-10[feature] make account sign-up / backlog limits configurable (#3768)Libravatar tobi1
2025-02-09[bugfix] Fix POST to create account endpoint (#3767)Libravatar tobi3
2025-02-08[bug] Add missing `group` property to the Account model (#3746)Libravatar pnwmatt2
* Stub out account's Group parameter * Fix typo in Contributing documentation * Update swagger and match style * Update the swagger command to mimic test/swagger.sh * Fix tests for new `group` param in Account model * More test changes for new `group` param in Account * Continuing test changes for `group` param stubbing * Another round of `group` stubbing for tests
2025-02-04[bugfix] wrong nodeinfo version (tobi is a boob) (#3735)Libravatar tobi1
2025-02-04[feature] Change `instance-stats-randomize` to `instance-stats-mode` with ↵Libravatar tobi1
multiple options; implement nodeinfo 2.1 (#3734) * [feature] Change `instance-stats-randomize` to `instance-stats-mode` with multiple options; implement nodeinfo 2.1 * swaggalaggadingdong
2025-02-03[feature] Implement Web Push notification policy (#3721)Libravatar Vyr Cossont2
* Web Push: add policy column to subscriptions * Web Push: add policy to API * Web Push: test notification policy * go-fmt unrelated file (how did this get thru?)
2025-01-31[feature] Add `instance-stats-randomize` config option (#3718)Libravatar tobi1
* [feature] Add `instance-stats-randomize` config option * don't use cache (overkill)
2025-01-24[feature] Process incoming Undo Announce properly (#3676)Libravatar tobi2
* [feature] Process incoming Undo Announce properly * test undo announce
2025-01-23[feature] Push notifications (#3587)Libravatar Vyr Cossont16
* Update push subscription API model to be Mastodon 4.0 compatible * Add webpush-go dependency # Conflicts: # go.sum * Single-row table for storing instance's VAPID key pair * Generate VAPID key pair during startup * Add VAPID public key to instance info API * Return VAPID public key when registering an app * Store Web Push subscriptions in DB * Add Web Push sender (similar to email sender) * Add no-op push senders to most processor tests * Test Web Push notifications from workers * Delete Web Push subscriptions when account is deleted * Implement push subscription API * Linter fixes * Update Swagger * Fix enum to int migration * Fix GetVAPIDKeyPair * Create web push subscriptions table with indexes * Log Web Push server error messages * Send instance URL as Web Push JWT subject * Accept any 2xx code as a success * Fix malformed VAPID sub claim * Use packed notification flags * Remove unused date columns * Add notification type for update notifications Not used yet * Make GetVAPIDKeyPair idempotent and remove PutVAPIDKeyPair * Post-rebase fixes * go mod tidy * Special-case 400 errors other than 408/429 Most client errors should remove the subscription. * Improve titles, trim body to reasonable length * Disallow cleartext HTTP for Web Push servers * Fix lint * Remove redundant index on unique column Also removes redundant unique and notnull tags on ID column since these are implied by pk * Make realsender.go more readable * Use Tobi's style for wrapping errors * Restore treating all 5xx codes as temporary problems * Always load target account settings * Stub `policy` and `standard` * webpush.Sender: take type converter as ctor param * Move webpush.MockSender and noopSender into testrig
2025-01-23[feature] Serve bot accounts over AP as Service instead of Person (#3672)Libravatar tobi2
* pepis * oopsie doopsie * bollocks
2025-01-08[bugfix] More permissive CSV parsing for perm subs, text parse fix (#3638)Libravatar tobi1
* [bugfix] More permissive CSV parsing for perm subs, text parse fix * wee * change the way dry works, slightly * me oh my, i'm just a little guy * we're just normal men
2025-01-08[feature] Fetch + create domain permissions from subscriptions nightly (#3635)Libravatar tobi21
* peepeepoopoo * test domain perm subs * swagger * envparsing * dries your wets * start on docs * finish up docs * copy paste errors * rename actions package * rename force -> skipCache * move obfuscate parse nearer to where err is checked * make higherPrios a simple slice * don't use receiver for permsFrom funcs * add more context to error logs * defer finished log * use switch for permType instead of if/else * thanks linter, love you <3 * validate csv headers before full read * use bufio scanner
2025-01-08[chore] replace statuses.updated_at column with statuses.edited_at (#3636)Libravatar kim4
* update statuses table to replace updated_at column with edited_at * code comment * better code comments, fix setting of status + edit + mention + poll database times * fix log to logf call * fix status.EditIDs not being carried over in dereferencer.encrichStatus() * move status.EditID setting into handleStatusEdit()
2025-01-05[feature] Create/update/remove domain permission subscriptions (#3623)Libravatar tobi2
* [feature] Create/update/remove domain permission subscriptions * lint * envparsing * remove errant fmt.Println * create drafts, subs, exclude, from snapshot models * name etag column correctly * remove count column * lint
2024-12-23[feature] add support for clients editing statuses and fetching status ↵Libravatar kim13
revision history (#3628) * start adding client support for making status edits and viewing history * modify 'freshest' freshness window to be 5s, add typeutils test for status -> api edits * only populate the status edits when specifically requested * start adding some simple processor status edit tests * add test editing status but adding a poll * test edits appropriately adding poll expiry handlers * finish adding status edit tests * store both new and old revision emojis in status * add code comment * ensure the requester's account is populated before status edits * add code comments for status edit tests * update status edit form swagger comments * remove unused function * fix status source test * add more code comments, move media description check back to media process in status create * fix tests, add necessary form struct tag
2024-12-05[feature] add support for receiving federated status edits (#3597)Libravatar kim8
* add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-12-04[feature] unending polls (#3592)Libravatar kim1
* adds support for unending polls to be created locally * remove unused argument
2024-12-02[feature] Add global instance CSS customization setting (#3352)Libravatar Victor Dyotte1
Allow instance admins to add custom CSS that will affect every page of their instance. This is done with a new CustomCSS instance setting that works pretty much exactly like the Users CustomCSS property. This custom CSS is then requested for every page load. User styles/themes take precedence over this CSS. Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-11-28send out poll votes as separate create activities given that no other AP ↵Libravatar kim1
servers support multiple objects in a single activity (#3582)
2024-11-28[chore] rename New___(string) int signature functions to Parse___(string) ↵Libravatar kim1
int (#3580) * rename New___(string) int {} signature functions to Parse___(string) int {} * remove test output
2024-11-26[bugfix] Allow unsetting filter expiration dates (#3560)Libravatar Vyr Cossont4
* Regression tests for #3497 (v1 and v2) * use Nullable type for v2 form.expires_in --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-11-25[bugfix] notification types missing from link header (#3571)Libravatar kim2
* ensure notification types get included in link header query for notifications * fix type query keys
2024-11-25[performance] convert enum strings to ints (#3558)Libravatar kim4
* convert statuses.visibility and notifications.notification_type columns from type string -> int for performance / space savings * fix test trying to compare string to int * fix instance count query using string literal instead of gtsmodel const type * ensure a default value is always set * also migrate the account settings and sin bin status tables * initialize maps outside loops and place into singular enum mapping creation func * use int16 for enum types * update sinbinstatus creation to be from a snapshot at initial creation * add snapshot of poll type at creation time
2024-11-21[feature] Add domain permission drafts and excludes (#3547)Libravatar tobi4
* [feature] Add domain permission drafts and excludes * fix typescript complaining * lint * make filenames more consistent * test own domain excluded
2024-11-11[performance] minimise log field allocations (#3529)Libravatar kim2
* when appending log field only do so by minimal amount * move slice utils to separate package to fix import cycle, add GrowJust() and AppendJust() functions * fix GrowJust() not returning slice of same length * improved xslices tests * make AppendJust() test check for slice contents, fix AppendJust() final copying behaviour * add a +1 with field growth to try minimise allocation for log 'msg' field