summaryrefslogtreecommitdiff
path: root/internal/processing/workers
AgeCommit message (Collapse)AuthorFiles
2025-02-20[bugfix] update fedi api to support multiple separate votes in same multiple ↵Libravatar kim1
choice poll (#3809)
2025-02-19[feature] Forward-compatibility with Approval objects (#3807)Libravatar tobi1
* vendor * [feature] Forward-compatibility with Approval objects * vendor the thing * fix leetle bug * lil syntax tweak for beloved kimb
2025-02-17adds more code comments and some small code formatting tweaks (#3799)Libravatar kim1
2025-02-12[feature] Implement backfilling statuses thru scheduled_at (#3685)Libravatar Vyr Cossont2
* Implement backfilling statuses thru scheduled_at * Forbid mentioning others in backfills * Update error messages & codes * Add new tests for backfilled statuses * Test that backfilling doesn't timeline or notify * Fix check for absence of notification * Test that backfills do not cause federation * Fix type of apimodel.StatusCreateRequest.ScheduledAt in tests * Add config file switch and min date check
2025-01-24[feature] Process incoming Undo Announce properly (#3676)Libravatar tobi2
* [feature] Process incoming Undo Announce properly * test undo announce
2025-01-23[feature] Push notifications (#3587)Libravatar Vyr Cossont6
* Update push subscription API model to be Mastodon 4.0 compatible * Add webpush-go dependency # Conflicts: # go.sum * Single-row table for storing instance's VAPID key pair * Generate VAPID key pair during startup * Add VAPID public key to instance info API * Return VAPID public key when registering an app * Store Web Push subscriptions in DB * Add Web Push sender (similar to email sender) * Add no-op push senders to most processor tests * Test Web Push notifications from workers * Delete Web Push subscriptions when account is deleted * Implement push subscription API * Linter fixes * Update Swagger * Fix enum to int migration * Fix GetVAPIDKeyPair * Create web push subscriptions table with indexes * Log Web Push server error messages * Send instance URL as Web Push JWT subject * Accept any 2xx code as a success * Fix malformed VAPID sub claim * Use packed notification flags * Remove unused date columns * Add notification type for update notifications Not used yet * Make GetVAPIDKeyPair idempotent and remove PutVAPIDKeyPair * Post-rebase fixes * go mod tidy * Special-case 400 errors other than 408/429 Most client errors should remove the subscription. * Improve titles, trim body to reasonable length * Disallow cleartext HTTP for Web Push servers * Fix lint * Remove redundant index on unique column Also removes redundant unique and notnull tags on ID column since these are implied by pk * Make realsender.go more readable * Use Tobi's style for wrapping errors * Restore treating all 5xx codes as temporary problems * Always load target account settings * Stub `policy` and `standard` * webpush.Sender: take type converter as ctor param * Move webpush.MockSender and noopSender into testrig
2025-01-23[feature] Serve bot accounts over AP as Service instead of Person (#3672)Libravatar tobi1
* pepis * oopsie doopsie * bollocks
2025-01-08[chore] replace statuses.updated_at column with statuses.edited_at (#3636)Libravatar kim1
* update statuses table to replace updated_at column with edited_at * code comment * better code comments, fix setting of status + edit + mention + poll database times * fix log to logf call * fix status.EditIDs not being carried over in dereferencer.encrichStatus() * move status.EditID setting into handleStatusEdit()
2024-12-23[feature] add support for clients editing statuses and fetching status ↵Libravatar kim1
revision history (#3628) * start adding client support for making status edits and viewing history * modify 'freshest' freshness window to be 5s, add typeutils test for status -> api edits * only populate the status edits when specifically requested * start adding some simple processor status edit tests * add test editing status but adding a poll * test edits appropriately adding poll expiry handlers * finish adding status edit tests * store both new and old revision emojis in status * add code comment * ensure the requester's account is populated before status edits * add code comments for status edit tests * update status edit form swagger comments * remove unused function * fix status source test * add more code comments, move media description check back to media process in status create * fix tests, add necessary form struct tag
2024-12-05[feature] add support for receiving federated status edits (#3597)Libravatar kim1
* add support for extracting Updated field from Statusable implementers * add support for status edits in the database, and update status dereferencer to handle them * remove unused AdditionalInfo{}.CreatedAt * remove unused AdditionalEmojiInfo{}.CreatedAt * update new mention creation to use status.UpdatedAt * remove mention.UpdatedAt, fixes related to NewULIDFromTime() change * add migration to remove Mention{}.UpdatedAt field * add migration to add the StatusEdit{} table * start adding tests, add delete function for status edits * add more of status edit migrations, fill in more of the necessary edit delete functionality * remove unused function * allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid` * add StatusEdit{} test models * fix new statusedits sql * use model instead of table name * actually remove the Mention.UpdatedAt field... * fix tests now new models are added, add more status edit DB tests * fix panic wording * add test for deleting status edits * don't automatically set `updated_at` field on updated statuses * flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses * remove media_attachments.updated_at column * fix up more tests, further complete the dereferencer status edit tests * update more status serialization tests not expecting 'updated' AS property * gah!! json serialization tests!! * undo some gtscontext wrapping changes * more serialization test fixing :smiling_face_with_tear: * more test fixing, ensure the edit.status_id field is actually set :facepalm: * fix status edit test * grrr linter * add edited_at field to apimodel status * remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...) * ensure that status.updated_at always fits chronologically * fix more serialization tests ... * add more code comments * fix envparsing * update swagger file * properly handle media description changes during status edits * slight formatting tweak * code comment
2024-11-28send out poll votes as separate create activities given that no other AP ↵Libravatar kim1
servers support multiple objects in a single activity (#3582)
2024-11-25[bugfix] notification types missing from link header (#3571)Libravatar kim1
* ensure notification types get included in link header query for notifications * fix type query keys
2024-11-25[performance] convert enum strings to ints (#3558)Libravatar kim1
* convert statuses.visibility and notifications.notification_type columns from type string -> int for performance / space savings * fix test trying to compare string to int * fix instance count query using string literal instead of gtsmodel const type * ensure a default value is always set * also migrate the account settings and sin bin status tables * initialize maps outside loops and place into singular enum mapping creation func * use int16 for enum types * update sinbinstatus creation to be from a snapshot at initial creation * add snapshot of poll type at creation time
2024-10-08[feature] Distribute + ingest Accepts to followers (#3404)Libravatar tobi1
2024-09-23[chore] local instance count query caching, improved status context endpoint ↵Libravatar kim1
logging, don't log ErrHideStatus when timelining (#3330) * ensure that errors checking status visibility / converting aren't dropped * add some more context to error messages * include calling function name in log entries * don't error on timelining hidden status * further code to ignore statusfilter.ErrHideStatus type errors * remove unused error type * add local instance status / domain / user counts * add checks for localhost * rename from InstanceCounts to LocalInstance * improved code comment
2024-09-20[chore] ensure consistent caller name fetching regardless of compiler ↵Libravatar kim2
inlining (#3323) * move logging levels into log package itself * ensure inconsistent inlining doesn't mess with log calling function name * remove unused global variable * fix log level
2024-09-16[bugfix] Ensure id set on outgoing Reject + Accept (#3312)Libravatar tobi1
2024-09-16[performance] cache more database calls, reduce required database calls ↵Libravatar kim3
overall (#3290) * improvements to caching for lists and relationship to accounts / follows * fix nil panic in AddToList() * ensure list related caches are correctly invalidated * ensure returned ID lists are ordered correctly * bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded) * remove zero checks in uncached key load functions (go-structr now handles this) * fix issues after rebase on upstream/main * update the expected return order of CSV exports (since list entries are now down by entry creation date) * rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests * use returning statements on delete to get cache invalidation info * fixes to recent database delete changes * fix broken list entries delete sql * remove unused db function * update remainder of delete functions to behave in similar way, some other small tweaks * fix delete user sql, allow returning on err no entries * uncomment + fix list database tests * update remaining list tests * update envparsing test * add comments to each specific key being invalidated * add more cache invalidation explanatory comments * whoops; actually delete poll votes from database in the DeletePollByID() func * remove added but-commented-out field * improved comment regarding paging being disabled * make cache invalidation comments match what's actually happening * fix up delete query comments to match what is happening * rename function to read a bit better * don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call) * update function name in test * move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic() * use the poll_id column in poll_votes for selecting votes in poll ID * fix function name
2024-09-10[feature] Process `Reject` of interaction via fedi API, put rejected ↵Libravatar tobi4
statuses in the "sin bin" 😈 (#3271) * [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin" * update test * move nil check back to `rejectStatusIRI`
2024-09-09[feature] Implement exclusive lists (#3280)Libravatar Vyr Cossont2
Fixes #2616
2024-08-24[chore/bugfix] Fix missing insertion of preapproved interaction requests (#3228)Libravatar tobi1
2024-08-24[feature] Interaction requests client api + settings panel (#3215)Libravatar tobi9
* [feature] Interaction requests client api + settings panel * test accept / reject * fmt * don't pin rejected interaction * use single db model for interaction accept, reject, and request * swaggor * env sharting * append errors * remove ErrNoEntries checks * change intReqID to reqID * rename "pend" to "request" * markIntsPending -> mark interactionsPending * use log instead of returning error when rejecting interaction * empty migration * jolly renaming * make interactionURI unique again * swag grr * remove unnecessary locks * invalidate as last step
2024-08-22[feature] Use `local_only` field, deprecate `federated` field (#3222)Libravatar tobi1
* [feature] Use `local_only` field, deprecate `federated` field * use `deprecated` comment for form.Federated * nolint
2024-08-13[security] harden account update logic (#3198)Libravatar kim1
* on account update, ensure that public key has not changed * change expected error message * also support the case of changing account keys when expired (not waiting for handshake) * tweak account update hardening logic, add tests for updating account with pubkey expired * add check for whether incoming data was via federator, accepting keys if so * use freshest window for federated account updates + comment about it
2024-08-11[bugfix] updated pinned counts on status delete (#3188)Libravatar kim3
* include pinned status when incrementing / decrementing status counts * remove the pinned increment on status creation * code comments * microoptimize decr
2024-08-02[chore] move PopulateAccountStats() nil check often performed into function ↵Libravatar kim1
itself (#3158) * move PopulateAccountStats() nil check often performed into function itself * fix test to take in mind we don't repopulate account stats if not-nil
2024-07-29[feature] Implement following hashtags (#3141)Libravatar Vyr Cossont2
* Implement followed tags API * Insert statuses with followed tags into home timelines * Test following and unfollowing tags * Correct Swagger path params * Trim conversation caches * Migration for followed_tags table * Followed tag caches and DB implementation * Lint and tests * Add missing tag info endpoint, reorganize tag API * Unwrap boosts when timelining based on tags * Apply visibility filters to tag followers * Address review comments
2024-07-26[feature] Federate interaction policies + Accepts; enforce policies (#3138)Libravatar tobi5
* [feature] Federate interaction policies + Accepts; enforce policies * use Acceptable type * fix index * remove appendIRIStrs * add GetAccept federatingdb function * lock on object IRI
2024-07-24[chore] Add interaction filter to complement existing visibility filter (#3111)Libravatar tobi6
* [chore] Add interaction filter to complement existing visibility filter * pass in ptr to visibility and interaction filters to Processor{} to ensure shared * use int constants for for match type, cache db calls in filterctx * function name typo :innocent: --------- Co-authored-by: kim <grufwub@gmail.com>
2024-07-23[feature] Conversations API (#3013)Libravatar Vyr Cossont7
* Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-11[chore] Add interaction policy gtsmodels (#3075)Libravatar tobi2
* [chore] introduce interaction policy gts models * update migration a smidge * fix copy paste typo * update migration * use int for InteractionType
2024-06-30[bugfix] update SignupURL routing in email template. (#3055)Libravatar CDN1
2024-06-17[feature] Implement types[] param for notifications (#3009)Libravatar Vyr Cossont1
Counterpart of exclude_types[]. Also updates Swagger spec for types[] to use the correct param name and enumerate possible values. Fixes #3003
2024-06-13[chore] Update move test (#3005)Libravatar tobi1
2024-06-12[bugfix] Ensure side effects handled for local unlocked follows (#2998)Libravatar tobi1
2024-06-06[feature] User muting (#2960)Libravatar Vyr Cossont3
* User muting * Address review feedback * Rename uniqueness constraint on user_mutes to match convention * Remove unused account_id from where clause * Add UserMute to NewTestDB * Update test/envparsing.sh with new and fixed cache stuff * Address tobi's review comments * Make compiledUserMuteListEntry.expired consistent with UserMute.Expired * Make sure mute_expires_at is serialized as an explicit null for indefinite mutes --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06[feature] Self-serve email change for users (#2957)Libravatar tobi4
* [feature] Email change * frontend stuff for changing email * docs * tests etc * differentiate more clearly between local user+account and account * populate user
2024-05-27[chore] Fix report email link (#2932)Libravatar tobi1
2024-05-06feature: filters v2 server-side warning/hiding (#2793)Libravatar Vyr Cossont3
* Remove dead code * Filter statuses when converting to frontend representation * status.filtered is an array * Make matching case-insensitive * Remove TODOs that don't need to be done now * Add missing filter check for notification * lint: rename ErrHideStatus * APIFilterActionToFilterAction not used yet * swaggerino docseroni * Address review comments * Add apimodel.FilterActionNone --------- Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com> Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-05-02[bugfix] Lock when checking/creating notifs to avoid race (#2890)Libravatar tobi10
* [bugfix] Lock when checking/creating notifs to avoid race * test notif spam
2024-04-29[chore] Fix conflict in workers tests (#2880)Libravatar tobi3
* [chore] Fix conflict in workers tests * commenty-wenty
2024-04-26[performance] update remaining worker pools to use queues (#2865)Libravatar kim9
* start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-16[feature/performance] Store account stats in separate table (#2831)Libravatar tobi5
* [feature/performance] Store account stats in separate table, get stats from remote * test account stats * add some missing increment / decrement calls * change stats function signatures * rejig logging a bit * use lock when updating stats
2024-04-13[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)Libravatar tobi2
* update settings panels, add pending overview + approve/deny functions * add admin accounts get, approve, reject * send approved/rejected emails * use signup URL * docs! * email * swagger * web linting * fix email tests * wee lil fixerinos * use new paging logic for GetAccounts() series of admin endpoints, small changes to query building * shuffle useAccountIDIn check *before* adding to query * fix parse from toot react error * use `netip.Addr` * put valid slices in globals * optimistic updates for account state --------- Co-authored-by: kim <grufwub@gmail.com>
2024-04-11[feature] New user sign-up via web page (#2796)Libravatar tobi4
* [feature] User sign-up form and admin notifs * add chosen + filtered languages to migration * remove stray comment * chosen languages schmosen schmanguages * proper error on local account missing
2024-04-11[performance] massively improved ActivityPub delivery worker efficiency (#2812)Libravatar kim2
* add delivery worker type that pulls from queue to httpclient package * finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller * hook up queue deletion logic * support deleting queued http requests by target ID * don't index APRequest by hostname in the queue * use gorun * use the original context's values when wrapping msg type as delivery{} * actually log in the AP delivery worker ... * add uncommitted changes * use errors.AsV2() * use errorsv2.AsV2() * finish adding some code comments, add bad host handling to delivery workers * slightly tweak deliveryworkerpool API, use advanced sender multiplier * remove PopCtx() method, let others instead rely on Wait() * shuffle things around to move delivery stuff into transport/ subpkg * remove dead code * formatting * validate request before queueing for delivery * finish adding code comments, fix up backoff code * finish adding more code comments * clamp minimum no. senders to 1 * add start/stop logging to delivery worker, some slight changes * remove double logging * use worker ptrs * expose the embedded log fields in httpclient.Request{} * ensure request context values are preserved when updating ctx * add delivery worker tests * fix linter issues * ensure delivery worker gets inited in testrig * fix tests to delivering messages to check worker delivery queue * update error type to use ptr instead of value receiver * fix test calling Workers{}.Start() instead of testrig.StartWorkers() * update docs for advanced-sender-multiplier * update to the latest activity library version * add comment about not using httptest.Server{}
2024-03-22[chore] Move local account settings to separate db table (#2770)Libravatar tobi1
* [chore] Move local account settings to separate database model * don't use separate settings_id
2024-03-13[feature] Process outgoing Move from clientAPI (#2750)Libravatar tobi7
* prevent moved accounts from taking create-type actions * update move logic * federate move out * indicate on web profile when an account has moved * [docs] Add migration docs section * lock while checking + setting move state * use redirectFollowers func for clientAPI as well * comment typo * linter? i barely know 'er! * Update internal/uris/uri.go Co-authored-by: Daenney <daenney@users.noreply.github.com> * add a couple tests for move * fix little mistake exposed by tests (thanks tests) * ensure Move marked as successful * attach shared util funcs to struct * lock whole account when doing move * move moving check to after error check * replace repeated text with error func * linterrrrrr!!!! * catch self follow case --------- Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-12[feature] Process incoming `Move` activity (#2724)Libravatar tobi3
* [feature] Process incoming account Move activity * fix targetAcct typo * put move origin account on fMsg * shift more move functionality back to the worker fn * simplify error logic
2024-03-04[bugfix] check remote status permissibility (#2703)Libravatar kim1
* add more stringent checks for remote status permissibility * add check for inreplyto of a remote status being a boost * do not permit inReplyTo boost wrapper statuses * change comment wording * fix calls to NewFederator() * add code comments for NotPermitted() and SetNotPermitted() * improve comment * check that existing != nil before attempting delete * ensure replying account isn't suspended * use a debug log instead of info. check for boost using ID * shorten log string length. make info level * add note that replying to boost wrapper status shouldn't be able to happen anyways * update to use onFail() function