summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2024-06-26[chore] media and emoji refactoring (#3000)Libravatar kim17
* start updating media manager interface ready for storing attachments / emoji right away * store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load() * remove now unused media workers * fix tests and issues * fix another test! * fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues * fix more tests * fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis * whoops, rebase issue * remove kim's whacky experiments * do some reshuffling, ensure emoji uri gets set * ensure marked as not cached on cleanup * tweaks to media / emoji processing to handle context canceled better * ensure newly fetched emojis actually get set in returned slice * use different varnames to be a bit more obvious * move emoji refresh rate limiting to dereferencer * add exported dereferencer functions for remote media, use these for recaching in processor * add check for nil attachment in updateAttachment() * remove unused emoji and media fields + columns * see previous commit * fix old migrations expecting image_updated_at to exists (from copies of old models) * remove freshness checking code (seems to be broken...) * fix error arg causing nil ptr exception * finish documentating functions with comments, slight tweaks to media / emoji deref error logic * remove some extra unneeded boolean checking * finish writing documentation (code comments) for exported media manager methods * undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot * move doesColumnExist() to util.go in migrations package
2024-06-24[chore]: Bump github.com/gorilla/feeds from 1.1.2 to 1.2.0 (#3035)Libravatar dependabot[bot]1
2024-06-18[bugfix] rename `include_types[]` to `types[]` (#3023)Libravatar tobi1
2024-06-18[feature/frontend] Reports frontend v2 (#3022)Libravatar tobi2
* use apiutil + paging in admin processor+handlers * we're making it happen * fix little whoopsie * styling for report list * don't youuuu forget about meee don't don't don't don't * last bits * sanitize content before showing in report statuses * update report docs
2024-06-17[feature] Implement types[] param for notifications (#3009)Libravatar Vyr Cossont2
Counterpart of exclude_types[]. Also updates Swagger spec for types[] to use the correct param name and enumerate possible values. Fixes #3003
2024-06-14[feature] filter API v2: Restore keywords_attributes and statuses_attributes ↵Libravatar Vyr Cossont2
(#2995) These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
2024-06-13[chore] Update move test (#3005)Libravatar tobi1
2024-06-12[bugfix] Ensure side effects handled for local unlocked follows (#2998)Libravatar tobi2
2024-06-11[bugfix] Deref stats async, serve stub collections if handshaking (#2990)v0.16.0-rc2Libravatar tobi3
* [bugfix] Deref stats async, allow peek if handshaking * don't return totalItems when handshaking or hiding collections * use GetLimit() * use StubAccountStats
2024-06-10[bugfix] boost and account recursion (#2982)Libravatar kim3
* fix possible infinite recursion if moved accounts are self-referential * adds a defensive check for a boost being a boost of a boost wrapper * add checks on input for a boost of a boost * remove unnecessary check * add protections on account move to prevent move recursion loops * separate status conversion without boost logic into separate function to remove risk of recursion * move boost check to boost function itself * formatting * use error 422 instead of 500 * use gtserror not standard errors package for error creation
2024-06-07[feature] Implement filters_changed stream event (#2972)Libravatar Vyr Cossont15
2024-06-06[feature] User muting (#2960)Libravatar Vyr Cossont16
* User muting * Address review feedback * Rename uniqueness constraint on user_mutes to match convention * Remove unused account_id from where clause * Add UserMute to NewTestDB * Update test/envparsing.sh with new and fixed cache stuff * Address tobi's review comments * Make compiledUserMuteListEntry.expired consistent with UserMute.Expired * Make sure mute_expires_at is serialized as an explicit null for indefinite mutes --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06[feature] Self-serve email change for users (#2957)Libravatar tobi22
* [feature] Email change * frontend stuff for changing email * docs * tests etc * differentiate more clearly between local user+account and account * populate user
2024-06-06[feature] do not uncache status / emoji media if attached status is ↵Libravatar kim1
bookmarked (#2956) * do not uncache status / emoji media if attached status is bookmarked * add status bookmark and bookmark IDs caches * update status bookmark tests * move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache * fix envparsing.sh test
2024-05-31[feature] Add from: search operator and account_id query param (#2943)Libravatar Vyr Cossont1
* Add from: search operator * Fix whitespace in Swagger YAML comment * Move query parsing into its own method * Document search * Clarify post search scope
2024-05-31[feature] Implement Filter API v2 (#2936)Libravatar Vyr Cossont16
* Use correct entity name * We support server-side filters now * Document filter v1 methods that can throw a 409 * Validate v1 filter phrase as filter title * Always check v1 filter API status codes in tests * Document keyword minimum requirement on filter API v1 * Make it possible to specify filter keyword update columns per filter keyword * Implement v2 filter API * Fix lint and tests * Update Swagger spec * Fix filter update test * Update Swagger spec *correctly* * Update actual files Swagger spec was generated from * Remove keywords_attributes and statuses_attributes * Add test for serialization of empty filter * More helpful messages when object is owned by wrong account
2024-05-29[feature] Implement profile API (#2926)Libravatar Vyr Cossont1
* Implement profile API This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header. See: https://docs.joinmastodon.org/methods/profile/ * Move profile media methods to media processor * Remove check for moved account
2024-05-27[feature/frontend] Let admins send test email to validate SMTP config (#2934)Libravatar tobi1
* [feature/frontend] Let admins send test email to validate SMTP config * wee
2024-05-27[chore] Fix report email link (#2932)Libravatar tobi1
2024-05-22[performance] update storage backend and make use of seek syscall when ↵Libravatar kim1
available (#2924) * update to use go-storage/ instead of go-store/v2/storage/ * pull in latest version from codeberg * remove test output :innocent: * add code comments * set the exclusive bit when creating new files in disk config * bump to actual release version * bump to v0.1.1 (tis a simple no-logic change) * update readme * only use a temporary read seeker when decoding video if required (should only be S3 now) * use fastcopy library to use memory pooled buffers when calling TempFileSeeker() * update to use seek call in serveFileRange()
2024-05-06feature: filters v2 server-side warning/hiding (#2793)Libravatar Vyr Cossont15
* Remove dead code * Filter statuses when converting to frontend representation * status.filtered is an array * Make matching case-insensitive * Remove TODOs that don't need to be done now * Add missing filter check for notification * lint: rename ErrHideStatus * APIFilterActionToFilterAction not used yet * swaggerino docseroni * Address review comments * Add apimodel.FilterActionNone --------- Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com> Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-05-02[bugfix] Lock when checking/creating notifs to avoid race (#2890)Libravatar tobi14
* [bugfix] Lock when checking/creating notifs to avoid race * test notif spam
2024-05-01[feature] Page through accounts as moderator (#2881)Libravatar tobi1
* [feature] Page through accounts as moderator * aaaaa * use COLLATE "C" for Postgres to ensure same ordering as SQLite * fix typo, test paging up * don't show moderation / info for our instance acct
2024-04-29[chore] Fix conflict in workers tests (#2880)Libravatar tobi3
* [chore] Fix conflict in workers tests * commenty-wenty
2024-04-26[bugfix] Fix error string typo (#2873)Libravatar kim1
2024-04-26[performance] update remaining worker pools to use queues (#2865)Libravatar kim34
* start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-17[feature] Status source endpoint (#2848)Libravatar tobi1
* [feature] statusSource endpoint * finish up
2024-04-17[feature] Stub status history endpoint (#2847)Libravatar tobi1
2024-04-16[feature/performance] Store account stats in separate table (#2831)Libravatar tobi13
* [feature/performance] Store account stats in separate table, get stats from remote * test account stats * add some missing increment / decrement calls * change stats function signatures * rejig logging a bit * use lock when updating stats
2024-04-15[performance] cached oauth database types (#2838)Libravatar kim1
* update token + client code to use struct caches * add code comments * slight tweak to default mem ratios * fix envparsing * add appropriate invalidate hooks * update the tokenstore sweeping function to rely on caches * update to use PutClient() * add ClientID to list of token struct indices
2024-04-13[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)Libravatar tobi9
* update settings panels, add pending overview + approve/deny functions * add admin accounts get, approve, reject * send approved/rejected emails * use signup URL * docs! * email * swagger * web linting * fix email tests * wee lil fixerinos * use new paging logic for GetAccounts() series of admin endpoints, small changes to query building * shuffle useAccountIDIn check *before* adding to query * fix parse from toot react error * use `netip.Addr` * put valid slices in globals * optimistic updates for account state --------- Co-authored-by: kim <grufwub@gmail.com>
2024-04-11[feature] New user sign-up via web page (#2796)Libravatar tobi10
* [feature] User sign-up form and admin notifs * add chosen + filtered languages to migration * remove stray comment * chosen languages schmosen schmanguages * proper error on local account missing
2024-04-11[performance] massively improved ActivityPub delivery worker efficiency (#2812)Libravatar kim4
* add delivery worker type that pulls from queue to httpclient package * finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller * hook up queue deletion logic * support deleting queued http requests by target ID * don't index APRequest by hostname in the queue * use gorun * use the original context's values when wrapping msg type as delivery{} * actually log in the AP delivery worker ... * add uncommitted changes * use errors.AsV2() * use errorsv2.AsV2() * finish adding some code comments, add bad host handling to delivery workers * slightly tweak deliveryworkerpool API, use advanced sender multiplier * remove PopCtx() method, let others instead rely on Wait() * shuffle things around to move delivery stuff into transport/ subpkg * remove dead code * formatting * validate request before queueing for delivery * finish adding code comments, fix up backoff code * finish adding more code comments * clamp minimum no. senders to 1 * add start/stop logging to delivery worker, some slight changes * remove double logging * use worker ptrs * expose the embedded log fields in httpclient.Request{} * ensure request context values are preserved when updating ctx * add delivery worker tests * fix linter issues * ensure delivery worker gets inited in testrig * fix tests to delivering messages to check worker delivery queue * update error type to use ptr instead of value receiver * fix test calling Workers{}.Start() instead of testrig.StartWorkers() * update docs for advanced-sender-multiplier * update to the latest activity library version * add comment about not using httptest.Server{}
2024-04-08[bugfix] Ensure side effects for local -> local follows get processed (#2820)v0.15.0-rc2Libravatar tobi2
2024-04-02[bugfix] Set the `Host` header within the signing transport (#2799)Libravatar kim1
2024-04-02[feature] Option to hide followers/following (#2788)Libravatar tobi4
2024-03-28[bugfix] Serve correct URI for AP following collection (#2787)Libravatar tobi1
2024-03-25[feature] User-selectable preset CSS themes for accounts (#2777)Libravatar tobi4
* [feature] User-selectable preset themes * docs, more theme stuff * lint, tests * fix css name * correct some little issues * add another theme * fix poll background * okay last theme i swear * make retrieval of apimodel themes more conventional * preallocate stylesheet slices
2024-03-25[bugfix] Avoid empty public/local timeline queries (#2784)Libravatar tobi3
2024-03-22[chore] Move local account settings to separate db table (#2770)Libravatar tobi8
* [chore] Move local account settings to separate database model * don't use separate settings_id
2024-03-14[bugfix] Don't return 500 when searching for unpermitted status (#2753)Libravatar tobi1
2024-03-13[chore] Expose move endpoint again, small settings panel fixes (#2752)Libravatar tobi1
2024-03-13[feature] Process outgoing Move from clientAPI (#2750)Libravatar tobi9
* prevent moved accounts from taking create-type actions * update move logic * federate move out * indicate on web profile when an account has moved * [docs] Add migration docs section * lock while checking + setting move state * use redirectFollowers func for clientAPI as well * comment typo * linter? i barely know 'er! * Update internal/uris/uri.go Co-authored-by: Daenney <daenney@users.noreply.github.com> * add a couple tests for move * fix little mistake exposed by tests (thanks tests) * ensure Move marked as successful * attach shared util funcs to struct * lock whole account when doing move * move moving check to after error check * replace repeated text with error func * linterrrrrr!!!! * catch self follow case --------- Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-12[feature] Process incoming `Move` activity (#2724)Libravatar tobi3
* [feature] Process incoming account Move activity * fix targetAcct typo * put move origin account on fMsg * shift more move functionality back to the worker fn * simplify error logic
2024-03-06[feature] Filters v1 (#2594)Libravatar Vyr Cossont8
* Implement client-side v1 filters * Exclude linter false positives * Update test/envparsing.sh * Fix minor Swagger, style, and Bun usage issues * Regenerate Swagger * De-generify filter keywords * Remove updating filter statuses This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered. * Add a test for filter statuses specifically * De-generify filter statuses * Inline FilterEntry * Use vertical style for Bun operations consistently * Add comment on Filter DB interface * Remove GoLand linter control comments Our existing linters should catch these, or they don't matter very much * Reduce memory ratio for filters
2024-03-04[bugfix] check remote status permissibility (#2703)Libravatar kim1
* add more stringent checks for remote status permissibility * add check for inreplyto of a remote status being a boost * do not permit inReplyTo boost wrapper statuses * change comment wording * fix calls to NewFederator() * add code comments for NotPermitted() and SetNotPermitted() * improve comment * check that existing != nil before attempting delete * ensure replying account isn't suspended * use a debug log instead of info. check for boost using ID * shorten log string length. make info level * add note that replying to boost wrapper status shouldn't be able to happen anyways * update to use onFail() function
2024-02-29[bugfix] unwrap boosts when checking in-reply-to status (#2702)Libravatar kim1
* add stronger checks on status being replied to * update error code test is expecting
2024-02-27[feature] Add experimental `instance-federation-spam-filter` option (#2685)Libravatar tobi17
* [chore] Move `visibility` to `filter/visibility` * [feature] Add experimental instance-federation-spam-filter option
2024-02-20[bugfix] fix possible mutex lockup during streaming code (#2633)Libravatar kim12
* rewrite Stream{} to use much less mutex locking, update related code * use new context for the stream context * ensure stream gets closed on return of writeTo / readFrom WSConn() * ensure stream write timeout gets cancelled * remove embedded context type from Stream{}, reformat log messages for consistency * use c.Request.Context() for context passed into Stream().Open() * only return 1 boolean, fix tests to expect multiple stream types in messages * changes to ping logic * further improved ping logic * don't export unused function types, update message sending to only include relevant stream type * ensure stream gets closed :facepalm: * update to error log on failed json marshal (instead of panic) * inverse websocket read error checking to _ignore_ expected close errors
2024-02-19[bugfix] Ensure local statuses always get a threadID so they can be muted ↵Libravatar tobi2
(#2665) * [chore/bugfix] Ensure threadID always set on local statuses * test