summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2023-06-10[bugfix] Delete from list timeline on status deletion (#1878)Libravatar tobi1
2023-06-04[chore] tidy + test timelines a bit better (#1865)Libravatar tobi2
* [chore] tidy + test timelines a bit better * thanks linter
2023-06-03[chore] Update versions, fix lint errors (#1860)Libravatar tobi2
2023-05-28[chore] tidy up media manager, add calling func to errors, build-script ↵Libravatar kim13
improvements (#1835) * media manager tidy-up: de-interface and remove unused PostDataFunc Signed-off-by: kim <grufwub@gmail.com> * remove last traces of media.Manager being an interface Signed-off-by: kim <grufwub@gmail.com> * update error to provide caller, allow tuneable via build tags Signed-off-by: kim <grufwub@gmail.com> * remove kim-specific build script changes Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * update build-script to support externally setting build variables Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-05-25[feature] Add List functionality (#1802)Libravatar tobi28
* start working on lists * further list work * test list db functions nicely * more work on lists * peepoopeepoo * poke * start list timeline func * we're getting there lads * couldn't be me working on stuff... could it? * hook up handlers * fiddling * weeee * woah * screaming, pissing * fix streaming being a whiny baby * lint, small test fix, swagger * tidying up, testing * fucked! by the linter * move timelines to state like a boss * add timeline start to tests using state * invalidate lists
2023-05-22[chore] update account statuses paging logic (#1814)Libravatar tobi1
2023-05-15[bugfix] Fix duplicating fields on profile edit (#1788)Libravatar tobi2
* [bugfix] Fix duplicating fields on profile edit * test non-duplicate fields
2023-05-12[feature] status refetch support (#1690)Libravatar kim13
* revamp http client to not limit requests, instead use sender worker Signed-off-by: kim <grufwub@gmail.com> * remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping Signed-off-by: kim <grufwub@gmail.com> * improve batch senders to keep popping recipients until remote URL found Signed-off-by: kim <grufwub@gmail.com> * fix recipient looping issue Signed-off-by: kim <grufwub@gmail.com> * move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP Signed-off-by: kim <grufwub@gmail.com> * first draft of status refetching logic Signed-off-by: kim <grufwub@gmail.com> * fix testrig to use new federation alloc func signature Signed-off-by: kim <grufwub@gmail.com> * fix log format directive Signed-off-by: kim <grufwub@gmail.com> * add status fetched_at migration Signed-off-by: kim <grufwub@gmail.com> * remove unused / unchecked for error types Signed-off-by: kim <grufwub@gmail.com> * add back the used type... Signed-off-by: kim <grufwub@gmail.com> * add separate internal getStatus() function for derefThread() that doesn't recurse Signed-off-by: kim <grufwub@gmail.com> * improved mention and media attachment error handling Signed-off-by: kim <grufwub@gmail.com> * fix log and error format directives Signed-off-by: kim <grufwub@gmail.com> * update account deref to match status deref changes Signed-off-by: kim <grufwub@gmail.com> * very small code formatting change to make things clearer Signed-off-by: kim <grufwub@gmail.com> * add more code comments Signed-off-by: kim <grufwub@gmail.com> * improved code commenting Signed-off-by: kim <grufwub@gmail.com> * only check for required further derefs if needed Signed-off-by: kim <grufwub@gmail.com> * improved cache invalidation Signed-off-by: kim <grufwub@gmail.com> * tweak cache restarting to use a (very small) backoff Signed-off-by: kim <grufwub@gmail.com> * small readability changes and fixes Signed-off-by: kim <grufwub@gmail.com> * fix account sync issues Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts + update account enrichment to accept already-passed accountable Signed-off-by: kim <grufwub@gmail.com> * remove secondary function declaration Signed-off-by: kim <grufwub@gmail.com> * normalise dereferencer get status / account behaviour, fix remaining tests Signed-off-by: kim <grufwub@gmail.com> * fix remaining rebase conflicts, finish commenting code Signed-off-by: kim <grufwub@gmail.com> * appease the linter Signed-off-by: kim <grufwub@gmail.com> * add source file header Signed-off-by: kim <grufwub@gmail.com> * update to use TIMESTAMPTZ column type instead of just TIMESTAMP Signed-off-by: kim <grufwub@gmail.com> * don't pass in 'updated_at' to UpdateEmoji() Signed-off-by: kim <grufwub@gmail.com> * use new ap.Resolve{Account,Status}able() functions Signed-off-by: kim <grufwub@gmail.com> * remove the somewhat confusing rescoping of the same variable names Signed-off-by: kim <grufwub@gmail.com> * update migration file name, improved database delete error returns Signed-off-by: kim <grufwub@gmail.com> * formatting Signed-off-by: kim <grufwub@gmail.com> * improved multi-delete database functions to minimise DB calls Signed-off-by: kim <grufwub@gmail.com> * remove unused type Signed-off-by: kim <grufwub@gmail.com> * fix delete statements Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-05-09[feature] Enable federation in/out of profile PropertyValue fields (#1722)Libravatar tobi11
Co-authored-by: kim <grufwub@gmail.com> Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-07[bugfix] Punycode fixes (#1743)Libravatar tobi1
Co-authored-by: kim <grufwub@gmail.com> Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-06[feature] Implement the preferences client API (#1740)Libravatar Daenney2
This adds the preferences endpoint to our Mastodon Client API implementation. It's a read-only endpoint that returns a number of user preferences. Applications can query these settings when logging in a user (for the first time) to configure themselves.
2023-05-04[bugfix] Rework notifs to use min_id for paging up (#1734)Libravatar tobi2
2023-04-29[feature] Add GET endpoint for single notification (#1719)Libravatar tobi1
2023-04-29[bugfix] add From to email header (#1717)Libravatar Mal Hancock1
* add From to email header * update tests
2023-04-28[performance] improved request batching (removes need for queueing) (#1687)Libravatar kim6
* revamp http client to not limit requests, instead use sender worker Signed-off-by: kim <grufwub@gmail.com> * remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping Signed-off-by: kim <grufwub@gmail.com> * improve batch senders to keep popping recipients until remote URL found Signed-off-by: kim <grufwub@gmail.com> * fix recipient looping issue Signed-off-by: kim <grufwub@gmail.com> * fix missing mutex unlock Signed-off-by: kim <grufwub@gmail.com> * move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP Signed-off-by: kim <grufwub@gmail.com> * slight code reformatting Signed-off-by: kim <grufwub@gmail.com> * a whitespace Signed-off-by: kim <grufwub@gmail.com> * remove unused code Signed-off-by: kim <grufwub@gmail.com> * add missing license headers Signed-off-by: kim <grufwub@gmail.com> * fix request backoff calculation Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-04-26[bugfix] Fix remaining mangled URI escaping issues in statuses + accounts ↵Libravatar tobi1
(#1712) * start fiddling with normalize + extract functions * normalize attachment name (image description) * NormalizeAccountableSummary * normalize summary + name
2023-04-10[feature] Receive notification when followed account posts (if desired) (#1680)Libravatar tobi8
* start working on notifs for new posts * tidy up a bit * update swagger * carry over show reblogs + notify from follow req * test notify on status post * update column slice * dedupe update logic + add tests * fix own boosts not being timelined * avoid type check, passing unnecessary accounts * remove unnecessary 'inReplyToID' check * add a couple todo's for future db functions
2023-04-06[bugfix/chore] Refactor timeline code (#1656)Libravatar tobi2
* start poking timelines * OK yes we're refactoring, but it's nothing like the last time so don't worry * more fiddling * update tests, simplify Get * thanks linter, you're the best, mwah mwah kisses * do a bit more tidying up * start buggering about with the prepare function * fix little oopsie * start merging lists into 1 * ik heb een heel zwaar leven nee nee echt waar * hey it works we did it reddit * regenerate swagger docs * tidy up a wee bit * adjust paging * fix little error, remove unused functions
2023-04-06[bugfix] Normalize status content (don't parse status content as IRI) (#1665)Libravatar tobi1
* start fannying about * finish up Normalize * tidy up * pin to tag * move errors about just a little bit
2023-04-06[bugfix] Always serialize orderedItems as array (#1673)Libravatar tobi1
2023-03-31[bugfix] Fix multiple "updated_at" columns for media updates (#1660)Libravatar tobi2
* [bugfix] Fix multiple "updated_at" columns for media updates * silly unrelated race condition
2023-03-31[bugfix] Fix report serialization errors caused by user delete (#1659)Libravatar tobi2
* [bugfix] Fix report serialization errors caused by user delete * fix tests
2023-03-28[performance] refactoring + add fave / follow / request / visibility caching ↵Libravatar kim27
(#1607) * refactor visibility checking, add caching for visibility * invalidate visibility cache items on account / status deletes * fix requester ID passed to visibility cache nil ptr * de-interface caches, fix home / public timeline caching + visibility * finish adding code comments for visibility filter * fix angry goconst linter warnings * actually finish adding filter visibility code comments for timeline functions * move home timeline status author check to after visibility * remove now-unused code * add more code comments * add TODO code comment, update printed cache start names * update printed cache names on stop * start adding separate follow(request) delete db functions, add specific visibility cache tests * add relationship type caching * fix getting local account follows / followed-bys, other small codebase improvements * simplify invalidation using cache hooks, add more GetAccountBy___() functions * fix boosting to return 404 if not boostable but no error (to not leak status ID) * remove dead code * improved placement of cache invalidation * update license headers * add example follow, follow-request config entries * add example visibility cache configuration to config file * use specific PutFollowRequest() instead of just Put() * add tests for all GetAccountBy() * add GetBlockBy() tests * update block to check primitive fields * update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests * fix copy-pasted code * update envparsing test * whitespace * fix bun struct tag * add license header to gtscontext * fix old license header * improved error creation to not use fmt.Errorf() when not needed * fix various rebase conflicts, fix account test * remove commented-out code, fix-up mention caching * fix mention select bun statement * ensure mention target account populated, pass in context to customrenderer logging * remove more uncommented code, fix typeutil test * add statusfave database model caching * add status fave cache configuration * add status fave cache example config * woops, catch missed error. nice catch linter! * add back testrig panic on nil db * update example configuration to match defaults, slight tweak to cache configuration defaults * update envparsing test with new defaults * fetch followingget to use the follow target account * use accounnt.IsLocal() instead of empty domain check * use constants for the cache visibility type check * use bun.In() for notification type restriction in db query * include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable()) * use bun query building for nested select statements to ensure working with postgres * update public timeline future status checks to match visibility filter * same as previous, for home timeline * update public timeline tests to dynamically check for appropriate statuses * migrate accounts to allow unique constraint on public_key * provide minimal account with publicKey --------- Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-20[chore] Refactor account deleting/block logic, tidy up some other processing ↵Libravatar tobi20
things (#1599) * start refactoring account deletion * update to use state.DB * further messing about * some more tidying up * more tidying, cleaning, nice-making * further adventures in refactoring and the woes of technical debt * update fr accept/reject * poking + prodding * fix up deleting * create fave uri * don't log using requestingAccount.ID because it might be nil * move getBookmarks function * use exists query to check for status bookmark * use deletenotifications func * fiddle * delete follow request notif * split up some db functions * Fix possible nil pointer panic * fix more possible nil pointers * fix license headers * warn when follow missing (target) account * return wrapped err when bookmark/fave models can't be retrieved * simplify self account delete * warn log likely race condition * de-sillify status delete loop * move error check due north * warn when unfollowSideEffects has no target account * warn when no boost account is found * warn + dump follow when no account * more warnings * warn on fave account not set * move for loop inside anonymous function * fix funky logic * don't remove mutual account items on block; do make sure unfollow occurs in both directions!
2023-03-19[feature] Email notifications for new / closed moderation reports (#1628)Libravatar tobi5
* start fiddling about with email sending to allow multiple recipients * do some fiddling * notifs working * notify on closed report * finishing up * envparsing * use strings.ContainsAny
2023-03-14[feature] Allow admins to send test emails (#1620)Libravatar tobi3
* [feature] Allow admins to send test emails * implement unwrap on new error type * add + use gtserror types * GoToSocial Email Test -> GoToSocial Test Email * add + use getInstance db call * removed unused "unknown" error type
2023-03-12[chore] Improve copyright header handling (#1608)Libravatar Daenney90
* [chore] Remove years from all license headers Years or year ranges aren't required in license headers. Many projects have removed them in recent years and it avoids a bit of yearly toil. In many cases our copyright claim was also a bit dodgy since we added the 2021-2023 header to files created after 2021 but you can't claim copyright into the past that way. * [chore] Add license header check This ensures a license header is always added to any new file. This avoids maintainers/reviewers needing to remember to check for and ask for it in case a contribution doesn't include it. * [chore] Add missing license headers * [chore] Further updates to license header * Use the more common // indentend comment format * Remove the hack we had for the linter now that we use the // format * Add SPDX license identifier
2023-03-11[feature] Support multiple subscriptions on single websocket connection (#1489)Libravatar darrinsmart2
- Allow Oauth authentication on websocket endpoint - Make streamType query parameter optional - Read websocket commands from client and update subscriptions
2023-03-09[feature] Provide .well-known/host-meta endpoint (#1604)Libravatar Daenney1
* [feature] Provide .well-known/host-meta endpoint This adds the host-meta endpoint as Mastodon clients use this to discover the API domain to use when the host and account domains aren't the same. * Address review comments
2023-03-06[feature] Add support for profile fields (#1483)Libravatar zowhoey1
* Add go-playground/form pkg * [feature] Add support for profile fields * Add field attributes test * Validate profile fields form * Add profile field validation tests * Add Field Attributes definition to swagger --------- Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2023-03-04use updateattachment when updating to ensure cache is invalidated (#1587)Libravatar kim1
Signed-off-by: kim <grufwub@gmail.com>
2023-03-03[performance] cache media attachments (#1525)Libravatar kim3
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly * add media attachment caching, slightly tweak default cache config * further tweak default cache config values * replace other media attachment db calls to go through cache * update envparsing test * fix delete media attachment sql * fix media sql query * invalidate cached media entries during status create / update * fix envparsing test * fix typo in panic log message... * add 'updated_at' column during UpdateAttachment * remove unused func --------- Signed-off-by: kim <grufwub@gmail.com>
2023-03-03[bugfix] Federate status delete using just the URI (#1584)Libravatar tobi1
2023-03-03[bugfix] Fix unpinning statuses not working (#1582)Libravatar tobi1
And also fix unpinning/pinning potentially leaking the ID of followers-only statuses through returning 422 instead of 404. Also tests!
2023-03-02[feature] Advertise rich text formats, support content_type field (#1370)Libravatar Vyr Cossont4
* Advertise rich text formats, support content_type field * Update JSON in instance patch tests * Replace format with content_type everywhere * update migration to work with both pg and sqlite * regenerate swagger docs * update instance serialization + tests * fix up * learn to code tobi please, i'm begging you --------- Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-01[chore] move client/federator workerpools to Workers{} (#1575)Libravatar kim64
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly * improve code comment * change back testrig default log level * un-comment-out TestAnnounceTwice() and fix --------- Signed-off-by: kim <grufwub@gmail.com> Reviewed-by: tobi
2023-03-01[feature] Federate pinned posts (aka `featuredCollection`) in and out (#1560)Libravatar tobi5
* start fiddling * the ol' fiddle + update * start working on fetching statuses * poopy doopy doo where r u uwu * further adventures in featuring statuses * finishing up * fmt * simply status unpin loop * move empty featured check back to caller function * remove unnecessary log.WithContext calls * remove unnecessary IsIRI() checks * add explanatory comment about status URIs * change log level to error * better test names
2023-02-25[feature] Client API endpoints + v. basic web view for pinned posts (#1547)Libravatar tobi6
* implement status pin client api + web handler * make test names + comments more descriptive * don't use separate table for status pins * remove unused add + remove checking * tidy up + add some more tests
2023-02-23[bugfix] Fix deleted status causing issues when getting bookmark (#1551)Libravatar tobi2
* [bugfix] Delete bookmark when status deleted * [chore] Give bookmark processing func some love * fix paging + embetter tests
2023-02-22[chore] Deinterface processor and subprocessors (#1501)Libravatar tobi120
* [chore] Deinterface processor and subprocessors * expose subprocessors via function calls * missing license header
2023-02-20[bugfix] Fix failure to look up remote profiles with duplicate emojis in ↵Libravatar Sam Lade1
some cases (#1534) * Tidy up emoji parsing on profile submission Don't bother reparsing for emoji unless one of the fields that can have emoji in it has changed. Deduplicate emoji between the display name and profile note - I'm not sure whether this was hurting anything, but better safe. * Deduplicate emoji when parsing remote accounts Some servers - Misskey at least - don't deduplicate emoji, so it's possible to get an account which has the same emoji used in both the display name and note and therefore includes that emoji twice in its metadata. When we start trying to put those into our database, we run into a uniqueness constraint and fall over. This change just deduplicates at the point of construction of an account.
2023-02-17[feature] Add a request ID and include it in logs (#1476)Libravatar Daenney12
This adds a lightweight form of tracing to GTS. Each incoming request is assigned a Request ID which we then pass on and log in all our log lines. Any function that gets called downstream from an HTTP handler should now emit a requestID=value pair whenever it logs something. Co-authored-by: kim <grufwub@gmail.com>
2023-02-16[bugfix] Set 'discoverable' properly on API accounts (#1511)v0.7.0Libravatar tobi1
2023-02-14[bugfix] Return empty result rather than 500 error when searching for ↵Libravatar tobi1
blocked domains (#1498) * [bugfix] Return empty result when searching for blocked domains * add tests
2023-02-13[performance] processing media and scheduled jobs improvements (#1482)Libravatar kim5
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling * improved code comment * fix worker tryUntil function, update go-runners/go-sched * make preprocess functions package public, use these where possible to stop doubled up processing * remove separate emoji worker pool * limit calls to time.Now() during media preprocessing * use Processor{} to manage singular runtime of processing media * ensure workers get started when media manager is used * improved error setting in processing media, fix media test * port changes from processingmedia to processing emoji * finish code commenting * finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced * linterrrrrrrrrrrrrrrr --------- Signed-off-by: kim <grufwub@gmail.com>
2023-02-12[bugfix] Set appropriate cache-control when using presigned s3 links (#1480)Libravatar tobi1
2023-02-11[chore/performance] Update media prune logic, add extra CLI command (#1474)v0.7.0-rc2Libravatar tobi2
* start updating media prune stuff a wee bit * continue prune / uncache work * more tidying + consistency stuff * add prune CLI command * docs * arg
2023-02-07[bugfix] don't trash emoji in profile fields on edit (#1440)Libravatar Sam Lade1
2023-02-07[bugfix] fix file range length calculation being off by 1 (#1448)Libravatar kim1
* small formatting change * fix range handling new length calculation --------- Signed-off-by: kim <grufwub@gmail.com>
2023-02-06Ignore missing files when cleaning up media (#1435)Libravatar Sam Lade1
generated by cgit v1.2.3 (git 2.46.0) at 2025-07-14 08:16:24 +0000