summaryrefslogtreecommitdiff
path: root/internal/processing/fromfederator_test.go
AgeCommit message (Collapse)AuthorFiles
2023-08-09[feature] List replies policy, refactor async workers (#2087)Libravatar tobi1
* Add/update some DB functions. * move async workers into subprocessor * rename FromFederator -> FromFediAPI * update home timeline check to include check for current status first before moving to parent status * change streamMap to pointer to mollify linter * update followtoas func signature * fix merge * remove errant debug log * don't use separate errs.Combine() check to wrap errs * wrap parts of workers functionality in sub-structs * populate report using new db funcs * embed federator (tiny bit tidier) * flesh out error msg, add continue(!) * fix other error messages to be more specific * better, nicer * give parseURI util function a bit more util * missing headers * use pointers for subprocessors
2023-08-07[chore] Use generic pointer function (#2080)Libravatar Daenney1
This replaces the different $TypePtr functions with a generic implementation.
2023-05-12[feature] status refetch support (#1690)Libravatar kim1
* revamp http client to not limit requests, instead use sender worker Signed-off-by: kim <grufwub@gmail.com> * remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping Signed-off-by: kim <grufwub@gmail.com> * improve batch senders to keep popping recipients until remote URL found Signed-off-by: kim <grufwub@gmail.com> * fix recipient looping issue Signed-off-by: kim <grufwub@gmail.com> * move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP Signed-off-by: kim <grufwub@gmail.com> * first draft of status refetching logic Signed-off-by: kim <grufwub@gmail.com> * fix testrig to use new federation alloc func signature Signed-off-by: kim <grufwub@gmail.com> * fix log format directive Signed-off-by: kim <grufwub@gmail.com> * add status fetched_at migration Signed-off-by: kim <grufwub@gmail.com> * remove unused / unchecked for error types Signed-off-by: kim <grufwub@gmail.com> * add back the used type... Signed-off-by: kim <grufwub@gmail.com> * add separate internal getStatus() function for derefThread() that doesn't recurse Signed-off-by: kim <grufwub@gmail.com> * improved mention and media attachment error handling Signed-off-by: kim <grufwub@gmail.com> * fix log and error format directives Signed-off-by: kim <grufwub@gmail.com> * update account deref to match status deref changes Signed-off-by: kim <grufwub@gmail.com> * very small code formatting change to make things clearer Signed-off-by: kim <grufwub@gmail.com> * add more code comments Signed-off-by: kim <grufwub@gmail.com> * improved code commenting Signed-off-by: kim <grufwub@gmail.com> * only check for required further derefs if needed Signed-off-by: kim <grufwub@gmail.com> * improved cache invalidation Signed-off-by: kim <grufwub@gmail.com> * tweak cache restarting to use a (very small) backoff Signed-off-by: kim <grufwub@gmail.com> * small readability changes and fixes Signed-off-by: kim <grufwub@gmail.com> * fix account sync issues Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts + update account enrichment to accept already-passed accountable Signed-off-by: kim <grufwub@gmail.com> * remove secondary function declaration Signed-off-by: kim <grufwub@gmail.com> * normalise dereferencer get status / account behaviour, fix remaining tests Signed-off-by: kim <grufwub@gmail.com> * fix remaining rebase conflicts, finish commenting code Signed-off-by: kim <grufwub@gmail.com> * appease the linter Signed-off-by: kim <grufwub@gmail.com> * add source file header Signed-off-by: kim <grufwub@gmail.com> * update to use TIMESTAMPTZ column type instead of just TIMESTAMP Signed-off-by: kim <grufwub@gmail.com> * don't pass in 'updated_at' to UpdateEmoji() Signed-off-by: kim <grufwub@gmail.com> * use new ap.Resolve{Account,Status}able() functions Signed-off-by: kim <grufwub@gmail.com> * remove the somewhat confusing rescoping of the same variable names Signed-off-by: kim <grufwub@gmail.com> * update migration file name, improved database delete error returns Signed-off-by: kim <grufwub@gmail.com> * formatting Signed-off-by: kim <grufwub@gmail.com> * improved multi-delete database functions to minimise DB calls Signed-off-by: kim <grufwub@gmail.com> * remove unused type Signed-off-by: kim <grufwub@gmail.com> * fix delete statements Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-03-28[performance] refactoring + add fave / follow / request / visibility caching ↵Libravatar kim1
(#1607) * refactor visibility checking, add caching for visibility * invalidate visibility cache items on account / status deletes * fix requester ID passed to visibility cache nil ptr * de-interface caches, fix home / public timeline caching + visibility * finish adding code comments for visibility filter * fix angry goconst linter warnings * actually finish adding filter visibility code comments for timeline functions * move home timeline status author check to after visibility * remove now-unused code * add more code comments * add TODO code comment, update printed cache start names * update printed cache names on stop * start adding separate follow(request) delete db functions, add specific visibility cache tests * add relationship type caching * fix getting local account follows / followed-bys, other small codebase improvements * simplify invalidation using cache hooks, add more GetAccountBy___() functions * fix boosting to return 404 if not boostable but no error (to not leak status ID) * remove dead code * improved placement of cache invalidation * update license headers * add example follow, follow-request config entries * add example visibility cache configuration to config file * use specific PutFollowRequest() instead of just Put() * add tests for all GetAccountBy() * add GetBlockBy() tests * update block to check primitive fields * update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests * fix copy-pasted code * update envparsing test * whitespace * fix bun struct tag * add license header to gtscontext * fix old license header * improved error creation to not use fmt.Errorf() when not needed * fix various rebase conflicts, fix account test * remove commented-out code, fix-up mention caching * fix mention select bun statement * ensure mention target account populated, pass in context to customrenderer logging * remove more uncommented code, fix typeutil test * add statusfave database model caching * add status fave cache configuration * add status fave cache example config * woops, catch missed error. nice catch linter! * add back testrig panic on nil db * update example configuration to match defaults, slight tweak to cache configuration defaults * update envparsing test with new defaults * fetch followingget to use the follow target account * use accounnt.IsLocal() instead of empty domain check * use constants for the cache visibility type check * use bun.In() for notification type restriction in db query * include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable()) * use bun query building for nested select statements to ensure working with postgres * update public timeline future status checks to match visibility filter * same as previous, for home timeline * update public timeline tests to dynamically check for appropriate statuses * migrate accounts to allow unique constraint on public_key * provide minimal account with publicKey --------- Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-12[chore] Improve copyright header handling (#1608)Libravatar Daenney1
* [chore] Remove years from all license headers Years or year ranges aren't required in license headers. Many projects have removed them in recent years and it avoids a bit of yearly toil. In many cases our copyright claim was also a bit dodgy since we added the 2021-2023 header to files created after 2021 but you can't claim copyright into the past that way. * [chore] Add license header check This ensures a license header is always added to any new file. This avoids maintainers/reviewers needing to remember to check for and ask for it in case a contribution doesn't include it. * [chore] Add missing license headers * [chore] Further updates to license header * Use the more common // indentend comment format * Remove the hack we had for the linter now that we use the // format * Add SPDX license identifier
2023-03-01[chore] move client/federator workerpools to Workers{} (#1575)Libravatar kim1
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly * improve code comment * change back testrig default log level * un-comment-out TestAnnounceTwice() and fix --------- Signed-off-by: kim <grufwub@gmail.com> Reviewed-by: tobi
2023-02-25[feature] Client API endpoints + v. basic web view for pinned posts (#1547)Libravatar tobi1
* implement status pin client api + web handler * make test names + comments more descriptive * don't use separate table for status pins * remove unused add + remove checking * tidy up + add some more tests
2023-02-22[chore] Deinterface processor and subprocessors (#1501)Libravatar tobi1
* [chore] Deinterface processor and subprocessors * expose subprocessors via function calls * missing license header
2023-01-05[chore] Update/add license headers for 2023 (#1304)Libravatar tobi1
2023-01-02[chore] The Big Middleware and API Refactor (tm) (#1250)Libravatar tobi1
* interim commit: start refactoring middlewares into package under router * another interim commit, this is becoming a big job * another fucking massive interim commit * refactor bookmarks to new style * ambassador, wiz zeze commits you are spoiling uz * she compiles, we're getting there * we're just normal men; we're just innocent men * apiutil * whoopsie * i'm glad noone reads commit msgs haha :blob_sweat: * use that weirdo go-bytesize library for maxMultipartMemory * fix media module paths
2022-11-30[bugfix] Don't call `strings.ToLower()` on usernames when selecting account ↵Libravatar tobi1
by domain+username (#1190) * don't lowercase account username when doing a select * test getting remote user with uppercase username
2022-10-03[performance] add user cache and database (#879)Libravatar tobi1
* go fmt * add + use user cache and database * fix import * update tests * remove unused relation
2022-09-23[feature] Allow delivery to sharedInboxes where possible (#847)Libravatar tobi1
* update Activity * add instance-deliver-to-shared-inboxes setting * update activity version again * add SharedInboxURI field to accounts * serdes for endpoints/sharedInbox * deliver to sharedInbox if one is available * update tests * only assign shared inbox if shared domain * look for shared inbox if currently nil * go fmt * finger to get params.RemoteAccountID if necessary * make comments clearer * compare dns more consistently
2022-09-21[chore] Tidy up status deletion, remove from cache too (#845)Libravatar tobi1
* add func for deleting status from db + cache * move deletes entirely back to processor and also only do a delete if the requesting account owns the item being deleted * tidy up unboost processing * delete status more efficiently * fix wrong account id on remote test attachments * fix federator test
2022-08-31[chore] Test fixes (#788)Libravatar tobi1
* use 'test' value for testrig storage backend * update test dependency * add WaitFor func in testrig * use WaitFor function instead of time.Sleep * tidy up tests * make SentMessages a sync.map * go fmt
2022-08-15[chore] Update bun / sqlite versions; update gtsmodels (#754)Libravatar tobi1
* upstep bun and sqlite versions * allow specific columns to be updated in the db * only update necessary columns for user * bit tidier * only update necessary fields of media_attachment * only update relevant instance fields * update tests * update only specific account columns * use bool pointers on gtsmodels includes attachment, status, account, user * update columns more selectively * test all default fields on new account insert * updating remaining bools on gtsmodels * initialize pointer fields when extracting AP emoji * copy bools properly * add copyBoolPtr convenience function + test it * initialize false bool ptrs a bit more neatly
2022-06-11[chore] Webfinger rework (#627)Libravatar tobi1
* move finger to dereferencer * totally break GetRemoteAccount * start reworking finger func a bit * start reworking getRemoteAccount a bit * move mention parts to namestring * rework webfingerget * use util function to extract webfinger parts * use accountDomain * rework finger again, final form * just a real nasty commit, the worst * remove refresh from account * use new ASRepToAccount signature * fix incorrect debug call * fix for new getRemoteAccount * rework GetRemoteAccount * start updating tests to remove repetition * break a lot of tests Move shared test logic into the testrig, rather than having it scattered all over the place. This allows us to just mock the transport controller once, and have all tests use it (unless they need not to for some other reason). * fix up tests to use main mock httpclient * webfinger only if necessary * cheeky linting with the lads * update mentionName regex recognize instance accounts * don't finger instance accounts * test webfinger part extraction * increase default worker count to 4 per cpu * don't repeat regex parsing * final search for discovered accountDomain * be more permissive in namestring lookup * add more extraction tests * simplify GetParseMentionFunc * skip long search if local account * fix broken test
2022-04-15[feature] Web profile pages for accounts (#449)Libravatar tobi1
* add default avatars * allow webModule to error * return errWithCode from account get * add AccountGetLocalByUsername * check nil requesting account * add timestampShort function for just month/year * move loading logic to New + add default avatars * add profile page view * update swagger docs * add excludeReblogs to GetAccountStatuses * ignore casing when selecting local account by username * appropriate redirects * css fiddling * add 'about' heading * adjust thread page to work with routing * return AP representation if requested + authorized * simplify auth check * go fmt * golangci-lint ignore math/rand
2021-12-20Extend license notices to 2022 (#354)Libravatar tobi1
2021-11-22Fix streamed messages ending up in wrong timeline(s) (#325)Libravatar tobi1
* define timeline consts * remove double stream of status * change test stream creation up a bit * stream messages more selectively * add test for streaming new status creation via clientAPI * tidy code + comments a bit * tidy up tests * make sure new status isn't streamed to public
2021-10-24Serve `outbox` for Actor (#289)Libravatar tobi1
* add statusesvisible convenience function * add minID + onlyPublic to account statuses get * move swagger collection stuff to common * start working on Outbox GETting * move functions into federationProcessor * outboxToASCollection * add statusesvisible convenience function * add minID + onlyPublic to account statuses get * move swagger collection stuff to common * start working on Outbox GETting * move functions into federationProcessor * outboxToASCollection * bit more work on outbox paging * wrapNoteInCreate function * test + hook up the processor functions * don't do prev + next links on empty reply * test get outbox through api * don't fail on no status entries * add outbox implementation doc * typo
2021-10-10Handle forwarded messages (#273)Libravatar tobi1
* correct path of foss_satan * add APIri and notes * test create forward note * rename target => receiving account * split up create into separate funcs * update extractFromCtx * tidy up from federator processing * foss satan => http not https * check if status in db * mock dereference of status from IRI * add forward message deref test * update test with activities * add remote_account_2 to test rig
2021-10-01Follow request auto approval (#259)Libravatar tobi1
* start messing about * fiddle more * Tests & fiddling
2021-09-30handle remote account deletion more systematically (#254)Libravatar tobi1
2021-09-27Weird notif issue (#248)Libravatar tobi1
* start working on weird issue * go fmt ./... * more tests
2021-09-14Fix mentions not notifying (#230)Libravatar tobi1
* set default privacy for new accounts * teshts * found it * tiny change * aaaa
2021-09-04tests + announce notification fix (#193)Libravatar tobi1