summaryrefslogtreecommitdiff
path: root/internal/federation
AgeCommit message (Collapse)AuthorFiles
2022-09-26[bugfix] add so, many, nil, checks (#853)Libravatar kim1
* add so, many, nil, checks. * remove comment
2022-09-26[bugfix] panic during child thread iteration (#852)Libravatar kim1
* *actually* start at top of stack loop on find remote child, fix iter indexing Signed-off-by: kim <grufwub@gmail.com> * add improved code comment Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-09-25[bugfix] update thread iterators to not use recursion (#851)Libravatar kim6
* update thread iterators to not use recursion, rewrote both Signed-off-by: kim <grufwub@gmail.com> * fix endless descendant deref, don't error if fetching existing status Signed-off-by: kim <grufwub@gmail.com> * don't refetch remote ancestor statuses, improve descendant iter commenting Signed-off-by: kim <grufwub@gmail.com> * move collection page next logic so we capture first page of entities Signed-off-by: kim <grufwub@gmail.com> * improve log format argument quoting Signed-off-by: kim <grufwub@gmail.com> * improve code commenting of collection paging Signed-off-by: kim <grufwub@gmail.com> * only dereference announce's originating status if _not_ local. update DereferenceThread() signature. cleanup searchStatusByURI() Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-09-23[feature] Allow delivery to sharedInboxes where possible (#847)Libravatar tobi5
* update Activity * add instance-deliver-to-shared-inboxes setting * update activity version again * add SharedInboxURI field to accounts * serdes for endpoints/sharedInbox * deliver to sharedInbox if one is available * update tests * only assign shared inbox if shared domain * look for shared inbox if currently nil * go fmt * finger to get params.RemoteAccountID if necessary * make comments clearer * compare dns more consistently
2022-09-21[chore] Tidy up status deletion, remove from cache too (#845)Libravatar tobi1
* add func for deleting status from db + cache * move deletes entirely back to processor and also only do a delete if the requesting account owns the item being deleted * tidy up unboost processing * delete status more efficiently * fix wrong account id on remote test attachments * fix federator test
2022-09-12[feature] Fetch + display custom emoji in statuses from remote instances (#807)Libravatar tobi5
* start implementing remote emoji fetcher * update status where pk * aaa * tidy up a little * check size limits for emojis * thank you linter, i love you <3 * update swagger docs * add emoji dereference test * make emoji max sizes configurable * normalize db.ErrAlreadyExists
2022-09-02[performance] cache account db lookups by public key URI (#795)Libravatar kim1
Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-09-02[performance] use GetAccountByUsernameDomain() for local account lookups to ↵Libravatar kim4
rely on cache (#793) Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-08-31[chore] Test fixes (#788)Libravatar tobi2
* use 'test' value for testrig storage backend * update test dependency * add WaitFor func in testrig * use WaitFor function instead of time.Sleep * tidy up tests * make SentMessages a sync.map * go fmt
2022-08-20[bugfix] Fix potential dereference of accounts on own instance (#757)Libravatar tobi2
* add GetAccountByUsernameDomain * simplify search * add escape to not deref accounts on own domain * check if local + we have account by ap uri
2022-08-15[chore] Update bun / sqlite versions; update gtsmodels (#754)Libravatar tobi3
* upstep bun and sqlite versions * allow specific columns to be updated in the db * only update necessary columns for user * bit tidier * only update necessary fields of media_attachment * only update relevant instance fields * update tests * update only specific account columns * use bool pointers on gtsmodels includes attachment, status, account, user * update columns more selectively * test all default fields on new account insert * updating remaining bools on gtsmodels * initialize pointer fields when extracting AP emoji * copy bools properly * add copyBoolPtr convenience function + test it * initialize false bool ptrs a bit more neatly
2022-07-19[chore] use our own logging implementation (#716)Libravatar kim19
* first commit Signed-off-by: kim <grufwub@gmail.com> * replace logging with our own log library Signed-off-by: kim <grufwub@gmail.com> * fix imports Signed-off-by: kim <grufwub@gmail.com> * fix log imports Signed-off-by: kim <grufwub@gmail.com> * add license text Signed-off-by: kim <grufwub@gmail.com> * fix package import cycle between config and log package Signed-off-by: kim <grufwub@gmail.com> * fix empty kv.Fields{} being passed to WithFields() Signed-off-by: kim <grufwub@gmail.com> * fix uses of log.WithFields() with whitespace issues and empty slices Signed-off-by: kim <grufwub@gmail.com> * *linter related grumbling* Signed-off-by: kim <grufwub@gmail.com> * gofmt the codebase! also fix more log.WithFields() formatting issues Signed-off-by: kim <grufwub@gmail.com> * update testrig code to match new changes Signed-off-by: kim <grufwub@gmail.com> * fix error wrapping in non fmt.Errorf function Signed-off-by: kim <grufwub@gmail.com> * add benchmarking of log.Caller() vs non-cached Signed-off-by: kim <grufwub@gmail.com> * fix syslog tests, add standard build tags to test runner to ensure consistency Signed-off-by: kim <grufwub@gmail.com> * make syslog tests more robust Signed-off-by: kim <grufwub@gmail.com> * fix caller depth arithmatic (is that how you spell it?) Signed-off-by: kim <grufwub@gmail.com> * update to use unkeyed fields in kv.Field{} instances Signed-off-by: kim <grufwub@gmail.com> * update go-kv library Signed-off-by: kim <grufwub@gmail.com> * update libraries list Signed-off-by: kim <grufwub@gmail.com> * fuck you linter get nerfed Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-03[feature] S3 support (#674)Libravatar Dominik Süß2
* feat: vendor minio client * feat: introduce storage package with s3 support * feat: serve s3 files directly this saves a lot of bandwith as the files are fetched from the object store directly * fix: use explicit local storage in tests * feat: integrate s3 storage with the main server * fix: add s3 config to cli tests * docs: explicitly set values in example config also adds license header to the storage package * fix: use better http status code on s3 redirect HTTP 302 Found is the best fit, as it signifies that the resource requested was found but not under its presumed URL 307/TemporaryRedirect would mean that this resource is usually located here, not in this case 303/SeeOther indicates that the redirection does not link to the requested resource but to another page * refactor: use context in storage driver interface
2022-06-19[chore] make tests more cacheable by avoiding time.Now() (#656)Libravatar tobi1
2022-06-11[chore] Refactor thread dereference a bit for clarity (#647)Libravatar tobi1
* refactor thread dereference a bit for clarity * lint for the lint gods
2022-06-11[chore] Webfinger rework (#627)Libravatar tobi13
* move finger to dereferencer * totally break GetRemoteAccount * start reworking finger func a bit * start reworking getRemoteAccount a bit * move mention parts to namestring * rework webfingerget * use util function to extract webfinger parts * use accountDomain * rework finger again, final form * just a real nasty commit, the worst * remove refresh from account * use new ASRepToAccount signature * fix incorrect debug call * fix for new getRemoteAccount * rework GetRemoteAccount * start updating tests to remove repetition * break a lot of tests Move shared test logic into the testrig, rather than having it scattered all over the place. This allows us to just mock the transport controller once, and have all tests use it (unless they need not to for some other reason). * fix up tests to use main mock httpclient * webfinger only if necessary * cheeky linting with the lads * update mentionName regex recognize instance accounts * don't finger instance accounts * test webfinger part extraction * increase default worker count to 4 per cpu * don't repeat regex parsing * final search for discovered accountDomain * be more permissive in namestring lookup * add more extraction tests * simplify GetParseMentionFunc * skip long search if local account * fix broken test
2022-06-08[feature] More consistent API error handling (#637)Libravatar tobi1
* update templates * start reworking api error handling * update template * return AP status at web endpoint if negotiated * start making api error handling much more consistent * update account endpoints to new error handling * use new api error handling in admin endpoints * go fmt ./... * use api error logic in app * use generic error handling in auth * don't export generic error handler * don't defer clearing session * user nicer error handling on oidc callback handler * tidy up the sign in handler * tidy up the token handler * use nicer error handling in blocksget * auth emojis endpoint * fix up remaining api endpoints * fix whoopsie during login flow * regenerate swagger docs * change http error logging to debug
2022-05-30[chore] Global server configuration overhaul (#575)Libravatar kim6
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct Signed-off-by: kim <grufwub@gmail.com> * improved code comment Signed-off-by: kim <grufwub@gmail.com> * linter Signed-off-by: kim <grufwub@gmail.com> * fix unmarshaling Signed-off-by: kim <grufwub@gmail.com> * remove kim's custom go compiler changes Signed-off-by: kim <grufwub@gmail.com> * generate setter and flag-name functions, implement these in codebase Signed-off-by: kim <grufwub@gmail.com> * update deps Signed-off-by: kim <grufwub@gmail.com> * small change Signed-off-by: kim <grufwub@gmail.com> * appease the linter... Signed-off-by: kim <grufwub@gmail.com> * move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync Signed-off-by: kim <grufwub@gmail.com> * lint Signed-off-by: kim <grufwub@gmail.com> * update code comments Signed-off-by: kim <grufwub@gmail.com> * fix merge issue Signed-off-by: kim <grufwub@gmail.com> * fix merge issue Signed-off-by: kim <grufwub@gmail.com> * improved version string (removes time + go version) Signed-off-by: kim <grufwub@gmail.com> * fix version string build to pass test script + consolidate logic in func Signed-off-by: kim <grufwub@gmail.com> * add license text, update config.Defaults comment Signed-off-by: kim <grufwub@gmail.com> * add license text to generated config helpers file Signed-off-by: kim <grufwub@gmail.com> * defer unlock on config.Set___(), to ensure unlocked on panic Signed-off-by: kim <grufwub@gmail.com> * make it more obvious which cmd flags are being attached Signed-off-by: kim <grufwub@gmail.com>
2022-05-27[chore] Add test for dereferencing Owncast service account (#613)Libravatar tobi2
2022-05-24[bugfix] Set refetch to true in iterateDescendants (#600)Libravatar tobi1
2022-05-23[bugfix] Fix multiple dereferences of boosted status causing media ↵Libravatar tobi9
duplication (#589) * add some announces to test models * start on announce test logic * test federatingDB.Announce * change signature of GetRemoteStatus * remove 'refresh' logic and replace it with refetch * go fmt * remove timeline manager from processor test * make zork created at determinate * test get account statuses * test get + serialize zork * make account keys determinate * make admin accountCreate time determinate * test account to as * init test config before test log * test status to frontend * remove daft Within check * hack around a bit * use index of slice
2022-05-23[security] Check all involved IRIs during block checking (#593)Libravatar tobi3
* tidy up context keys, add otherInvolvedIRIs * add ReplyToable interface * skip block check if we own the requesting domain * add block check for other involved IRIs * use cacheable status fetch * remove unused ContextActivity * remove unused ContextActivity * add helper for unique URIs * check through CCs and clean slice * add GetAccountIDForStatusURI * add GetAccountIDForAccountURI * check blocks on involved account * add statuses to tests * add some blocked tests * go fmt * extract Tos as well as CCs * test PostInboxRequestBodyHook * add some more testActivities * deduplicate involvedAccountIDs * go fmt * use cacheable db functions, remove new functions
2022-05-15[security] transport.Controller{} and transport.Transport{} security and ↵Libravatar kim5
performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-02[chore] Fix inbox iris test to use 'contains' instead of index (#529)Libravatar tobi1
2022-05-02[performance] Speed up some of the slower db queries (#523)Libravatar tobi1
* remove unnecessary LOWER() db calls * warn during slow db queries * use bundb built-in exists function * add db block test * update account block query * add domain block db test * optimize domain block query * fix implementing wrong test * exclude most columns when checking block * go fmt * remote more unnecessary use of LOWER()
2022-04-29[chore] Add Federatingactor.Send() tests and log call (#509)Libravatar tobi4
* expose testrig util functions * add tests for federatingActor * rename some suite vars
2022-04-29[chore] Inboxes for iri test (#508)Libravatar tobi4
* tidy up some federation tests * add missing license to following.go * give zork some followers, as a treat * test InboxesForIRI * Go fmt * update timeline tests
2022-04-28replace async client API / federator msg processing with worker pools (#497)Libravatar kim14
* replace async client API / federator msg processing with worker pools * appease our lord-and-saviour, the linter
2022-04-28[bugfix] Fix possible race condition in federatingdb (#490)Libravatar kim2
Signed-off-by: kim <grufwub@gmail.com>
2022-04-26[chore] Return more useful errors from auth failure (#494)v0.3.1Libravatar tobi3
* try rsa_sha256 sig algo first * return more informative errors from auth * adapt to reworked auth function
2022-04-24Revert "[chore] Tidy up federating db locks a tiny bit (#472)" (#479)Libravatar tobi1
This reverts commit 31c1067a548039e7d95f131b3972ecfe1f57f835.
2022-04-19[chore] Tidy up federating db locks a tiny bit (#472)Libravatar tobi1
2022-04-05[dependency] Update superseriousbusiness/activity dependency (#446)Libravatar tobi1
* bump superseriousbusiness/activity version * fetch list of follower inbox iris
2022-03-21[bugfix] Fix images not being processed correctly sometimes (#437)Libravatar tobi2
* bump exif-terminator to latest version * add and test giant turnip from turnip.farm * don't error if content property is nil
2022-03-07[feature] Clean up/uncache remote media (#407)Libravatar tobi1
* Add whereNotEmptyAndNotNull * Add GetRemoteOlderThanDays * Add GetRemoteOlderThanDays * Add PruneRemote to Manager interface * Start implementing PruneRemote * add new attachment + status to tests * fix up and test GetRemoteOlderThan * fix bad import * PruneRemote: return number pruned * add Cached column to mediaattachment * update + test pruneRemote * update mediaTest * use Cached column * upstep bun to latest version * embed structs in mediaAttachment * migrate mediaAttachment to new format * don't default cached to true * select only remote media * update db dependencies * step bun back to last working version * update pruneRemote to use Cached field * fix storage path of test attachments * add recache logic to manager * fix trimmed aspect ratio * test prune and recache * return errwithcode * tidy up different paths for emoji vs attachment * fix incorrect thumbnail type being stored * expose TransportController to media processor * implement tee-ing recached content * add thoughts of dog to test fedi attachments * test get remote files * add comment on PruneRemote * add postData cleanup to recache * test thumbnail fetching * add incredible diagram * go mod tidy * buffer pipes for recache streaming * test for client stops reading after 1kb * add media-remote-cache-days to config * add cron package * wrap logrus so it's available to cron * start and stop cron jobs gracefully
2022-02-22[feature] Add postDataCallbackFunc to allow cleanup (#408)Libravatar tobi2
2022-02-08hopefully fix potential race conditionLibravatar tsmethurst1
2022-02-08Merge branch 'main' into media_refactorLibravatar tsmethurst1
2022-02-08update outdated commentLibravatar tsmethurst1
2022-02-07[feature] add authorization to the already-existing authentication (#365)Libravatar Forest Johnson2
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize * adding authorization (email confirm, account approve, etc) to TokenCheck * revert un-needed changes to signin.go * oops what happened here * error css * add account.SuspendedAt check * remove redundant checks from oauth util Authed function * wip tests * tests passing * stop stripping useful information from ErrAlreadyExists * that feeling of scraping the dryer LINT off the screen * oops I didn't mean to get rid of this NewTestRouter function * make tests work with recorder * re-add ConfigureTemplatesWithGin to handle template path err Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-01-25ensure blocking calls to getRemoteAccount before showing stuff to clientLibravatar tsmethurst1
2022-01-25only update account in db if changedLibravatar tsmethurst1
2022-01-24change getaccount function signatureLibravatar tsmethurst1
2022-01-24use background context w/deadlineLibravatar tsmethurst1
2022-01-24update remote account get/deref logicLibravatar tsmethurst7
2022-01-23rework data function to provide filesizeLibravatar tsmethurst2
2022-01-16pass reader around instead of []byteLibravatar tsmethurst2
2022-01-11pass a function into the manager, start work on emojiLibravatar tsmethurst5
2022-01-10test the media manager a bit, add shutdown logicLibravatar tsmethurst5
2022-01-09add async testLibravatar tsmethurst1