summaryrefslogtreecommitdiff
path: root/testrig
AgeCommit message (Collapse)AuthorFiles
2023-05-28[chore] tidy up media manager, add calling func to errors, build-script ↵Libravatar kim3
improvements (#1835) * media manager tidy-up: de-interface and remove unused PostDataFunc Signed-off-by: kim <grufwub@gmail.com> * remove last traces of media.Manager being an interface Signed-off-by: kim <grufwub@gmail.com> * update error to provide caller, allow tuneable via build tags Signed-off-by: kim <grufwub@gmail.com> * remove kim-specific build script changes Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * update build-script to support externally setting build variables Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-05-25[feature] Configurable custom css length (#1827)Libravatar tobi1
* [feature] Make accounts custom css length configurable * test custom css validation
2023-05-25[feature] Add List functionality (#1802)Libravatar tobi4
* start working on lists * further list work * test list db functions nicely * more work on lists * peepoopeepoo * poke * start list timeline func * we're getting there lads * couldn't be me working on stuff... could it? * hook up handlers * fiddling * weeee * woah * screaming, pissing * fix streaming being a whiny baby * lint, small test fix, swagger * tidying up, testing * fucked! by the linter * move timelines to state like a boss * add timeline start to tests using state * invalidate lists
2023-05-21[bugfix] Add back removed ValidateRequest() before backoff-retry loop (#1805)v0.9.0-rc2Libravatar kim1
* add back removed ValidateRequest() before backoff-retry loop Signed-off-by: kim <grufwub@gmail.com> * include response body in error response log Signed-off-by: kim <grufwub@gmail.com> * improved error response body draining Signed-off-by: kim <grufwub@gmail.com> * add more code commenting Signed-off-by: kim <grufwub@gmail.com> * move new error response logic to gtserror, handle instead in transport.Transport{} impl Signed-off-by: kim <grufwub@gmail.com> * appease ye oh mighty linter Signed-off-by: kim <grufwub@gmail.com> * fix mockhttpclient not setting request in http response Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-05-15[bugfix] Fix duplicating fields on profile edit (#1788)Libravatar tobi1
* [bugfix] Fix duplicating fields on profile edit * test non-duplicate fields
2023-05-12[feature] status refetch support (#1690)Libravatar kim2
* revamp http client to not limit requests, instead use sender worker Signed-off-by: kim <grufwub@gmail.com> * remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping Signed-off-by: kim <grufwub@gmail.com> * improve batch senders to keep popping recipients until remote URL found Signed-off-by: kim <grufwub@gmail.com> * fix recipient looping issue Signed-off-by: kim <grufwub@gmail.com> * move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP Signed-off-by: kim <grufwub@gmail.com> * first draft of status refetching logic Signed-off-by: kim <grufwub@gmail.com> * fix testrig to use new federation alloc func signature Signed-off-by: kim <grufwub@gmail.com> * fix log format directive Signed-off-by: kim <grufwub@gmail.com> * add status fetched_at migration Signed-off-by: kim <grufwub@gmail.com> * remove unused / unchecked for error types Signed-off-by: kim <grufwub@gmail.com> * add back the used type... Signed-off-by: kim <grufwub@gmail.com> * add separate internal getStatus() function for derefThread() that doesn't recurse Signed-off-by: kim <grufwub@gmail.com> * improved mention and media attachment error handling Signed-off-by: kim <grufwub@gmail.com> * fix log and error format directives Signed-off-by: kim <grufwub@gmail.com> * update account deref to match status deref changes Signed-off-by: kim <grufwub@gmail.com> * very small code formatting change to make things clearer Signed-off-by: kim <grufwub@gmail.com> * add more code comments Signed-off-by: kim <grufwub@gmail.com> * improved code commenting Signed-off-by: kim <grufwub@gmail.com> * only check for required further derefs if needed Signed-off-by: kim <grufwub@gmail.com> * improved cache invalidation Signed-off-by: kim <grufwub@gmail.com> * tweak cache restarting to use a (very small) backoff Signed-off-by: kim <grufwub@gmail.com> * small readability changes and fixes Signed-off-by: kim <grufwub@gmail.com> * fix account sync issues Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts + update account enrichment to accept already-passed accountable Signed-off-by: kim <grufwub@gmail.com> * remove secondary function declaration Signed-off-by: kim <grufwub@gmail.com> * normalise dereferencer get status / account behaviour, fix remaining tests Signed-off-by: kim <grufwub@gmail.com> * fix remaining rebase conflicts, finish commenting code Signed-off-by: kim <grufwub@gmail.com> * appease the linter Signed-off-by: kim <grufwub@gmail.com> * add source file header Signed-off-by: kim <grufwub@gmail.com> * update to use TIMESTAMPTZ column type instead of just TIMESTAMP Signed-off-by: kim <grufwub@gmail.com> * don't pass in 'updated_at' to UpdateEmoji() Signed-off-by: kim <grufwub@gmail.com> * use new ap.Resolve{Account,Status}able() functions Signed-off-by: kim <grufwub@gmail.com> * remove the somewhat confusing rescoping of the same variable names Signed-off-by: kim <grufwub@gmail.com> * update migration file name, improved database delete error returns Signed-off-by: kim <grufwub@gmail.com> * formatting Signed-off-by: kim <grufwub@gmail.com> * improved multi-delete database functions to minimise DB calls Signed-off-by: kim <grufwub@gmail.com> * remove unused type Signed-off-by: kim <grufwub@gmail.com> * fix delete statements Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-05-09feat: initial tracing support (#1623)Libravatar Dominik Süß1
2023-05-09[feature] Enable federation in/out of profile PropertyValue fields (#1722)Libravatar tobi1
Co-authored-by: kim <grufwub@gmail.com> Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-08[chore/performance] Make sender multiplier configurable (#1750)Libravatar tobi1
2023-05-07[bugfix] Punycode fixes (#1743)Libravatar tobi1
Co-authored-by: kim <grufwub@gmail.com> Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-06[feature] Implement the preferences client API (#1740)Libravatar Daenney1
This adds the preferences endpoint to our Mastodon Client API implementation. It's a read-only endpoint that returns a number of user preferences. Applications can query these settings when logging in a user (for the first time) to configure themselves.
2023-04-28[performance] improved request batching (removes need for queueing) (#1687)Libravatar kim1
* revamp http client to not limit requests, instead use sender worker Signed-off-by: kim <grufwub@gmail.com> * remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping Signed-off-by: kim <grufwub@gmail.com> * improve batch senders to keep popping recipients until remote URL found Signed-off-by: kim <grufwub@gmail.com> * fix recipient looping issue Signed-off-by: kim <grufwub@gmail.com> * fix missing mutex unlock Signed-off-by: kim <grufwub@gmail.com> * move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP Signed-off-by: kim <grufwub@gmail.com> * slight code reformatting Signed-off-by: kim <grufwub@gmail.com> * a whitespace Signed-off-by: kim <grufwub@gmail.com> * remove unused code Signed-off-by: kim <grufwub@gmail.com> * add missing license headers Signed-off-by: kim <grufwub@gmail.com> * fix request backoff calculation Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-03-20[chore] Refactor account deleting/block logic, tidy up some other processing ↵Libravatar tobi2
things (#1599) * start refactoring account deletion * update to use state.DB * further messing about * some more tidying up * more tidying, cleaning, nice-making * further adventures in refactoring and the woes of technical debt * update fr accept/reject * poking + prodding * fix up deleting * create fave uri * don't log using requestingAccount.ID because it might be nil * move getBookmarks function * use exists query to check for status bookmark * use deletenotifications func * fiddle * delete follow request notif * split up some db functions * Fix possible nil pointer panic * fix more possible nil pointers * fix license headers * warn when follow missing (target) account * return wrapped err when bookmark/fave models can't be retrieved * simplify self account delete * warn log likely race condition * de-sillify status delete loop * move error check due north * warn when unfollowSideEffects has no target account * warn when no boost account is found * warn + dump follow when no account * more warnings * warn on fave account not set * move for loop inside anonymous function * fix funky logic * don't remove mutual account items on block; do make sure unfollow occurs in both directions!
2023-03-19[feature] Email notifications for new / closed moderation reports (#1628)Libravatar tobi1
* start fiddling about with email sending to allow multiple recipients * do some fiddling * notifs working * notify on closed report * finishing up * envparsing * use strings.ContainsAny
2023-03-12[chore] Improve copyright header handling (#1608)Libravatar Daenney16
* [chore] Remove years from all license headers Years or year ranges aren't required in license headers. Many projects have removed them in recent years and it avoids a bit of yearly toil. In many cases our copyright claim was also a bit dodgy since we added the 2021-2023 header to files created after 2021 but you can't claim copyright into the past that way. * [chore] Add license header check This ensures a license header is always added to any new file. This avoids maintainers/reviewers needing to remember to check for and ask for it in case a contribution doesn't include it. * [chore] Add missing license headers * [chore] Further updates to license header * Use the more common // indentend comment format * Remove the hack we had for the linter now that we use the // format * Add SPDX license identifier
2023-03-08[feature] Discover webfinger through host-meta (#1588)Libravatar Daenney1
* [feature] Discover webfinger through host-meta This implements a fallback for discovering the webfinger endpoint in case the /.well-known/webfinger endpoint wasn't properly redirected. Some instances do this because the recommendation used to be to use host-meta for the webfinger redirect in the before times. Closes #1558. * [bug] Ensure we only ever update cache on success * [chore] Move finger tests to their own place This adds a test suite for transport and moves the finger cache tests into there instead of abusing the search test suite. * [chore] cleanup the test a bit more We don't really need a separate function for the oddly located webfinger response as we check the full URL string anyway * Address review comments * [chore] update config example * [chore] access DB only through state in controller
2023-03-03[performance] cache media attachments (#1525)Libravatar kim1
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly * add media attachment caching, slightly tweak default cache config * further tweak default cache config values * replace other media attachment db calls to go through cache * update envparsing test * fix delete media attachment sql * fix media sql query * invalidate cached media entries during status create / update * fix envparsing test * fix typo in panic log message... * add 'updated_at' column during UpdateAttachment * remove unused func --------- Signed-off-by: kim <grufwub@gmail.com>
2023-03-01[chore] move client/federator workerpools to Workers{} (#1575)Libravatar kim7
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly * improve code comment * change back testrig default log level * un-comment-out TestAnnounceTwice() and fix --------- Signed-off-by: kim <grufwub@gmail.com> Reviewed-by: tobi
2023-03-01[chore/performance] simplify storage driver to use storage.Storage directly ↵Libravatar kim1
(#1576) * simply use storage.Storage, removing wrapping KVStore as we don't need KV store locking functionality Signed-off-by: kim <grufwub@gmail.com> * fix missing unwrapped function Signed-off-by: kim <grufwub@gmail.com> * add code comment Signed-off-by: kim <grufwub@gmail.com> * linter, please take my offering in peace Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-02-25[feature] Client API endpoints + v. basic web view for pinned posts (#1547)Libravatar tobi1
* implement status pin client api + web handler * make test names + comments more descriptive * don't use separate table for status pins * remove unused add + remove checking * tidy up + add some more tests
2023-02-22[chore] Deinterface processor and subprocessors (#1501)Libravatar tobi1
* [chore] Deinterface processor and subprocessors * expose subprocessors via function calls * missing license header
2023-02-17[feature] Add a request ID and include it in logs (#1476)Libravatar Daenney3
This adds a lightweight form of tracing to GTS. Each incoming request is assigned a Request ID which we then pass on and log in all our log lines. Any function that gets called downstream from an HTTP handler should now emit a requestID=value pair whenever it logs something. Co-authored-by: kim <grufwub@gmail.com>
2023-02-13[performance] processing media and scheduled jobs improvements (#1482)Libravatar kim2
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling * improved code comment * fix worker tryUntil function, update go-runners/go-sched * make preprocess functions package public, use these where possible to stop doubled up processing * remove separate emoji worker pool * limit calls to time.Now() during media preprocessing * use Processor{} to manage singular runtime of processing media * ensure workers get started when media manager is used * improved error setting in processing media, fix media test * port changes from processingmedia to processing emoji * finish code commenting * finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced * linterrrrrrrrrrrrrrrr --------- Signed-off-by: kim <grufwub@gmail.com>
2023-02-11[chore/performance] Update media prune logic, add extra CLI command (#1474)v0.7.0-rc2Libravatar tobi1
* start updating media prune stuff a wee bit * continue prune / uncache work * more tidying + consistency stuff * add prune CLI command * docs * arg
2023-02-03[chore] reformat GetAccount() functionality, support updating accounts based ↵Libravatar kim3
on last_fetch (#1411) * reformat GetAccount() functionality, and add UpdateAccount() function. * use fetched_at instead of last_webfingered_at * catch local "not found" errors. small formatting / error string changes * remove now unused error type * return nil when wrapping nil error * update expected error messages * return correct url for foss satan webfinger * add AP model for Some_User * normalize local domain * return notretrievable where appropriate * expose NewErrNotRetrievable * ensure webfinger for new accounts searched by uri * update local account short circuit * allow enrich to fail for already-known accounts * remove unused LastWebfingeredAt * expose test maps on mock http client * update Update test * reformat GetAccount() functionality, and add UpdateAccount() function. * use fetched_at instead of last_webfingered_at * catch local "not found" errors. small formatting / error string changes * remove nil error checks (we shouldn't be passing nil errors to newError() initializers) * remove mutex unlock on transport init fail (it hasn't yet been locked!) * woops add back the error wrapping to use ErrNotRetrievable * caches were never being started... :see_no_evil: --------- Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-01-27[feature] Federate reports to remote instance as Flag (if desired) (#1386)Libravatar tobi1
* reports federate out, we did it lxds * fix optional line start (should be optional slash)
2023-01-26[chore] set max open / idle conns + conn max lifetime for both postgres and ↵Libravatar tobi1
sqlite (#1369) * [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite * reduce cache size default to 8MiB, reduce connections to 2 * cpu * introduce max open conns multiplier, tune sqlite and pg separately * go fmt
2023-01-25[feature] Public list of suspended domains (#1362)Libravatar f0x521
* basic rendered domain blocklist (unauthenticated!) * style basic domain block list * better formatting for domain blocklist * add opt-in config option for showing suspended domains * format/linter * re-use InstancePeersGet for web-accessible domain blocklist * reword explanation, border styling * always attach blocklist handler, update error message * domain blocklist error message grammar
2023-01-25[feature] Implement reports admin API so admins can view + close reports (#1378)Libravatar tobi1
* add admin report api endpoints + tests * [chore] remove funky duplicate attachment in testrig
2023-01-23[chore] remove funky duplicate attachment in testrig (#1379)Libravatar tobi1
* [chore] remove funky duplicate attachment in testrig * whoops
2023-01-18[chore] Change default sqlite busy timeout to 5m (#1352)Libravatar tobi1
2023-01-17[feature] Tune sqlite pragmas (#1349)Libravatar tobi2
* sqlite pragma tuning * use formatuint * add sqlite busy timeout * fix incorrect cache size format * update envparsing test * add sqlite tuning flags to cli * set sqlite timeout to 30s default
2023-01-11[performance] media processing improvements (#1288)Libravatar kim14
* media processor consolidation and reformatting, reduce amount of required syscalls Signed-off-by: kim <grufwub@gmail.com> * update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling Signed-off-by: kim <grufwub@gmail.com> * fix duration not being set, fix mp4 test expecting error Signed-off-by: kim <grufwub@gmail.com> * fix test expecting media files with different extension Signed-off-by: kim <grufwub@gmail.com> * remove unused code Signed-off-by: kim <grufwub@gmail.com> * fix expected storage paths in tests, update expected test thumbnails Signed-off-by: kim <grufwub@gmail.com> * remove dead code Signed-off-by: kim <grufwub@gmail.com> * fix cached presigned s3 url fetching Signed-off-by: kim <grufwub@gmail.com> * fix tests Signed-off-by: kim <grufwub@gmail.com> * fix test models Signed-off-by: kim <grufwub@gmail.com> * update media processing to use sync.Once{} for concurrency protection Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * fix passing in KVStore GetStream() as stream to PutStream() Signed-off-by: kim <grufwub@gmail.com> * fix unlocks of storage keys Signed-off-by: kim <grufwub@gmail.com> * whoops, return the error... Signed-off-by: kim <grufwub@gmail.com> * pour one out for tobi's code <3 Signed-off-by: kim <grufwub@gmail.com> * add back the byte slurping code Signed-off-by: kim <grufwub@gmail.com> * check for both ErrUnexpectedEOF and EOF Signed-off-by: kim <grufwub@gmail.com> * add back links to file format header information Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2023-01-10[feature] Implement Report database model and utility functions (#1310)Libravatar tobi2
* implement report database model * implement report cache + config changes * implement report database functions * report uri / regex functions * update envparsing test * remove unnecessary uri index * remove unused function + cache lookup * process error when storing report
2023-01-05[chore] Update/add license headers for 2023 (#1304)Libravatar tobi16
2023-01-04[feature] HTTP request throttling middleware (#1297)Libravatar tobi1
* [feature] Add throttling middleware to AP endpoints * refactor a lil bit * use config setting, start updating docs * doc updates * use relative links in faq doc * small docs fixes * return code 503 instead of 429 when throttled * throttle other endpoints too * simplify token channel prefills
2023-01-02[chore] The Big Middleware and API Refactor (tm) (#1250)Libravatar tobi1
* interim commit: start refactoring middlewares into package under router * another interim commit, this is becoming a big job * another fucking massive interim commit * refactor bookmarks to new style * ambassador, wiz zeze commits you are spoiling uz * she compiles, we're getting there * we're just normal men; we're just innocent men * apiutil * whoopsie * i'm glad noone reads commit msgs haha :blob_sweat: * use that weirdo go-bytesize library for maxMultipartMemory * fix media module paths
2022-12-22[feature] For video attachments, store + return fps, bitrate, duration (#1282)Libravatar tobi3
* start messing about with different mp4 metadata extraction * heyyooo it works * add test cow * move useful multierror to gtserror package * error out if video doesn't seem to be a real mp4 * test parsing mkv in disguise as mp4 * tidy up error handling * remove extraneous line * update framerate formatting * use float32 for aspect * fixy mctesterson
2022-12-13[bugfix] fix unordered favorites (#1245)Libravatar Christoph Voigt1
* [bugfix] fix unordered favorites * add test for favouritesget * add license to new test files
2022-12-12[feature] allow uncaching of other media types (#1234)Libravatar kim1
* simplify pruneRemote, remove unncecessary media trace logging, update RemoteOlderThan() to include headers/avis Signed-off-by: kim <grufwub@gmail.com> * cleanup pruneallmeta, add remote header to pruneremote tests Signed-off-by: kim <grufwub@gmail.com> * fix olderthan duration additions Signed-off-by: kim <grufwub@gmail.com> * fix broken test now that test model header changed Signed-off-by: kim <grufwub@gmail.com> * instead use new remote test account for new header model Signed-off-by: kim <grufwub@gmail.com> * use newer generated ULID for remote_account_3 to ensure it is sorted last Signed-off-by: kim <grufwub@gmail.com> * reorganize serialized keys to match expected test account model order Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-12-11[feature] support configuring database caches (#1246)Libravatar kim1
* update config generator to support nested structs, add cache configuration options * update envparsing test * add cache configuration to config parse tests * set cache configuration in testrig * move caches to sub-cache "gts" namespace, update envparsing, add cache config docs to example config Signed-off-by: kim <grufwub@gmail.com>
2022-12-09[feature] Allow users to create + delete bookbarks, and view bookmarked ↵Libravatar Matthew Phillips2
statuses (#1168) * Implement Bookmarks * Update based on review comments * Update swagger doc * Fix argument passing to status.Bookmark * Update changed test * Updates based on latest PR review
2022-12-08[chore] move caches to a separate State{} structure (#1078)Libravatar kim1
* move caches to a separate State{} structure Signed-off-by: kim <grufwub@gmail.com> * fix call to log.Panic not using formatted call Signed-off-by: kim <grufwub@gmail.com> * move caches to use interfaces, to make switchouts easier in future Signed-off-by: kim <grufwub@gmail.com> * fix rebase issue Signed-off-by: kim <grufwub@gmail.com> * improve code comment Signed-off-by: kim <grufwub@gmail.com> * fix further issues after rebase Signed-off-by: kim <grufwub@gmail.com> * heh Signed-off-by: kim <grufwub@gmail.com> * add missing license text Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-12-06[feature] overhaul the oidc system (#961)Libravatar Dominik Süß1
* [feature] overhaul the oidc system this allows for more flexible username handling and prevents account takeover using old email addresses * [feature] add migration path for old OIDC users * [feature] nicer error reporting for users * [docs] document the new OIDC flow * [fix] return early on oidc error * [docs]: add comments on the finalization logic
2022-12-04[chore] Fix a few possible cases of int truncation (#1207)Libravatar Daniele Sluijters1
This fixes a couple of cases where due to int being platform dependent a value could get truncated if running on 32bits.
2022-12-02[performance]: make s3 urls cacheable (#1194)v0.6.0-rc3Libravatar Dominik Süß1
Implements #864 and should speed up s3 based installations by a lot. With more static urls, we can then also implement #1026 for even better performance when used in conjunction with CDNs
2022-11-30[bugfix] Don't call `strings.ToLower()` on usernames when selecting account ↵Libravatar tobi2
by domain+username (#1190) * don't lowercase account username when doing a select * test getting remote user with uppercase username
2022-11-30[bugfix] Avoid accidentally marking changed emojis as orphaned + pruning ↵Libravatar tobi1
them (#1188) * add predictable instance account to tests, use it in emoji urls + paths * use static image url to select emojis when pruning orphaned
2022-11-24[chore] cleanup storage implementation, no need for multiple interface types ↵Libravatar kim4
(#1131) Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-11-22[chore] use kv.KVStore also for S3 storage (#1113)Libravatar kim1
* replace s3 storage implementation to also use kv.KVStore Signed-off-by: kim <grufwub@gmail.com> * pull in latest `go-store` fix Signed-off-by: kim <grufwub@gmail.com> * pull-in go-store v2.0.9 fixes, update s3 put chunk size to 5MiB Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
generated by cgit v1.2.3 (git 2.46.0) at 2025-07-28 13:51:47 +0000