| Age | Commit message (Collapse) | Author | Files |
|---|
|
|
|
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup
* ensure the tasks are sorted by creation time before pushing them
* add migration to insert WorkerTask{} into database, add test for worker task persistence
* add test for recovering worker queues from database
* quick tweak
* whoops we ended up with double cleaner job scheduling
* insert each task separately, because bun is throwing some reflection error??
* add specific checking of cancelled worker contexts
* add http request signing to deliveries recovered from database
* add test for outgoing public key ID being correctly set on delivery
* replace select with Queue.PopCtx()
* get rid of loop now we don't use it
* remove field now we don't use it
* ensure that signing func is set
* header values weren't being copied over :facepalm:
* use ptr for httpclient.Request in delivery
* move worker queue filling to later in server init process
* fix rebase issues
* make logging less shouty
* use slices.Delete() instead of copying / reslicing
* have database return tasks in ascending order instead of sorting them
* add a 1 minute timeout to persisting worker queues
|
|
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it
* move delivery and workers into separate files
* add worker task model and Serialize() / Deserialize() methods for message types
* start adding message serialize / deserialize tests
* start adding test cases
* update body rewinding to rely on standard library mechanism of r.GetBody()
* remove request rewinding (http.Client{} should already handle this)
* standard library already handles rewinding
* improved code comment
* move the newPOST() function contents to prepare(), fits better with current API
* add Serialize() / Deserialize() implementations for Delivery{} type
* finish writing FromClientAPI sserialize / deserialize tests
* start adding FromFediAPI{} serialize / deserialize test cases
* fix FromFediAPI{} tests
* add tests for delivery message type
* fix repeat code
* missing license header
* use testrig status and accounts for marshal / unmarshaling tests
* add a specific test for checking account RSA keys are preserved
|
|
* add delivery worker type that pulls from queue to httpclient package
* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller
* hook up queue deletion logic
* support deleting queued http requests by target ID
* don't index APRequest by hostname in the queue
* use gorun
* use the original context's values when wrapping msg type as delivery{}
* actually log in the AP delivery worker ...
* add uncommitted changes
* use errors.AsV2()
* use errorsv2.AsV2()
* finish adding some code comments, add bad host handling to delivery workers
* slightly tweak deliveryworkerpool API, use advanced sender multiplier
* remove PopCtx() method, let others instead rely on Wait()
* shuffle things around to move delivery stuff into transport/ subpkg
* remove dead code
* formatting
* validate request before queueing for delivery
* finish adding code comments, fix up backoff code
* finish adding more code comments
* clamp minimum no. senders to 1
* add start/stop logging to delivery worker, some slight changes
* remove double logging
* use worker ptrs
* expose the embedded log fields in httpclient.Request{}
* ensure request context values are preserved when updating ctx
* add delivery worker tests
* fix linter issues
* ensure delivery worker gets inited in testrig
* fix tests to delivering messages to check worker delivery queue
* update error type to use ptr instead of value receiver
* fix test calling Workers{}.Start() instead of testrig.StartWorkers()
* update docs for advanced-sender-multiplier
* update to the latest activity library version
* add comment about not using httptest.Server{}
|
|
|
|
performance and multiple-result lookups (#2535)
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial
* use actual go-structr release version (not just commit hash)
* revert go toolchain changes (damn you go for auto changing this)
* fix go mod woes
* ensure %w is used in calls to errs.Appendf()
* fix error checking
* fix possible panic
* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop
* fix copy-paste artifact... :innocent:
* fix all comment copy-paste artifacts
* remove dropID() function, now we can just use slices.DeleteFunc()
* use util.Deduplicate() instead of collate(), move collate to util
* move orderByIDs() to util package and "generify"
* add a util.DeleteIf() function, use this to delete entries on failed population
* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)
* add note about how collate differs from deduplicate
|
|
improvements (#1835)
* media manager tidy-up: de-interface and remove unused PostDataFunc
Signed-off-by: kim <grufwub@gmail.com>
* remove last traces of media.Manager being an interface
Signed-off-by: kim <grufwub@gmail.com>
* update error to provide caller, allow tuneable via build tags
Signed-off-by: kim <grufwub@gmail.com>
* remove kim-specific build script changes
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* update build-script to support externally setting build variables
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
|
|
* add back removed ValidateRequest() before backoff-retry loop
Signed-off-by: kim <grufwub@gmail.com>
* include response body in error response log
Signed-off-by: kim <grufwub@gmail.com>
* improved error response body draining
Signed-off-by: kim <grufwub@gmail.com>
* add more code commenting
Signed-off-by: kim <grufwub@gmail.com>
* move new error response logic to gtserror, handle instead in transport.Transport{} impl
Signed-off-by: kim <grufwub@gmail.com>
* appease ye oh mighty linter
Signed-off-by: kim <grufwub@gmail.com>
* fix mockhttpclient not setting request in http response
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
|
|
* revamp http client to not limit requests, instead use sender worker
Signed-off-by: kim <grufwub@gmail.com>
* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping
Signed-off-by: kim <grufwub@gmail.com>
* improve batch senders to keep popping recipients until remote URL found
Signed-off-by: kim <grufwub@gmail.com>
* fix recipient looping issue
Signed-off-by: kim <grufwub@gmail.com>
* fix missing mutex unlock
Signed-off-by: kim <grufwub@gmail.com>
* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP
Signed-off-by: kim <grufwub@gmail.com>
* slight code reformatting
Signed-off-by: kim <grufwub@gmail.com>
* a whitespace
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* add missing license headers
Signed-off-by: kim <grufwub@gmail.com>
* fix request backoff calculation
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
|
|
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
|
|
* add error value wrapping, include status code / not found flags from transport errors, update error usages
Signed-off-by: kim <grufwub@gmail.com>
* add code commenting for gtserror functions
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
|
|
* improve error readability, mark "bad hosts" as fastFail
Signed-off-by: kim <grufwub@gmail.com>
* pull in latest go-byteutil version with byteutil.Reader{}
Signed-off-by: kim <grufwub@gmail.com>
* use rewindable body reader for post requests
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
|
|
|
|
* interim commit: start refactoring middlewares into package under router
* another interim commit, this is becoming a big job
* another fucking massive interim commit
* refactor bookmarks to new style
* ambassador, wiz zeze commits you are spoiling uz
* she compiles, we're getting there
* we're just normal men; we're just innocent men
* apiutil
* whoopsie
* i'm glad noone reads commit msgs haha :blob_sweat:
* use that weirdo go-bytesize library for maxMultipartMemory
* fix media module paths
|
|
Currently requests set their own User-Agent. This moves it down to set
it in the transport's do() method, to guarantee it's always set on all
requests.
|
|
* update templates
* start reworking api error handling
* update template
* return AP status at web endpoint if negotiated
* start making api error handling much more consistent
* update account endpoints to new error handling
* use new api error handling in admin endpoints
* go fmt ./...
* use api error logic in app
* use generic error handling in auth
* don't export generic error handler
* don't defer clearing session
* user nicer error handling on oidc callback handler
* tidy up the sign in handler
* tidy up the token handler
* use nicer error handling in blocksget
* auth emojis endpoint
* fix up remaining api endpoints
* fix whoopsie during login flow
* regenerate swagger docs
* change http error logging to debug
|
|
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct
Signed-off-by: kim <grufwub@gmail.com>
* improved code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter
Signed-off-by: kim <grufwub@gmail.com>
* fix unmarshaling
Signed-off-by: kim <grufwub@gmail.com>
* remove kim's custom go compiler changes
Signed-off-by: kim <grufwub@gmail.com>
* generate setter and flag-name functions, implement these in codebase
Signed-off-by: kim <grufwub@gmail.com>
* update deps
Signed-off-by: kim <grufwub@gmail.com>
* small change
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter...
Signed-off-by: kim <grufwub@gmail.com>
* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync
Signed-off-by: kim <grufwub@gmail.com>
* lint
Signed-off-by: kim <grufwub@gmail.com>
* update code comments
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* improved version string (removes time + go version)
Signed-off-by: kim <grufwub@gmail.com>
* fix version string build to pass test script + consolidate logic in func
Signed-off-by: kim <grufwub@gmail.com>
* add license text, update config.Defaults comment
Signed-off-by: kim <grufwub@gmail.com>
* add license text to generated config helpers file
Signed-off-by: kim <grufwub@gmail.com>
* defer unlock on config.Set___(), to ensure unlocked on panic
Signed-off-by: kim <grufwub@gmail.com>
* make it more obvious which cmd flags are being attached
Signed-off-by: kim <grufwub@gmail.com>
|
|
performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests
Signed-off-by: kim <grufwub@gmail.com>
* update code comments, defer mutex unlocks
Signed-off-by: kim <grufwub@gmail.com>
* add count to 'performing request' log message
Signed-off-by: kim <grufwub@gmail.com>
* reduce repeated conversions of same url.URL object
Signed-off-by: kim <grufwub@gmail.com>
* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue
Signed-off-by: kim <grufwub@gmail.com>
* fix security advisories regarding max outgoing conns, max rsp body size
- implemented by a new httpclient.Client{} that wraps an underlying
client with a queue to limit connections, and limit reader wrapping
a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}
Signed-off-by: kim <grufwub@gmail.com>
* add httpclient tests, move ip validation to separate package + change mechanism
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* use singular mutex in transport rather than separate signer mus
Signed-off-by: kim <grufwub@gmail.com>
* improved useragent string
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding missing test
Signed-off-by: kim <grufwub@gmail.com>
* remove useragent field from transport (instead store in controller)
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* reset other signing headers on each loop iteration
Signed-off-by: kim <grufwub@gmail.com>
* respect request ctx during retry-backoff sleep period
Signed-off-by: kim <grufwub@gmail.com>
* use external pkg with docs explaining performance "hack"
Signed-off-by: kim <grufwub@gmail.com>
* use http package constants instead of string method literals
Signed-off-by: kim <grufwub@gmail.com>
* add license file headers
Signed-off-by: kim <grufwub@gmail.com>
* update code comment to match new func names
Signed-off-by: kim <grufwub@gmail.com>
* updates to user-agent string
Signed-off-by: kim <grufwub@gmail.com>
* update signed testrig models to fit with new transport logic (instead uses separate signer now)
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter
Signed-off-by: kim <grufwub@gmail.com>
|
|
|
|
(#430)
* update transport (controller) to allow shortcuts
* go fmt
* expose underlying sig transport to allow test sigs
|
|
|
|
* reference logrus' global logger instead of passing and storing a logger reference everywhere
* always directly use global logrus logger instead of referencing an instance
* test suites should also directly use the global logrus logger
* rename gin logging function to clarify that it's middleware
* correct comments which erroneously referenced removed logger parameter
* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
|
|
* start moving to bun
* changing more stuff
* more
* and yet more
* tests passing
* seems stable now
* more big changes
* small fix
* little fixes
|
|
Remote instances are now dereferenced when they post to an inbox on a GtS instance.
Dereferencing will be done first by checking the /api/v1/instance endpoint of an instance.
If that doesn't work, /.well-known/nodeinfo will be checked.
If that doesn't work, only a minimal representation of the instance will be stored.
A new field was added to the Instance database model. To create it:
alter table instances add column contact_account_username text;
|
tobi
kim
Daenney
R. Aidan Campbell