summaryrefslogtreecommitdiff
path: root/internal/processing/processor.go
AgeCommit message (Collapse)AuthorFiles
2022-05-15[security] transport.Controller{} and transport.Transport{} security and ↵Libravatar kim1
performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-03[chore] Move initial MediaRemotePrune to cmd.server function (#536)Libravatar tobi1
* move initial prune up to the server start function * don't do initial prune when starting media manager * add admin/mediaremoteprune in processor
2022-04-28replace async client API / federator msg processing with worker pools (#497)Libravatar kim1
* replace async client API / federator msg processing with worker pools * appease our lord-and-saviour, the linter
2022-04-15[feature] Web profile pages for accounts (#449)Libravatar tobi1
* add default avatars * allow webModule to error * return errWithCode from account get * add AccountGetLocalByUsername * check nil requesting account * add timestampShort function for just month/year * move loading logic to New + add default avatars * add profile page view * update swagger docs * add excludeReblogs to GetAccountStatuses * ignore casing when selecting local account by username * appropriate redirects * css fiddling * add 'about' heading * adjust thread page to work with routing * return AP representation if requested + authorized * simplify auth check * go fmt * golangci-lint ignore math/rand
2022-03-29[feature] Dereference remote mentions when the account is not already known ↵v0.2.2Libravatar tobi1
(#442) * remove mention util function from db * add ParseMentionFunc to gtsmodel * add parseMentionFunc to processor * refactor search to simplify it a bit * add parseMentionFunc to account * add parseMentionFunc to status * some renaming for clarity * test dereference of unknown mentioned account
2022-03-19[feature] Admin account actions (#432)Libravatar tobi1
* add accountAction to the admin API * model admin account action * add admin account action to the processor * add migration for new AdminAccountActions table * fix accounts admin path * Update swagger docs
2022-03-15[feature] Federate local account deletion (#431)Libravatar tobi1
* add account delete to API * model account delete request * add AccountDeleteLocal * federate local account deletes * add DeleteLocal * update transport (controller) to allow shortcuts * delete logic + testing * update swagger docs * more tests + fixes
2022-03-07[feature] Clean up/uncache remote media (#407)Libravatar tobi1
* Add whereNotEmptyAndNotNull * Add GetRemoteOlderThanDays * Add GetRemoteOlderThanDays * Add PruneRemote to Manager interface * Start implementing PruneRemote * add new attachment + status to tests * fix up and test GetRemoteOlderThan * fix bad import * PruneRemote: return number pruned * add Cached column to mediaattachment * update + test pruneRemote * update mediaTest * use Cached column * upstep bun to latest version * embed structs in mediaAttachment * migrate mediaAttachment to new format * don't default cached to true * select only remote media * update db dependencies * step bun back to last working version * update pruneRemote to use Cached field * fix storage path of test attachments * add recache logic to manager * fix trimmed aspect ratio * test prune and recache * return errwithcode * tidy up different paths for emoji vs attachment * fix incorrect thumbnail type being stored * expose TransportController to media processor * implement tee-ing recached content * add thoughts of dog to test fedi attachments * test get remote files * add comment on PruneRemote * add postData cleanup to recache * test thumbnail fetching * add incredible diagram * go mod tidy * buffer pipes for recache streaming * test for client stops reading after 1kb * add media-remote-cache-days to config * add cron package * wrap logrus so it's available to cron * start and stop cron jobs gracefully
2022-02-08Merge branch 'main' into media_refactorLibravatar tsmethurst1
2022-02-05[feature] Rework timeline code to make it useful for more than just statuses ↵Libravatar tobi1
(#373) * add preparable and timelineable interfaces * initialize timeline manager within the processor * generic renaming * move status-specific timeline logic into the processor * refactor timeline to make it useful for more than statuses
2022-01-15emoji code passing musterLibravatar tsmethurst1
2021-12-28more refactoring, media handler => managerLibravatar tsmethurst1
2021-12-20Extend license notices to 2022 (#354)Libravatar tobi1
2021-12-07Implement Cobra CLI tooling, Viper config tooling (#336)Libravatar tobi1
* start pulling out + replacing urfave and config * replace many many instances of config * move more stuff => viper * properly remove urfave * move some flags to root command * add testrig commands to root * alias config file keys * start adding cli parsing tests * reorder viper init * remove config path alias * fmt * change config file keys to non-nested * we're more or less in business now * tidy up the common func * go fmt * get tests passing again * add note about the cliparsing tests * reorganize * update docs with changes * structure cmd dir better * rename + move some files around * fix dangling comma
2021-11-13update dependencies (#296)Libravatar tobi1
2021-10-31smtp + email confirmation (#285)Libravatar tobi1
* add smtp configuration * add email confirm + reset templates * add email sender to testrig * flesh out the email sender interface * go fmt * golint * update from field with more clarity * tidy up the email formatting * fix tests * add email sender to processor * tidy client api processing a bit * further tidying in fromClientAPI * pin new account to user * send msg to processor on new account creation * generate confirm email uri * remove emailer from account processor again * add processCreateAccountFromClientAPI * move emailer accountprocessor => userprocessor * add email sender to user processor * SendConfirmEmail function * add noop email sender * use noop email sender in tests * only assemble message if callback is not nil * use noop email sender if no smtp host is defined * minify email html before sending * fix wrong email address * email confirm test * fmt * serve web hndler * add email confirm handler * init test log properly on testrig * log emails that *would* have been sent * go fmt ./... * unexport confirm email handler * updatedAt * test confirm email function * don't allow tokens older than 7 days * change error message a bit * add basic smtp docs * add a few more snippets * typo * add email sender to outbox tests * don't use dutch wikipedia link * don't minify email html
2021-10-24Serve `outbox` for Actor (#289)Libravatar tobi1
* add statusesvisible convenience function * add minID + onlyPublic to account statuses get * move swagger collection stuff to common * start working on Outbox GETting * move functions into federationProcessor * outboxToASCollection * add statusesvisible convenience function * add minID + onlyPublic to account statuses get * move swagger collection stuff to common * start working on Outbox GETting * move functions into federationProcessor * outboxToASCollection * bit more work on outbox paging * wrapNoteInCreate function * test + hook up the processor functions * don't do prev + next links on empty reply * test get outbox through api * don't fail on no status entries * add outbox implementation doc * typo
2021-10-16Follow request improvements (#282)Libravatar tobi1
* tiny doc update * add rejectfollowrequest to db * add follow request reject to processor * add reject handler * tidy up follow request api * tidy up federation call * regenerate swagger docs * api endpoint tests * processor test * add reject federatingdb handler * start writing reject tests * test reject follow request * go fmt * increase sleep for slow test setups * more relaxed time.sleep
2021-10-14User password change (#280)Libravatar tobi1
* start passwordChangeHandler * add user scope * add user module / api path * add password change request * make comment clearer * add user to processor * required true * add processor call to handler * don't pass tc or channel * change password func + tests * add some first docs about password management * update swagger docs * add api tests * go fmt * test fixes
2021-10-11reference global logrus (#274)Libravatar R. Aidan Campbell1
* reference logrus' global logger instead of passing and storing a logger reference everywhere * always directly use global logrus logger instead of referencing an instance * test suites should also directly use the global logrus logger * rename gin logging function to clarify that it's middleware * correct comments which erroneously referenced removed logger parameter * setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-04Refactor/tidy (#261)Libravatar tobi1
* tidy up streaming * cut down code duplication * test get followers/following * test streaming processor * fix some test models * add TimeMustParse * fix uri / url typo * make trace logging less verbose * make logging more consistent * disable quote on logging * remove context.Background * remove many extraneous mastodon references * regenerate swagger * don't log query on no rows result * log latency first for easier reading
2021-09-20Update webfingering a little, add tests (#236)Libravatar tobi1
* Update webfingering a little, add tests * fix broken tests oops
2021-09-11rename KVStore uses as storage to differentiate between upcoming store packageLibravatar kim (grufwub)1
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11add git.iim.gay/grufwub/go-store for storage backend, replacing blob.StorageLibravatar kim (grufwub)1
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-04tests + announce notification fix (#193)Libravatar tobi1
2021-09-01moving stuff aroundLibravatar tsmethurst1
2021-09-01change muchos thingsLibravatar tsmethurst1
2021-08-25Pg to bun (#148)Libravatar tobi1
* start moving to bun * changing more stuff * more * and yet more * tests passing * seems stable now * more big changes * small fix * little fixes
2021-08-15Timeline loop fix (#140)Libravatar Tobi Smethurst1
* uwu we made a fucky wucky * uwu we made a fucky wucky * work on timeline fixes a little * fiddle with tests some more * bleep bloop more tests * more tests * update drone yml * update some sturf * make the timeline code a bit lazier * go fmt * fix drone.yml
2021-08-10Dereference remote replies (#132)Libravatar Tobi Smethurst1
* decided where to put reply dereferencing * fiddling with dereferencing threads * further adventures * tidy up some stuff * move dereferencing functionality * a bunch of refactoring * go fmt * more refactoring * bleep bloop * docs and linting * start implementing replies collection on gts side * fiddling around * allow dereferencing our replies * lint, fmt
2021-07-11Block/unblock (#96)Libravatar Tobi Smethurst1
* remote + local block logic, incl. federation * improve blocking stuff * fiddle with display of blocked profiles * go fmt
2021-07-09favourites GET implementation (#95)Libravatar Tobi Smethurst1
2021-07-06Blocklist import (#77)Libravatar Tobi Smethurst1
* first steps on importing blocklists * unblock domains properly
2021-07-05Domain block (#76)Libravatar Tobi Smethurst1
* start work on admin domain blocking * move stuff around + further work on domain blocks * move + restructure processor * prep work for deleting account * tidy * go fmt * formatting * domain blocking more work * check domain blocks way earlier on * progress on delete account * delete more stuff when an account is gone * and more... * domain blocky block block * get individual domain block, delete a block
2021-06-24nodeinfo compliance (#61)Libravatar Tobi Smethurst1
2021-06-23Instance settings updates (#59)Libravatar Tobi Smethurst1
Allow admins to set instance settings through a PATCH to /api/v1/instance Update templates to reflect some of the new fields
2021-06-21Deletes+unboosts (#52)Libravatar Tobi Smethurst1
* Status deletes properly streamed now. * Unboosts now work locally and federated. * Documentation updates.
2021-06-19Streaming (#49)Libravatar Tobi Smethurst1
Add new status and notification websocket streaming capabilities
2021-06-17Timeline improvements (#41)Libravatar Tobi Smethurst1
Tidying up. Parent/child statuses now display correctly in status/id/context.
2021-06-13Timeline manager (#40)Libravatar Tobi Smethurst1
* start messing about with timeline manager * i have no idea what i'm doing * i continue to not know what i'm doing * it's coming along * bit more progress * update timeline with new posts as they come in * lint and fmt * Select accounts where empty string * restructure a bunch, get unfaves working * moving stuff around * federate status deletes properly * mention regex better but not 100% there * fix regex * some more hacking away at the timeline code phew * fix up some little things * i can't even * more timeline stuff * move to ulid * fiddley * some lil fixes for kibou compatibility * timelines working pretty alright! * tidy + lint
2021-05-31Tidy + timeline embetterment (#38)Libravatar Tobi Smethurst1
* tidy up timelines a bit + stub out some endpoints * who's faved and who's boosted, reblog notifs * linting * Update progress with new endpoints
2021-05-30Move a lot of stuff + tidy stuff (#37)Libravatar Tobi Smethurst1
Lots of renaming and moving stuff, some bug fixes, more lenient parsing of notifications and home timeline.
generated by cgit v1.2.3 (git 2.46.0) at 2025-07-12 23:16:45 +0000