summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2022-09-02[performance] cache recently allowed/denied domains to cut down on db calls ↵Libravatar kim2
(#794) * fetch creation and fetching domain blocks from db Signed-off-by: kim <grufwub@gmail.com> * add separate domainblock cache type, handle removing block from cache on delete Signed-off-by: kim <grufwub@gmail.com> * fix sentinel nil values being passed into cache Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-09-02[performance] use GetAccountByUsernameDomain() for local account lookups to ↵Libravatar kim11
rely on cache (#793) Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-08-31[feature] Add support for the exclude_types[] parameter on the notifications ↵Libravatar Blackle Morisanchetto3
endpoint (#784) * Add support for the exclude_types[] parameter on the notifications endpoint * Add swagger docs to notifications
2022-08-31[chore] Test fixes (#788)Libravatar tobi4
* use 'test' value for testrig storage backend * update test dependency * add WaitFor func in testrig * use WaitFor function instead of time.Sleep * tidy up tests * make SentMessages a sync.map * go fmt
2022-08-31[bugfix] Fix tusky search issue by returning empty if offset is greater than ↵Libravatar Blackle Morisanchetto1
zero (#786)
2022-08-30[bugfix] Do not generate a notification when favouriting your own post (#780)Libravatar Blackle Morisanchetto1
2022-08-27[bugfix] Status visibility + `excludeReplies` fixes (#769)Libravatar Blackle Morisanchetto1
* Fix some bugs when viewing a user's posts: include their self-replies (threads) even when excludeReplies is set, and use in_reply_to_uri instead of in_reply_to_id to filter out replies * Assign values to InReplyToURI when creating statuses. Add index and update old statuses with a migration
2022-08-23[bugfix] Fix boost of boost issue (#764)Libravatar tobi2
2022-08-20[bugfix] Fix potential dereference of accounts on own instance (#757)Libravatar tobi1
* add GetAccountByUsernameDomain * simplify search * add escape to not deref accounts on own domain * check if local + we have account by ap uri
2022-08-15[chore] Update bun / sqlite versions; update gtsmodels (#754)Libravatar tobi24
* upstep bun and sqlite versions * allow specific columns to be updated in the db * only update necessary columns for user * bit tidier * only update necessary fields of media_attachment * only update relevant instance fields * update tests * update only specific account columns * use bool pointers on gtsmodels includes attachment, status, account, user * update columns more selectively * test all default fields on new account insert * updating remaining bools on gtsmodels * initialize pointer fields when extracting AP emoji * copy bools properly * add copyBoolPtr convenience function + test it * initialize false bool ptrs a bit more neatly
2022-08-08[feature] Add first iteration of a user panel at `/user` (#736)Libravatar tobi2
* start work on user panel * parse source first before checking if empty form * newline * set avi + header nicely * add posts settings * render signin a bit nicer on mobile * return OK json on successful change * return unauthorized on bad password * clarify message on insecure password * make login a bit prettier * add alt text + border round image previews * add logout button * add password change * styling updates * redirect /auth/edit to /user * update tests * fix validation tests * better labels, link to more info * make submit button generic component * move submit button inside forms * add autocomplete labels to password fields * fix indentation (thx eslint) * update eslintrc * eslint: no-unescaped-entities * initial deduplication between user and admin panel * add default status/post format setting * user panel styling for inputs * update user panel styling, include normalize css * add placeholder text * input padding Co-authored-by: f0x <f0x@cthu.lu>
2022-08-06[feature] Let accounts set default status format, and use this when ↵Libravatar tobi2
processing new statuses (#739) * add post_format to acct & use it when making post * update swagger docs * add status_format updating to frontend * fix up tests * post_format => status_format * add status_format to account validation
2022-08-01[feature] Implemented notification clear (#720)Libravatar Artémis2
* Implemented notification clear * Added the cache clear mechanism * added multi user check test
2022-07-19[bugfix] html escape special characters in text instead of totally removing ↵Libravatar tobi2
them (#719) * remove minify dependency * tidy up some tests * remove pre + postformat funcs * rework sanitization + formatting * update tests * add some more markdown tests
2022-07-19[chore] use our own logging implementation (#716)Libravatar kim15
* first commit Signed-off-by: kim <grufwub@gmail.com> * replace logging with our own log library Signed-off-by: kim <grufwub@gmail.com> * fix imports Signed-off-by: kim <grufwub@gmail.com> * fix log imports Signed-off-by: kim <grufwub@gmail.com> * add license text Signed-off-by: kim <grufwub@gmail.com> * fix package import cycle between config and log package Signed-off-by: kim <grufwub@gmail.com> * fix empty kv.Fields{} being passed to WithFields() Signed-off-by: kim <grufwub@gmail.com> * fix uses of log.WithFields() with whitespace issues and empty slices Signed-off-by: kim <grufwub@gmail.com> * *linter related grumbling* Signed-off-by: kim <grufwub@gmail.com> * gofmt the codebase! also fix more log.WithFields() formatting issues Signed-off-by: kim <grufwub@gmail.com> * update testrig code to match new changes Signed-off-by: kim <grufwub@gmail.com> * fix error wrapping in non fmt.Errorf function Signed-off-by: kim <grufwub@gmail.com> * add benchmarking of log.Caller() vs non-cached Signed-off-by: kim <grufwub@gmail.com> * fix syslog tests, add standard build tags to test runner to ensure consistency Signed-off-by: kim <grufwub@gmail.com> * make syslog tests more robust Signed-off-by: kim <grufwub@gmail.com> * fix caller depth arithmatic (is that how you spell it?) Signed-off-by: kim <grufwub@gmail.com> * update to use unkeyed fields in kv.Field{} instances Signed-off-by: kim <grufwub@gmail.com> * update go-kv library Signed-off-by: kim <grufwub@gmail.com> * update libraries list Signed-off-by: kim <grufwub@gmail.com> * fuck you linter get nerfed Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-13[feature] Add back/next buttons to profiles for paging through statuses (#708)Libravatar tobi4
* add GetAccountWebStatuses to db * add WebStatusesGet func to processor * don't add limit to next/prev links if 0 * take query params for next/prev statuses * add separate next + prev links for convenience * show 'nothing here' message if no statuses exist * add back / next links to profiles * allow paging down only * go fmt ./... * 'recent public toots' -> 'latest public toots'
2022-07-10[chore] consolidate caching libraries (#704)Libravatar kim2
* add miekg/dns dependency * set/validate accountDomain * move finger to dereferencer * totally break GetRemoteAccount * start reworking finger func a bit * start reworking getRemoteAccount a bit * move mention parts to namestring * rework webfingerget * use util function to extract webfinger parts * use accountDomain * rework finger again, final form * just a real nasty commit, the worst * remove refresh from account * use new ASRepToAccount signature * fix incorrect debug call * fix for new getRemoteAccount * rework GetRemoteAccount * start updating tests to remove repetition * break a lot of tests Move shared test logic into the testrig, rather than having it scattered all over the place. This allows us to just mock the transport controller once, and have all tests use it (unless they need not to for some other reason). * fix up tests to use main mock httpclient * webfinger only if necessary * cheeky linting with the lads * update mentionName regex recognize instance accounts * don't finger instance accounts * test webfinger part extraction * increase default worker count to 4 per cpu * don't repeat regex parsing * final search for discovered accountDomain * be more permissive in namestring lookup * add more extraction tests * simplify GetParseMentionFunc * skip long search if local account * fix broken test * consolidate to all use same caching libraries Signed-off-by: kim <grufwub@gmail.com> * perform more caching in the database layer Signed-off-by: kim <grufwub@gmail.com> * remove ASNote cache Signed-off-by: kim <grufwub@gmail.com> * update cache library, improve db tracing hooks Signed-off-by: kim <grufwub@gmail.com> * return ErrNoEntries if no account status IDs found, small formatting changes Signed-off-by: kim <grufwub@gmail.com> * fix tests, thanks tobi! Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-07-03[feature] S3 support (#674)Libravatar Dominik Süß9
* feat: vendor minio client * feat: introduce storage package with s3 support * feat: serve s3 files directly this saves a lot of bandwith as the files are fetched from the object store directly * fix: use explicit local storage in tests * feat: integrate s3 storage with the main server * fix: add s3 config to cli tests * docs: explicitly set values in example config also adds license header to the storage package * fix: use better http status code on s3 redirect HTTP 302 Found is the best fit, as it signifies that the resource requested was found but not under its presumed URL 307/TemporaryRedirect would mean that this resource is usually located here, not in this case 303/SeeOther indicates that the redirection does not link to the requested resource but to another page * refactor: use context in storage driver interface
2022-06-30[feature] Cleanup unattached local media (#680)Libravatar tobi1
* add localUnattached db function * add parseOlderThan util function * add pruneunusedlocalattachments to media manager * add unusedlocal pruning to schedule + admin call * set number of days to keep as a const * fix test
2022-06-25[bugfix] disallow following or blocking yoursel (#667)Libravatar ugla2
Closes #664
2022-06-24[bugfix] Fix 404 on status delete redraft (#668)Libravatar tobi6
* add unattach function to media processor * call delete or unattach appropriately unattach from client api, delete from federated api * typo fix
2022-06-24[bugfix] allow setting empty email via instance patch (#665)Libravatar tobi1
2022-06-23[feature] Implement `/api/v1/instance/peers` endpoint (#660)Libravatar tobi3
* add missing license headers * start adding instance peers get * rename domainblock.go * embed domain in domainblock so it can be reused * update swagger docs * add test instances to db * update tests * add/update instancepeersget * update domain model * add getinstancepeers to db * instance-expose-peers, instance-expose-suspended * add auth checks for both current filters * attach endpoint to router * include public comment * obfuscate domain if required * go mod tidy * update swagger docs * remove unnecessary comment * return 'flat' peerlist if no query params provided
2022-06-20[bugfix] Account self finger fix (#658)v0.3.6Libravatar tobi1
* check only 1 of status/account when search by uri * don't try to resolve local uris when searching
2022-06-11[chore] Webfinger rework (#627)Libravatar tobi17
* move finger to dereferencer * totally break GetRemoteAccount * start reworking finger func a bit * start reworking getRemoteAccount a bit * move mention parts to namestring * rework webfingerget * use util function to extract webfinger parts * use accountDomain * rework finger again, final form * just a real nasty commit, the worst * remove refresh from account * use new ASRepToAccount signature * fix incorrect debug call * fix for new getRemoteAccount * rework GetRemoteAccount * start updating tests to remove repetition * break a lot of tests Move shared test logic into the testrig, rather than having it scattered all over the place. This allows us to just mock the transport controller once, and have all tests use it (unless they need not to for some other reason). * fix up tests to use main mock httpclient * webfinger only if necessary * cheeky linting with the lads * update mentionName regex recognize instance accounts * don't finger instance accounts * test webfinger part extraction * increase default worker count to 4 per cpu * don't repeat regex parsing * final search for discovered accountDomain * be more permissive in namestring lookup * add more extraction tests * simplify GetParseMentionFunc * skip long search if local account * fix broken test
2022-06-11[feature] Add `created_at` and `error_description` to `/oauth/token` ↵Libravatar tobi2
endpoint (#645) * start fiddling about with oauth server * start returning more helpful errors from oauth * test helpful(ish) token errors * add missing license header
2022-06-08[feature] More consistent API error handling (#637)Libravatar tobi27
* update templates * start reworking api error handling * update template * return AP status at web endpoint if negotiated * start making api error handling much more consistent * update account endpoints to new error handling * use new api error handling in admin endpoints * go fmt ./... * use api error logic in app * use generic error handling in auth * don't export generic error handler * don't defer clearing session * user nicer error handling on oidc callback handler * tidy up the sign in handler * tidy up the token handler * use nicer error handling in blocksget * auth emojis endpoint * fix up remaining api endpoints * fix whoopsie during login flow * regenerate swagger docs * change http error logging to debug
2022-06-08[feature] Add paging via `Link` header for notifications and account ↵Libravatar tobi7
statuses (#629) * test link headers * page get account statuses properly * page get notifications * add util func for packaging timeline responses * return timelined stuff from accountstatusesget * rename timeline response * use new convenience function * go fmt
2022-06-04[bugfix] Update time marshalling format to provide 3 digits of ms (#630)Libravatar tobi1
2022-05-30[chore] Global server configuration overhaul (#575)Libravatar kim9
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct Signed-off-by: kim <grufwub@gmail.com> * improved code comment Signed-off-by: kim <grufwub@gmail.com> * linter Signed-off-by: kim <grufwub@gmail.com> * fix unmarshaling Signed-off-by: kim <grufwub@gmail.com> * remove kim's custom go compiler changes Signed-off-by: kim <grufwub@gmail.com> * generate setter and flag-name functions, implement these in codebase Signed-off-by: kim <grufwub@gmail.com> * update deps Signed-off-by: kim <grufwub@gmail.com> * small change Signed-off-by: kim <grufwub@gmail.com> * appease the linter... Signed-off-by: kim <grufwub@gmail.com> * move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync Signed-off-by: kim <grufwub@gmail.com> * lint Signed-off-by: kim <grufwub@gmail.com> * update code comments Signed-off-by: kim <grufwub@gmail.com> * fix merge issue Signed-off-by: kim <grufwub@gmail.com> * fix merge issue Signed-off-by: kim <grufwub@gmail.com> * improved version string (removes time + go version) Signed-off-by: kim <grufwub@gmail.com> * fix version string build to pass test script + consolidate logic in func Signed-off-by: kim <grufwub@gmail.com> * add license text, update config.Defaults comment Signed-off-by: kim <grufwub@gmail.com> * add license text to generated config helpers file Signed-off-by: kim <grufwub@gmail.com> * defer unlock on config.Set___(), to ensure unlocked on panic Signed-off-by: kim <grufwub@gmail.com> * make it more obvious which cmd flags are being attached Signed-off-by: kim <grufwub@gmail.com>
2022-05-28[chore] Mastodon api fixups (#617)Libravatar tobi1
* don't omitempty on description * don't omitempty on any fields * add ms to timestamp format * don't omitempty on text_url * rearrange attachment fields a bit * just give URL again as attachment text url * update tests * fix accidental replace
2022-05-27[chore] Debug failed account get (#612)Libravatar tobi1
2022-05-26[bugfix] Fix HTML escaping in instance title (#607)Libravatar tobi7
* move caption sanitization -> sanitize.go * use sanitizeplaintext rather than removehtml * rename sanitizecaption to sanitizeplaintext * avoid removing html twice from statuses * unexport remoteHTML it's no longer used outside the text package so this makes it less confusing * test instance PATCH
2022-05-24[chore] Serialize times as UTC ISO8601 instead of RFC3339 (#602)Libravatar tobi1
* add time util to mimic utc ISO8601 * use ISO8601 when serializing to frontend * update test notification
2022-05-24[bugfix] Only search remote if protocol is http(s) (#601)Libravatar tobi1
2022-05-23[bugfix] Fix multiple dereferences of boosted status causing media ↵Libravatar tobi3
duplication (#589) * add some announces to test models * start on announce test logic * test federatingDB.Announce * change signature of GetRemoteStatus * remove 'refresh' logic and replace it with refetch * go fmt * remove timeline manager from processor test * make zork created at determinate * test get account statuses * test get + serialize zork * make account keys determinate * make admin accountCreate time determinate * test account to as * init test config before test log * test status to frontend * remove daft Within check * hack around a bit * use index of slice
2022-05-20[feature] implement custom_emojis endpoint (#563)Libravatar Adelie Paull5
* implement custom_emojis api endpoint * add tests for getting custom emoji out of the database and converting to api emoji * change sort direction of emoji query * change logging level and initialize array with known length as per kim's suggestions * add continue to lessen risk of making a malformed struct during conversion from db to api emojis
2022-05-18[bugfix] Clean up boosts of status when the status itself is deleted (#579)Libravatar tobi4
* move status wiping logic to fromcommon.go * delete reblogs of status when a status is deleted * add admin boost of zork to test model * update tests to make them more determinate * Merge branch 'main' into status_reblog_cleanup * move status wiping logic to fromcommon.go * delete reblogs of status when a status is deleted * add admin boost of zork to test model * update tests to make them more determinate * Merge branch 'main' into status_reblog_cleanup * test status delete via client api * go fmt
2022-05-15[feature] Unused avatar and header cleanup (#574)Libravatar tobi4
* rename + tidy up remote pruning * fix media attachment account join see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships * update logging to new function name * add get avatars and headers to bun * add pruneallmeta function * don't set uncached since we're deleting anyway * fix totalPruned being in wrong place * test pruning meta * go fmt ./... * rename mediaprune * add meta pruning to routine mediaprune * tidy up cleanup job scheduling * rename adminmediaremoteprune * update mediacleanup to use renamed prune func * update swagger docs a little bit * reuse cancel + context
2022-05-15[security] transport.Controller{} and transport.Transport{} security and ↵Libravatar kim8
performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-09[feature] Include password strength in error message when password strength ↵Libravatar Eamonn O'Brien-Strain1
is too low (#550) * When password validation fails, return how close to enough entropy it has. * Shorter version of low-strength password error message
2022-05-07[bugfix] Fix existing bio text showing as HTML (#531)Libravatar kim3
* fix existing bio text showing as HTML - updated replaced mentions to include instance - strips HTML from account source note in Verify handler - update text formatter to use buffers for string writes Signed-off-by: kim <grufwub@gmail.com> * go away linter Signed-off-by: kim <grufwub@gmail.com> * change buf reset location, change html mention tags Signed-off-by: kim <grufwub@gmail.com> * reduce FindLinks code complexity Signed-off-by: kim <grufwub@gmail.com> * fix HTML to text conversion Signed-off-by: kim <grufwub@gmail.com> * Update internal/regexes/regexes.go Co-authored-by: Mina Galić <mina.galic@puppet.com> * use improved html2text lib with more options Signed-off-by: kim <grufwub@gmail.com> * fix to produce actual plaintext from html Signed-off-by: kim <grufwub@gmail.com> * fix span tags instead written as space Signed-off-by: kim <grufwub@gmail.com> * performance improvements to regex replacements, fix link replace logic for un-html-ing in the future Signed-off-by: kim <grufwub@gmail.com> * fix tag/mention replacements to use input string, fix link replace to not include scheme Signed-off-by: kim <grufwub@gmail.com> * use matched input string for link replace href text Signed-off-by: kim <grufwub@gmail.com> * remove unused code (to appease linter :sobs:) Signed-off-by: kim <grufwub@gmail.com> * improve hashtagFinger regex to be more compliant Signed-off-by: kim <grufwub@gmail.com> * update breakReplacer to include both unix and windows line endings Signed-off-by: kim <grufwub@gmail.com> * add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts Signed-off-by: kim <grufwub@gmail.com> * drop unnecessary code Signed-off-by: kim <grufwub@gmail.com> * update text package tests to fix logic changes Signed-off-by: kim <grufwub@gmail.com> * add raw note content testing to account update and account verify Signed-off-by: kim <grufwub@gmail.com> * remove unused modules Signed-off-by: kim <grufwub@gmail.com> * fix emoji regex Signed-off-by: kim <grufwub@gmail.com> * fix replacement of hashtags Signed-off-by: kim <grufwub@gmail.com> * update code comment Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: Mina Galić <mina.galic@puppet.com>
2022-05-03[chore] Move initial MediaRemotePrune to cmd.server function (#536)Libravatar tobi4
* move initial prune up to the server start function * don't do initial prune when starting media manager * add admin/mediaremoteprune in processor
2022-05-02[bugfix] Allow self-boosting for any visibility but direct (#510)Libravatar Adelie Paull1
* create visibility filter for boostability and allow self-boosting for any visbility but direct messages * add a followers-only status to local_account_2 * fix typo in comment * add license header, unwrap errors, be explicit about non-boostable visibility settings to avoid rogue boosting from miscoded clients, use ID compare for checking if self-boosting * add tests for statusboostable filter * fix tests that were affected by adding a new status to the test data * fix the rest of tests affected by adding a status to the textrig data
2022-05-02[performance] Speed up some of the slower db queries (#523)Libravatar tobi2
* remove unnecessary LOWER() db calls * warn during slow db queries * use bundb built-in exists function * add db block test * update account block query * add domain block db test * optimize domain block query * fix implementing wrong test * exclude most columns when checking block * go fmt * remote more unnecessary use of LOWER()
2022-04-28replace async client API / federator msg processing with worker pools (#497)Libravatar kim26
* replace async client API / federator msg processing with worker pools * appease our lord-and-saviour, the linter
2022-04-26[chore] Return more useful errors from auth failure (#494)v0.3.1Libravatar tobi6
* try rsa_sha256 sig algo first * return more informative errors from auth * adapt to reworked auth function
2022-04-24[bug] Fix status API / status web API being case sensitive (#481)Libravatar tobi1
* make getStatus by id case-insensitive * test get status case insensitive * init config before log
2022-04-16[bugfix] Use background ctx for domain block side effects (#457)Libravatar tobi1
2022-04-15[feature] Web profile pages for accounts (#449)Libravatar tobi9
* add default avatars * allow webModule to error * return errWithCode from account get * add AccountGetLocalByUsername * check nil requesting account * add timestampShort function for just month/year * move loading logic to New + add default avatars * add profile page view * update swagger docs * add excludeReblogs to GetAccountStatuses * ignore casing when selecting local account by username * appropriate redirects * css fiddling * add 'about' heading * adjust thread page to work with routing * return AP representation if requested + authorized * simplify auth check * go fmt * golangci-lint ignore math/rand
generated by cgit v1.2.3 (git 2.46.0) at 2025-07-15 04:29:07 +0000