summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2022-05-24[chore] Serialize times as UTC ISO8601 instead of RFC3339 (#602)Libravatar tobi1
* add time util to mimic utc ISO8601 * use ISO8601 when serializing to frontend * update test notification
2022-05-24[bugfix] Only search remote if protocol is http(s) (#601)Libravatar tobi1
2022-05-23[bugfix] Fix multiple dereferences of boosted status causing media ↵Libravatar tobi3
duplication (#589) * add some announces to test models * start on announce test logic * test federatingDB.Announce * change signature of GetRemoteStatus * remove 'refresh' logic and replace it with refetch * go fmt * remove timeline manager from processor test * make zork created at determinate * test get account statuses * test get + serialize zork * make account keys determinate * make admin accountCreate time determinate * test account to as * init test config before test log * test status to frontend * remove daft Within check * hack around a bit * use index of slice
2022-05-20[feature] implement custom_emojis endpoint (#563)Libravatar Adelie Paull5
* implement custom_emojis api endpoint * add tests for getting custom emoji out of the database and converting to api emoji * change sort direction of emoji query * change logging level and initialize array with known length as per kim's suggestions * add continue to lessen risk of making a malformed struct during conversion from db to api emojis
2022-05-18[bugfix] Clean up boosts of status when the status itself is deleted (#579)Libravatar tobi4
* move status wiping logic to fromcommon.go * delete reblogs of status when a status is deleted * add admin boost of zork to test model * update tests to make them more determinate * Merge branch 'main' into status_reblog_cleanup * move status wiping logic to fromcommon.go * delete reblogs of status when a status is deleted * add admin boost of zork to test model * update tests to make them more determinate * Merge branch 'main' into status_reblog_cleanup * test status delete via client api * go fmt
2022-05-15[feature] Unused avatar and header cleanup (#574)Libravatar tobi4
* rename + tidy up remote pruning * fix media attachment account join see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships * update logging to new function name * add get avatars and headers to bun * add pruneallmeta function * don't set uncached since we're deleting anyway * fix totalPruned being in wrong place * test pruning meta * go fmt ./... * rename mediaprune * add meta pruning to routine mediaprune * tidy up cleanup job scheduling * rename adminmediaremoteprune * update mediacleanup to use renamed prune func * update swagger docs a little bit * reuse cancel + context
2022-05-15[security] transport.Controller{} and transport.Transport{} security and ↵Libravatar kim8
performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-09[feature] Include password strength in error message when password strength ↵Libravatar Eamonn O'Brien-Strain1
is too low (#550) * When password validation fails, return how close to enough entropy it has. * Shorter version of low-strength password error message
2022-05-07[bugfix] Fix existing bio text showing as HTML (#531)Libravatar kim3
* fix existing bio text showing as HTML - updated replaced mentions to include instance - strips HTML from account source note in Verify handler - update text formatter to use buffers for string writes Signed-off-by: kim <grufwub@gmail.com> * go away linter Signed-off-by: kim <grufwub@gmail.com> * change buf reset location, change html mention tags Signed-off-by: kim <grufwub@gmail.com> * reduce FindLinks code complexity Signed-off-by: kim <grufwub@gmail.com> * fix HTML to text conversion Signed-off-by: kim <grufwub@gmail.com> * Update internal/regexes/regexes.go Co-authored-by: Mina Galić <mina.galic@puppet.com> * use improved html2text lib with more options Signed-off-by: kim <grufwub@gmail.com> * fix to produce actual plaintext from html Signed-off-by: kim <grufwub@gmail.com> * fix span tags instead written as space Signed-off-by: kim <grufwub@gmail.com> * performance improvements to regex replacements, fix link replace logic for un-html-ing in the future Signed-off-by: kim <grufwub@gmail.com> * fix tag/mention replacements to use input string, fix link replace to not include scheme Signed-off-by: kim <grufwub@gmail.com> * use matched input string for link replace href text Signed-off-by: kim <grufwub@gmail.com> * remove unused code (to appease linter :sobs:) Signed-off-by: kim <grufwub@gmail.com> * improve hashtagFinger regex to be more compliant Signed-off-by: kim <grufwub@gmail.com> * update breakReplacer to include both unix and windows line endings Signed-off-by: kim <grufwub@gmail.com> * add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts Signed-off-by: kim <grufwub@gmail.com> * drop unnecessary code Signed-off-by: kim <grufwub@gmail.com> * update text package tests to fix logic changes Signed-off-by: kim <grufwub@gmail.com> * add raw note content testing to account update and account verify Signed-off-by: kim <grufwub@gmail.com> * remove unused modules Signed-off-by: kim <grufwub@gmail.com> * fix emoji regex Signed-off-by: kim <grufwub@gmail.com> * fix replacement of hashtags Signed-off-by: kim <grufwub@gmail.com> * update code comment Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: Mina Galić <mina.galic@puppet.com>
2022-05-03[chore] Move initial MediaRemotePrune to cmd.server function (#536)Libravatar tobi4
* move initial prune up to the server start function * don't do initial prune when starting media manager * add admin/mediaremoteprune in processor
2022-05-02[bugfix] Allow self-boosting for any visibility but direct (#510)Libravatar Adelie Paull1
* create visibility filter for boostability and allow self-boosting for any visbility but direct messages * add a followers-only status to local_account_2 * fix typo in comment * add license header, unwrap errors, be explicit about non-boostable visibility settings to avoid rogue boosting from miscoded clients, use ID compare for checking if self-boosting * add tests for statusboostable filter * fix tests that were affected by adding a new status to the test data * fix the rest of tests affected by adding a status to the textrig data
2022-05-02[performance] Speed up some of the slower db queries (#523)Libravatar tobi2
* remove unnecessary LOWER() db calls * warn during slow db queries * use bundb built-in exists function * add db block test * update account block query * add domain block db test * optimize domain block query * fix implementing wrong test * exclude most columns when checking block * go fmt * remote more unnecessary use of LOWER()
2022-04-28replace async client API / federator msg processing with worker pools (#497)Libravatar kim26
* replace async client API / federator msg processing with worker pools * appease our lord-and-saviour, the linter
2022-04-26[chore] Return more useful errors from auth failure (#494)v0.3.1Libravatar tobi6
* try rsa_sha256 sig algo first * return more informative errors from auth * adapt to reworked auth function
2022-04-24[bug] Fix status API / status web API being case sensitive (#481)Libravatar tobi1
* make getStatus by id case-insensitive * test get status case insensitive * init config before log
2022-04-16[bugfix] Use background ctx for domain block side effects (#457)Libravatar tobi1
2022-04-15[feature] Web profile pages for accounts (#449)Libravatar tobi9
* add default avatars * allow webModule to error * return errWithCode from account get * add AccountGetLocalByUsername * check nil requesting account * add timestampShort function for just month/year * move loading logic to New + add default avatars * add profile page view * update swagger docs * add excludeReblogs to GetAccountStatuses * ignore casing when selecting local account by username * appropriate redirects * css fiddling * add 'about' heading * adjust thread page to work with routing * return AP representation if requested + authorized * simplify auth check * go fmt * golangci-lint ignore math/rand
2022-04-02[chore] Update Go version to 1.18 (#444)Libravatar tobi2
* linting with new golangci-lint version * update go to 1.18 * bump versions in drone.yml * use new runtime/debug package for version info * remove Commit build flag from goreleaser * remove mock commit + version from build script * go fmt * add dummy version env flag to test container * install git in golang container for testing * only set versionString if Version is defined
2022-03-29[feature] Dereference remote mentions when the account is not already known ↵v0.2.2Libravatar tobi9
(#442) * remove mention util function from db * add ParseMentionFunc to gtsmodel * add parseMentionFunc to processor * refactor search to simplify it a bit * add parseMentionFunc to account * add parseMentionFunc to status * some renaming for clarity * test dereference of unknown mentioned account
2022-03-21[bugfix] Close ReadClosers properly in the media package (#434)Libravatar tobi2
* defer lock reader * close readers when finished with them * close the reader in the teereader when finished
2022-03-19[feature] Admin account actions (#432)Libravatar tobi4
* add accountAction to the admin API * model admin account action * add admin account action to the processor * add migration for new AdminAccountActions table * fix accounts admin path * Update swagger docs
2022-03-15[feature] Federate local account deletion (#431)Libravatar tobi7
* add account delete to API * model account delete request * add AccountDeleteLocal * federate local account deletes * add DeleteLocal * update transport (controller) to allow shortcuts * delete logic + testing * update swagger docs * more tests + fixes
2022-03-13[bugfix] Fix html-escaped characters in content warnings (#426)Libravatar tobi2
* test status create with odd CWs * use SanitizeCaption for content warning escaping
2022-03-07[feature] Clean up/uncache remote media (#407)Libravatar tobi7
* Add whereNotEmptyAndNotNull * Add GetRemoteOlderThanDays * Add GetRemoteOlderThanDays * Add PruneRemote to Manager interface * Start implementing PruneRemote * add new attachment + status to tests * fix up and test GetRemoteOlderThan * fix bad import * PruneRemote: return number pruned * add Cached column to mediaattachment * update + test pruneRemote * update mediaTest * use Cached column * upstep bun to latest version * embed structs in mediaAttachment * migrate mediaAttachment to new format * don't default cached to true * select only remote media * update db dependencies * step bun back to last working version * update pruneRemote to use Cached field * fix storage path of test attachments * add recache logic to manager * fix trimmed aspect ratio * test prune and recache * return errwithcode * tidy up different paths for emoji vs attachment * fix incorrect thumbnail type being stored * expose TransportController to media processor * implement tee-ing recached content * add thoughts of dog to test fedi attachments * test get remote files * add comment on PruneRemote * add postData cleanup to recache * test thumbnail fetching * add incredible diagram * go mod tidy * buffer pipes for recache streaming * test for client stops reading after 1kb * add media-remote-cache-days to config * add cron package * wrap logrus so it's available to cron * start and stop cron jobs gracefully
2022-02-22[feature] Add postDataCallbackFunc to allow cleanup (#408)Libravatar tobi3
2022-02-19[feature] Stream files via reader (#404)Libravatar tobi1
* serve files via reader rather than byte slice * close readcloser when we're done with it * cast reader to readcloser
2022-02-08merge fixupLibravatar tsmethurst1
2022-02-08Merge branch 'main' into media_refactorLibravatar tsmethurst9
2022-02-07[feature] add authorization to the already-existing authentication (#365)Libravatar Forest Johnson1
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize * adding authorization (email confirm, account approve, etc) to TokenCheck * revert un-needed changes to signin.go * oops what happened here * error css * add account.SuspendedAt check * remove redundant checks from oauth util Authed function * wip tests * tests passing * stop stripping useful information from ErrAlreadyExists * that feeling of scraping the dryer LINT off the screen * oops I didn't mean to get rid of this NewTestRouter function * make tests work with recorder * re-add ConfigureTemplatesWithGin to handle template path err Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-02-06if accountDomain isn't set, use Host value instead (#380)Libravatar tobi1
2022-02-05[feature] Rework timeline code to make it useful for more than just statuses ↵Libravatar tobi6
(#373) * add preparable and timelineable interfaces * initialize timeline manager within the processor * generic renaming * move status-specific timeline logic into the processor * refactor timeline to make it useful for more than statuses
2022-01-31[bug] Send plaintext emails to fix "message refused: Message is not RFC 2822 ↵Libravatar Forest Johnson1
compliant" (#366) * trying to fix "message refused: Message is not RFC 2822 compliant" * fix "message refused: Message is not RFC 2822 compliant" 550 5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant * remove silly regex * lint * fix tests * we should use text/template instead of html/template now
2022-01-25ensure blocking calls to getRemoteAccount before showing stuff to clientLibravatar tsmethurst1
2022-01-24ctx => innerctxLibravatar tsmethurst1
2022-01-24update remote account get/deref logicLibravatar tsmethurst9
2022-01-23rework data function to provide filesizeLibravatar tsmethurst3
2022-01-16pass reader around instead of []byteLibravatar tsmethurst3
2022-01-15emoji code passing musterLibravatar tsmethurst4
2022-01-15start fixing up emoji processing codeLibravatar tsmethurst1
2022-01-11pass a function into the manager, start work on emojiLibravatar tsmethurst3
2022-01-10test the media manager a bit, add shutdown logicLibravatar tsmethurst4
2022-01-09tests are passing, but there's still much to be doneLibravatar tsmethurst3
2022-01-08compiling nowLibravatar tsmethurst3
2022-01-08further refinementsLibravatar tsmethurst1
2021-12-28more refactoring, media handler => managerLibravatar tsmethurst9
2021-12-20Extend license notices to 2022 (#354)Libravatar tobi92
2021-12-20Remove unnecessary storage config variables (#344)Libravatar tobi9
* rewire config to not use extraneous serve vars * rename 'file' to 'local' for consistency * use Type and Size again
2021-12-07Implement Cobra CLI tooling, Viper config tooling (#336)Libravatar tobi26
* start pulling out + replacing urfave and config * replace many many instances of config * move more stuff => viper * properly remove urfave * move some flags to root command * add testrig commands to root * alias config file keys * start adding cli parsing tests * reorder viper init * remove config path alias * fmt * change config file keys to non-nested * we're more or less in business now * tidy up the common func * go fmt * get tests passing again * add note about the cliparsing tests * reorganize * update docs with changes * structure cmd dir better * rename + move some files around * fix dangling comma
2021-11-22Fix streamed messages ending up in wrong timeline(s) (#325)Libravatar tobi9
* define timeline consts * remove double stream of status * change test stream creation up a bit * stream messages more selectively * add test for streaming new status creation via clientAPI * tidy code + comments a bit * tidy up tests * make sure new status isn't streamed to public
2021-11-22Fix image description unnecessarily html-escaping innocent characters (#321)Libravatar tobi2
* implement SanitizeCaption function * tidy up text test setup
generated by cgit v1.2.3 (git 2.46.0) at 2025-07-14 02:58:44 +0000