summaryrefslogtreecommitdiff
path: root/internal/processing
AgeCommit message (Collapse)AuthorFiles
2024-07-26[feature] Federate interaction policies + Accepts; enforce policies (#3138)Libravatar tobi10
* [feature] Federate interaction policies + Accepts; enforce policies * use Acceptable type * fix index * remove appendIRIStrs * add GetAccept federatingdb function * lock on object IRI
2024-07-24[chore] Add interaction filter to complement existing visibility filter (#3111)Libravatar tobi34
* [chore] Add interaction filter to complement existing visibility filter * pass in ptr to visibility and interaction filters to Processor{} to ensure shared * use int constants for for match type, cache db calls in filterctx * function name typo :innocent: --------- Co-authored-by: kim <grufwub@gmail.com>
2024-07-23[feature] Conversations API (#3013)Libravatar Vyr Cossont23
* Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-22[bugfix] media.Processor{}.GetFile() returning 404s on first call, correctly ↵Libravatar kim1
loading on 2nd (#3129) * refactor file handling a tiny bit * whoops * make processing media / emoji defers a bit clear to see that it's the "on finished processing" path * some wording * add some debug logging * add mutex locks for processing remote media * try removing freshness check * fix derefMedia not being allocated * fix log format string * handle case of empty file paths (i.e. not stored) * remove media / emoji once finished processing from dereferencer maps * whoops, fix the cached / force checks * move url parsing outside of 'process___Safely()' funcs to prevalidate url * use emoji.ShortcodeDomain() * update RefreshEmoji() to also match RefreshMedia() changes --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-21[feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, ↵Libravatar tobi2
and emojis (#3118) * [feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis * go fmt * fix tests * use static version of instance thumbnail when appropriate * use prefers-reduced-motion * simplify account conversion a bit * fix c&p error
2024-07-21[bugfix] update common get target account / status doing refresh async (#3124)Libravatar kim2
2024-07-20[chore/frontend] Use smaller webp images for logo and default avatars (#3119)Libravatar tobi2
2024-07-19[feature] use webp for thumbnails (#3116)Libravatar kim3
* update to use webp for thumbnails * bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting) * update to use yuva colorspace, and use thumbnail=n=10 to select frame * fix missing comma in ffmpeg args * add links to appropriate ffmpeg docs * update tests * add file size tests for thumbnails --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-17[chore] Fix typo (#3113)Libravatar tobi1
2024-07-17[chore] media pipeline improvements (#3110)Libravatar kim5
* don't set emoji / media image paths on failed download, migrate FileType from string to integer * fix incorrect uses of util.PtrOr, fix returned frontend media * fix migration not setting arguments correctly in where clause * fix not providing default with not null column * whoops * ensure a default gets set for media attachment file type * remove the exclusive flag from writing files in disk storage * rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match * slight wording changes * use singular / plural word forms (no parentheses), is better for screen readers * update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests * store first instance in ffmpeg wasm pool, fill remaining with closed instances
2024-07-17[feature] Allow users to set default interaction policies per status ↵Libravatar tobi3
visibility (#3108) * [feature] Allow users to set default interaction policies * use vars for default policies * avoid some code repetition * unfuck form binding * avoid bonkers loop * beep boop * put policyValsToAPIPolicyVals in separate function * don't bother with slices.Grow * oops
2024-07-15[feature/frontend] Add player for audio files; use thumbnail for `poster` ↵Libravatar tobi1
(#3099) * [feature/frontend] Audio player for audio media types * use video preview images for previews instead of video itself * don't preload * update tests for new zork status * collapse media gallery into single row when small
2024-07-13[chore/frontend] Tweak threading a bit, inform about hidden replies (#3097)Libravatar tobi1
* [chore/frontend] Tweak threading a bit, inform about hidden replies * whoops * round off bottom of replies col-header if no replies visible
2024-07-12[feature/frontend] Better visual separation between "main" thread and ↵Libravatar tobi4
"replies" (#3093) * [feature/frontend] Better web threading model * fix test * bwap * tweaks * more tweaks to wording * typo * indenting * adjust wording * aaa
2024-07-12[feature] support processing of (many) more media types (#3090)Libravatar kim5
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg * specify the video codec to use when generating static image from emoji * update go-storage library (fixes incompatibility after updating go-iotools) * maintain image aspect ratio when generating a thumbnail for it * update readme to show go-ffmpreg * fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility * remove extra debug from error message * fix up incorrect function signatures * update PutFile to just use regular file copy, as changes are file is on separate partition * fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe * update more tests, add more code comments * add utilities to generate processed emoji / media outputs * fix remaining tests * add test for opus media file, add license header to utility cmds * limit the number of concurrently available ffmpeg / ffprobe instances * reduce number of instances * further reduce number of instances * fix envparsing test with configuration variables * update docs and configuration with new media-{local,remote}-max-size variables
2024-07-11[chore] Add interaction policy gtsmodels (#3075)Libravatar tobi5
* [chore] introduce interaction policy gts models * update migration a smidge * fix copy paste typo * update migration * use int for InteractionType
2024-07-08[feature/frontend] Allow setting alt-text for avatar + header (#3086)Libravatar tobi2
2024-07-04[bugfix] Handle ErrHideStatus when preparing timeline statuses (#3071)Libravatar Vyr Cossont3
2024-07-03[chore] Allow gtsmodel to depend on util (#3068)Libravatar Vyr Cossont1
Convert one free function into a gtsmodel.Emoji method so that util does not depend on gtsmodel and can be used from it in the future
2024-06-30[bugfix] update SignupURL routing in email template. (#3055)Libravatar CDN1
2024-06-26[chore] media and emoji refactoring (#3000)Libravatar kim17
* start updating media manager interface ready for storing attachments / emoji right away * store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load() * remove now unused media workers * fix tests and issues * fix another test! * fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues * fix more tests * fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis * whoops, rebase issue * remove kim's whacky experiments * do some reshuffling, ensure emoji uri gets set * ensure marked as not cached on cleanup * tweaks to media / emoji processing to handle context canceled better * ensure newly fetched emojis actually get set in returned slice * use different varnames to be a bit more obvious * move emoji refresh rate limiting to dereferencer * add exported dereferencer functions for remote media, use these for recaching in processor * add check for nil attachment in updateAttachment() * remove unused emoji and media fields + columns * see previous commit * fix old migrations expecting image_updated_at to exists (from copies of old models) * remove freshness checking code (seems to be broken...) * fix error arg causing nil ptr exception * finish documentating functions with comments, slight tweaks to media / emoji deref error logic * remove some extra unneeded boolean checking * finish writing documentation (code comments) for exported media manager methods * undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot * move doesColumnExist() to util.go in migrations package
2024-06-24[chore]: Bump github.com/gorilla/feeds from 1.1.2 to 1.2.0 (#3035)Libravatar dependabot[bot]1
2024-06-18[bugfix] rename `include_types[]` to `types[]` (#3023)Libravatar tobi1
2024-06-18[feature/frontend] Reports frontend v2 (#3022)Libravatar tobi2
* use apiutil + paging in admin processor+handlers * we're making it happen * fix little whoopsie * styling for report list * don't youuuu forget about meee don't don't don't don't * last bits * sanitize content before showing in report statuses * update report docs
2024-06-17[feature] Implement types[] param for notifications (#3009)Libravatar Vyr Cossont2
Counterpart of exclude_types[]. Also updates Swagger spec for types[] to use the correct param name and enumerate possible values. Fixes #3003
2024-06-14[feature] filter API v2: Restore keywords_attributes and statuses_attributes ↵Libravatar Vyr Cossont2
(#2995) These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
2024-06-13[chore] Update move test (#3005)Libravatar tobi1
2024-06-12[bugfix] Ensure side effects handled for local unlocked follows (#2998)Libravatar tobi2
2024-06-11[bugfix] Deref stats async, serve stub collections if handshaking (#2990)v0.16.0-rc2Libravatar tobi3
* [bugfix] Deref stats async, allow peek if handshaking * don't return totalItems when handshaking or hiding collections * use GetLimit() * use StubAccountStats
2024-06-10[bugfix] boost and account recursion (#2982)Libravatar kim3
* fix possible infinite recursion if moved accounts are self-referential * adds a defensive check for a boost being a boost of a boost wrapper * add checks on input for a boost of a boost * remove unnecessary check * add protections on account move to prevent move recursion loops * separate status conversion without boost logic into separate function to remove risk of recursion * move boost check to boost function itself * formatting * use error 422 instead of 500 * use gtserror not standard errors package for error creation
2024-06-07[feature] Implement filters_changed stream event (#2972)Libravatar Vyr Cossont15
2024-06-06[feature] User muting (#2960)Libravatar Vyr Cossont16
* User muting * Address review feedback * Rename uniqueness constraint on user_mutes to match convention * Remove unused account_id from where clause * Add UserMute to NewTestDB * Update test/envparsing.sh with new and fixed cache stuff * Address tobi's review comments * Make compiledUserMuteListEntry.expired consistent with UserMute.Expired * Make sure mute_expires_at is serialized as an explicit null for indefinite mutes --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06[feature] Self-serve email change for users (#2957)Libravatar tobi22
* [feature] Email change * frontend stuff for changing email * docs * tests etc * differentiate more clearly between local user+account and account * populate user
2024-06-06[feature] do not uncache status / emoji media if attached status is ↵Libravatar kim1
bookmarked (#2956) * do not uncache status / emoji media if attached status is bookmarked * add status bookmark and bookmark IDs caches * update status bookmark tests * move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache * fix envparsing.sh test
2024-05-31[feature] Add from: search operator and account_id query param (#2943)Libravatar Vyr Cossont1
* Add from: search operator * Fix whitespace in Swagger YAML comment * Move query parsing into its own method * Document search * Clarify post search scope
2024-05-31[feature] Implement Filter API v2 (#2936)Libravatar Vyr Cossont16
* Use correct entity name * We support server-side filters now * Document filter v1 methods that can throw a 409 * Validate v1 filter phrase as filter title * Always check v1 filter API status codes in tests * Document keyword minimum requirement on filter API v1 * Make it possible to specify filter keyword update columns per filter keyword * Implement v2 filter API * Fix lint and tests * Update Swagger spec * Fix filter update test * Update Swagger spec *correctly* * Update actual files Swagger spec was generated from * Remove keywords_attributes and statuses_attributes * Add test for serialization of empty filter * More helpful messages when object is owned by wrong account
2024-05-29[feature] Implement profile API (#2926)Libravatar Vyr Cossont1
* Implement profile API This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header. See: https://docs.joinmastodon.org/methods/profile/ * Move profile media methods to media processor * Remove check for moved account
2024-05-27[feature/frontend] Let admins send test email to validate SMTP config (#2934)Libravatar tobi1
* [feature/frontend] Let admins send test email to validate SMTP config * wee
2024-05-27[chore] Fix report email link (#2932)Libravatar tobi1
2024-05-22[performance] update storage backend and make use of seek syscall when ↵Libravatar kim1
available (#2924) * update to use go-storage/ instead of go-store/v2/storage/ * pull in latest version from codeberg * remove test output :innocent: * add code comments * set the exclusive bit when creating new files in disk config * bump to actual release version * bump to v0.1.1 (tis a simple no-logic change) * update readme * only use a temporary read seeker when decoding video if required (should only be S3 now) * use fastcopy library to use memory pooled buffers when calling TempFileSeeker() * update to use seek call in serveFileRange()
2024-05-06feature: filters v2 server-side warning/hiding (#2793)Libravatar Vyr Cossont15
* Remove dead code * Filter statuses when converting to frontend representation * status.filtered is an array * Make matching case-insensitive * Remove TODOs that don't need to be done now * Add missing filter check for notification * lint: rename ErrHideStatus * APIFilterActionToFilterAction not used yet * swaggerino docseroni * Address review comments * Add apimodel.FilterActionNone --------- Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com> Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-05-02[bugfix] Lock when checking/creating notifs to avoid race (#2890)Libravatar tobi14
* [bugfix] Lock when checking/creating notifs to avoid race * test notif spam
2024-05-01[feature] Page through accounts as moderator (#2881)Libravatar tobi1
* [feature] Page through accounts as moderator * aaaaa * use COLLATE "C" for Postgres to ensure same ordering as SQLite * fix typo, test paging up * don't show moderation / info for our instance acct
2024-04-29[chore] Fix conflict in workers tests (#2880)Libravatar tobi3
* [chore] Fix conflict in workers tests * commenty-wenty
2024-04-26[bugfix] Fix error string typo (#2873)Libravatar kim1
2024-04-26[performance] update remaining worker pools to use queues (#2865)Libravatar kim34
* start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-04-17[feature] Status source endpoint (#2848)Libravatar tobi1
* [feature] statusSource endpoint * finish up
2024-04-17[feature] Stub status history endpoint (#2847)Libravatar tobi1
2024-04-16[feature/performance] Store account stats in separate table (#2831)Libravatar tobi13
* [feature/performance] Store account stats in separate table, get stats from remote * test account stats * add some missing increment / decrement calls * change stats function signatures * rejig logging a bit * use lock when updating stats
2024-04-15[performance] cached oauth database types (#2838)Libravatar kim1
* update token + client code to use struct caches * add code comments * slight tweak to default mem ratios * fix envparsing * add appropriate invalidate hooks * update the tokenstore sweeping function to rely on caches * update to use PutClient() * add ClientID to list of token struct indices