summaryrefslogtreecommitdiff
path: root/internal/media/processingmedia.go
AgeCommit message (Collapse)AuthorFiles
2024-11-04[bugfix] determine mime-type to use during ffprobe evaluation stage, don't ↵Libravatar kim1
bother checking against file extension (#3506) * determine mime-type to use during ffprobe evaluation stage, don't bother rechecking by file extension * set mjpeg content-type * fix up tests expecting differing default values
2024-10-28[bugfix] support classifying correct video codec without audio as webm (#3494)Libravatar kim1
* for webm support video:[vp8,vp9,av1] and audio:[NONE,vorbis,opus] * improved unsupported data type error output
2024-09-26[bugfix] s3 media uploaded without content-type (#3353)Libravatar kim1
* update go-storage dependency, for S3Storage manually call PutObject() so we can set content-type * update calls to PutFile() to include the contentType
2024-08-19[performance] use native Go code to probe JPEGs (#3206)Libravatar kim1
* use native Go code to probe JPEGs * add note about copying from github.com/disintegration/imaging * add more code comments
2024-08-08[performance] move thumbnail generation to go code where possible (#3183)Libravatar kim1
* wrap thumbnailing code to handle generation natively where possible * more code comments! * add even more code comments! * add code comments about blurhash generation * maintain image rotation if contained in exif data * move rotation before resizing * ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs * use linear instead of nearest-neighbour for resizing * work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation * add header to new file * use thumb extension when getting thumb mime type * update test models and tests with new media processing * add suggested code comments * add note about thumbnail filter count reducing memory usage
2024-08-08[feature] Use gifv type for short soundless mp4 videos (#3182)Libravatar tobi1
2024-08-02[chore] add back exif-terminator and use only for jpeg,png,webp (#3161)Libravatar kim1
* add back exif-terminator and use only for jpeg,png,webp * fix arguments passed to terminateExif() * pull in latest exif-terminator * fix test * update processed img --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-31[chore] Take account of rotation data when calculating full size image ↵Libravatar tobi1
dimensions (#3159) * [chore] Take account of rotation data when calculating full size image dimensions * boobies
2024-07-30[chore] add some more slice related utility functions + remove duplicated ↵Libravatar kim1
functions (#3149)
2024-07-28[bugfix] take into account rotation when generating thumbnail (#3147)Libravatar kim1
* take into account rotation when generating thumbnail, simplify ffprobe output to show only fields we need * only show rotation side data * remove unnecessary comment * fix code comments * remove debug logging
2024-07-28[bugfix] moves file rename to earlier in media pipeline so ffmpeg calls ↵Libravatar kim1
ALWAYS have extension (#3146)
2024-07-22[bugfix] media.Processor{}.GetFile() returning 404s on first call, correctly ↵Libravatar kim1
loading on 2nd (#3129) * refactor file handling a tiny bit * whoops * make processing media / emoji defers a bit clear to see that it's the "on finished processing" path * some wording * add some debug logging * add mutex locks for processing remote media * try removing freshness check * fix derefMedia not being allocated * fix log format string * handle case of empty file paths (i.e. not stored) * remove media / emoji once finished processing from dereferencer maps * whoops, fix the cached / force checks * move url parsing outside of 'process___Safely()' funcs to prevalidate url * use emoji.ShortcodeDomain() * update RefreshEmoji() to also match RefreshMedia() changes --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-19[feature] use webp for thumbnails (#3116)Libravatar kim1
* update to use webp for thumbnails * bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting) * update to use yuva colorspace, and use thumbnail=n=10 to select frame * fix missing comma in ffmpeg args * add links to appropriate ffmpeg docs * update tests * add file size tests for thumbnails --------- Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-17[chore] media pipeline improvements (#3110)Libravatar kim1
* don't set emoji / media image paths on failed download, migrate FileType from string to integer * fix incorrect uses of util.PtrOr, fix returned frontend media * fix migration not setting arguments correctly in where clause * fix not providing default with not null column * whoops * ensure a default gets set for media attachment file type * remove the exclusive flag from writing files in disk storage * rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match * slight wording changes * use singular / plural word forms (no parentheses), is better for screen readers * update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests * store first instance in ffmpeg wasm pool, fill remaining with closed instances
2024-07-15[feature] more filetype support! (#3107)Libravatar kim1
* add more supported file types to our media processor that ffmpeg supports, update supported mime type lists * add code comments to the supported mime types slice * don't check for zero value string, just parse * remove some unneeded consts which make the code a bit harder to read * fix test expected instance media mime types, use compact ffprobe json, simple media processing by type * final tweaks to media processing code * don't use safe divide where we don't need to
2024-07-15[feature/frontend] Add player for audio files; use thumbnail for `poster` ↵Libravatar tobi1
(#3099) * [feature/frontend] Audio player for audio media types * use video preview images for previews instead of video itself * don't preload * update tests for new zork status * collapse media gallery into single row when small
2024-07-12[feature] support processing of (many) more media types (#3090)Libravatar kim1
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg * specify the video codec to use when generating static image from emoji * update go-storage library (fixes incompatibility after updating go-iotools) * maintain image aspect ratio when generating a thumbnail for it * update readme to show go-ffmpreg * fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility * remove extra debug from error message * fix up incorrect function signatures * update PutFile to just use regular file copy, as changes are file is on separate partition * fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe * update more tests, add more code comments * add utilities to generate processed emoji / media outputs * fix remaining tests * add test for opus media file, add license header to utility cmds * limit the number of concurrently available ffmpeg / ffprobe instances * reduce number of instances * further reduce number of instances * fix envparsing test with configuration variables * update docs and configuration with new media-{local,remote}-max-size variables
2024-06-26[chore] media and emoji refactoring (#3000)Libravatar kim1
* start updating media manager interface ready for storing attachments / emoji right away * store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load() * remove now unused media workers * fix tests and issues * fix another test! * fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues * fix more tests * fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis * whoops, rebase issue * remove kim's whacky experiments * do some reshuffling, ensure emoji uri gets set * ensure marked as not cached on cleanup * tweaks to media / emoji processing to handle context canceled better * ensure newly fetched emojis actually get set in returned slice * use different varnames to be a bit more obvious * move emoji refresh rate limiting to dereferencer * add exported dereferencer functions for remote media, use these for recaching in processor * add check for nil attachment in updateAttachment() * remove unused emoji and media fields + columns * see previous commit * fix old migrations expecting image_updated_at to exists (from copies of old models) * remove freshness checking code (seems to be broken...) * fix error arg causing nil ptr exception * finish documentating functions with comments, slight tweaks to media / emoji deref error logic * remove some extra unneeded boolean checking * finish writing documentation (code comments) for exported media manager methods * undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot * move doesColumnExist() to util.go in migrations package
2024-05-22[performance] update storage backend and make use of seek syscall when ↵Libravatar kim1
available (#2924) * update to use go-storage/ instead of go-store/v2/storage/ * pull in latest version from codeberg * remove test output :innocent: * add code comments * set the exclusive bit when creating new files in disk config * bump to actual release version * bump to v0.1.1 (tis a simple no-logic change) * update readme * only use a temporary read seeker when decoding video if required (should only be S3 now) * use fastcopy library to use memory pooled buffers when calling TempFileSeeker() * update to use seek call in serveFileRange()
2024-04-26[performance] update remaining worker pools to use queues (#2865)Libravatar kim1
* start replacing client + federator + media workers with new worker + queue types * refactor federatingDB.Delete(), drop queued messages when deleting account / status * move all queue purging to the processor workers * undo toolchain updates * code comments, ensure dereferencer worker pool gets started * update gruf libraries in readme * start the job scheduler separately to the worker pools * reshuffle ordering or server.go + remove duplicate worker start / stop * update go-list version * fix vendoring * move queue invalidation to before wipeing / deletion, to ensure queued work not dropped * add logging to worker processing functions in testrig, don't start workers in unexpected places * update go-structr to add (+then rely on) QueueCtx{} type * ensure more worker pools get started properly in tests * fix remaining broken tests relying on worker queue logic * fix account test suite queue popping logic, ensure noop workers do not pull from queue * move back accidentally shuffled account deletion order * ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete() * silently drop deletes from accounts not permitted to * don't warn log on forwarded deletes * make if else clauses easier to parse * use getFederatorMsg() * improved code comment * improved code comment re: requesting account delete checks * remove boolean result from worker start / stop since false = already running or already stopped * remove optional passed-in http.client * remove worker starting from the admin CLI commands (we don't need to handle side-effects) * update prune cli to start scheduler but not all of the workers * fix rebase issues * remove redundant return statements * i'm sorry sir linter
2024-01-15[chore] Move to codeberg's exif-terminator (#2536)Libravatar tobi1
2023-11-30[bugfix] return 400 Bad Request on more cases of malformed AS data (#2399)Libravatar kim1
2023-11-30[bugfix] Update exif-terminator (fix png issue) (#2391)Libravatar tobi1
* [bugfix] Update exif-terminator (fix png issue) * bump exif terminator * fix tests
2023-11-10[feature] Media attachment placeholders (#2331)Libravatar tobi1
* [feature] Use placeholders for unknown media types * fix read of underreported small files * switch to reduce nesting * simplify cleanup
2023-10-25[bugfix] allow store smaller PNG image than 261 bytes (#2263) (#2298)Libravatar KEINOS1
* chore: add test of golden cases before fix of #2263 * chore: add test case to reproduce error of #2263 * [bugfix] allow store smaller PNG image than 261 bytes (#2263)
2023-06-22[bugfix/chore] oauth entropy fix + media cleanup tasks rewrite (#1853)Libravatar kim1
2023-05-28[chore] tidy up media manager, add calling func to errors, build-script ↵Libravatar kim1
improvements (#1835) * media manager tidy-up: de-interface and remove unused PostDataFunc Signed-off-by: kim <grufwub@gmail.com> * remove last traces of media.Manager being an interface Signed-off-by: kim <grufwub@gmail.com> * update error to provide caller, allow tuneable via build tags Signed-off-by: kim <grufwub@gmail.com> * remove kim-specific build script changes Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * update build-script to support externally setting build variables Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-05-12[feature] status refetch support (#1690)Libravatar kim1
* revamp http client to not limit requests, instead use sender worker Signed-off-by: kim <grufwub@gmail.com> * remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping Signed-off-by: kim <grufwub@gmail.com> * improve batch senders to keep popping recipients until remote URL found Signed-off-by: kim <grufwub@gmail.com> * fix recipient looping issue Signed-off-by: kim <grufwub@gmail.com> * move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP Signed-off-by: kim <grufwub@gmail.com> * first draft of status refetching logic Signed-off-by: kim <grufwub@gmail.com> * fix testrig to use new federation alloc func signature Signed-off-by: kim <grufwub@gmail.com> * fix log format directive Signed-off-by: kim <grufwub@gmail.com> * add status fetched_at migration Signed-off-by: kim <grufwub@gmail.com> * remove unused / unchecked for error types Signed-off-by: kim <grufwub@gmail.com> * add back the used type... Signed-off-by: kim <grufwub@gmail.com> * add separate internal getStatus() function for derefThread() that doesn't recurse Signed-off-by: kim <grufwub@gmail.com> * improved mention and media attachment error handling Signed-off-by: kim <grufwub@gmail.com> * fix log and error format directives Signed-off-by: kim <grufwub@gmail.com> * update account deref to match status deref changes Signed-off-by: kim <grufwub@gmail.com> * very small code formatting change to make things clearer Signed-off-by: kim <grufwub@gmail.com> * add more code comments Signed-off-by: kim <grufwub@gmail.com> * improved code commenting Signed-off-by: kim <grufwub@gmail.com> * only check for required further derefs if needed Signed-off-by: kim <grufwub@gmail.com> * improved cache invalidation Signed-off-by: kim <grufwub@gmail.com> * tweak cache restarting to use a (very small) backoff Signed-off-by: kim <grufwub@gmail.com> * small readability changes and fixes Signed-off-by: kim <grufwub@gmail.com> * fix account sync issues Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts + update account enrichment to accept already-passed accountable Signed-off-by: kim <grufwub@gmail.com> * remove secondary function declaration Signed-off-by: kim <grufwub@gmail.com> * normalise dereferencer get status / account behaviour, fix remaining tests Signed-off-by: kim <grufwub@gmail.com> * fix remaining rebase conflicts, finish commenting code Signed-off-by: kim <grufwub@gmail.com> * appease the linter Signed-off-by: kim <grufwub@gmail.com> * add source file header Signed-off-by: kim <grufwub@gmail.com> * update to use TIMESTAMPTZ column type instead of just TIMESTAMP Signed-off-by: kim <grufwub@gmail.com> * don't pass in 'updated_at' to UpdateEmoji() Signed-off-by: kim <grufwub@gmail.com> * use new ap.Resolve{Account,Status}able() functions Signed-off-by: kim <grufwub@gmail.com> * remove the somewhat confusing rescoping of the same variable names Signed-off-by: kim <grufwub@gmail.com> * update migration file name, improved database delete error returns Signed-off-by: kim <grufwub@gmail.com> * formatting Signed-off-by: kim <grufwub@gmail.com> * improved multi-delete database functions to minimise DB calls Signed-off-by: kim <grufwub@gmail.com> * remove unused type Signed-off-by: kim <grufwub@gmail.com> * fix delete statements Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-04-29[performance] tweak http client error handling (#1718)Libravatar kim1
* update errors library, check for more TLS type error in http client Signed-off-by: kim <grufwub@gmail.com> * bump cache library version to match errors library Signed-off-by: kim <grufwub@gmail.com> --------- Signed-off-by: kim <grufwub@gmail.com>
2023-03-12[chore] Improve copyright header handling (#1608)Libravatar Daenney1
* [chore] Remove years from all license headers Years or year ranges aren't required in license headers. Many projects have removed them in recent years and it avoids a bit of yearly toil. In many cases our copyright claim was also a bit dodgy since we added the 2021-2023 header to files created after 2021 but you can't claim copyright into the past that way. * [chore] Add license header check This ensures a license header is always added to any new file. This avoids maintainers/reviewers needing to remember to check for and ask for it in case a contribution doesn't include it. * [chore] Add missing license headers * [chore] Further updates to license header * Use the more common // indentend comment format * Remove the hack we had for the linter now that we use the // format * Add SPDX license identifier
2023-03-03[performance] cache media attachments (#1525)Libravatar kim1
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly * add media attachment caching, slightly tweak default cache config * further tweak default cache config values * replace other media attachment db calls to go through cache * update envparsing test * fix delete media attachment sql * fix media sql query * invalidate cached media entries during status create / update * fix envparsing test * fix typo in panic log message... * add 'updated_at' column during UpdateAttachment * remove unused func --------- Signed-off-by: kim <grufwub@gmail.com>
2023-02-17[feature] Add a request ID and include it in logs (#1476)Libravatar Daenney1
This adds a lightweight form of tracing to GTS. Each incoming request is assigned a Request ID which we then pass on and log in all our log lines. Any function that gets called downstream from an HTTP handler should now emit a requestID=value pair whenever it logs something. Co-authored-by: kim <grufwub@gmail.com>
2023-02-13[performance] processing media and scheduled jobs improvements (#1482)Libravatar kim1
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling * improved code comment * fix worker tryUntil function, update go-runners/go-sched * make preprocess functions package public, use these where possible to stop doubled up processing * remove separate emoji worker pool * limit calls to time.Now() during media preprocessing * use Processor{} to manage singular runtime of processing media * ensure workers get started when media manager is used * improved error setting in processing media, fix media test * port changes from processingmedia to processing emoji * finish code commenting * finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced * linterrrrrrrrrrrrrrrr --------- Signed-off-by: kim <grufwub@gmail.com>
2023-02-11[chore/performance] Update media prune logic, add extra CLI command (#1474)v0.7.0-rc2Libravatar tobi1
* start updating media prune stuff a wee bit * continue prune / uncache work * more tidying + consistency stuff * add prune CLI command * docs * arg
2023-01-11[performance] media processing improvements (#1288)Libravatar kim1
* media processor consolidation and reformatting, reduce amount of required syscalls Signed-off-by: kim <grufwub@gmail.com> * update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling Signed-off-by: kim <grufwub@gmail.com> * fix duration not being set, fix mp4 test expecting error Signed-off-by: kim <grufwub@gmail.com> * fix test expecting media files with different extension Signed-off-by: kim <grufwub@gmail.com> * remove unused code Signed-off-by: kim <grufwub@gmail.com> * fix expected storage paths in tests, update expected test thumbnails Signed-off-by: kim <grufwub@gmail.com> * remove dead code Signed-off-by: kim <grufwub@gmail.com> * fix cached presigned s3 url fetching Signed-off-by: kim <grufwub@gmail.com> * fix tests Signed-off-by: kim <grufwub@gmail.com> * fix test models Signed-off-by: kim <grufwub@gmail.com> * update media processing to use sync.Once{} for concurrency protection Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * fix passing in KVStore GetStream() as stream to PutStream() Signed-off-by: kim <grufwub@gmail.com> * fix unlocks of storage keys Signed-off-by: kim <grufwub@gmail.com> * whoops, return the error... Signed-off-by: kim <grufwub@gmail.com> * pour one out for tobi's code <3 Signed-off-by: kim <grufwub@gmail.com> * add back the byte slurping code Signed-off-by: kim <grufwub@gmail.com> * check for both ErrUnexpectedEOF and EOF Signed-off-by: kim <grufwub@gmail.com> * add back links to file format header information Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2023-01-05[chore] Update/add license headers for 2023 (#1304)Libravatar tobi1
2022-12-22[feature] For video attachments, store + return fps, bitrate, duration (#1282)Libravatar tobi1
* start messing about with different mp4 metadata extraction * heyyooo it works * add test cow * move useful multierror to gtserror package * error out if video doesn't seem to be a real mp4 * test parsing mkv in disguise as mp4 * tidy up error handling * remove extraneous line * update framerate formatting * use float32 for aspect * fixy mctesterson
2022-12-17[feature] Enable basic video support (mp4 only) (#1274)Libravatar tobi1
* [feature] basic video support * fix missing semicolon * replace text shadow with stacked icons Co-authored-by: f0x <f0x@cthu.lu>
2022-12-12[feature] allow uncaching of other media types (#1234)Libravatar kim1
* simplify pruneRemote, remove unncecessary media trace logging, update RemoteOlderThan() to include headers/avis Signed-off-by: kim <grufwub@gmail.com> * cleanup pruneallmeta, add remote header to pruneremote tests Signed-off-by: kim <grufwub@gmail.com> * fix olderthan duration additions Signed-off-by: kim <grufwub@gmail.com> * fix broken test now that test model header changed Signed-off-by: kim <grufwub@gmail.com> * instead use new remote test account for new header model Signed-off-by: kim <grufwub@gmail.com> * use newer generated ULID for remote_account_3 to ensure it is sorted last Signed-off-by: kim <grufwub@gmail.com> * reorganize serialized keys to match expected test account model order Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-12-06[feature] media: add webp support (#1155)Libravatar Sigrid Solveig Haflínudóttir1
* media: add webp support Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se> * bump exif-terminator to v0.5.0 Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se> Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
2022-11-24[chore] cleanup storage implementation, no need for multiple interface types ↵Libravatar kim1
(#1131) Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-11-11[chore] Fixes + updates in emoji dereferencing logic (#1022)Libravatar tobi1
* fix incorrect static remote url use for emojis * warn when emoji/attachment already exists * defer emoji postdata execution * rename ctx to innerCtx for clarity * warn on emoji too large * small efficiency fix in fetchRemoteAccountEmojis * tidy up lock+load * lock processing emojis * fix little fucky wucky * this wasn't go fmted for some reason
2022-11-11[chore] close in-storage media reader _before_ opening write, no need to ↵Libravatar kim1
leave it hanging around (#1016) Signed-off-by: kim <grufwub@gmail.com> Signed-off-by: kim <grufwub@gmail.com>
2022-11-03[bugfix] Wrap media in read closer (#941)Libravatar tobi1
* use readcloser for content.Content * call media postdata function no matter what * return a readcloser from data func * tidy of logic of readertostore * fix whoopsie
2022-10-08[chore] Standardize database queries, use `bun.Ident()` properly (#886)Libravatar tobi1
* use bun.Ident for user queries * use bun.Ident for account queries * use bun.Ident for media queries * add DeleteAccount func * remove CaseInsensitive in Where+use Ident ipv Safe * update admin db * update domain, use ident * update emoji, use ident * update instance queries, use bun.Ident * fix media * update mentions, use bun ident * update relationship + tests * use tableexpr * add test follows to bun db test suite * update notifications * updatebyprimarykey => updatebyid * fix session * prefer explicit ID to pk * fix little fucky wucky * remove workaround * use proper db func for attachment selection * update status db * add m2m entries in test rig * fix up timeline * go fmt * fix status put issue * update GetAccountStatuses
2022-09-29[feature] update config types to use bytesize.Size (#828)Libravatar kim1
* update config size types to use bytesize.Size * submit unchecked-out file ... :facepalm: * fix bytesize config var decoding * bump bytesize version * update kim's libraries in readme * update envparse.sh to output more useful errors * improve envparse.sh * remove reliance on jq * instead, use uint64 for bytesize flag types * remove redundant type * fix viper unmarshaling * Update envparsing.sh * fix envparsing test Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-09-24[bugfix] Wrap media reader in length reader to determine length if no ↵Libravatar tobi1
`content-length` given (#848) * use lengthReader 2 determine fileSize if not given * update tests * small fixes * go fmt
2022-09-19don't error out if storage key already exists (#840)Libravatar tobi1
2022-09-19[bugfix] Server and closer bugfixes (#839)Libravatar tobi1
* defer streaming from storage more forcefully * shut down Server more gracefully * use command context as server BaseContext
2022-09-04[bugfix] Use 'Image' instead of unrecognized 'Gif' type for media ↵Libravatar tobi1
attachments (#801) * Store gifs as Image type * remove Gif attachment type, add Gifv type * update test
generated by cgit v1.2.3 (git 2.46.0) at 2025-07-12 22:46:10 +0000