summaryrefslogtreecommitdiff
path: root/internal/media
AgeCommit message (Collapse)AuthorFiles
2022-06-30[feature] Cleanup unattached local media (#680)Libravatar tobi5
* add localUnattached db function * add parseOlderThan util function * add pruneunusedlocalattachments to media manager * add unusedlocal pruning to schedule + admin call * set number of days to keep as a const * fix test
2022-06-26[feature] add configuration to `/api/v1/instance` response (#670)Libravatar tobi1
* add configuration object to api instance model * regenerate swagger docs * add func to return all supported mimes for media * add instance configuration to api serialization * fix json tags * update instance endpoint tests * fix typeutils tests * final regen of swagger docs * omitempty instance configuration
2022-05-30[chore] Global server configuration overhaul (#575)Libravatar kim1
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct Signed-off-by: kim <grufwub@gmail.com> * improved code comment Signed-off-by: kim <grufwub@gmail.com> * linter Signed-off-by: kim <grufwub@gmail.com> * fix unmarshaling Signed-off-by: kim <grufwub@gmail.com> * remove kim's custom go compiler changes Signed-off-by: kim <grufwub@gmail.com> * generate setter and flag-name functions, implement these in codebase Signed-off-by: kim <grufwub@gmail.com> * update deps Signed-off-by: kim <grufwub@gmail.com> * small change Signed-off-by: kim <grufwub@gmail.com> * appease the linter... Signed-off-by: kim <grufwub@gmail.com> * move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync Signed-off-by: kim <grufwub@gmail.com> * lint Signed-off-by: kim <grufwub@gmail.com> * update code comments Signed-off-by: kim <grufwub@gmail.com> * fix merge issue Signed-off-by: kim <grufwub@gmail.com> * fix merge issue Signed-off-by: kim <grufwub@gmail.com> * improved version string (removes time + go version) Signed-off-by: kim <grufwub@gmail.com> * fix version string build to pass test script + consolidate logic in func Signed-off-by: kim <grufwub@gmail.com> * add license text, update config.Defaults comment Signed-off-by: kim <grufwub@gmail.com> * add license text to generated config helpers file Signed-off-by: kim <grufwub@gmail.com> * defer unlock on config.Set___(), to ensure unlocked on panic Signed-off-by: kim <grufwub@gmail.com> * make it more obvious which cmd flags are being attached Signed-off-by: kim <grufwub@gmail.com>
2022-05-15[feature] Unused avatar and header cleanup (#574)Libravatar tobi6
* rename + tidy up remote pruning * fix media attachment account join see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships * update logging to new function name * add get avatars and headers to bun * add pruneallmeta function * don't set uncached since we're deleting anyway * fix totalPruned being in wrong place * test pruning meta * go fmt ./... * rename mediaprune * add meta pruning to routine mediaprune * tidy up cleanup job scheduling * rename adminmediaremoteprune * update mediacleanup to use renamed prune func * update swagger docs a little bit * reuse cancel + context
2022-05-15[feature] Media cleanup endpoint (#560)Libravatar Sashanoraa1
Adds an admin endpoint to trigger a remote media cleanup. Fixed #348 Signed-off-by: Sashanoraa <sasha@noraa.gay>
2022-05-15[security] transport.Controller{} and transport.Transport{} security and ↵Libravatar kim1
performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-08[bugfix] Fix remote media pruning failing if media already gone (#548)Libravatar tobi1
* fix error check of prune to allow missing files * update go-store library, add test for pruning item with db entry but no file Signed-off-by: kim <grufwub@gmail.com> * remove now-unneccessary error check Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: kim <grufwub@gmail.com>
2022-05-07[feature] Update media manager to use internal/worker package (#543)Libravatar kim2
* update media manager to use internal/worker package, update worker with better logging Signed-off-by: kim <grufwub@gmail.com> * fix Queue() trace log message format operators Signed-off-by: kim <grufwub@gmail.com> * update media manager comment to match updated worker implementation Signed-off-by: kim <grufwub@gmail.com>
2022-05-03[chore] Move initial MediaRemotePrune to cmd.server function (#536)Libravatar tobi1
* move initial prune up to the server start function * don't do initial prune when starting media manager * add admin/mediaremoteprune in processor
2022-05-03[feature] Run initial cache cleanup on gotosocial start in case max-age ↵Libravatar kim1
changed (#533) Signed-off-by: kim <grufwub@gmail.com>
2022-04-25[bugfix] Allow processing of .png files where checksum is not correct (#487)Libravatar tobi7
* add png stripping code from google/wuffs * experiment with stripping data from pngs * add test images * use StrippedPngDecode for pngs * add StrippedPngDecode func * update tests for (no)alphachannel pngs * nolint on copied function
2022-04-24[chore] Add two new test images for media processing (#424)Libravatar 0hlov32
2022-04-02[chore] Update Go version to 1.18 (#444)Libravatar tobi1
* linting with new golangci-lint version * update go to 1.18 * bump versions in drone.yml * use new runtime/debug package for version info * remove Commit build flag from goreleaser * remove mock commit + version from build script * go fmt * add dummy version env flag to test container * install git in golang container for testing * only set versionString if Version is defined
2022-03-22 [bugfix] Fix media uploads blocking after a failed upload #438 Libravatar tobi1
safer in case of failure
2022-03-21[bugfix] Fix images not being processed correctly sometimes (#437)Libravatar tobi2
* bump exif-terminator to latest version * add and test giant turnip from turnip.farm * don't error if content property is nil
2022-03-21[bugfix] Close ReadClosers properly in the media package (#434)Libravatar tobi2
* defer lock reader * close readers when finished with them * close the reader in the teereader when finished
2022-03-07[feature] Clean up/uncache remote media (#407)Libravatar tobi7
* Add whereNotEmptyAndNotNull * Add GetRemoteOlderThanDays * Add GetRemoteOlderThanDays * Add PruneRemote to Manager interface * Start implementing PruneRemote * add new attachment + status to tests * fix up and test GetRemoteOlderThan * fix bad import * PruneRemote: return number pruned * add Cached column to mediaattachment * update + test pruneRemote * update mediaTest * use Cached column * upstep bun to latest version * embed structs in mediaAttachment * migrate mediaAttachment to new format * don't default cached to true * select only remote media * update db dependencies * step bun back to last working version * update pruneRemote to use Cached field * fix storage path of test attachments * add recache logic to manager * fix trimmed aspect ratio * test prune and recache * return errwithcode * tidy up different paths for emoji vs attachment * fix incorrect thumbnail type being stored * expose TransportController to media processor * implement tee-ing recached content * add thoughts of dog to test fedi attachments * test get remote files * add comment on PruneRemote * add postData cleanup to recache * test thumbnail fetching * add incredible diagram * go mod tidy * buffer pipes for recache streaming * test for client stops reading after 1kb * add media-remote-cache-days to config * add cron package * wrap logrus so it's available to cron * start and stop cron jobs gracefully
2022-02-22[feature] Add postDataCallbackFunc to allow cleanup (#408)Libravatar tobi5
2022-02-21[bugfix] Fix thumbnail image type (#406)Libravatar tobi2
* fix thumbnail content-type * test fix thumbnail content-type
2022-02-08store and retrieve processState atomicallyLibravatar tsmethurst3
2022-01-29put store lock file in base of storageLibravatar tsmethurst1
2022-01-24test with disk storage as wellLibravatar tsmethurst1
2022-01-23add file size checksLibravatar tsmethurst1
2022-01-23rework data function to provide filesizeLibravatar tsmethurst5
2022-01-16pass reader around instead of []byteLibravatar tsmethurst6
2022-01-15lint, fmtLibravatar tsmethurst2
2022-01-15emoji code passing musterLibravatar tsmethurst2
2022-01-15start fixing up emoji processing codeLibravatar tsmethurst6
2022-01-11pin instanceAccountID to in-process emojiLibravatar tsmethurst1
2022-01-11pass a function into the manager, start work on emojiLibravatar tsmethurst6
2022-01-10test the media manager a bit, add shutdown logicLibravatar tsmethurst9
2022-01-09tests are passing, but there's still much to be doneLibravatar tsmethurst6
2022-01-08compiling nowLibravatar tsmethurst4
2022-01-08further refinementsLibravatar tsmethurst3
2022-01-04start working on thumb + full funcsLibravatar tsmethurst3
2022-01-03return very partial image on first uploadLibravatar tsmethurst5
2022-01-02fiddle around with workersLibravatar tsmethurst7
2021-12-28more refactoring, media handler => managerLibravatar tsmethurst7
2021-12-27start refactor of media packageLibravatar tsmethurst6
2021-12-20Extend license notices to 2022 (#354)Libravatar tobi6
2021-12-20Remove unnecessary storage config variables (#344)Libravatar tobi4
* rewire config to not use extraneous serve vars * rename 'file' to 'local' for consistency * use Type and Size again
2021-12-12Add optional syslog logrus hook (#343)Libravatar tobi1
* add optional syslog logrus hook * document syslog
2021-12-07Implement Cobra CLI tooling, Viper config tooling (#336)Libravatar tobi3
* start pulling out + replacing urfave and config * replace many many instances of config * move more stuff => viper * properly remove urfave * move some flags to root command * add testrig commands to root * alias config file keys * start adding cli parsing tests * reorder viper init * remove config path alias * fmt * change config file keys to non-nested * we're more or less in business now * tidy up the common func * go fmt * get tests passing again * add note about the cliparsing tests * reorganize * update docs with changes * structure cmd dir better * rename + move some files around * fix dangling comma
2021-11-13update dependencies (#296)Libravatar tobi1
2021-10-11reference global logrus (#274)Libravatar R. Aidan Campbell2
* reference logrus' global logger instead of passing and storing a logger reference everywhere * always directly use global logrus logger instead of referencing an instance * test suites should also directly use the global logrus logger * rename gin logging function to clarify that it's middleware * correct comments which erroneously referenced removed logger parameter * setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-04Refactor/tidy (#261)Libravatar tobi1
* tidy up streaming * cut down code duplication * test get followers/following * test streaming processor * fix some test models * add TimeMustParse * fix uri / url typo * make trace logging less verbose * make logging more consistent * disable quote on logging * remove context.Background * remove many extraneous mastodon references * regenerate swagger * don't log query on no rows result * log latency first for easier reading
2021-09-23update media processing (#244)Libravatar tobi6
2021-09-11fix keys used to access storage itemsLibravatar kim (grufwub)3
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11rename KVStore uses as storage to differentiate between upcoming store packageLibravatar kim (grufwub)3
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11add git.iim.gay/grufwub/go-store for storage backend, replacing blob.StorageLibravatar kim (grufwub)3
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
generated by cgit v1.2.3 (git 2.46.0) at 2025-09-25 20:33:42 +0000