| Age | Commit message (Collapse) | Author | Files |
|---|
|
should help to debug https://codeberg.org/superseriousbusiness/gotosocial/issues/4309
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4387
Reviewed-by: Daenney <daenney@noreply.codeberg.org>
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
Adds 2FA management to the admin CLI. Also does some CLI refactoring so the functions we pass around are exported functions instead of changeable global variables.
closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4320
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4368
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
An implementation of [`scheduled_statuses`](https://docs.joinmastodon.org/methods/scheduled_statuses/). Will fix #1006.
this is heavily WIP and I need to reorganize some of the code, working on this made me somehow familiar with the codebase and led to my other recent contributions
i told some fops on fedi i'd work on this so i have no choice but to complete it 🤷♀️
btw iirc my avatar presents me working on this branch
Signed-off-by: nicole mikołajczyk <git@mkljczk.pl>
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4274
Co-authored-by: nicole mikołajczyk <git@mkljczk.pl>
Co-committed-by: nicole mikołajczyk <git@mkljczk.pl>
|
|
This finalizes the moving status filtering out of the type converter, and into its own `./internal/filter/` subpkg :)
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4306
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
- adds configuration for thumbnail maximum dimensions with warning on exceeding recommendations
- moves the media configuration vars into their own sub-struct
- replaces the configuration flag funcs with simple string consts
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4258
Reviewed-by: tobi <kipvandenbos@noreply.codeberg.org>
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
This separates our the user mute handling from the typeconverter code, and creates a new "mutes" filter type (in a similar vein to the visibility filter) subpkg with its own result cache. This is a heavy mix of both chore given that mute calculation shouldn't have been handled in the conversion to frontend API types, and a performance bonus since we don't need to load and calculate so many things each time, just the single result each time with all necessary invalidation handled by database cache hooks.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4202
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
after seeing a potential reported federating worker lockup i decided to start digging into the federatingdb code. this PR encompasses:
- removes one of our last unused interface types `federatingdb.DB{}`, replacing it with a struct type `*federatingdb.DB{}`
- in `transport.dereferenceLocal()` differentiates between an unsupported lookup type and ErrNoEntries to reduce unnecessary calls, and reduce potential lockups that may occur while trying to call our own endpoints that then call `federatingdb.Lock()`
- removes a bunch of the locks on follow state changes since the DB already synchronizes that
- removes the unnecessary `pub.Clock{}` struct field and type passed to the transport controller
frankly it would be great if we could remove the locking in `federatingdb.Lock()` and instead handle it ourselves as it gets very confusing trying to figure out what functions will have locks held. but i guess that's one for when we move further away from the go-fed/activity/pub package usage.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4178
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
# Description
This simplifies our OTEL setup by:
* Getting rid of some deprecated things.
* Using `autoexport` and letting things get configured by the `OTEL_` environment variables.
* Removing all the unnecessary config options.
## Checklist
Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]`
If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [ ] I/we have commented the added code, particularly in hard-to-understand areas.
- [x] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [x] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4110
Reviewed-by: tobi <kipvandenbos@noreply.codeberg.org>
Co-authored-by: Daenney <daenney@noreply.codeberg.org>
Co-committed-by: Daenney <daenney@noreply.codeberg.org>
|
|
# Description
- tweaks the NoLLaMas proof-of-work algorithm to further granularity on time spent computing solutions
- standardizes GoToSocial cookie security directive setting in a CookiePolicy{} type
## Checklist
- [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md).
- [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
- [x] I/we have not leveraged AI to create the proposed changes.
- [x] I/we have performed a self-review of added code.
- [x] I/we have written code that is legible and maintainable by others.
- [x] I/we have commented the added code, particularly in hard-to-understand areas.
- [ ] I/we have made any necessary changes to documentation.
- [ ] I/we have added tests that cover new code.
- [ ] I/we have run tests and they pass locally with the changes.
- [x] I/we have run `go fmt ./...` and `golangci-lint run`.
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4090
Co-authored-by: kim <grufwub@gmail.com>
Co-committed-by: kim <grufwub@gmail.com>
|
|
|
|
* start work rewriting timeline cache type
* further work rewriting timeline caching
* more work integration new timeline code
* remove old code
* add local timeline, fix up merge conflicts
* remove old use of go-bytes
* implement new timeline code into more areas of codebase, pull in latest go-mangler, go-mutexes, go-structr
* remove old timeline package, add local timeline cache
* remove references to old timeline types that needed starting up in tests
* start adding page validation
* fix test-identified timeline cache package issues
* fix up more tests, fix missing required changes, etc
* add exclusion for test.out in gitignore
* clarify some things better in code comments
* tweak cache size limits
* fix list timeline cache fetching
* further list timeline fixes
* linter, ssssssssshhhhhhhhhhhh please
* fix linter hints
* reslice the output if it's beyond length of 'lim'
* remove old timeline initialization code, bump go-structr to v0.9.4
* continued from previous commit
* improved code comments
* don't allow multiple entries for BoostOfID values to prevent repeated boosts of same boosts
* finish writing more code comments
* some variable renaming, for ease of following
* change the way we update lo,hi paging values during timeline load
* improved code comments for updated / returned lo , hi paging values
* finish writing code comments for the StatusTimeline{} type itself
* fill in more code comments
* update go-structr version to latest with changed timeline unique indexing logic
* have a local and public timeline *per user*
* rewrite calls to public / local timeline calls
* remove the zero length check, as lo, hi values might still be set
* simplify timeline cache loading, fix lo/hi returns, fix timeline invalidation side-effects missing for some federated actions
* swap the lo, hi values :facepalm:
* add (now) missing slice reverse of tag timeline statuses when paging ASC
* remove local / public caches (is out of scope for this work), share more timeline code
* remove unnecessary change
* again, remove more unused code
* remove unused function to appease the linter
* move boost checking to prepare function
* fix use of timeline.lastOrder, fix incorrect range functions used
* remove comments for repeat code
* remove the boost logic from prepare function
* do a maximum of 5 loads, not 10
* add repeat boost filtering logic, update go-structr, general improvements
* more code comments
* add important note
* fix timeline tests now that timelines are returned in page order
* remove unused field
* add StatusTimeline{} tests
* add more status timeline tests
* start adding preloading support
* ensure repeat boosts are marked in preloaded entries
* share a bunch of the database load code in timeline cache, don't clear timelines on relationship change
* add logic to allow dynamic clear / preloading of timelines
* comment-out unused functions, but leave in place as we might end-up using them
* fix timeline preload state check
* much improved status timeline code comments
* more code comments, don't bother inserting statuses if timeline not preloaded
* shift around some logic to make sure things aren't accidentally left set
* finish writing code comments
* remove trim-after-insert behaviour
* fix-up some comments referring to old logic
* remove unsetting of lo, hi
* fix preload repeatBoost checking logic
* don't return on status filter errors, these are usually transient
* better concurrency safety in Clear() and Done()
* fix test broken due to addition of preloader
* fix repeatBoost logic that doesn't account for already-hidden repeatBoosts
* ensure edit submodels are dropped on cache insertion
* update code-comment to expand CAS accronym
* use a plus1hULID() instead of 24h
* remove unused functions
* add note that public / local timeline requester can be nil
* fix incorrect visibility filtering of tag timeline statuses
* ensure we filter home timeline statuses on local only
* some small re-orderings to confirm query params in correct places
* fix the local only home timeline filter func
|
|
field (#3982)
* add IPPrefixes type so we don't need separate rate limit parsed field
* sshhhh please linter, mommy's working
|
|
* [feature] add TOTP two-factor authentication (2FA)
* use byteutil.S2B to avoid allocations when comparing + generating password hashes
* don't bother with string conversion for consts
* use io.ReadFull
* use MustGenerateSecret for backup codes
* rename util functions
|
|
* [bugfix] Return useful err on `server start` failure
* remove scheduler started func
* remove tryUntil
|
|
* [feature] Refactor tokens, allow multiple app redirect_uris
* move + tweak handlers a bit
* return error for unset oauth2.ClientStore funcs
* wrap UpdateToken with cache
* panic handling
* cheeky little time optimization
* unlock on error
|
|
* chore: update otel dependencies
* refactor: combine tracing & metrics in observability package
* chore: update example tracing compose file
|
|
* [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers
* use switch for RobotsHeaders
|
|
starting/migrating (#3705)
* [feature] Use maintenance router to serve 503 while server is starting/migrating
* love you linter, kissies
|
|
* [chore] skip `trusted-proxies` warning if ip excepted from rate limiting
* weep
* typo
* fix env parsing test
|
|
* Update push subscription API model to be Mastodon 4.0 compatible
* Add webpush-go dependency
# Conflicts:
# go.sum
* Single-row table for storing instance's VAPID key pair
* Generate VAPID key pair during startup
* Add VAPID public key to instance info API
* Return VAPID public key when registering an app
* Store Web Push subscriptions in DB
* Add Web Push sender (similar to email sender)
* Add no-op push senders to most processor tests
* Test Web Push notifications from workers
* Delete Web Push subscriptions when account is deleted
* Implement push subscription API
* Linter fixes
* Update Swagger
* Fix enum to int migration
* Fix GetVAPIDKeyPair
* Create web push subscriptions table with indexes
* Log Web Push server error messages
* Send instance URL as Web Push JWT subject
* Accept any 2xx code as a success
* Fix malformed VAPID sub claim
* Use packed notification flags
* Remove unused date columns
* Add notification type for update notifications
Not used yet
* Make GetVAPIDKeyPair idempotent
and remove PutVAPIDKeyPair
* Post-rebase fixes
* go mod tidy
* Special-case 400 errors other than 408/429
Most client errors should remove the subscription.
* Improve titles, trim body to reasonable length
* Disallow cleartext HTTP for Web Push servers
* Fix lint
* Remove redundant index on unique column
Also removes redundant unique and notnull tags on ID column since these are implied by pk
* Make realsender.go more readable
* Use Tobi's style for wrapping errors
* Restore treating all 5xx codes as temporary problems
* Always load target account settings
* Stub `policy` and `standard`
* webpush.Sender: take type converter as ctor param
* Move webpush.MockSender and noopSender into testrig
|
|
* peepeepoopoo
* test domain perm subs
* swagger
* envparsing
* dries your wets
* start on docs
* finish up docs
* copy paste errors
* rename actions package
* rename force -> skipCache
* move obfuscate parse nearer to where err is checked
* make higherPrios a simple slice
* don't use receiver for permsFrom funcs
* add more context to error logs
* defer finished log
* use switch for permType instead of if/else
* thanks linter, love you <3
* validate csv headers before full read
* use bufio scanner
|
|
|
|
during db migrations (#3242)
|
|
|
|
|
|
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup
* ensure the tasks are sorted by creation time before pushing them
* add migration to insert WorkerTask{} into database, add test for worker task persistence
* add test for recovering worker queues from database
* quick tweak
* whoops we ended up with double cleaner job scheduling
* insert each task separately, because bun is throwing some reflection error??
* add specific checking of cancelled worker contexts
* add http request signing to deliveries recovered from database
* add test for outgoing public key ID being correctly set on delivery
* replace select with Queue.PopCtx()
* get rid of loop now we don't use it
* remove field now we don't use it
* ensure that signing func is set
* header values weren't being copied over :facepalm:
* use ptr for httpclient.Request in delivery
* move worker queue filling to later in server init process
* fix rebase issues
* make logging less shouty
* use slices.Delete() instead of copying / reslicing
* have database return tasks in ascending order instead of sorting them
* add a 1 minute timeout to persisting worker queues
|
|
* [chore] Add interaction filter to complement existing visibility filter
* pass in ptr to visibility and interaction filters to Processor{} to ensure shared
* use int constants for for match type, cache db calls in filterctx
* function name typo :innocent:
---------
Co-authored-by: kim <grufwub@gmail.com>
|
|
* Implement conversations API
* Sort and page conversations by last status ID
* Appease linter
* Fix deleting conversations and statuses
* Refactor to make migrations automatic
* Lint
* Update tests post-merge
* Fixes from live-fire testing
* Linter caught a format problem
* Refactor tests, fix cache
* Negative test for non-DMs
* Run conversations advanced migration on testrig startup as well as regular server startup
* Document (lack of) side effects of API method for deleting a conversation
* Make not-found check less nested for readability
* Rename PutConversation to UpsertConversation
* Use util.Ptr instead of IIFE
* Reduce cache used by conversations
* Remove unnecessary TableExpr/ColumnExpr
* Use struct tags for both unique constraints on Conversation
* Make it clear how paging with GetDirectStatusIDsBatch should be used
* Let conversation paging skip conversations it can't render
* Use Bun NewDropTable
* Convert delete raw query to Bun
* Convert update raw query to Bun
* Convert latestConversationStatusesTempTable raw query partially to Bun
* Convert conversationStatusesTempTable raw query partially to Bun
* Rename field used to store result of MaxDirectStatusID
* Move advanced migrations to their own tiny processor
* Catch up util function name with main
* Remove json.… wrappers
* Remove redundant check
* Combine error checks
* Replace map with slice of structs
* Address processor/type converter comments
- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation
* Add error context when dropping temp tables
|
|
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg
* specify the video codec to use when generating static image from emoji
* update go-storage library (fixes incompatibility after updating go-iotools)
* maintain image aspect ratio when generating a thumbnail for it
* update readme to show go-ffmpreg
* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility
* remove extra debug from error message
* fix up incorrect function signatures
* update PutFile to just use regular file copy, as changes are file is on separate partition
* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe
* update more tests, add more code comments
* add utilities to generate processed emoji / media outputs
* fix remaining tests
* add test for opus media file, add license header to utility cmds
* limit the number of concurrently available ffmpeg / ffprobe instances
* reduce number of instances
* further reduce number of instances
* fix envparsing test with configuration variables
* update docs and configuration with new media-{local,remote}-max-size variables
|
|
The memlimit package started to log any error returned by automemlimit.
This updates our implementation to call SetGoMemLimitWithOpts() instead
which uses the same defaults as automemlimit except for being
initialised with a noop logger.
We check the returned error for a particular substring, as when cgroups
isn't available even when running on a Linux system that's not a
problem. If it's anything but that error, we log it at the warning
level so that admins can still diagnose other cgroup related issues.
Fixes #2983
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
|
|
* [chore] little startup tweaks
* go fmt
|
|
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it
* make the same changes to the testrig server
* use testrig specific func
* update variable name to fix nilptr
* fix removal of setting db on state
|
|
* [feature] Debug admin endpoint to clear caches
* go fmt
|
|
* start replacing client + federator + media workers with new worker + queue types
* refactor federatingDB.Delete(), drop queued messages when deleting account / status
* move all queue purging to the processor workers
* undo toolchain updates
* code comments, ensure dereferencer worker pool gets started
* update gruf libraries in readme
* start the job scheduler separately to the worker pools
* reshuffle ordering or server.go + remove duplicate worker start / stop
* update go-list version
* fix vendoring
* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped
* add logging to worker processing functions in testrig, don't start workers in unexpected places
* update go-structr to add (+then rely on) QueueCtx{} type
* ensure more worker pools get started properly in tests
* fix remaining broken tests relying on worker queue logic
* fix account test suite queue popping logic, ensure noop workers do not pull from queue
* move back accidentally shuffled account deletion order
* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()
* silently drop deletes from accounts not permitted to
* don't warn log on forwarded deletes
* make if else clauses easier to parse
* use getFederatorMsg()
* improved code comment
* improved code comment re: requesting account delete checks
* remove boolean result from worker start / stop since false = already running or already stopped
* remove optional passed-in http.client
* remove worker starting from the admin CLI commands (we don't need to handle side-effects)
* update prune cli to start scheduler but not all of the workers
* fix rebase issues
* remove redundant return statements
* i'm sorry sir linter
|
|
* [feature] User sign-up form and admin notifs
* add chosen + filtered languages to migration
* remove stray comment
* chosen languages schmosen schmanguages
* proper error on local account missing
|
|
* add delivery worker type that pulls from queue to httpclient package
* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller
* hook up queue deletion logic
* support deleting queued http requests by target ID
* don't index APRequest by hostname in the queue
* use gorun
* use the original context's values when wrapping msg type as delivery{}
* actually log in the AP delivery worker ...
* add uncommitted changes
* use errors.AsV2()
* use errorsv2.AsV2()
* finish adding some code comments, add bad host handling to delivery workers
* slightly tweak deliveryworkerpool API, use advanced sender multiplier
* remove PopCtx() method, let others instead rely on Wait()
* shuffle things around to move delivery stuff into transport/ subpkg
* remove dead code
* formatting
* validate request before queueing for delivery
* finish adding code comments, fix up backoff code
* finish adding more code comments
* clamp minimum no. senders to 1
* add start/stop logging to delivery worker, some slight changes
* remove double logging
* use worker ptrs
* expose the embedded log fields in httpclient.Request{}
* ensure request context values are preserved when updating ctx
* add delivery worker tests
* fix linter issues
* ensure delivery worker gets inited in testrig
* fix tests to delivering messages to check worker delivery queue
* update error type to use ptr instead of value receiver
* fix test calling Workers{}.Start() instead of testrig.StartWorkers()
* update docs for advanced-sender-multiplier
* update to the latest activity library version
* add comment about not using httptest.Server{}
|
|
* [feature] Add healthcheck endpoints `/livez` and `/readyz`
* use select that returns no data
|
|
* add more stringent checks for remote status permissibility
* add check for inreplyto of a remote status being a boost
* do not permit inReplyTo boost wrapper statuses
* change comment wording
* fix calls to NewFederator()
* add code comments for NotPermitted() and SetNotPermitted()
* improve comment
* check that existing != nil before attempting delete
* ensure replying account isn't suspended
* use a debug log instead of info. check for boost using ID
* shorten log string length. make info level
* add note that replying to boost wrapper status shouldn't be able to happen anyways
* update to use onFail() function
|
|
* [chore] Move `visibility` to `filter/visibility`
* [feature] Add experimental instance-federation-spam-filter option
|
|
instances count (#2592)
Co-authored-by: Tsuribori <none@example.org>
|
|
* [feature] Serve + rate limit emoji files separately from attachments
* add a wee little warning about uploading loads of emojis
|
|
performance and multiple-result lookups (#2535)
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial
* use actual go-structr release version (not just commit hash)
* revert go toolchain changes (damn you go for auto changing this)
* fix go mod woes
* ensure %w is used in calls to errs.Appendf()
* fix error checking
* fix possible panic
* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop
* fix copy-paste artifact... :innocent:
* fix all comment copy-paste artifacts
* remove dropID() function, now we can just use slices.DeleteFunc()
* use util.Deduplicate() instead of collate(), move collate to util
* move orderByIDs() to util package and "generify"
* add a util.DeleteIf() function, use this to delete entries on failed population
* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)
* add note about how collate differs from deduplicate
|
|
|
|
* [chore] Move "/metrics" into separate API module
* use our own gzip middleware for prom
|
|
* feat: Initial OTEL metrics
* docs: add metrics documentation
* fix: metrics endpoint conditional check
* feat: metrics endpoint basic auth
* fix: make metrics-auth-enabled default false
* fix: go fmt helpers.gen.go
* fix: add metric-related env vars to envparsing.sh
* fix: metrics docs
* fix: metrics related stuff in envparsing.sh
* fix: metrics docs
* chore: metrics docs wording
* fix: metrics stuff in envparsing?
* bump otel versions
---------
Co-authored-by: Tsuribori <user@acertaindebian>
Co-authored-by: Tsuribori <none@example.org>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
|
|
|
|
improvements (#2329)
|
|
|
|
structure (#2217)
* update typeconverter to use state structure
* deinterface the typeutils.TypeConverter -> typeutils.Converter
* finish copying over old type converter code comments
* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
|
|
|
kim
nicole mikołajczyk
Daenney
Daenney
kim
tobi
Dominik Süß
Vyr Cossont
Tsuribori