summaryrefslogtreecommitdiff
path: root/internal/ap/extract.go
AgeCommit message (Collapse)AuthorFiles
2025-08-09[feature + performance] add JSON logging format (#4355)Libravatar kim1
# Description Adds JSON logging as an optional alternative log output format. In the process this moves our log formatting itself into a separate subpkg to make it more easily modular, and improves caller name getting with some calling function name caching. ## Checklist - [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md). - [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat. - [x] I/we have not leveraged AI to create the proposed changes. - [x] I/we have performed a self-review of added code. - [x] I/we have written code that is legible and maintainable by others. - [x] I/we have commented the added code, particularly in hard-to-understand areas. - [x] I/we have made any necessary changes to documentation. - [ ] I/we have added tests that cover new code. - [x] I/we have run tests and they pass locally with the changes. - [x] I/we have run `go fmt ./...` and `golangci-lint run`. Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4355 Co-authored-by: kim <grufwub@gmail.com> Co-committed-by: kim <grufwub@gmail.com>
2025-06-19[feature] Support incoming avatar/header descriptions (#4275)Libravatar nicole mikołajczyk1
Signed-off-by: nicole mikołajczyk <git@mkljczk.pl> # Description Follow-up to #4270 Closes https://codeberg.org/superseriousbusiness/gotosocial/issues/3450 ## Checklist - [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md). - [ ] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat. - [x] I/we have not leveraged AI to create the proposed changes. - [x] I/we have performed a self-review of added code. - [x] I/we have written code that is legible and maintainable by others. - [x] I/we have commented the added code, particularly in hard-to-understand areas. - [ ] I/we have made any necessary changes to documentation. - [x] I/we have added tests that cover new code. - [x] I/we have run tests and they pass locally with the changes. - [x] I/we have run `go fmt ./...` and `golangci-lint run`. Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4275 Co-authored-by: nicole mikołajczyk <git@mkljczk.pl> Co-committed-by: nicole mikołajczyk <git@mkljczk.pl>
2025-06-10[chore] Update interactionPolicy sub-policy parsing in line with documented ↵Libravatar tobi1
defaults (#4229) # Description > If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements. > > If this is a documentation change, please briefly describe what you've changed and why. Brings our parsing of unset sub-policies in line with the defaults documented here: https://docs.gotosocial.org/en/v0.19.1/federation/interaction_policy/#defaults-per-sub-policy Closes https://codeberg.org/superseriousbusiness/gotosocial/issues/4146 Part of https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 ## Checklist Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]` If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want). - [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md). - [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat. - [x] I/we have not leveraged AI to create the proposed changes. - [x] I/we have performed a self-review of added code. - [x] I/we have written code that is legible and maintainable by others. - [x] I/we have commented the added code, particularly in hard-to-understand areas. - [ ] I/we have made any necessary changes to documentation. - [x] I/we have added tests that cover new code. - [x] I/we have run tests and they pass locally with the changes. - [x] I/we have run `go fmt ./...` and `golangci-lint run`. Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4229 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-06-09[bugfix] Update `GetURL` to extract url from Link objects with href (#4249)Libravatar tobi1
# Description > If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements. > > If this is a documentation change, please briefly describe what you've changed and why. This pull request updates our parsing of the `url` property in incoming ActivityPub items to also include Link items, and not just bare URIs. The first discovered url is still used as the *gtsmodel.Account or *gtsmodel.Status `url` property, so this change only really affects our dereference URL anti-spoof check thingy. ~~Should fix https://codeberg.org/superseriousbusiness/gotosocial/issues/4248 but I need to run it and test it myself first to be sure.~~ Fixes https://codeberg.org/superseriousbusiness/gotosocial/issues/4248 ## Checklist Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]` If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want). - [x] I/we have read the [GoToSocial contribution guidelines](https://codeberg.org/superseriousbusiness/gotosocial/src/branch/main/CONTRIBUTING.md). - [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat. - [x] I/we have not leveraged AI to create the proposed changes. - [x] I/we have performed a self-review of added code. - [x] I/we have written code that is legible and maintainable by others. - [x] I/we have commented the added code, particularly in hard-to-understand areas. - [ ] I/we have made any necessary changes to documentation. - [x] I/we have added tests that cover new code. - [x] I/we have run tests and they pass locally with the changes. - [x] I/we have run `go fmt ./...` and `golangci-lint run`. Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4249 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-05-13[chore] Deprecate `with_approval`, `always` (client API), ↵Libravatar tobi1
`approvalRequired`, `always` (fedi API) (#4173) This pull request deprecates `with_approval` and `always` on the client API side, and `approvalRequired` and `always` on the fedi API side, replacing them with `automatic_approval` and `manual_approval` and `automaticApproval` and `manualApproval`, respectively. Back-compat is kept with these deprecated fields, and they're still serialized to the client API and fedi APIs respectively, in addition to the new non-deprecated properties. This will stay the case until v0.21.0 when they'll be removed. For the sake of not doing a massive database migration, the fields are still named `Always` and `WithApproval` in storage. I think this is probably fine! Part of https://codeberg.org/superseriousbusiness/gotosocial/issues/4026 Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4173 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-04-27[chore] Rewrite all remaining Github linksLibravatar Daenney1
2025-04-26[feature] Move to code.superseriousbusiness.orgLibravatar Daenney1
2025-04-26[feature] Update attachment format, receive + send `focalPoint` prop + use ↵Libravatar tobi1
it on the frontend (#4052) * [feature] Update attachment format, receive + send `focalPoint` prop + use it on the frontend * whoops * boop * restore function signature of ExtractAttachments
2025-04-25[chore] Move deps to code.superseriousbusiness.org (#4054)Libravatar tobi1
2025-03-24[feature] Parse funkwhale `Album` as Statusable to allow barebones ↵Libravatar tobi1
interacting with bandwagon (#3931) * bump activity version * parse funkwhale / bandwagon album as statusable
2025-03-02[chore] github.com/superseriousbusiness/activity -> ↵Libravatar tobi1
codeberg.org/superseriousbusiness/activity (#3853)
2025-01-28[bugfix] Allow processing null ID emojis (#3702)Libravatar tobi1
* [bugfix] Allow processing null ID emojis * document emojis * blah * typo * array thingy
2024-11-25[performance] convert enum strings to ints (#3558)Libravatar kim1
* convert statuses.visibility and notifications.notification_type columns from type string -> int for performance / space savings * fix test trying to compare string to int * fix instance count query using string literal instead of gtsmodel const type * ensure a default value is always set * also migrate the account settings and sin bin status tables * initialize maps outside loops and place into singular enum mapping creation func * use int16 for enum types * update sinbinstatus creation to be from a snapshot at initial creation * add snapshot of poll type at creation time
2024-09-17[bugfix] Fix incorrect policy value parsing (#3315)Libravatar tobi1
2024-08-13[bugfix] relax missing preferred_username, instead using webfingered ↵Libravatar kim1
username (#3189) * support no preferred_username, instead using webfingered username * add tests for the new preferred_username behaviour
2024-07-26[feature] Federate interaction policies + Accepts; enforce policies (#3138)Libravatar tobi1
* [feature] Federate interaction policies + Accepts; enforce policies * use Acceptable type * fix index * remove appendIRIStrs * add GetAccept federatingdb function * lock on object IRI
2024-04-02[chore] Try to parse public key as both Actor + bare key (#2710)Libravatar tobi1
* [chore] Try to parse public key as both Actor + bare key * fix weird test fail
2024-02-27[feature] Add experimental `instance-federation-spam-filter` option (#2685)Libravatar tobi1
* [chore] Move `visibility` to `filter/visibility` * [feature] Add experimental instance-federation-spam-filter option
2024-01-19[chore] Harden up boolptr logic on Accounts, warn if not set (#2544)Libravatar tobi1
2023-12-12[bugfix] poll vote count fixes (#2444)Libravatar kim1
* don't drop all vote counts if hideCounts is set, refactors poll option extraction slightly * omit voters_count when not set * make voters_count a ptr to ensure it is omit unless definitely needed * handle case of expires_at, voters_count and option.votes_count being nilable * faster isNil check * remove omitempty tags since mastodon API marks things as nullable but still sets them in outgoing json
2023-11-30[bugfix] return 400 Bad Request on more cases of malformed AS data (#2399)Libravatar kim1
2023-11-21[feature] Federate status language in and out (#2366)Libravatar tobi1
* [feature] Federate status language in + out * go fmt * tests, little fix * improve comments * unnest a bit * avoid unnecessary nil check * use more descriptive variable for contentMap * prefer instance languages when selecting from contentMap * update docs to reflect lang selection * rename rdfLangString -> rdfLangs * update comments to mention Pollable * iter through slice instead of map
2023-11-11[bugfix] support endless polls, and misskey's' method of inferring expiry in ↵Libravatar kim1
closed polls (#2349)
2023-11-08[feature] add support for polls + receiving federated status edits (#2330)Libravatar kim1
2023-11-04[feature] support canceling scheduled tasks, some federation API performance ↵Libravatar kim1
improvements (#2329)
2023-10-31[bugfix] Relax `Mention` parsing, allowing either href or name (#2320)Libravatar tobi1
2023-10-26[bugfix] Extract description as `summary` first, fall back to `name` (#2303)Libravatar tobi1
2023-10-04[feature] tentatively start adding polls support (#2249)Libravatar kim1
2023-10-03[chore] internal/ap: add pollable AS types, code reformatting, general ↵Libravatar kim1
niceties (#2248)
2023-07-31[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)Libravatar tobi1
* update go-fed * do the things * remove unused columns from tags * update to latest lingo from main * further tag shenanigans * serve stub page at tag endpoint * we did it lads * tests, oh tests, ohhh tests, oh tests (doo doo doo doo) * swagger docs * document hashtag usage + federation * instanceGet * don't bother parsing tag href * rename whereStartsWith -> whereStartsLike * remove GetOrCreateTag * dont cache status tag timelineability
2023-06-17[chore/bugfix] Refactor `ap/extract.go` functions, return URIs more reliably ↵Libravatar tobi1
(#1897)
2023-06-13[chore] Refactor AP authentication, other small bits of tidying up (#1874)Libravatar tobi1
2023-05-09[feature] Enable federation in/out of profile PropertyValue fields (#1722)Libravatar tobi1
Co-authored-by: kim <grufwub@gmail.com> Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-04-26[bugfix] Fix remaining mangled URI escaping issues in statuses + accounts ↵Libravatar tobi1
(#1712) * start fiddling with normalize + extract functions * normalize attachment name (image description) * NormalizeAccountableSummary * normalize summary + name
2023-03-12[chore] Improve copyright header handling (#1608)Libravatar Daenney1
* [chore] Remove years from all license headers Years or year ranges aren't required in license headers. Many projects have removed them in recent years and it avoids a bit of yearly toil. In many cases our copyright claim was also a bit dodgy since we added the 2021-2023 header to files created after 2021 but you can't claim copyright into the past that way. * [chore] Add license header check This ensures a license header is always added to any new file. This avoids maintainers/reviewers needing to remember to check for and ask for it in case a contribution doesn't include it. * [chore] Add missing license headers * [chore] Further updates to license header * Use the more common // indentend comment format * Remove the hack we had for the linter now that we use the // format * Add SPDX license identifier
2023-02-20[bugfix] Fix failure to look up remote profiles with duplicate emojis in ↵Libravatar Sam Lade1
some cases (#1534) * Tidy up emoji parsing on profile submission Don't bother reparsing for emoji unless one of the fields that can have emoji in it has changed. Deduplicate emoji between the display name and profile note - I'm not sure whether this was hurting anything, but better safe. * Deduplicate emoji when parsing remote accounts Some servers - Misskey at least - don't deduplicate emoji, so it's possible to get an account which has the same emoji used in both the display name and note and therefore includes that emoji twice in its metadata. When we start trying to put those into our database, we run into a uniqueness constraint and fall over. This change just deduplicates at the point of construction of an account.
2023-02-02[bugfix] Read Bookwyrm Articles more thoroughly (#1410)Libravatar tobi1
2023-01-27[feature] Federate reports to remote instance as Flag (if desired) (#1386)Libravatar tobi1
* reports federate out, we did it lxds * fix optional line start (should be optional slash)
2023-01-25[feature] Accept incoming federated Flag activity (#1382)Libravatar tobi1
* start working on handling incoming Flag activity * interim commit * federate Flag in successfully
2023-01-05[chore] Update/add license headers for 2023 (#1304)Libravatar tobi1
2022-11-30[feature] Support PKCS1 "RSA PUBLIC KEY" pem block type (#1179)Libravatar Sigrid Solveig Haflínudóttir1
* ap: add support for PKCS1 "RSA PUBLIC KEY" pem block type Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se> * ap: report no PEM data or unknown pem block type Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se> Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
2022-10-13[feature] Refetch emojis when they change on remote instances (#905)Libravatar tobi1
* select emoji using image_static_url * use updated on AP emojis * allow refetch of updated emojis * cheeky workaround for test * clean up old files for refreshed emoji * check error for originalPostData * shorten GetEmojiByStaticImageURL * delete kirby (sorry nintendo)
2022-09-28[chore] update dependencies, bump to Go 1.19.1 (#826)Libravatar kim1
* update dependencies, bump Go version to 1.19 * bump test image Go version * update golangci-lint * update gotosocial-drone-build * sign * linting, go fmt * update swagger docs * update swagger docs * whitespace * update contributing.md * fuckin whoopsie doopsie * linterino, linteroni * fix followrequest test not starting processor * fix other api/client tests not starting processor * fix remaining tests where processor not started * bump go-runners version * don't check last-webfingered-at, processor may have updated this * update swagger command * update bun to latest version * fix embed to work the same as before with new bun Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-09-23[feature] Allow delivery to sharedInboxes where possible (#847)Libravatar tobi1
* update Activity * add instance-deliver-to-shared-inboxes setting * update activity version again * add SharedInboxURI field to accounts * serdes for endpoints/sharedInbox * deliver to sharedInbox if one is available * update tests * only assign shared inbox if shared domain * look for shared inbox if currently nil * go fmt * finger to get params.RemoteAccountID if necessary * make comments clearer * compare dns more consistently
2022-08-15[chore] Update bun / sqlite versions; update gtsmodels (#754)Libravatar tobi1
* upstep bun and sqlite versions * allow specific columns to be updated in the db * only update necessary columns for user * bit tidier * only update necessary fields of media_attachment * only update relevant instance fields * update tests * update only specific account columns * use bool pointers on gtsmodels includes attachment, status, account, user * update columns more selectively * test all default fields on new account insert * updating remaining bools on gtsmodels * initialize pointer fields when extracting AP emoji * copy bools properly * add copyBoolPtr convenience function + test it * initialize false bool ptrs a bit more neatly
2022-06-11[chore] Webfinger rework (#627)Libravatar tobi1
* move finger to dereferencer * totally break GetRemoteAccount * start reworking finger func a bit * start reworking getRemoteAccount a bit * move mention parts to namestring * rework webfingerget * use util function to extract webfinger parts * use accountDomain * rework finger again, final form * just a real nasty commit, the worst * remove refresh from account * use new ASRepToAccount signature * fix incorrect debug call * fix for new getRemoteAccount * rework GetRemoteAccount * start updating tests to remove repetition * break a lot of tests Move shared test logic into the testrig, rather than having it scattered all over the place. This allows us to just mock the transport controller once, and have all tests use it (unless they need not to for some other reason). * fix up tests to use main mock httpclient * webfinger only if necessary * cheeky linting with the lads * update mentionName regex recognize instance accounts * don't finger instance accounts * test webfinger part extraction * increase default worker count to 4 per cpu * don't repeat regex parsing * final search for discovered accountDomain * be more permissive in namestring lookup * add more extraction tests * simplify GetParseMentionFunc * skip long search if local account * fix broken test
2022-05-23[bugfix] Fix `error extracting status content: no content found` (#598)Libravatar tobi1
* don't return error if no content found in Activity * add test for content extraction * go fmt
2022-04-26[bugfix] Fix CWs not showing sometimes (#488)Libravatar tobi1
* allow summaries that are parsed as iris * test parsing a status with iri summary
2022-03-21don't error if content property is nil (#436)Libravatar tobi1
2022-01-09tests are passing, but there's still much to be doneLibravatar tsmethurst1