summaryrefslogtreecommitdiff
path: root/internal/api/util/robots.go
AgeCommit message (Collapse)AuthorFiles
2026-01-22[chore] Update robots.txt md5 hashes for etags (#4623)Libravatar tobi1
Forgot to update these in https://codeberg.org/superseriousbusiness/gotosocial/pulls/4622 Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4623 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2026-01-22[chore] Update robots.txt with latest list of "ai" bullshit user-agents (#4622)Libravatar tobi1
Love this timeline. Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4622 Co-authored-by: tobi <tobi.smethurst@protonmail.com> Co-committed-by: tobi <tobi.smethurst@protonmail.com>
2025-04-19[chore] Update robots.txt with latest ai bullshit (#4024)Libravatar tobi1
2025-02-24[feature] Use ETag for robots.txt to prevent mishaps (#3829)v0.18.1Libravatar tobi1
* [feature] Use ETag for robots.txt to prevent mishaps * check incoming if-none-match header
2025-02-05[feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers (#3737)Libravatar tobi1
* [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers * use switch for RobotsHeaders
generated by cgit v1.3 (git 2.53.0) at 2026-08-15 06:37:59 +0000