From bad427e7f0e654835ece503e4666d42769ed4f58 Mon Sep 17 00:00:00 2001 From: cdn0x12 Date: Sat, 3 May 2025 09:28:43 +0000 Subject: [chore/docs] fix relative link to scraper deterrence (#4111) # Description While working on the doc translation update, I found a broken link. So I;m opening this separate PR to keep it clean from the translation stuff. Marked as draft currently for checking for any other typos :) Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4111 Co-authored-by: cdn0x12 Co-committed-by: cdn0x12 --- docs/admin/robots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/admin') diff --git a/docs/admin/robots.md b/docs/admin/robots.md index e4b3d27ce..29a02db42 100644 --- a/docs/admin/robots.md +++ b/docs/admin/robots.md @@ -10,6 +10,6 @@ You can allow or disallow crawlers from collecting stats about your instance fro The AI scrapers come from a [community maintained repository][airobots]. It's manually kept in sync for the time being. If you know of any missing robots, please send them a PR! -A number of AI scrapers are known to ignore entries in `robots.txt` even if it explicitly matches their User-Agent. This means the `robots.txt` file is not a foolproof way of ensuring AI scrapers don't grab your content. In addition to this you might want to look into blocking User-Agents via [requester header filtering](request_filtering_modes.md), and enabling a proof-of-work [scraper deterrence](scraper_deterrence.md). +A number of AI scrapers are known to ignore entries in `robots.txt` even if it explicitly matches their User-Agent. This means the `robots.txt` file is not a foolproof way of ensuring AI scrapers don't grab your content. In addition to this you might want to look into blocking User-Agents via [requester header filtering](request_filtering_modes.md), and enabling a proof-of-work [scraper deterrence](../advanced/scraper_deterrence.md). [airobots]: https://github.com/ai-robots-txt/ai.robots.txt/ -- cgit v1.2.3