diff options
| author |  Daenney <daenney@users.noreply.github.com> | 2024-02-27 14:25:08 +0100 |
|---|---|---|
| committer |  GitHub <noreply@github.com> | 2024-02-27 13:25:08 +0000 |
| commit | 6528592dd289f9fda5538be5c54c028ddd9a09f1 (patch) | |
| tree | 44174fa09534a97ebd7d6c879f8581b2685d0346 /internal/web/robots.go | |
| parent | [feature] Add experimental `instance-federation-spam-filter` option (#2685) (diff) | |
| download | gotosocial-6528592dd289f9fda5538be5c54c028ddd9a09f1.tar.xz | |
[feature] Block Amazonbot (#2692)
Blocks the Amazon crawler bot.
Closes: #2686
Diffstat (limited to 'internal/web/robots.go')
| -rw-r--r-- | internal/web/robots.go | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/internal/web/robots.go b/internal/web/robots.go index 92bd33014..cfd1758a4 100644 --- a/internal/web/robots.go +++ b/internal/web/robots.go @@ -64,6 +64,11 @@ Disallow: / User-agent: WellKnownBot Disallow: / +# Block Amazonbot, because Amazon. +# https://developer.amazon.com/amazonbot +User-agent: Amazonbot +Disallow: / + # Rules for everything else. User-agent: * Crawl-delay: 500 |