Age | Commit message (Collapse) | Author | Files | |
---|---|---|---|---|
2025-02-24 | [feature] Use ETag for robots.txt to prevent mishaps (#3829)v0.18.1 | 1 | ||
* [feature] Use ETag for robots.txt to prevent mishaps * check incoming if-none-match header | ||||
2025-02-05 | [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers (#3737) | 1 | ||
* [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers * use switch for RobotsHeaders |