diff options
| author |  Daenney <daenney@users.noreply.github.com> | 2023-09-30 21:44:57 +0200 |
|---|---|---|
| committer |  GitHub <noreply@github.com> | 2023-09-30 20:44:57 +0100 |
| commit | 0cce2c083809ad6e64182fd4dc05ddec7d73323b (patch) | |
| tree | af369493e0c9259875c1f4081f74accdb8981219 /internal/ap/extract.go | |
| parent | [bugfix] Fix paging for empty items (#2236) (diff) | |
| download | gotosocial-0cce2c083809ad6e64182fd4dc05ddec7d73323b.tar.xz | |
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers
* [feature] Block the other OpenAI crawler
* [feature] Block Common Crawl crawler
This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.
Fixes: #2240
* [feature] Block Omgilikebot
Used by some shady big web data engine company.
* [feature] Block Meta's language model crawler
* [feature] Block well-known.dev crawler
Diffstat (limited to 'internal/ap/extract.go')
0 files changed, 0 insertions, 0 deletions