ai2bot
Allen Institute for AI crawler, type: ai training.
- 952,483sites name it in robots.txt
- 112,352block it fully (11.8%)
- 802,445block some paths
- 37,686allow it explicitly
About this crawler
AI2Bot explores web domains to collect content used for training open language models.
- User-Agent
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)- robots.txt token
ai2bot- Respects robots.txt
- unknown
- Operator page
- https://allenai.org/crawler
- Page says the bot can be filtered or rejected via its user-agent string.
- Page does not mention robots.txt directives, the AI2Bot-Dolma token or an IP list.
Including sites that block it only through the * group, 4.3% of all sites with a robots.txt keep it out.