opencrawlers

ai2bot

Allen Institute for AI crawler, type: ai training.

About this crawler

AI2Bot explores web domains to collect content used for training open language models.

User-Agent
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)
robots.txt token
ai2bot
Respects robots.txt
unknown
Operator page
https://allenai.org/crawler

Including sites that block it only through the * group, 4.3% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind