opencrawlers

ai2bot-dolma

Allen Institute for AI crawler, type: ai training.

About this crawler

Downloads website content to include in datasets used for training AI models such as LLMs. Operated by Ai2, a non-profit AI research institute.

User-Agent
Mozilla/5.0 (compatible; AI2Bot-Dolma; +https://allenai.org/dolma)
robots.txt token
ai2bot-dolma
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/ai2bot-dolma

Including sites that block it only through the * group, 4.28% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind