ai2bot-dolma
Allen Institute for AI crawler, type: ai training.
- 931,652sites name it in robots.txt
- 99,641block it fully (10.7%)
- 799,889block some paths
- 32,122allow it explicitly
About this crawler
Downloads website content to include in datasets used for training AI models such as LLMs. Operated by Ai2, a non-profit AI research institute.
- User-Agent
Mozilla/5.0 (compatible; AI2Bot-Dolma; +https://allenai.org/dolma)- robots.txt token
ai2bot-dolma- Respects robots.txt
- yes
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/ai2bot-dolma
- Third-party source (Known Agents, formerly Dark Visitors; darkvisitors.com redirects there); Ai2 does not confirm on this page.
- Source states it is 'expected to follow robots.txt rules' and normally visits from the United States.
- Source states no verification method is published, so a UA match in logs is only a clue.
Including sites that block it only through the * group, 4.28% of all sites with a robots.txt keep it out.