anthropic-ai
Anthropic crawler, type: ai training.
- 1,314,920sites name it in robots.txt
- 276,265block it fully (21%)
- 854,151block some paths
- 184,504allow it explicitly
About this crawler
Unconfirmed agent possibly used by Anthropic to download training data for its LLMs.
- robots.txt token
anthropic-ai- Respects robots.txt
- yes
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/anthropic-ai
- Third-party source (Known Agents / Dark Visitors); Anthropic has not publicly disclosed this agent's purpose, so the operator does not confirm it.
- Source lists the user agent as just the token anthropic-ai; it does not call it deprecated but classifies it as an undocumented agent.
- Source says it is expected to follow robots.txt.
Including sites that block it only through the * group, 4.49% of all sites with a robots.txt keep it out.