claude-web
Anthropic crawler, type: ai training.
- 398,318sites name it in robots.txt
- 184,643block it fully (46.4%)
- 56,072block some paths
- 157,603allow it explicitly
About this crawler
Anthropic-attributed agent of undisclosed purpose; the source says it is unclear what it is used for since there is no official documentation.
- User-Agent
Claude-Web/1.0- robots.txt token
claude-web- Respects robots.txt
- yes
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/claude-web
- Third-party source (Known Agents / Dark Visitors); Anthropic does not confirm it there.
- Source says it is expected to follow robots.txt and does not call it deprecated.
- Source classifies it as an Undocumented AI Agent.
Including sites that block it only through the * group, 4.4% of all sites with a robots.txt keep it out.