opencrawlers

claude-web

Anthropic crawler, type: ai training.

About this crawler

Anthropic-attributed agent of undisclosed purpose; the source says it is unclear what it is used for since there is no official documentation.

User-Agent
Claude-Web/1.0
robots.txt token
claude-web
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/claude-web

Including sites that block it only through the * group, 4.4% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind