claude-user
Anthropic crawler, type: ai assistant.
- 151,621sites name it in robots.txt
- 52,231block it fully (34.4%)
- 43,197block some paths
- 56,193allow it explicitly
About this crawler
Supports Claude users by accessing websites when individuals ask Claude questions.
- User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)(third party: source)- robots.txt token
claude-user- Respects robots.txt
- yes
- Published IP ranges
- https://claude.com/crawling/bots.json
- Operator page
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Page states it respects do-not-crawl signals.
Including sites that block it only through the * group, 4.23% of all sites with a robots.txt keep it out.