ClaudeBot
Anthropic crawler, type: ai training.
- 3,570,070sites name it in robots.txt
- 2,258,964block it fully (63.3%)
- 998,488block some paths
- 312,618allow it explicitly
About this crawler
Collects web content to help enhance the utility and safety of Anthropic's generative AI models.
- User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)(third party: source)- robots.txt token
claudebot- Respects robots.txt
- yes
- Published IP ranges
- https://claude.com/crawling/bots.json
- Operator page
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Page gives robots.txt examples: Disallow: / and Crawl-delay: 1.
- IP list at the bots.json URL; it lists prefixes only, without per-bot labels.
Including sites that block it only through the * group, 7.39% of all sites with a robots.txt keep it out.