opencrawlers

ClaudeBot

Anthropic crawler, type: ai training.

About this crawler

Collects web content to help enhance the utility and safety of Anthropic's generative AI models.

User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) (third party: source)
robots.txt token
claudebot
Respects robots.txt
yes
Published IP ranges
https://claude.com/crawling/bots.json
Operator page
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Including sites that block it only through the * group, 7.39% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind