opencrawlers

claude-user

Anthropic crawler, type: ai assistant.

About this crawler

Supports Claude users by accessing websites when individuals ask Claude questions.

User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com) (third party: source)
robots.txt token
claude-user
Respects robots.txt
yes
Published IP ranges
https://claude.com/crawling/bots.json
Operator page
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Including sites that block it only through the * group, 4.23% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind