opencrawlers

AI crawlers

Blocked means a group naming the bot disallows everything. Counts are sites, not files.

All tracked AI crawlers, most blocked first
CrawlerOperatorTypeSites naming itBlock it fullyShare of naming sites that block
GPTBotOpenAIai training4,002,0472,591,83464.8%
AmazonbotAmazonai training3,516,8532,456,30369.8%
BytespiderByteDanceai training3,396,1882,316,63368.2%
ClaudeBotAnthropicai training3,570,0702,258,96463.3%
CCBotCommon Crawlai training3,367,4812,243,50966.6%
Google-ExtendedGoogleai training3,283,9772,111,81964.3%
meta-externalagentMetaai training3,184,3012,108,75966.2%
applebot-extendedAppleai training3,036,5591,959,28564.5%
chatgpt-userOpenAIai assistant690,278306,52944.4%
facebookbotMetaai training1,295,241295,04522.8%
anthropic-aiAnthropicai training1,314,920276,26521%
omgiliWebz.ioai training1,096,438260,05223.7%
diffbotDiffbotai training304,734221,07172.5%
oai-searchbotOpenAIai search522,887205,89239.4%
cohere-aiCohereai training1,127,199202,62218%
imagesiftbotHiveai training331,143197,98559.8%
claude-webAnthropicai training398,318184,64346.4%
perplexitybotPerplexityai search600,248173,40028.9%
omgilibotWebz.ioai training1,016,525172,35117%
youbotYou.comai search1,046,420157,61215.1%
timpibotTimpiai training181,652117,82364.9%
ai2botAllen Institute for AIai training952,483112,35211.8%
velenpublicwebcrawlerVelenai training149,611110,40573.8%
cohere-training-data-crawlerCohereai training907,998106,34111.7%
ai2bot-dolmaAllen Institute for AIai training931,65299,64110.7%
googleotherGoogleai training1,068,88599,2949.3%
img2datasetimg2datasetai training923,27691,4269.9%
webzio-extendedWebz.ioai training124,65584,01467.4%
pangubotHuaweiai training88,73881,46891.8%
tiktokspiderByteDanceai training882,26376,8798.7%
icc-crawlerNICTai training114,80775,37165.7%
meta-externalfetcherMetaai assistant133,93374,91955.9%
duckassistbotDuckDuckGoai search912,28470,8687.8%
claude-searchbotAnthropicai search152,12863,95042%
friendlycrawlerunknownai training99,30361,26861.7%
kangaroo botKangaroo LLMai training99,21560,16260.6%
iaskspideriAskai search94,16856,16159.6%
claude-userAnthropicai assistant151,62152,23134.4%
google-cloudvertexbotGoogleai training62,08948,79178.6%
perplexity-userPerplexityai assistant146,14846,93032.1%
sidetrade indexer botSidetradeai training77,53540,42752.1%
phindbotPhindai search41,00032,48279.2%
mistralai-userMistralai assistant57,92231,14353.8%
firecrawlagentFirecrawlai scraper39,57830,19676.3%
brightbotBright Dataai training11,60211,11695.8%
novellumNovellumai training2118439.8%

What we collect, and how to change your mind