AI crawlers
Blocked means a group naming the bot disallows everything. Counts are sites, not files.
| Crawler | Operator | Type | Sites naming it | Block it fully | Share of naming sites that block |
|---|---|---|---|---|---|
| GPTBot | OpenAI | ai training | 4,002,047 | 2,591,834 | 64.8% |
| Amazonbot | Amazon | ai training | 3,516,853 | 2,456,303 | 69.8% |
| Bytespider | ByteDance | ai training | 3,396,188 | 2,316,633 | 68.2% |
| ClaudeBot | Anthropic | ai training | 3,570,070 | 2,258,964 | 63.3% |
| CCBot | Common Crawl | ai training | 3,367,481 | 2,243,509 | 66.6% |
| Google-Extended | ai training | 3,283,977 | 2,111,819 | 64.3% | |
| meta-externalagent | Meta | ai training | 3,184,301 | 2,108,759 | 66.2% |
| applebot-extended | Apple | ai training | 3,036,559 | 1,959,285 | 64.5% |
| chatgpt-user | OpenAI | ai assistant | 690,278 | 306,529 | 44.4% |
| facebookbot | Meta | ai training | 1,295,241 | 295,045 | 22.8% |
| anthropic-ai | Anthropic | ai training | 1,314,920 | 276,265 | 21% |
| omgili | Webz.io | ai training | 1,096,438 | 260,052 | 23.7% |
| diffbot | Diffbot | ai training | 304,734 | 221,071 | 72.5% |
| oai-searchbot | OpenAI | ai search | 522,887 | 205,892 | 39.4% |
| cohere-ai | Cohere | ai training | 1,127,199 | 202,622 | 18% |
| imagesiftbot | Hive | ai training | 331,143 | 197,985 | 59.8% |
| claude-web | Anthropic | ai training | 398,318 | 184,643 | 46.4% |
| perplexitybot | Perplexity | ai search | 600,248 | 173,400 | 28.9% |
| omgilibot | Webz.io | ai training | 1,016,525 | 172,351 | 17% |
| youbot | You.com | ai search | 1,046,420 | 157,612 | 15.1% |
| timpibot | Timpi | ai training | 181,652 | 117,823 | 64.9% |
| ai2bot | Allen Institute for AI | ai training | 952,483 | 112,352 | 11.8% |
| velenpublicwebcrawler | Velen | ai training | 149,611 | 110,405 | 73.8% |
| cohere-training-data-crawler | Cohere | ai training | 907,998 | 106,341 | 11.7% |
| ai2bot-dolma | Allen Institute for AI | ai training | 931,652 | 99,641 | 10.7% |
| googleother | ai training | 1,068,885 | 99,294 | 9.3% | |
| img2dataset | img2dataset | ai training | 923,276 | 91,426 | 9.9% |
| webzio-extended | Webz.io | ai training | 124,655 | 84,014 | 67.4% |
| pangubot | Huawei | ai training | 88,738 | 81,468 | 91.8% |
| tiktokspider | ByteDance | ai training | 882,263 | 76,879 | 8.7% |
| icc-crawler | NICT | ai training | 114,807 | 75,371 | 65.7% |
| meta-externalfetcher | Meta | ai assistant | 133,933 | 74,919 | 55.9% |
| duckassistbot | DuckDuckGo | ai search | 912,284 | 70,868 | 7.8% |
| claude-searchbot | Anthropic | ai search | 152,128 | 63,950 | 42% |
| friendlycrawler | unknown | ai training | 99,303 | 61,268 | 61.7% |
| kangaroo bot | Kangaroo LLM | ai training | 99,215 | 60,162 | 60.6% |
| iaskspider | iAsk | ai search | 94,168 | 56,161 | 59.6% |
| claude-user | Anthropic | ai assistant | 151,621 | 52,231 | 34.4% |
| google-cloudvertexbot | ai training | 62,089 | 48,791 | 78.6% | |
| perplexity-user | Perplexity | ai assistant | 146,148 | 46,930 | 32.1% |
| sidetrade indexer bot | Sidetrade | ai training | 77,535 | 40,427 | 52.1% |
| phindbot | Phind | ai search | 41,000 | 32,482 | 79.2% |
| mistralai-user | Mistral | ai assistant | 57,922 | 31,143 | 53.8% |
| firecrawlagent | Firecrawl | ai scraper | 39,578 | 30,196 | 76.3% |
| brightbot | Bright Data | ai training | 11,602 | 11,116 | 95.8% |
| novellum | Novellum | ai training | 211 | 84 | 39.8% |