How we count
Source: robots.txt captures published by Common Crawl, 8 crawls in 2026, plus a live sample of 20,000 Tranco top-1M hostnames fetched by sus.bot in October 2026. The live sample is shown separately and is not part of the totals.
The unit is the hostname of the robots.txt URL. Each host gets one final capture. Percentages use sites with a robots.txt as the denominator.
A site "blocks" a crawler when a group naming that crawler disallows / with no Allow rules. Sites that block only through the * group are not counted in the block column.
Misspelled or invented crawler names are not counted. The list of tracked crawlers is a fixed catalog.