brightbot
Bright Data crawler, type: ai training.
- 11,602sites name it in robots.txt
- 11,116block it fully (95.8%)
- 107block some paths
- 379allow it explicitly
About this crawler
BrightBot is Bright Data's crawler; domain owners control its crawl behavior with a collectors.txt file set in the Bright Data Webmaster Console.
- User-Agent
Brightbot 1.0- robots.txt token
brightbot- Respects robots.txt
- unknown
- Operator page
- https://docs.brightdata.com/general/webmaster-console.md
- Page governs behavior through collectors.txt (directives pii, disallow, copyright, private), not robots.txt; page says to allow 82.97.199.0/24 (stated on page, no separate list URL).
- Page states the BrightBot User-Agent and collectors.txt rules apply only to traffic routed through Web Unlocker, not Browser API or browser-based Data Collector jobs.
- Page states blocking BrightBot requests triggers a 7-day suspension of the user-agent for that domain.
Including sites that block it only through the * group, 4.2% of all sites with a robots.txt keep it out.