Bytespider
ByteDance crawler, type: ai training.
- 3,396,188sites name it in robots.txt
- 2,316,633block it fully (68.2%)
- 934,295block some paths
- 145,260allow it explicitly
About this crawler
Web crawler operated by ByteDance, the Chinese owner of TikTok, used to download training data for its LLMs including those powering Doubao.
- User-Agent
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36- robots.txt token
bytespider- Respects robots.txt
- yes
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/bytespider
- Third-party source (Dark Visitors / Known Agents, darkvisitors.com redirects to knownagents.com); ByteDance did not confirm on this page.
- Source says it is expected to follow robots.txt but that Bytespider does not publish a verification method.
- Source lists no IP ranges.
Including sites that block it only through the * group, 7.44% of all sites with a robots.txt keep it out.