opencrawlers

Bytespider

ByteDance crawler, type: ai training.

About this crawler

Web crawler operated by ByteDance, the Chinese owner of TikTok, used to download training data for its LLMs including those powering Doubao.

User-Agent
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
robots.txt token
bytespider
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/bytespider

Including sites that block it only through the * group, 7.44% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind