tiktokspider
ByteDance crawler, type: ai training.
- 882,263sites name it in robots.txt
- 76,879block it fully (8.7%)
- 802,773block some paths
- 2,611allow it explicitly
About this crawler
ByteDance crawler that fetches content from websites when users share external URLs on TikTok, retrieving metadata and thumbnails for link previews.
- User-Agent
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)- robots.txt token
tiktokspider- Respects robots.txt
- no
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/tiktokspider
- Third-party source (Known Agents / Dark Visitors); not confirmed by the operator here.
- Source states it is not expected to follow robots.txt; firewall or server rules are needed to block it.
- Source says it typically makes a single request after a URL is shared.
Including sites that block it only through the * group, 4.25% of all sites with a robots.txt keep it out.