opencrawlers

sidetrade indexer bot

Sidetrade crawler, type: ai training.

About this crawler

Uncategorized agent with unclear purpose.

User-Agent
Sidetrade indexer bot
robots.txt token
sidetrade indexer bot
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/sidetrade-indexer-bot

Including sites that block it only through the * group, 4.23% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind