opencrawlers

webzio-extended

Webz.io crawler, type: ai training.

About this crawler

Webz.io describes a crawler pair (Webzio and Webzio-extended) that collects news, blogs, discussions and reviews for its data products; Webzio-extended tags data as usable or not usable for AI/ML training.

User-Agent
Webzio-Extended/1.0 (+https://webz.io/bot.html) (third party: source)
robots.txt token
webzio-extended
Respects robots.txt
yes
Operator page
https://webz.io/bot.html

Including sites that block it only through the * group, 4.29% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind