webzio-extended
Webz.io crawler, type: ai training.
- 124,655sites name it in robots.txt
- 84,014block it fully (67.4%)
- 7,205block some paths
- 33,436allow it explicitly
About this crawler
Webz.io describes a crawler pair (Webzio and Webzio-extended) that collects news, blogs, discussions and reviews for its data products; Webzio-extended tags data as usable or not usable for AI/ML training.
- User-Agent
Webzio-Extended/1.0 (+https://webz.io/bot.html)(third party: source)- robots.txt token
webzio-extended- Respects robots.txt
- yes
- Operator page
- https://webz.io/bot.html
- Page names Omgilibot as the legacy crawler, Webzio as current crawler and Webzio-extended as the validation bot.
- States it only provides data tagged as explicitly permitted for AI purposes.
- Page gives no full user-agent strings and no IP list.
Including sites that block it only through the * group, 4.29% of all sites with a robots.txt keep it out.