omgili
Webz.io crawler, type: ai training.
- 1,096,438sites name it in robots.txt
- 260,052block it fully (23.7%)
- 800,201block some paths
- 36,185allow it explicitly
About this crawler
Webz.io describes a crawler pair (Webzio and Webzio-extended) that collects news, blogs, discussions and reviews for its data products; Webzio-extended tags data as usable or not usable for AI/ML training.
- User-Agent
omgili(third party: source)- robots.txt token
omgili- Respects robots.txt
- yes
- Operator page
- https://webz.io/bot.html
- Page names Omgilibot as the legacy crawler, Webzio as current crawler and Webzio-extended as the validation bot.
- States it only provides data tagged as explicitly permitted for AI purposes.
- Page gives no full user-agent strings and no IP list.
Including sites that block it only through the * group, 4.49% of all sites with a robots.txt keep it out.