opencrawlers

pangubot

Huawei crawler, type: ai training.

About this crawler

Downloads website content to train Huawei's multimodal LLM called PanGu.

User-Agent
Mozilla/5.0 (compatible; PanguBot/1.0; +https://www.huaweicloud.com/)
robots.txt token
pangubot
Respects robots.txt
yes
Source (third party, not confirmed by the operator)
https://knownagents.com/agents/pangubot

Including sites that block it only through the * group, 4.29% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind