pangubot
Huawei crawler, type: ai training.
- 88,738sites name it in robots.txt
- 81,468block it fully (91.8%)
- 3,770block some paths
- 3,500allow it explicitly
About this crawler
Downloads website content to train Huawei's multimodal LLM called PanGu.
- User-Agent
Mozilla/5.0 (compatible; PanguBot/1.0; +https://www.huaweicloud.com/)- robots.txt token
pangubot- Respects robots.txt
- yes
- Source (third party, not confirmed by the operator)
- https://knownagents.com/agents/pangubot
- Third-party source (Known Agents, formerly Dark Visitors); Huawei does not confirm on this page.
- Source states it is expected to follow robots.txt rules and normally visits from the United States.
- UA info URL points to the generic Huawei Cloud site, not a crawler page.
Including sites that block it only through the * group, 4.29% of all sites with a robots.txt keep it out.