icc-crawler
NICT crawler, type: ai training.
- 114,807sites name it in robots.txt
- 75,371block it fully (65.7%)
- 7,116block some paths
- 32,320allow it explicitly
About this crawler
Research and development of advanced information processing including multilingual translation, information analysis and AI technologies.
- User-Agent
ICC-Crawler- robots.txt token
icc-crawler- Respects robots.txt
- yes
- Operator page
- https://ucri.nict.go.jp/en/icccrawler.html
- Supports REP/RFC 9309 and Crawl-Delay; operated by NICT Universal Communication Research Institute.
- Page lists IPs 202.180.34.186 and 61.86.246.72 but no IP list URL.
- Collected data may be shared with third parties for research; stops access on host operator request.
Including sites that block it only through the * group, 4.28% of all sites with a robots.txt keep it out.