opencrawlers

icc-crawler

NICT crawler, type: ai training.

About this crawler

Research and development of advanced information processing including multilingual translation, information analysis and AI technologies.

User-Agent
ICC-Crawler
robots.txt token
icc-crawler
Respects robots.txt
yes
Operator page
https://ucri.nict.go.jp/en/icccrawler.html

Including sites that block it only through the * group, 4.28% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind