img2dataset
img2dataset crawler, type: ai training.
- 923,276sites name it in robots.txt
- 91,426block it fully (9.9%)
- 799,810block some paths
- 32,040allow it explicitly
About this crawler
Open source tool that downloads images from lists of URLs for machine learning datasets and web indexing.
- robots.txt token
img2dataset- Respects robots.txt
- unknown
- Operator page
- https://github.com/rom1504/img2dataset
- By default ignores images served with X-Robots-Tag noai, noindex, noimageai or noimageindex; disable with --disallowed_header_directives '[]'.
- README does not mention robots.txt and gives no default user agent; --user_agent_token adds an identifying token to the User-Agent header.
- Claims about 100M urls in 20h on one machine.
Including sites that block it only through the * group, 4.27% of all sites with a robots.txt keep it out.