opencrawlers

img2dataset

img2dataset crawler, type: ai training.

About this crawler

Open source tool that downloads images from lists of URLs for machine learning datasets and web indexing.

robots.txt token
img2dataset
Respects robots.txt
unknown
Operator page
https://github.com/rom1504/img2dataset

Including sites that block it only through the * group, 4.27% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind