opencrawlers

firecrawlagent

Firecrawl crawler, type: ai scraper.

About this crawler

Firecrawl is a service that scrapes and crawls web pages for its customers. The page says it reads robots.txt before crawling a site and applies rules written for the user-agent token FirecrawlAgent as well as rules for *.

User-Agent
FirecrawlAgent (third party: source)
robots.txt token
firecrawlagent
Respects robots.txt
yes
Operator page
https://docs.firecrawl.dev/advanced-scraping-guide

Including sites that block it only through the * group, 4.21% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind