firecrawlagent
Firecrawl crawler, type: ai scraper.
- 39,578sites name it in robots.txt
- 30,196block it fully (76.3%)
- 2,899block some paths
- 6,483allow it explicitly
About this crawler
Firecrawl is a service that scrapes and crawls web pages for its customers. The page says it reads robots.txt before crawling a site and applies rules written for the user-agent token FirecrawlAgent as well as rules for *.
- User-Agent
FirecrawlAgent(third party: source)- robots.txt token
firecrawlagent- Respects robots.txt
- yes
- Operator page
- https://docs.firecrawl.dev/advanced-scraping-guide
- Page states Firecrawl does not use a fixed set of outbound IPs when crawling; 35.245.250.27 is listed only for calling the Firecrawl API from behind a firewall.
- A search snippet from Firecrawl docs says Enterprise users can set a custom User-Agent for robots.txt evaluation.
Including sites that block it only through the * group, 4.21% of all sites with a robots.txt keep it out.