opencrawlers

meta-externalfetcher

Meta crawler, type: ai assistant.

About this crawler

Fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities.

User-Agent
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
robots.txt token
meta-externalfetcher
Respects robots.txt
no
Operator page
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers

Including sites that block it only through the * group, 4.27% of all sites with a robots.txt keep it out.

All AI crawlers

What we collect, and how to change your mind