meta-externalfetcher
Meta-ExternalFetcher fetches individual links when a user asks for them, for example to support agentic AI that helps people complete tasks on websites. Meta says it may bypass robots.txt because it performs fetches that users requested, so only a firewall rule reliably stops it.
What it does
Meta says Meta-ExternalFetcher performs user-initiated fetches of individual links to support product functions, such as agentic AI that helps users complete tasks on websites. It is not a general crawler.
Meta states that because these fetches were requested by a user, Meta-ExternalFetcher may bypass robots.txt. Blocking it at the firewall makes those fetches fail.
robots.txt token
meta-externalfetcher
User-agent string
meta-externalfetcher/1.1How to allow it
User-agent: meta-externalfetcher
Allow: /How to block it
User-agent: meta-externalfetcher
Disallow: /How to verify real visits
Meta does not publish IP ranges or a reverse DNS method for this fetcher in its crawler documentation, so a user agent match alone cannot prove a request is genuine. Meta directs questions about its crawlers to webmasters@meta.com.