AI crawler
meta-externalagent — AI crawler guide
What meta-externalagent (Meta) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | meta-externalagent |
| Operator | Meta |
| Category | training |
Purpose
Meta AI crawler user-agent used for AI-related fetching.
robots.txt guidance
Document Meta AI crawling preference in robots.txt rather than relying on defaults.
Measurement
BatSignal reports meta-externalagent policy in multi-bot audits.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- anthropic-ai — training
Related
FAQ
What is meta-externalagent?
meta-externalagent is operated by Meta. Category: training. Meta AI crawler user-agent used for AI-related fetching.
Should I allow meta-externalagent in robots.txt?
Document Meta AI crawling preference in robots.txt rather than relying on defaults.
How does BatSignal treat meta-externalagent?
BatSignal reports meta-externalagent policy in multi-bot audits.