AI crawler
anthropic-ai — AI crawler guide
What anthropic-ai (Anthropic) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | anthropic-ai |
| Operator | Anthropic |
| Category | training |
Purpose
Additional Anthropic-related crawler token seen in robots policies and bot lists.
robots.txt guidance
Mirror your ClaudeBot stance unless you have a documented reason to differ.
Measurement
BatSignal’s audit bot list includes anthropic-ai for completeness.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- PerplexityBot — search
Related
FAQ
What is anthropic-ai?
anthropic-ai is operated by Anthropic. Category: training. Additional Anthropic-related crawler token seen in robots policies and bot lists.
Should I allow anthropic-ai in robots.txt?
Mirror your ClaudeBot stance unless you have a documented reason to differ.
How does BatSignal treat anthropic-ai?
BatSignal’s audit bot list includes anthropic-ai for completeness.