AI crawler
ClaudeBot — AI crawler guide
What ClaudeBot (Anthropic) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | ClaudeBot |
| Operator | Anthropic |
| Category | training |
Purpose
Anthropic training-oriented crawler that fetches public web content subject to robots.txt.
robots.txt guidance
Encode your Anthropic training preference explicitly. Do not assume a Googlebot rule covers ClaudeBot.
Measurement
ClaudeBot Allow/Disallow state appears in BatSignal crawl-access reporting.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- Claude-Web — search
- anthropic-ai — training
- PerplexityBot — search
Related
FAQ
What is ClaudeBot?
ClaudeBot is operated by Anthropic. Category: training. Anthropic training-oriented crawler that fetches public web content subject to robots.txt.
Should I allow ClaudeBot in robots.txt?
Encode your Anthropic training preference explicitly. Do not assume a Googlebot rule covers ClaudeBot.
How does BatSignal treat ClaudeBot?
ClaudeBot Allow/Disallow state appears in BatSignal crawl-access reporting.