AI crawler
Claude-Web — AI crawler guide
What Claude-Web (Anthropic) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | Claude-Web |
| Operator | Anthropic |
| Category | search |
Purpose
Associated with Claude web retrieval rather than bulk training crawl alone.
robots.txt guidance
If you want Claude web answers to access your docs, allow Claude-Web even when ClaudeBot is restricted.
Measurement
Included in BatSignal’s multi-agent robots and policy review.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- anthropic-ai — training
- PerplexityBot — search
Related
FAQ
What is Claude-Web?
Claude-Web is operated by Anthropic. Category: search. Associated with Claude web retrieval rather than bulk training crawl alone.
Should I allow Claude-Web in robots.txt?
If you want Claude web answers to access your docs, allow Claude-Web even when ClaudeBot is restricted.
How does BatSignal treat Claude-Web?
Included in BatSignal’s multi-agent robots and policy review.