AI crawler
CCBot — AI crawler guide
What CCBot (Common Crawl) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | CCBot |
| Operator | Common Crawl |
| Category | training |
Purpose
Builds the Common Crawl public web archive used widely in research and some training corpora.
robots.txt guidance
Allowing CCBot increases odds of archival snapshots; disallowing it is a valid policy if you reject archival crawling.
Measurement
Common Crawl domain presence is a small training-presence signal in BatSignal scores.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- anthropic-ai — training
Related
FAQ
What is CCBot?
CCBot is operated by Common Crawl. Category: training. Builds the Common Crawl public web archive used widely in research and some training corpora.
Should I allow CCBot in robots.txt?
Allowing CCBot increases odds of archival snapshots; disallowing it is a valid policy if you reject archival crawling.
How does BatSignal treat CCBot?
Common Crawl domain presence is a small training-presence signal in BatSignal scores.