Guide
robots.txt for AI crawlers
How to allow or block GPTBot, OAI-SearchBot, ClaudeBot, and other AI agents without accidentally hiding your site from answer engines.
Why AI bots need an intentional policy
Many sites inherit a default robots.txt meant for classic search—or a CDN “AI crawl control” preamble that blocks training bots. That can be fine for training opt-out, but a blanket block can also prevent answer engines from fetching the pages you want cited.
Common AI user-agents
| User-agent | Typical role |
|---|---|
| GPTBot | OpenAI training / fetch crawler |
| OAI-SearchBot | OpenAI search indexing |
| ChatGPT-User | User-initiated fetches via ChatGPT |
| ClaudeBot / anthropic-ai | Anthropic crawlers |
| PerplexityBot | Perplexity retrieval |
| Google-Extended | Gemini training control (separate from Googlebot) |
| CCBot | Common Crawl |
A sensible public-site pattern
- Allow public marketing, docs, blog, and legal pages
- Disallow `/api/`, `/account`, `/dashboard`, checkout, and other private surfaces
- Declare a Sitemap: line with your canonical sitemap URL
- Keep training opt-outs (if any) explicit—don’t confuse them with search crawl
What BatSignal checks
BatSignal’s crawlability audit reads your robots rules and attempts live fetches (including as GPTBot when possible). Results show up in the crawl-access pillar of the methodology.
Next: make sure those allowed URLs return crawlable HTML, not an empty SPA shell.
FAQ
Should I allow GPTBot?
It depends on your content policy. If you want ChatGPT search and related systems to fetch public pages, allow GPTBot (and often OAI-SearchBot) on public marketing and docs paths while still Disallowing private app routes.
Does Allow in robots.txt force AI to cite me?
No. robots.txt only governs crawl permission. Citations still depend on relevance, content quality, and how answer engines choose sources.