AI crawler
GPTBot — AI crawler guide
What GPTBot (OpenAI) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | GPTBot |
| Operator | OpenAI |
| Category | training |
Purpose
Fetches public pages for OpenAI training-related crawling. Often managed separately from ChatGPT search bots in robots.txt.
robots.txt guidance
Decide deliberately whether training on your public content is acceptable. If you disallow GPTBot, still consider allowing OAI-SearchBot if you want ChatGPT search retrieval.
Measurement
BatSignal checks GPTBot robots rules and attempts a live homepage fetch with an AI crawler identity as part of crawl access.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- anthropic-ai — training
- PerplexityBot — search
Related
FAQ
What is GPTBot?
GPTBot is operated by OpenAI. Category: training. Fetches public pages for OpenAI training-related crawling. Often managed separately from ChatGPT search bots in robots.txt.
Should I allow GPTBot in robots.txt?
Decide deliberately whether training on your public content is acceptable. If you disallow GPTBot, still consider allowing OAI-SearchBot if you want ChatGPT search retrieval.
How does BatSignal treat GPTBot?
BatSignal checks GPTBot robots rules and attempts a live homepage fetch with an AI crawler identity as part of crawl access.