AI crawler
Google-Extended — AI crawler guide
What Google-Extended (Google) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | Google-Extended |
| Operator | |
| Category | training |
Purpose
Used in robots.txt to signal preferences related to Google AI / Gemini training controls, separate from Googlebot search.
robots.txt guidance
Configure Google-Extended based on AI training policy; keep Googlebot allowed if you still want classic Search.
Measurement
BatSignal surfaces Google-Extended alongside other AI training agents.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- anthropic-ai — training
Related
FAQ
What is Google-Extended?
Google-Extended is operated by Google. Category: training. Used in robots.txt to signal preferences related to Google AI / Gemini training controls, separate from Googlebot search.
Should I allow Google-Extended in robots.txt?
Configure Google-Extended based on AI training policy; keep Googlebot allowed if you still want classic Search.
How does BatSignal treat Google-Extended?
BatSignal surfaces Google-Extended alongside other AI training agents.