AI crawler
Slurp — AI crawler guide
What Slurp (Yahoo) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | Slurp |
| Operator | Yahoo |
| Category | search |
Purpose
Yahoo’s crawler user-agent, still seen in logs and robots examples.
robots.txt guidance
Usually mirror your Bing policy; Yahoo search is Bing-powered in many regions.
Measurement
Lower priority than GPTBot/OAI-SearchBot but useful for complete robots documentation.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- anthropic-ai — training
Related
FAQ
What is Slurp?
Slurp is operated by Yahoo. Category: search. Yahoo’s crawler user-agent, still seen in logs and robots examples.
Should I allow Slurp in robots.txt?
Usually mirror your Bing policy; Yahoo search is Bing-powered in many regions.
How does BatSignal treat Slurp?
Lower priority than GPTBot/OAI-SearchBot but useful for complete robots documentation.