AI crawler
Bytespider — AI crawler guide
What Bytespider (ByteDance) does, how to configure robots.txt, and how BatSignal measures access.
Operator and category
| Field | Value |
|---|---|
| User-agent | Bytespider |
| Operator | ByteDance |
| Category | training |
Purpose
ByteDance crawler that appears in AI-related robots policies.
robots.txt guidance
Set an explicit rule matching your jurisdictional and product policy for ByteDance crawling.
Measurement
Included in BatSignal’s AI bot audit roster.
Also read robots.txt for AI crawlers and AI crawl access.
Related bots
- GPTBot — training
- OAI-SearchBot — search
- ChatGPT-User — agent
- ClaudeBot — training
- Claude-Web — search
- anthropic-ai — training
Related
FAQ
What is Bytespider?
Bytespider is operated by ByteDance. Category: training. ByteDance crawler that appears in AI-related robots policies.
Should I allow Bytespider in robots.txt?
Set an explicit rule matching your jurisdictional and product policy for ByteDance crawling.
How does BatSignal treat Bytespider?
Included in BatSignal’s AI bot audit roster.