Glossary
Disallow (robots)
A robots.txt rule telling a crawler not to fetch matching paths.
Definition
Disallow is powerful and easy to misconfigure. A broad Disallow: / silently zeros AI crawl access for that agent.
In practice
- Use precise language on public pages so assistants and crawlers share the same meaning
- Pair definitions with crawlable HTML, metadata, and intentional robots policy
- Measure outcomes (mentions, citations, pillar scores) after you change signals
See also
Next step
Browse the glossary index, read what is AI visibility?, or measure your site.
Related
FAQ
What is Disallow (robots)?
A robots.txt rule telling a crawler not to fetch matching paths.
Why does Disallow (robots) matter for AI visibility?
Disallow is powerful and easy to misconfigure. A broad Disallow: / silently zeros AI crawl access for that agent.
How can BatSignal help?
BatSignal measures crawl access, content readiness, ChatGPT presence, and related signals so teams can improve AI visibility with evidence—not guesswork.