Glossary
robots.txt
A root file that tells crawlers which paths they may fetch.
Definition
For AI visibility, robots.txt should encode an intentional policy per agent class (training, search, browsing), include a Sitemap line, and avoid accidental sitewide blocks on pages you want cited.
In practice
- Use precise language on public pages so assistants and crawlers share the same meaning
- Pair definitions with crawlable HTML, metadata, and intentional robots policy
- Measure outcomes (mentions, citations, pillar scores) after you change signals
Related guide: robots.txt.
See also
Next step
Browse the glossary index, read what is AI visibility?, or measure your site.
Related
FAQ
What is robots.txt?
A root file that tells crawlers which paths they may fetch.
Why does robots.txt matter for AI visibility?
For AI visibility, robots.txt should encode an intentional policy per agent class (training, search, browsing), include a Sitemap line, and avoid accidental sitewide blocks on pages you want cited.
How can BatSignal help?
BatSignal measures crawl access, content readiness, ChatGPT presence, and related signals so teams can improve AI visibility with evidence—not guesswork.