OAI-SearchBot vs GPTBot: Which Should a Documentation Site Allow?

The short answer: treat them as separate policy decisions

A documentation site does not have to make one blanket decision for every OpenAI crawler. OAI-SearchBot and GPTBot represent different uses of access, so the right choice depends on what you are comfortable making available for search retrieval versus model development.

For many public documentation sites, a defensible starting position is to allow OAI-SearchBot and make a separate decision about GPTBot. If your documentation is intentionally public and you want it to be eligible for ChatGPT search retrieval, allowing OAI-SearchBot is consistent with that goal. If you do not want public docs collected for potential model training or improvement, you may block GPTBot while keeping search access open.

That policy is not a ranking tactic. Allowing a crawler does not promise inclusion, citations, traffic, or leads. It simply removes one access barrier. The rest of the visibility chain still matters, as explained in BatSignal’s guide to AI visibility.

What OAI-SearchBot and GPTBot are used for

The names are easy to confuse, but the implications are different. OpenAI’s published crawler documentation describes OAI-SearchBot as a crawler used to surface websites in ChatGPT search results. GPTBot is associated with crawling content that may be used to improve or train OpenAI’s foundation models. The exact products, processing rules, and policies can change, so treat the current OpenAI documentation as the source of truth for implementation details.

The distinction between retrieval and training is important. A page can be available to a search crawler without being supplied for training, and a page may have historical training presence without being available for a current search response. These are different states, not interchangeable labels for “AI visibility.”

Why public documentation needs a more careful policy than marketing pages

Documentation sites often contain several kinds of information under one domain: stable API references, implementation examples, release notes, troubleshooting advice, pricing constraints, and private customer material. A single allow or disallow rule can therefore be too blunt.

• Public reference pages may benefit from being retrievable when developers ask product-specific questions.
• Examples and tutorials may contain valuable intellectual property or operational know-how that your organization does not want used for training.
• Release notes and deprecation notices can become stale quickly, making freshness and source attribution more important than broad crawl access.
• Private, customer-specific, or contractually restricted content should not be treated as public merely because it is reachable through a URL.
• Generated or low-value pages can increase crawl load without improving the chance of a useful answer.

Before changing robots.txt, classify the content. A policy for a public developer portal may be different from a policy for an authenticated support center. If the site mixes both, use authentication and authorization controls for private content; robots.txt is a crawler preference, not a security boundary.

The main tradeoff: retrieval reach versus training control

Allowing OAI-SearchBot can support a practical discoverability objective: when a user asks a current, product-specific question, a retrieval system may be able to find and use your documentation. That is most useful when pages are accessible HTML, clearly scoped, internally linked, and explicit about product names and versions. See the guide to crawlable HTML versus SPAs for the technical side.

Allowing GPTBot is a separate policy choice about potential model development use. Some publishers view broad training access as part of being on the open web. Others want to reserve documentation, examples, or proprietary explanations for their own users, licensing arrangements, or retrieval-controlled experiences. Neither position is automatically correct.

How to express the policy in robots.txt

A robots.txt file can address the crawler names independently. A common example for a public documentation site that allows search retrieval but not GPTBot collection is:

User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: /

The inverse, or a policy that allows both, is also possible. However, syntax and precedence deserve testing. Rules can be affected by user-agent matching, path-specific directives, generated files, CDN behavior, and conflicting blocks. Do not assume that placing a rule in a source repository means the same rule is being served at https://your-domain.example/robots.txt.

1. Fetch the production robots.txt over HTTPS, including redirects and the exact host used by the documentation site.
2. Check that the user-agent strings match the current crawler documentation and are not accidentally merged into a broader rule.
3. Test representative public paths, versioned paths, assets, and any directories containing restricted material.
4. Confirm that the sitemap URL is present and that the sitemap lists canonical, indexable documentation URLs.
5. Review server logs for crawler requests, response codes, request volume, and unexpected paths.

Robots.txt is advisory. It does not authenticate users, delete previously crawled content, control third-party archives, or guarantee that a page will never appear in an answer. Use access controls, noindex directives where appropriate, content removal procedures, and contractual controls for stronger requirements.

Gated content, private docs, and versioned documentation

If a page is genuinely private, enforce that privacy at the application or server layer. A crawler that cannot authenticate should receive an appropriate denial or login response, not the full document with a robots.txt instruction attached. Do not publish secrets, customer data, private URLs, or sensitive configuration in HTML, JavaScript bundles, source maps, or downloadable assets.

For documentation with public and gated tiers, a useful structure is to keep a durable public reference layer open and place customer-specific procedures behind authentication. The public layer can explain concepts, supported versions, and integration requirements without exposing account details. This also gives retrieval systems a clear source for general questions while preserving private operational content.

Versioning adds another concern. If old versions remain publicly crawlable, a retrieval system may find an obsolete instruction. Use canonical links, visible version labels, redirects where appropriate, and clear deprecation notices. A well-formed JSON-LD and AI discovery setup can help machines understand page type and relationships, but structured data cannot force a system to prefer the newest page.

What to measure after changing access

The useful question is not “Did we allow the bot?” It is “What changed, and can we distinguish access from outcomes?” Track the following as separate measurements.

• Crawler access: requests by user-agent, response status, blocked paths, crawl frequency, and server load.
• Content readiness: indexable HTML, title and description quality, canonical URLs, internal links, XML sitemap coverage, and JSON-LD validity.
• Search presence: whether relevant pages are retrieved for representative ChatGPT queries and whether source links are accurate.
• Training presence: evidence that content was available to open-web datasets or historical snapshots, which is not the same as current live retrieval.
• Answer outcomes: mention, recommendation, citation, and competitor share of voice across a fixed query set.

Use a baseline before changing robots.txt. Record the policy, crawl logs, important URLs, and a repeatable set of buyer-intent and technical questions. Recheck after a reasonable interval rather than interpreting a single answer as proof. BatSignal’s measurement guide and AI share-of-voice guide describe this separation in more detail.

A scan can make the mechanical checks easier by flagging crawler access, HTML and metadata issues, sitemap coverage, JSON-LD, and related readiness signals. BatSignal’s features explain the checks included in a Visibility Scan; using one is optional, and it should complement—not replace—your own logs and policy review.

A practical decision framework for documentation teams

Use this sequence when deciding whether to allow one or both crawlers:

1. Define the asset. List the public docs, private docs, generated pages, support content, examples, and downloadable files covered by the domain.
2. Define the intended use. Decide whether you want eligibility for search retrieval, potential training collection, both, or neither.
3. Check obligations. Review customer agreements, licensing terms, contributor agreements, privacy requirements, and internal IP policy.
4. Choose the narrowest workable rule. Separate OAI-SearchBot and GPTBot rather than assuming they have identical purposes.
5. Protect private content technically. Require authentication and authorization; do not depend on robots.txt for confidentiality.
6. Improve the source before expecting visibility. Make important answers available in crawlable HTML, link related concepts, expose current versions, and maintain the sitemap.
7. Measure outcomes over time. Compare access logs and representative answer tests against the baseline.

For teams starting from uncertainty, “allow OAI-SearchBot, review GPTBot deliberately” is often a more informative first policy than allowing or blocking everything by habit. It preserves a clear distinction between search retrieval and training access. But a different policy may be right for a regulated, licensed, private, or highly proprietary documentation set.

A policy is not a substitute for documentation quality

Crawler access is only one layer of discoverability. A page that returns a 200 status can still be difficult for a retrieval system to use if the key answer is hidden behind client-side rendering, scattered across tabs, contradicted by another page, or missing a clear product and version context.

Prioritize pages that answer real questions directly: setup requirements, supported environments, limits, compatibility, migration steps, pricing conditions, and failure recovery. Use descriptive headings and stable URLs. Keep important facts in visible HTML. Add citations to primary sources where relevant. These practices help human readers and machines regardless of which crawler policy you choose.

For broader context, compare GEO and SEO, review the AEO checklist, and use the buyer-intent content guide to prioritize pages that matter to evaluation and adoption.

Bottom line: make two decisions, then verify both

FAQ