Should You Allow AI Crawlers on Product Pages but Block Internal Search Results?

The useful answer is usually not “allow” or “block” across the whole site. It is a policy by content section. Public product pages may contain the facts an answer engine needs, while internal search results, filter combinations, account areas, and duplicate URLs add crawl volume without adding much independent value. A section-level policy lets you preserve useful discovery without treating every URL as equally valuable.

Start by separating access, indexing, training, and citation

Robots policy discussions often collapse several different outcomes into one. They are not the same. A crawler being permitted to fetch a page does not mean the page will be included in a training dataset, shown in a search index, cited in a live answer, or used in a recommendation.

This distinction matters when writing a policy. If your goal is to reduce waste from faceted URLs, robots.txt may help with compliant crawlers. If your goal is to keep confidential information out of public systems, use authentication and access controls. If your goal is to improve the chance that a product fact is understood, focus on crawlable HTML, clear metadata, structured data, and accurate content as well as access.

BatSignal’s AI visibility guide covers this broader model, including why crawl access, Common Crawl presence, and live answer visibility should be measured separately.

Create a URL inventory before writing directives

A robots.txt file is only as good as the URL model behind it. Before adding rules, group URLs by their purpose rather than by how they look in a navigation menu. Review a crawl export, XML sitemaps, analytics or server logs, and application route definitions. Include URLs that are not linked in the main navigation but are generated by filters, search, personalization, or campaigns.

1. List public content sections: products, categories, documentation, editorial content, comparison pages, and help articles.
2. List generated sections: internal search, faceted navigation, sorting, pagination, calendars, tag archives, and recommendation parameters.
3. List restricted sections: account pages, checkout, carts, admin routes, previews, staging paths, and customer-specific resources.
4. Identify duplicate URL forms, including tracking parameters, alternate casing, session IDs, print views, and legacy routes.
5. For each group, record whether it has unique user value, whether it should appear in search, and whether it contains information suitable for public retrieval.

The key question is not “Could a crawler fetch this?” It is “Does this URL represent a durable, public piece of information that deserves independent discovery?” A product page usually does. A search for “blue shoes under $80 sorted by popularity” often does not, unless the business deliberately maintains and publishes that result as a landing page.

A practical default policy for common website sections

The following is a starting point, not a copy-and-paste universal policy. Apply it to the URL patterns your application actually emits, and verify behavior with logs and repeated scans.

For product pages, permission is only the first step. A crawler that receives a thin JavaScript shell, contradictory pricing, missing specifications, or an inaccessible variant may not have enough usable evidence to represent the product correctly. The crawlable HTML versus SPA guide explains why rendering reliability belongs in the same review as robots policy.

Use robots.txt for broad access policy, not URL security

robots.txt is a public set of instructions for compliant crawlers. Anyone can request the file, and not every bot will honor it. It should therefore express crawl preferences, not enforce confidentiality. Private areas need authentication, authorization, network controls, and careful handling of URLs that may leak through referrals or logs.

A simple conceptual file might look like this:

User-agent: * Disallow: /search Disallow: /account/ Disallow: /cart Disallow: /checkout/ Disallow: /admin/ Disallow: /preview/ User-agent: ExampleAI-Crawler Allow: /products/ Allow: /guides/ Disallow: /search Disallow: /account/ Disallow: /checkout/

The exact syntax and crawler names require care. Standard robots.txt matching supports paths, but query-string handling, wildcard support, and bot behavior can vary. A rule intended to block every URL containing a parameter may not work as expected across all consumers. Test representative URLs rather than assuming that a visually plausible directive is effective.

• Put the file at the site root and keep it valid, readable, and version controlled.
• Use the narrowest path rules that match the intended section.
• Avoid accidentally disallowing product assets, documentation, or API endpoints required to understand public pages.
• Review User-agent-specific groups carefully; rules may not combine the way a team expects.
• Keep a change log explaining why each high-impact rule exists.
• Validate the result with crawler documentation, server logs, and a robots testing tool where available.

The robots.txt for AI crawlers guide goes deeper into bot identity, directives, and the limits of relying on robots policy alone.

Treat internal search and faceted navigation as separate problems

Internal search results are commonly poor crawler targets because they are generated on demand, can reflect arbitrary user input, and often create many equivalent URLs. Blocking a predictable search path is sensible when the site does not intentionally publish search-result pages. It is less sensible if the same route serves carefully edited, indexable destination pages. In that case, give the curated pages stable URLs and distinguish them from arbitrary searches.

Faceted navigation needs more nuance. A filter combination can be low value when it differs from another URL only by ordering or a parameter. But a maintained “waterproof hiking boots” collection may be valuable if it has unique copy, a stable canonical URL, appropriate inventory, and a reason for users to land there. Do not block every filtered URL merely because some filters are unhelpful.

• Define a finite set of valuable landing pages instead of exposing unlimited combinations.
• Normalize parameter order and remove session IDs, tracking parameters, and duplicate separators.
• Use canonical URLs for equivalent versions, while remembering that canonicalization is a hint rather than a crawler prohibition.
• Use noindex when a crawler can access a page and you want to communicate that it should not be indexed; do not expect a blocked crawler to read that noindex signal.
• Monitor crawl patterns after changes. A sharp reduction in requests is not proof that valuable discovery improved.

For AI systems, the practical goal is to make the preferred, stable page the easiest source to interpret. The JSON-LD and AI discovery guide can help with product, organization, and other structured facts, but structured data does not override an inaccessible page or guarantee a citation.

Apply bot-specific rules only when the policy is intentional

Some organizations want one policy for all crawlers. Others distinguish search crawlers, user-triggered retrieval bots, training crawlers, and unknown agents. Either approach can be defensible, but it should follow a documented purpose rather than a theory that one named bot automatically produces business value.

Before creating a bot-specific group, answer four questions:

1. Do we know who operates the crawler and what it is used for?
2. Do we have a clear reason to allow or restrict it for this section?
3. Will the rule affect public discovery, training datasets, or only a particular retrieval path?
4. How will we verify that the claimed bot is genuine and that the rule changed the intended requests?

User-agent strings can be spoofed, and crawler names change. Server logs, reverse DNS or documented verification methods, request rates, and published crawler policies provide stronger evidence than a name alone. If licensing, privacy, or regulatory concerns drive the decision, involve the appropriate legal and security owners. This is operational policy, not just an SEO setting.

A cautious default is to permit compliant crawlers to access high-value public pages unless there is a concrete reason not to, while restricting low-value generated areas consistently. Revisit that default when you have evidence about resource consumption, content misuse, contractual requirements, or actual answer visibility.

Pair access rules with page and sitemap hygiene

Section-level access works best as part of a larger technical system. A crawler that is allowed into a section should find coherent signals when it arrives. A crawler that is excluded should not be continually invited through internal links and XML sitemaps.

• Include preferred public URLs in XML sitemaps, not arbitrary search results or parameter combinations.
• Use self-referencing or deliberate canonical tags on pages that should stand on their own.
• Keep titles, headings, descriptions, product attributes, and availability information consistent and specific.
• Expose important facts in crawlable HTML instead of requiring every detail to be assembled after client-side execution.
• Use llms.txt only as an optional guidance file; it does not replace robots.txt, authentication, sitemaps, or page quality work.
• Review links from public pages so blocked or disposable URL patterns are not unnecessarily promoted.

There is also a distinction between being present in an open-web crawl and being cited in a current answer. The Common Crawl and training presence guide covers the former, while the ChatGPT citations guide addresses the latter. Neither outcome should be inferred from a robots file alone.

Measure the policy with repeatable checks

A crawler policy is a hypothesis about which requests should happen and which pages should remain discoverable. Test it like one. Establish a baseline before changing the file, then compare the same URL sets and time periods afterward.

1. Export representative URLs from every section, including valid product pages, blocked search results, filter combinations, private routes, assets, and legacy URLs.
2. Check robots matching for each crawler class you care about. Record the expected result and the actual result.
3. Inspect server logs for requests to disallowed paths, unexpected product blocks, excessive parameter crawling, and crawler identity anomalies.
4. Crawl a sample of allowed pages to check status codes, redirects, rendered HTML, metadata, canonical tags, structured data, and sitemap inclusion.
5. Compare bot request volume and error rates after deployment. Look for accidental 403, 5xx, redirect loops, and blocked resources.
6. Recheck live answer visibility separately with a fixed prompt set if that is a business goal. Do not use crawler access as a proxy for citations or recommendations.

A Visibility Scan can be an optional way to organize checks across crawler access, content readiness, open-web coverage, buyer-intent visibility, and competitor share of voice. It is most useful when treated as a repeatable baseline and not as a promise of rankings, citations, traffic, or leads. BatSignal documents its measurement approach in the methodology. Pricing and scope are available on the pricing page.

FAQ