Why Your CDN or WAF May Be Blocking AI Crawlers Even When robots.txt Allows Them

An open robots.txt is not the same as an open site

A robots.txt file answers a narrow question: may a compliant crawler request this path? It does not force a CDN, WAF, bot-management product, hosting platform, or origin server to deliver the response. The request must first pass the network and security controls in front of the application.

That distinction matters for AI visibility. A crawler may be permitted in robots.txt but stopped at the edge with a 403, 429, 503, connection reset, TLS failure, or browser challenge. It may also receive a successful status code containing a block page rather than the requested HTML. From the crawler's perspective, the content was not available even though the policy file looked permissive.

This is why AI crawler and robots access should be checked separately from content readiness and from live answer citations. Crawl access, training presence, and citations in generated answers are related, but they are not interchangeable outcomes.

The request path: where an AI crawler can be stopped

A useful investigation follows the request from the network boundary to the document. The exact architecture varies, but the common path is:

1. DNS returns an address for the hostname. A DNS security policy or incorrect record can send the crawler to the wrong service.
2. The crawler negotiates TCP and TLS with the edge. Cipher requirements, certificate-chain problems, SNI handling, HTTP/2 behavior, or aggressive TLS fingerprinting can cause failure before HTTP begins.
3. The CDN receives the request and applies caching, rate limits, geography rules, bot management, and edge functions.
4. The WAF evaluates the IP, headers, method, path, request body, and behavioral signals. It may allow, block, rate-limit, or issue a challenge.
5. The edge either serves a cached response, forwards the request to the origin, or returns an error. Origin access controls can still reject it.
6. The crawler retrieves robots.txt, follows permitted links, and requests the final page. A page-level application rule can block it even when the hostname is reachable.

Do not begin by changing robots.txt if the evidence shows the request never reaches that file. First identify the layer that made the decision and the action it took.

The most common CDN and WAF failure modes

Why user-agent rules and IP allowlists are not enough

Many teams start with a rule such as “allow this user agent.” That is useful for diagnosis, but it is not reliable authentication. User-agent strings are easy to copy, and different retrieval systems may use different identifiers. Conversely, a legitimate crawler can use a shared infrastructure range or an identifier your rule does not recognize.

IP allowlisting has the opposite tradeoff. It can lower false positives when the crawler publishes verifiable address ranges, but ranges can change and may differ by product, region, or retrieval function. A broad allowlist can also bypass protections for traffic that only resembles the crawler.

Use layered verification instead:

• Confirm the crawler's documented identity and verification process. Prefer reverse-and-forward DNS verification or a provider-supported mechanism over a user-agent-only match.
• Match narrowly on hostname, path, and action. An exception for GET requests to public HTML is safer than a blanket bypass for every method and endpoint.
• Keep security logging enabled for allowed requests. An allow rule that removes observability makes later troubleshooting harder.
• Set an owner and review date for exceptions. Record why the rule exists, which evidence supports it, and how it will be tested after a vendor change.
• Never bypass authentication, private content controls, or origin protections merely to improve discoverability. Public AI access should be an intentional content policy.

A repeatable troubleshooting procedure

The goal is not to prove that a particular crawler can fetch one page from one laptop. The goal is to locate the failing control and confirm behavior across representative paths.

1. Define the expected request. Record the hostname, URL, method, user-agent, approximate time, region, and whether the test is for robots.txt, HTML, or an asset. Avoid testing only a cached homepage.
2. Check the public policy files. Fetch robots.txt, sitemap URLs, and, where used, llms.txt. Validate status codes and content, but treat these as policy evidence rather than proof of end-to-end access.
3. Run a plain HTTP request without a browser. Capture status, redirects, headers, content type, response length, and a short body sample. A 200 response with a challenge page is not a successful content retrieval.
4. Correlate the timestamp with CDN and WAF logs. Search by request ID, client IP, user-agent, host, URL, and edge location. Identify the exact rule, action, and response generated.
5. Check whether the origin saw the request. If there is no origin record, the block is probably at DNS, TLS, CDN, or WAF level. If the origin saw it, compare the upstream status and body with the edge response.
6. Repeat with a control request. Use a normal browser or a known permitted fetcher, then vary one signal at a time: region, IP range, user-agent, HTTP version, and URL.
7. Make the smallest rule change possible, preferably in a staging or monitored production policy. Retest the original failing case and a negative case that should remain blocked.
8. Document the result and add a periodic check. Security policies drift as vendors change bot scores, IP ranges, managed rules, TLS defaults, and edge-function code.

What good evidence looks like

A useful incident record contains more than “the AI bot was blocked.” Capture enough detail for another person to reproduce the finding:

• UTC timestamp and request ID
• Hostname, full path, method, and redirect chain
• Observed user-agent, source IP or verified range, ASN, country, and edge location
• DNS result, TLS outcome, HTTP version, status code, headers, and response-body classification
• CDN cache status and origin response status
• WAF rule ID, managed rule category, bot score, and action
• Whether robots.txt and the target HTML behaved differently
• The policy change made, its scope, expiry or review date, and the negative test used to prevent over-broad access

Classify outcomes precisely. “Allowed” should mean the requested document was returned, not merely that the TCP connection succeeded. “Blocked” should distinguish an explicit deny from a timeout, a challenge, a rate limit, a TLS failure, and an origin error. Those causes require different fixes.

How access relates to AI visibility—and what it does not prove

Fixing a WAF block removes one possible access failure. It does not guarantee that a model will train on the content, retrieve it for a user query, cite it, or recommend the business. Some systems use live retrieval; others rely on previously collected data, licensed sources, indexes, or internal pipelines. Common Crawl and training presence is a separate question from whether a live request currently succeeds.

After access is confirmed, check the document itself. Can the important text be retrieved in server-rendered HTML? Are canonical tags, metadata, structured data, and sitemaps coherent? Is the page useful for the buyer-intent question it is meant to answer? The crawlable HTML versus SPA guide, JSON-LD guide, and buyer-intent content guide cover those adjacent checks.

Finally, measure live answer behavior separately. A clean fetch does not imply a citation, and a citation can occur through a source path you did not expect. Use a defined query set and record mention, recommendation, citation, and competitor visibility over time. The AI share of voice guide explains why one successful retrieval is not a visibility metric.

A practical prevention checklist

Treat crawler access as an operational control, not a one-time SEO setting. Add these checks to ordinary site reliability and security review:

• Monitor robots.txt and representative public HTML from more than one region and network.
• Alert on unusual increases in 403, 429, 5xx, timeouts, and challenge responses for documented crawlers.
• Review managed WAF rule updates and bot-score changes before they reach production when possible.
• Keep public informational pages on a path that does not require JavaScript clearance, cookies, or browser verification.
• Use cache and rate-limit settings that account for legitimate low-volume crawlers without disabling abuse controls.
• Retest after CDN migrations, DNS changes, certificate renewals, origin changes, and major WAF policy updates.
• Maintain a written inventory of intended AI access, restricted paths, and exceptions. Include whether each policy concerns live retrieval, training use, or both.
• Run a full AI visibility measurement review instead of treating crawler access as the final KPI.

The bottom line

robots.txt can express permission, but it cannot override the infrastructure that receives the request. If AI crawlers appear absent despite an open policy file, inspect DNS and TLS, then CDN decisions, WAF events, origin logs, and the actual response body. Look for rate limits, challenges, reputation scores, geography rules, protocol mismatches, and custom firewall logic.

The safest fix is evidence-led and narrow: identify the failing layer, verify the crawler rather than trusting its label, allow only the public traffic you intend, and keep a negative test for protected areas. Then evaluate content readiness and live answer visibility as separate measurements. BatSignal's methodology describes this separation; a Visibility Scan is optional if you want a repeatable external check rather than a one-off manual test.

Frequently asked questions

Can a WAF block robots.txt while allowing normal pages?

Yes. A WAF rule can target the robots.txt path, apply a different cache policy, or rate-limit it separately. A crawler may also retrieve robots.txt but be blocked when it requests the HTML paths listed or linked from the site. Test both the policy files and representative content URLs.

Is a 403 proof that the crawler is intentionally blocked?

No. A 403 may come from a WAF deny rule, origin authorization, a bad Host header, an IP reputation decision, or an application-level policy. The response body and correlated rule or origin logs are needed to identify the cause.

Should I remove all bot protection from public pages?

Usually not. Removing protection can expose the site to scraping, abuse, and denial-of-service traffic. Prefer a narrowly scoped exception for verified, read-only access to genuinely public content, while retaining controls for login, checkout, APIs, administration, and sensitive paths.

How often should AI crawler access be retested?

FAQ