Which AI Crawlers Can Render JavaScript—and Which Cannot?

The short answer: do not make JavaScript a prerequisite

AI crawlers do not form one consistent technical category. Some are simple HTTP fetchers. Some belong to search systems that may render pages in a separate pipeline. Some are browser-based agents that can execute JavaScript while completing a task. A crawler’s published user-agent name does not, by itself, tell you which of these behaviors applies.

The practical default is therefore conservative: assume that the first HTML response matters. Put the page’s primary answer, meaningful headings, internal links, canonical URL, and important metadata in HTML that arrives without requiring a click, a delayed script, or a successful client-side API request. JavaScript can enhance that foundation, but it should not be the only path to the content.

This is the same distinction covered in BatSignal’s guide to crawlable HTML versus SPAs: a page can work perfectly for a human using a modern browser while exposing very little to a basic fetcher.

Crawler access, JavaScript rendering, and citation are different questions

Teams often collapse several stages into the phrase “AI visibility.” They are related, but they are not interchangeable.

For a fuller framework, see the AI visibility guide and BatSignal’s methodology, which separate crawler access, content readiness, open-web coverage, training presence, and live answer visibility.

What the major crawler categories generally do

Specific implementations change, and a service can operate more than one crawler. The table below is a planning guide, not a permanent capability list. Verify behavior using your own logs and test pages rather than treating a user-agent label as proof.

Why client-rendered content is fragile

A single-page application is not automatically a problem. The risk comes from what the server returns before JavaScript runs. A minimal document containing only a root element and script tags gives a crawler little to work with if it does not render, if rendering times out, or if the application fails outside a full browser.

Common failure patterns include:

• The server returns an empty <div id="root"> and puts the article text into a later API response.
• The page title and meta description are changed only after hydration, while the initial HTML has generic metadata.
• Internal links are created by click handlers or JavaScript routing rather than normal anchor elements.
• The content request requires cookies, a consent decision, a token, or a browser-generated header.
• Important sections appear only after scrolling, opening an accordion, submitting a form, or waiting for a timer.
• A JavaScript bundle fails because of a Content Security Policy, an expired asset, a blocked third-party domain, or a framework error.
• The page displays a loading state in the initial response and never exposes a useful fallback to non-browser clients.

These problems can also affect accessibility, link previews, caching, translation tools, and conventional search. Improving the HTML is therefore a robustness measure, not an attempt to exploit a particular AI system.

What an AI-ready HTML fallback should contain

The fallback does not need to reproduce every interactive feature. It needs to make the page’s purpose and substantive answer understandable without the application booting.

• A specific, descriptive title and a useful meta description.
• One clear H1 and a logical heading structure.
• The main explanatory text, product or service details, and important qualifications in the response HTML.
• Normal <a href> links to related pages, category pages, authors, documentation, and sources.
• A canonical URL and appropriate robots meta directives.
• Structured data where it accurately describes the visible page; see the guide to JSON-LD for AI discovery.
• A sitemap that exposes indexable URLs, with consistent canonical and internal-link signals.
• Visible author, date, organization, and contact or policy information where those facts matter.
• A readable response even when CSS and JavaScript are disabled.

Do not fill the fallback with keyword-heavy duplicate text that users are not meant to see. The goal is parity: the no-script version should communicate the same core answer as the enhanced version, with appropriate changes only for interactive controls.

An llms.txt file can provide an additional navigational aid in some workflows, but it does not repair a page whose main content is inaccessible. Treat it as a supplement to HTML, robots.txt, sitemaps, and normal information architecture. BatSignal’s llms.txt guide explains that distinction.

A repeatable test for JavaScript-dependent content

You can test this without guessing which AI company uses which rendering stack. Start with a representative sample: your homepage, a high-value guide, a product or service page, a comparison page, and a few deep URLs.

1. Fetch each URL with a plain HTTP client such as curl or a small script. Record status code, redirects, response size, title, headings, visible text, links, canonical URL, robots directives, and JSON-LD.
2. Save the raw HTML. Do not inspect only the browser’s Elements panel, because that panel shows the post-JavaScript DOM rather than necessarily the server response.
3. Load the same URLs in a controlled browser and compare the rendered text with the raw response. Identify content that appears only after hydration or an API request.
4. Run the browser test with JavaScript disabled. Note whether the article, navigation, metadata, and links remain usable.
5. Check network requests and timing. A page that needs several seconds or a third-party API to reveal its answer is more fragile than one that returns the answer immediately.
6. Test important routes with a mobile viewport, a fresh session, no cookies, and blocked third-party scripts. This approximates conditions that differ from your own logged-in browser.
7. Review server and CDN logs for crawler requests, response codes, blocked paths, unusual latency, and whether the requested URL received the same HTML as a normal visitor.
8. Repeat after deployments. Rendering regressions often come from framework, routing, consent, or asset changes rather than from editorial updates.

A useful comparison is the text available at time zero versus the text available after rendering. If the answer to a user’s question is absent at time zero, mark the URL as JavaScript-dependent even if Chrome eventually displays it.

How to interpret crawler evidence

Access logs are valuable, but they require skepticism. A request with a familiar user-agent string is not conclusive proof that the request came from that organization; bots can impersonate one another. Conversely, the absence of a request does not prove that a service cannot render JavaScript. Collection may be infrequent, routed through another infrastructure layer, or controlled by a separate product identity.

Look for patterns rather than a single request:

• Does the crawler receive a 200 response rather than a redirect loop, 403, 429, or soft error?
• Does robots.txt permit the relevant path and its required assets where rendering is expected?
• Does the server return the same substantive HTML to a clean crawler request and an ordinary anonymous request?
• Are important CSS, JavaScript, API, or image resources blocked by robots rules or authentication?
• Does the page remain available when query parameters, cookies, and browser headers are absent?
• Are crawler requests failing because of rate limits, bot challenges, geofencing, or a slow origin?

For robots-specific checks, use the robots.txt and AI crawlers guide. Remember that robots.txt governs access instructions; it does not force a permitted crawler to execute JavaScript, and it cannot grant access to content hidden behind authentication.

A practical decision framework for site teams

The right implementation depends on your stack and publishing workflow, but the following sequence avoids most unnecessary debates about individual bots.

1. Identify the pages that must be discoverable: core explanations, product pages, documentation, comparisons, and evidence pages.
2. Inspect their raw HTML and classify each as HTML-complete, partially hydrated, or shell-only.
3. Move the primary answer and important links into server-side rendering, static generation, or a reliable edge-rendered response.
4. Keep JavaScript for filters, personalization, live data, and other enhancements that genuinely require it.
5. Remove avoidable gates before the first meaningful content: cookie walls for public pages, interaction-only navigation, and API calls required for basic copy.
6. Confirm that canonical URLs, headings, metadata, JSON-LD, sitemap entries, and internal links agree.
7. Run the same checks after framework upgrades and major design changes.

What this can and cannot improve

Making content available in HTML can improve the probability that a crawler retrieves, parses, and associates the page with its topic. It can also improve conventional search resilience and make audits more straightforward. It cannot guarantee that an AI system will train on the page, select it for retrieval, cite it in an answer, recommend the business, or send traffic.

Keep three records separate in your measurement plan: current crawler access, presence in a known open-web or training corpus, and live answer outcomes. A page can pass the first test and fail the second because the collection happened earlier. It can pass both and still not appear in an answer because retrieval and source selection are query-dependent. BatSignal’s guides on Common Crawl and training presence and measuring AI visibility cover these differences.

If you want a repeatable baseline across URLs, a BatSignal Visibility Scan is optional. It can help organize checks for crawler and robots access, HTML and metadata readiness, JSON-LD, sitemaps, llms.txt, open-web coverage, and selected buyer-intent visibility. Treat the result as a diagnostic snapshot, not a promise of citations, rankings, traffic, or leads.

The durable rule

Do not build your discovery strategy around the hope that an AI crawler will behave like Chrome. Build pages that return a complete, understandable answer in HTML, use JavaScript to enhance rather than conceal it, and verify the result with raw-response tests, no-script checks, and server logs.

FAQ