How to Run AI Visibility Regression Tests After a Website Migration

A migration is not complete when the new pages return 200 status codes. It is complete when the important information remains reachable, renderable, understandable, and attributable across the systems that may use it. That includes search crawlers, AI crawlers, Common Crawl, and live answer systems such as ChatGPT and other assistants.

The practical way to test this is to treat the migration as a software release: define a pre-migration baseline, run deterministic checks after launch, compare representative prompts, and investigate meaningful deltas instead of relying on anecdotes. This is a measurement exercise, not a promise that a particular model will cite a page.

Define what “AI visibility” means before testing

AI visibility is not one score and it is not a synonym for rankings. At minimum, separate three layers that are often mixed together:

• Crawl access: whether a crawler can fetch the site and its important resources under the relevant robots.txt rules, status codes, redirects, and network conditions.
• Training or open-web presence: whether content may have been available to datasets such as Common Crawl. This is historical coverage, not evidence that a model learned a specific page or will use it in an answer.
• Live answer visibility: whether a model mentions, recommends, or cites the site for a defined prompt at a defined time, region, account state, and model version.

A migration can improve one layer and weaken another. For example, a cleaner site may be easier to crawl while a rewritten product page no longer contains the exact facts that support a useful answer. Your test plan should report these layers separately rather than combining them into a misleading pass/fail result.

Build a before-and-after test inventory

Create the baseline before changing DNS, routing, templates, or content. Export the URLs and attributes that matter to the business, not just the URLs in the XML sitemap. Include product pages, service pages, comparison pages, documentation, locations, author pages, and high-value articles.

1. Select critical URLs. Mark the pages that describe your products, category, audience, differentiators, pricing, availability, policies, and company entities.
2. Map old URLs to new URLs. Record the expected destination, redirect status, canonical URL, title, headings, word or content blocks, and key facts for each mapping.
3. Save representative HTML. Capture the response HTML and, where relevant, a rendered DOM or screenshot. A browser view alone can hide differences in the server response that a crawler receives.
4. Record technical signals. Include robots.txt, XML sitemaps, HTTP status, redirect chains, canonical tags, hreflang, JSON-LD, metadata, internal links, and important asset requests.
5. Freeze a prompt set. Use the same prompts before and after launch, with timestamps, model names, region, and any relevant settings recorded.

A useful inventory has both URL-level and prompt-level rows. The URL rows answer “can the important source be reached and understood?” The prompt rows answer “does the system still use or describe that source in the same way?” Keep the raw captures; summaries without evidence are difficult to debug.

Run the URL and crawl-access regression checks

The first post-launch pass should be mechanical. It should not depend on an AI model’s current answer. Start at the old URL list and verify the expected path to the new URL.

Check robots.txt from the production hostname and protocol, including any redirects to it. Review disallow rules for AI crawlers separately from general search rules, but do not assume that allowing a crawler guarantees a citation. For a practical implementation checklist, see the guides to robots.txt and AI crawlers, crawlable HTML versus SPAs, and llms.txt.

Also inspect the final response, not only the source URL. A page that returns a successful status to a browser but depends on a client-side route, delayed API call, cookie, or geo condition may not expose the same information to a crawler.

Compare page meaning, entities, and facts—not just templates

A migration can pass every URL check while changing what a page actually says. Compare the semantic content that supports the organization’s identity and buyer decisions.

• Entity identity: organization name, parent or subsidiary relationships, locations, authors, product names, and official domains.
• Product and service facts: capabilities, compatibility, dimensions, plans, pricing language, availability, restrictions, and supported use cases.
• Differentiators: verifiable claims that distinguish the business from competitors, including evidence, certifications, integrations, or service boundaries.
• Page purpose: whether a product page still answers product questions, a comparison page still compares alternatives, and a documentation page still contains the procedure or specification.
• Evidence and attribution: author information, dates, references, customer proof, first-party documentation, and consistent claims across pages.

Use a fact ledger for high-value entities. Each row can contain a fact, its preferred wording, the source URL, the structured-data property if applicable, and whether the fact changed during migration. This catches subtle regressions such as a product losing its supported platform, an organization name changing in one template, or a location page inheriting the wrong address.

JSON-LD is useful when it accurately describes visible page content and connects related entities. It is not a hidden control channel for answer engines. Validate it alongside the visible HTML using the JSON-LD and AI discovery guide, rather than treating valid syntax as proof of discoverability.

Test live prompts with a controlled comparison set

Live answer testing is inherently less deterministic than HTTP checks. Models can change retrieval sources, wording, context windows, and citation behavior. You can still make the comparison useful by controlling the test conditions.

1. Create prompt groups: category discovery, buyer-intent recommendations, product comparisons, factual entity questions, use-case questions, and support or policy questions.
2. Write prompts that do not assume your brand. Include a separate set that names the brand or product so you can distinguish discovery from confirmation.
3. Run the same prompts before and after migration. Preserve exact wording, model, model version where available, region, language, browsing setting, account state, and date.
4. Capture the complete answer and cited sources. Record whether the site was mentioned, recommended, linked, or cited, and whether the answer stated a relevant fact correctly.
5. Repeat selected prompts. One answer is an observation, not a stable rate. Use repeated runs or scheduled samples to identify a pattern rather than overreacting to one response.

For prompt design, the buyer-intent content guide, ChatGPT citations guide, and AEO checklist provide useful starting points. Keep the prompt set stable during the migration comparison; changing prompts at the same time makes the result uninterpretable.

Account for model-specific and time-dependent changes

A post-migration answer may differ for reasons unrelated to the website. Models update, retrieval indexes refresh at different times, browsing may be enabled or disabled, and answer interfaces can select different sources for the same question. Your report should expose those variables.

• Name the model and interface used, including whether web browsing or search was enabled.
• Record the run date, time zone, country or region, language, and logged-in state where those affect results.
• Keep a control group of prompts about stable, unrelated topics. If those answers change broadly, the model or test environment may have shifted.
• Compare source types. A live citation, a search result, a Common Crawl record, and a model-training claim are different observations.
• Mark uncertainty explicitly. Use labels such as “not observed,” “not tested,” or “inconclusive” instead of converting missing evidence into a failure.

Common Crawl presence can be checked as an open-web coverage signal, but it cannot establish that a current model trained on the page or that the page will appear in a live response. The Common Crawl and training presence guide explains that distinction.

Set regression thresholds and investigate deltas

Do not choose thresholds after looking at the results. Before launch, decide which failures block release, which require investigation, and which are normal variation.

A useful report shows absolute and relative changes, sample counts, and confidence limits where the sample is large enough. For small prompt sets, show the raw runs. “Three of ten prompts changed” is more honest and actionable than a single composite visibility score.

When a delta appears, trace it in order: URL resolution, access rules, rendered HTML, canonical and sitemap signals, entity facts, internal links, and only then prompt behavior. This sequence prevents teams from rewriting content to solve what is actually a redirect or rendering failure.

Turn the migration test into an ongoing monitor

A migration regression test is most valuable when it becomes a repeatable operating process. Keep the pre-migration captures as a reference, then schedule smaller checks after launch.

1. Run a daily or per-deploy technical smoke test for critical URLs, robots.txt, sitemaps, status codes, rendering, and structured data.
2. Run a weekly or biweekly prompt sample with the same models and conditions where possible.
3. Review changes to templates, navigation, product facts, and robots rules as release inputs, not only after a visibility drop.
4. Maintain a changelog connecting deploys to observed changes in crawl access, content, and live answers.
5. Segment results by model, prompt type, product, country, and page template so an aggregate number does not hide a localized regression.

A scan can make the mechanical portion easier to repeat. BatSignal’s features cover checks such as AI crawler and robots access, content readiness, metadata, JSON-LD, sitemap and llms.txt signals, open-web coverage, and sampled AI visibility. You can review the methodology to understand what is measured and what is not.

A practical migration test worksheet

Use the following minimum worksheet for each migration. Store it with the release record so future changes can be compared against the same baseline.

FAQ