Check the technical foundations of AI search visibility.

A robots.txt rule governs crawler access; it is not an authentication system or a reliable way to remove a URL from Google’s index. A blocked URL can still be known through links. Sensitive content belongs behind access controls. See Google’s robots.txt introduction.

For a public page that should be excluded from Google Search, review the applicable noindex documentation. Google must be able to crawl the page to see the directive. A Disallow combined with noindex is not a shortcut to having that instruction processed.

• Inspect both response headers and the HTML robots meta tag. Record the actual URL and response, including redirects; a directive on a different page is not the same evidence.
• Confirm that production pages intended for discovery do not retain staging noindex directives or a login challenge. Preserve intentional exclusions instead of removing them site-wide.
• Inspect CDN, firewall and security-plugin logs for legitimate requests that are challenged or denied. Record what was verified; an invented crawler user agent in a manual request does not prove a genuine crawler can enter.

Treat robots, indexing, preview and training decisions as separate policy choices. A page can be publicly reachable while intentionally excluded from one search system. A crawler visit alone is not evidence that the page entered training data.

Link important public pages from relevant navigation and related articles using ordinary crawlable links. Keep the page reachable through the actual site architecture. There is no universal three-click rule asserted by this checklist.

Use a sitemap to describe the preferred URLs and genuine modification dates. Google’s sitemap guidance describes discovery support, not an indexing guarantee. Remove retired destinations from the current list when a migration is complete.

• Check that sitemap URLs resolve to the intended pages and agree with their canonical signals. Investigate broken URLs, accidental variants and redirect chains.
• Keep service, author, entity and evidence pages connected where the relationship is useful. An isolated page in a sitemap is not a substitute for understandable navigation.
• For real translated equivalents, verify that each language version exists, is accessible and has appropriate language annotations. Do not invent an alternate version or a hreflang relationship to an unrelated homepage.

Separate discovery checks from answer tests. Search Console information, server logs and an observed AI response answer different questions; retain their dates and scope.

Download the blank technical-check register. It contains column names only: no audit observations, scores or customer data. Populate it with authorized checks and retain the supporting evidence securely.

• Record page, provider context, check date, method, observed status and evidence location. Use “not checked” when a source of evidence is unavailable.
• Assign the issue, intended fix and owner. After a change, repeat the relevant access, rendering or indexing check and record the result separately from the original observation.
• Then test the audience question where appropriate. A technical fix followed by a different AI answer is not sufficient to prove causation. Use the prompt-set workflow to preserve the test context.
• Report what changed, what remains unknown and what needs another review. Keep technical checks separate from mentions, cited sources, visits and qualified enquiries in the reputation dashboard.

Illustrative example: a public service page returns a firewall challenge to a verified search crawler. The team corrects that unintended rule and records a successful fetch. The conclusion is restored access, not proven indexing or a guaranteed citation. This is a fictional scenario, not a client result.

Documentation checked on 29 October 2026. Recheck the relevant provider guidance before applying a policy change. For provider differences, read the AI search engines comparison.

What is the first technical check for AI search visibility?

Define the public page and the product context, then verify whether the relevant crawler can fetch the intended content. Inspect the documented access policy, actual response and verified logs when available. A successful fetch establishes access, not indexing, retrieval or a citation.

Must GPTBot be allowed for a site to appear in ChatGPT Search?

No. OpenAI documents OAI-SearchBot for search and GPTBot for potential model training as independent controls. Choose the appropriate policy for each purpose. Allowing the search crawler does not guarantee selection, and blocking training is not the same as blocking search.

Can robots.txt remove a page from Google Search?

A robots.txt Disallow is a crawling control, not a reliable removal instruction. Google can know a blocked URL through links. For a public page intended to be excluded from Search, review the applicable noindex guidance and allow Google to see that instruction. Protect confidential content with access controls.

Must all important content work without JavaScript?

Inspect both the returned HTML and the rendered page. Google can render JavaScript with documented constraints, while this checklist establishes no universal rendering behavior for AI crawlers. Making essential public answers and links reliably readable is a practical implementation goal, not a guarantee of inclusion.

Do structured data, Markdown or llms.txt guarantee AI citations?

No. Structured data should match visible facts; Markdown should repeat the corresponding page accurately; llms.txt can provide navigation. These files do not establish selection or a citation. Google’s AI Search guidance does not require a special AI file or special schema for its features.

How should a team prove that a technical issue was fixed?

Retain the original observation, the change and a dated retest using the relevant evidence source. A corrected status, readable response or processed directive proves only the corresponding check. Record indexing and AI-answer observations separately and declare anything that has not been verified.