Each engine has its own rules. Most of them are short.
These pages report what each provider publishes about how a site can be included, and say plainly where the documentation stops. None of them reports a test result.
Each page below covers one AI search surface: what its provider documents, what it does not, which controls a site has, and a test protocol. Documentation was last checked on October 4, 2026, and is re-checked every quarter. The pages quote providers; they do not describe how any engine selects sources, because none of the sources cited here describes it.
- Google AI Overviews Eligibility through Google Search, and the snippet controls.
- Google AI Mode Follow-up questions and query fan-out, with the same eligibility as Search.
- Gemini Google-Extended, a control separate from those of Google Search.
Assistants
- ChatGPT OAI-SearchBot, the referral parameter, and what OpenAI does not publish.
- Perplexity PerplexityBot, Perplexity-User and the recommendation to allow the first.
- Claude Three Anthropic crawlers with three different effects.
The controls, side by side
| Surface | Control for search or answers | Control for training | Source checked |
|---|---|---|---|
| ChatGPT | OAI-SearchBot in robots.txt |
GPTBot in robots.txt |
OpenAI |
| Google AI Overviews and AI Mode | Snippet controls and noindex |
Google-Extended (see Gemini) |
Google Search Central |
| Gemini | Google-Extended (grounding) |
Google-Extended (training) |
Google Search Central |
| Perplexity | PerplexityBot in robots.txt, and IP ranges |
PerplexityBot is stated not to crawl content for AI foundation models | Perplexity |
| Claude | Claude-SearchBot and Claude-User |
ClaudeBot |
Anthropic |
A rule that blocks every AI user agent treats these cases as one. They are not.
Microsoft Copilot has no page here yet: the only documentation that could be read is a 2023 post about Bing Chat, which is not enough to describe Copilot today. Copilot remains one of the surfaces observed in an assessment and in monthly intelligence: what is recorded there is what Copilot answers, with its date and conditions, not how Microsoft selects its sources.
How to use these pages
Start with access: it is what the sources cited here cover. Then follow the test protocol on the page concerned, and record what you observe as described in AI search intelligence. The disciplines behind the work are set out in AI visibility.
Understand how search and AI currently interpret your authority.
Request a confidential assessment of your visibility, entity signals, narrative consistency and source authority.
Request a Private AssessmentSelective engagements for leaders and organizations with material reputational stakes.