Claude: three crawlers, three different effects.
Anthropic documents three user agents and says what blocking each one does. Two of them concern whether a site can be found when Claude searches; the third concerns training.
Appearing in Claude means being retrieved and used when Claude searches the web for a user. Anthropic documents three user agents: one for training, one for user-directed retrieval and one for search quality, and states the effect of blocking each.
Documentation last checked: October 4, 2026. It is re-checked every quarter, and each check is entered in the change log. These pages can change; the statements below are quoted or paraphrased from the sources linked, as they read on that date.
What is documented
| Statement | Source |
|---|---|
| ClaudeBot collects web content "that could potentially contribute to their training"; restricting it signals that future materials should be excluded from training datasets | Anthropic, web crawling |
| Claude-User: "When individuals ask questions to Claude, it may access websites using a Claude-User agent." | Anthropic, web crawling |
| Disabling Claude-User "may reduce your site's visibility for user-directed web search" | Anthropic, web crawling |
| Claude-SearchBot "navigates the web to improve search result quality for users" | Anthropic, web crawling |
| Disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" | Anthropic, web crawling |
| Anthropic's bots honor robots.txt directives | Anthropic, web crawling |
What the cited sources do not cover
- How sources are chosen for a given answer.
- Any way to request or guarantee a mention.
- A report of appearances for site owners.
Where the documentation is silent, this page says so and does not fill the gap.
Access and controls
The three agents are set separately in robots.txt: ClaudeBot, Claude-User, Claude-SearchBot. Blocking the first is a statement about training. Blocking either of the other two is, in Anthropic's words, something that may reduce visibility. Anthropic's page was last updated on April 7, 2026.
A test protocol for Claude
This is a procedure to follow, not a result. No test of Claude is reported on this page.
- Read robots.txt for the three user agents, one by one.
- Decide each on purpose. Training and search visibility are separate decisions.
- Write entity, category and comparison prompts.
- Run each several times, note for each run whether sources are shown, and keep the answer and those sources.
- Note the answers given without sources separately: they are not evidence of retrieval.
- Repeat at a fixed interval.
What to record for each run is listed in what is recorded; how many prompts a reading needs is in how many prompts make a reading. The rules a measurement must meet before it is published on this site are in the research methodology.
What to do with the result
- Claude-SearchBot or Claude-User is blocked by a general rule Decide whether that was intended.
- Answers about you show no sources They may not rest on retrieval; record them as such.
- The description is wrong See inaccurate or outdated AI answers.
- Nothing independent is cited See weak independent sources.
No one can guarantee a mention, a citation or a position in Claude. What can be worked on is access, the clarity of the entity, the evidence on the page and the independent sources. The method is The Authority Architecture System.
Frequently asked questions
If I block ClaudeBot, do I disappear from Claude?
Anthropic describes ClaudeBot as collecting content that could contribute to training. The agents it associates with search visibility are Claude-User and Claude-SearchBot, which are set separately.
Does Anthropic respect robots.txt?
Anthropic states that its bots honor industry-standard directives in robots.txt.
Can I ask to be included in Claude's answers?
The documentation cited here describes crawlers and how to block them. It does not describe a way to request inclusion.
Other platforms
- ChatGPT OAI-SearchBot, the referral parameter, and what OpenAI does not publish.
- Google AI Overviews Eligibility through Google Search, and the snippet controls.
- Google AI Mode Follow-up questions and query fan-out, with the same eligibility as Search.
- Gemini Google-Extended, a control separate from those of Google Search.
- Perplexity PerplexityBot, Perplexity-User and the recommendation to allow the first.
How the engines differ, and why each is measured separately, is set out in AI visibility. Related analyses are collected in Insights.
Understand how search and AI currently interpret your authority.
Request a confidential assessment of your visibility, entity signals, narrative consistency and source authority.
Request a Private AssessmentSelective engagements for leaders and organizations with material reputational stakes.