# Claude: three crawlers, three different effects.
> How site content relates to Claude, according to Anthropic: ClaudeBot, Claude-User and Claude-SearchBot, what blocking each one does, and robots.txt.
- Page: https://mohammedteto.com/platforms/claude-visibility/
- Language: en — French version: https://mohammedteto.com/fr/visibilite-ia/
- Author: Mohammed Teto (https://mohammedteto.com/#person)
- Published: 2026-10-07
- Updated: 2026-10-04
Anthropic documents three user agents and says what blocking each one does. Two of them concern whether a site can be found when Claude searches; the third concerns training.
Appearing in Claude means being retrieved and used when Claude searches the web for a user. Anthropic documents three user agents: one for training, one for user-directed retrieval and one for search quality, and states the effect of blocking each.
Documentation last checked: October 4, 2026. It is re-checked every quarter, and each check is entered in the [change log](https://mohammedteto.com/research/change-log/). These pages can change; the statements below are quoted or paraphrased from the sources linked, as they read on that date.
## What is documented
| Statement | Source |
|---|---|
| ClaudeBot collects web content "that could potentially contribute to their training"; restricting it signals that future materials should be excluded from training datasets | [Anthropic, web crawling](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-User: "When individuals ask questions to Claude, it may access websites using a Claude-User agent." | [Anthropic, web crawling](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Disabling Claude-User "may reduce your site's visibility for user-directed web search" | [Anthropic, web crawling](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-SearchBot "navigates the web to improve search result quality for users" | [Anthropic, web crawling](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" | [Anthropic, web crawling](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Anthropic's bots honor robots.txt directives | [Anthropic, web crawling](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
## What the cited sources do not cover
- How sources are chosen for a given answer.
- Any way to request or guarantee a mention.
- A report of appearances for site owners.
Where the documentation is silent, this page says so and does not fill the gap.
## Access and controls
The three agents are set separately in robots.txt: `ClaudeBot`, `Claude-User`, `Claude-SearchBot`. Blocking the first is a statement about training. Blocking either of the other two is, in Anthropic's words, something that may reduce visibility. Anthropic's page was last updated on April 7, 2026.
## A test protocol for Claude
This is a procedure to follow, not a result. No test of Claude is reported on this page.
1. **Read robots.txt** for the three user agents, one by one.
2. **Decide each on purpose.** Training and search visibility are separate decisions.
3. **Write entity, category and comparison prompts.**
4. **Run each several times**, note for each run whether sources are shown, and keep the answer and those sources.
5. **Note the answers given without sources** separately: they are not evidence of retrieval.
6. **Repeat at a fixed interval.**
What to record for each run is listed in [what is recorded](https://mohammedteto.com/ai-search-intelligence/#record); how many prompts a reading needs is in [how many prompts make a reading](https://mohammedteto.com/ai-search-intelligence/#sample). The rules a measurement must meet before it is published on this site are in the [research methodology](https://mohammedteto.com/research/methodology/).
## What to do with the result
- **Claude-SearchBot or Claude-User is blocked by a general rule** Decide whether that was intended.
- **Answers about you show no sources** They may not rest on retrieval; record them as such.
- **The description is wrong** See [inaccurate or outdated AI answers](https://mohammedteto.com/problems/inaccurate-ai-results/).
- **Nothing independent is cited** See [weak independent sources](https://mohammedteto.com/problems/weak-source-authority/).
No one can guarantee a mention, a citation or a position in Claude. What can be worked on is access, the clarity of the entity, the evidence on the page and the independent sources. The method is [The Authority Architecture System](https://mohammedteto.com/method/).
## Frequently asked questions
### If I block ClaudeBot, do I disappear from Claude?
Anthropic describes ClaudeBot as collecting content that could contribute to training. The agents it associates with search visibility are Claude-User and Claude-SearchBot, which are set separately.
### Does Anthropic respect robots.txt?
Anthropic states that its bots honor industry-standard directives in robots.txt.
### Can I ask to be included in Claude's answers?
The documentation cited here describes crawlers and how to block them. It does not describe a way to request inclusion.
## Other platforms
- [ChatGPT](https://mohammedteto.com/platforms/chatgpt-visibility/) OAI-SearchBot, the referral parameter, and what OpenAI does not publish.
- [Google AI Overviews](https://mohammedteto.com/platforms/google-ai-overviews/) Eligibility through Google Search, and the snippet controls.
- [Google AI Mode](https://mohammedteto.com/platforms/google-ai-mode/) Follow-up questions and query fan-out, with the same eligibility as Search.
- [Gemini](https://mohammedteto.com/platforms/gemini-visibility/) Google-Extended, a control separate from those of Google Search.
- [Perplexity](https://mohammedteto.com/platforms/perplexity-citations/) PerplexityBot, Perplexity-User and the recommendation to allow the first.
How the engines differ, and why each is measured separately, is set out in [AI visibility](https://mohammedteto.com/ai-visibility/). Related analyses are collected in [Insights](https://mohammedteto.com/insights/).
---
Understand how search and AI currently interpret your authority. Request a confidential assessment of your visibility, entity signals, narrative consistency and source authority.
[Request a Private Assessment](https://mohammedteto.com/private-assessment/)