How Claude finds web pages: ClaudeBot, Claude-User and Claude-SearchBot
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
Anthropic uses three user agents: ClaudeBot collects public web content that may be used for model training, Claude-User fetches pages when a Claude user asks it to, and Claude-SearchBot crawls to improve the quality of search results Claude shows. To be readable in Claude's answers while opting out of training, allow Claude-User and Claude-SearchBot and disallow ClaudeBot.
Anthropic's bots
| User agent | What Anthropic says it does | Blocking it means |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Your future content isn't used for training |
| Claude-User | Retrieves a page when a user asks Claude a question that needs it | Claude can't read your page for a user |
| Claude-SearchBot | Navigates the web to improve search result quality for users | Your pages may be less visible in Claude's search results |
robots.txt for 'answers yes, training no'
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /Anthropic says its bots honour robots.txt and that blocking by IP address doesn't work reliably because it uses public cloud IPs; use robots.txt instead. Confirm the result with the free AI crawler checker.
What helps Claude cite you
- Plain HTML text: Claude-User fetches the page like a simple client; content that only appears after heavy JavaScript may be missed.
- A clear H1 and a direct first paragraph that answers one question.
- A Markdown version of key pages or an llms.txt, which agentic tools often read; see Markdown twins.
- Stable URLs: answers keep linking to the page they first found.
Worked example
A developer-tools company found Claude described its API limits from a 2024 blog post. The current limits were only in a JavaScript-rendered docs page. Publishing a static /docs/limits page (with an .md version) and linking it from llms.txt gave Claude-User a readable source with today's numbers.
Mistakes that block Claude by accident
- A robots.txt group for 'ClaudeBot' that you assume also covers Claude-User; each user agent is matched separately.
- A blanket 'User-agent: * / Disallow: /' left from a staging site.
- Bot protection that challenges unknown clients, which can stop user-requested fetches.
- Key facts only inside PDFs or images.
- Moving pages without redirects, so links Claude learned earlier break.
Questions
Does blocking ClaudeBot remove me from Claude's answers?
Not by itself. Claude-User and Claude-SearchBot are separate user agents with their own robots.txt groups.
Does Anthropic respect crawl-delay?
Anthropic's support article says its bots honour robots.txt directives, and lists Crawl-delay among the supported extensions.
Where are Anthropic's rules published?
In Anthropic's help centre article on whether Anthropic crawls the web and how site owners can block the crawler.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- Anthropic: how site owners can block the crawler (checked 10 Nov 2026)
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)