How Claude finds web pages: ClaudeBot, Claude-User and Claude-SearchBot

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

Anthropic uses three user agents: ClaudeBot collects public web content that may be used for model training, Claude-User fetches pages when a Claude user asks it to, and Claude-SearchBot crawls to improve the quality of search results Claude shows. To be readable in Claude's answers while opting out of training, allow Claude-User and Claude-SearchBot and disallow ClaudeBot.

Anthropic's bots

User agentWhat Anthropic says it doesBlocking it means
ClaudeBotCollects web content that could contribute to model trainingYour future content isn't used for training
Claude-UserRetrieves a page when a user asks Claude a question that needs itClaude can't read your page for a user
Claude-SearchBotNavigates the web to improve search result quality for usersYour pages may be less visible in Claude's search results

robots.txt for 'answers yes, training no'

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Anthropic says its bots honour robots.txt and that blocking by IP address doesn't work reliably because it uses public cloud IPs; use robots.txt instead. Confirm the result with the free AI crawler checker.

What helps Claude cite you

  • Plain HTML text: Claude-User fetches the page like a simple client; content that only appears after heavy JavaScript may be missed.
  • A clear H1 and a direct first paragraph that answers one question.
  • A Markdown version of key pages or an llms.txt, which agentic tools often read; see Markdown twins.
  • Stable URLs: answers keep linking to the page they first found.

Worked example

A developer-tools company found Claude described its API limits from a 2024 blog post. The current limits were only in a JavaScript-rendered docs page. Publishing a static /docs/limits page (with an .md version) and linking it from llms.txt gave Claude-User a readable source with today's numbers.

Mistakes that block Claude by accident

  • A robots.txt group for 'ClaudeBot' that you assume also covers Claude-User; each user agent is matched separately.
  • A blanket 'User-agent: * / Disallow: /' left from a staging site.
  • Bot protection that challenges unknown clients, which can stop user-requested fetches.
  • Key facts only inside PDFs or images.
  • Moving pages without redirects, so links Claude learned earlier break.

Questions

Does blocking ClaudeBot remove me from Claude's answers?

Not by itself. Claude-User and Claude-SearchBot are separate user agents with their own robots.txt groups.

Does Anthropic respect crawl-delay?

Anthropic's support article says its bots honour robots.txt directives, and lists Crawl-delay among the supported extensions.

Where are Anthropic's rules published?

In Anthropic's help centre article on whether Anthropic crawls the web and how site owners can block the crawler.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

Sources

Related

How Claude web search finds pages: Anthropic's bots