AI crawler user agents: the list, what each does and how to allow it

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

The AI crawlers that matter fall into three groups: search crawlers that decide whether you can be cited (OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) and training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent). Google-Extended and Applebot-Extended are not crawlers at all but robots.txt tokens that control training use.

The list (read from each operator's page)

TokenOperatorTypeWhat it does
OAI-SearchBotOpenAISearchSurfaces sites in ChatGPT search answers; blocked sites are not shown there.
ChatGPT-UserOpenAIUser fetchVisits pages for actions a ChatGPT user triggers; OpenAI says robots.txt may not apply.
GPTBotOpenAITrainingCrawls content that may be used to train OpenAI's models.
Claude-SearchBotAnthropicSearchCrawls to improve Claude's search results.
Claude-UserAnthropicUser fetchFetches pages when Claude users ask; honours robots.txt.
ClaudeBotAnthropicTrainingCollects content that could be used for training.
PerplexityBotPerplexitySearchSurfaces and links sites in Perplexity results; not used for training.
Perplexity-UserPerplexityUser fetchFetches for a user's question; generally ignores robots.txt.
Google-ExtendedGoogleControl tokenNo crawler of its own; controls Gemini training and grounding use. No effect on Search.
Applebot-ExtendedAppleControl tokenDoes not crawl; opts content out of Apple model training.
meta-webindexerMetaSearchImproves Meta AI search; lets Meta AI cite your pages.
meta-externalagentMetaTrainingCrawls for training AI models and indexing.
meta-externalfetcherMetaUser fetchFetches links a user asks about; may bypass robots.txt.
CCBotCommon CrawlTrainingBuilds the open Common Crawl archive many models are trained on.

How to use the list

If you want to be cited in AI answers, the search crawlers are the ones that must be allowed. Training crawlers are a business choice: allowing them may help models learn your brand over time, blocking them keeps your content out of future training sets.

User-triggered fetchers act for a person who asked about your page. Several operators say robots.txt may not apply to them, so a robots.txt block is not a reliable way to stop them; use authentication for content that must stay private.

Example: allow everything except training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) have no group of their own here, so they follow the * group and may read everything. Check the result with the AI crawler checker.

Questions

Does blocking GPTBot remove me from ChatGPT search?

No. ChatGPT search uses OAI-SearchBot, which is controlled separately. OpenAI says each setting is independent.

How fast does OpenAI pick up robots.txt changes?

OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update.

Is user-agent matching case-sensitive?

No. RFC 9309 says crawlers must match the product token case-insensitively.

Do new AI crawlers appear often?

Yes. Re-check the operators' pages a few times a year; this list was read on 10 Nov 2026.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

Sources

Related

AI crawler user agents list 2026: GPTBot, ClaudeBot & more