AI crawler user agents: the list, what each does and how to allow it
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
The AI crawlers that matter fall into three groups: search crawlers that decide whether you can be cited (OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) and training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent). Google-Extended and Applebot-Extended are not crawlers at all but robots.txt tokens that control training use.
The list (read from each operator's page)
| Token | Operator | Type | What it does |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | Surfaces sites in ChatGPT search answers; blocked sites are not shown there. |
| ChatGPT-User | OpenAI | User fetch | Visits pages for actions a ChatGPT user triggers; OpenAI says robots.txt may not apply. |
| GPTBot | OpenAI | Training | Crawls content that may be used to train OpenAI's models. |
| Claude-SearchBot | Anthropic | Search | Crawls to improve Claude's search results. |
| Claude-User | Anthropic | User fetch | Fetches pages when Claude users ask; honours robots.txt. |
| ClaudeBot | Anthropic | Training | Collects content that could be used for training. |
| PerplexityBot | Perplexity | Search | Surfaces and links sites in Perplexity results; not used for training. |
| Perplexity-User | Perplexity | User fetch | Fetches for a user's question; generally ignores robots.txt. |
| Google-Extended | Control token | No crawler of its own; controls Gemini training and grounding use. No effect on Search. | |
| Applebot-Extended | Apple | Control token | Does not crawl; opts content out of Apple model training. |
| meta-webindexer | Meta | Search | Improves Meta AI search; lets Meta AI cite your pages. |
| meta-externalagent | Meta | Training | Crawls for training AI models and indexing. |
| meta-externalfetcher | Meta | User fetch | Fetches links a user asks about; may bypass robots.txt. |
| CCBot | Common Crawl | Training | Builds the open Common Crawl archive many models are trained on. |
How to use the list
If you want to be cited in AI answers, the search crawlers are the ones that must be allowed. Training crawlers are a business choice: allowing them may help models learn your brand over time, blocking them keeps your content out of future training sets.
User-triggered fetchers act for a person who asked about your page. Several operators say robots.txt may not apply to them, so a robots.txt block is not a reliable way to stop them; use authentication for content that must stay private.
Example: allow everything except training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) have no group of their own here, so they follow the * group and may read everything. Check the result with the AI crawler checker.
Questions
Does blocking GPTBot remove me from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot, which is controlled separately. OpenAI says each setting is independent.
How fast does OpenAI pick up robots.txt changes?
OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update.
Is user-agent matching case-sensitive?
No. RFC 9309 says crawlers must match the product token case-insensitively.
Do new AI crawlers appear often?
Yes. Re-check the operators' pages a few times a year; this list was read on 10 Nov 2026.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- OpenAI: crawlers and user agents (checked 10 Nov 2026)
- Anthropic: how site owners can block the crawler (checked 10 Nov 2026)
- Perplexity: crawlers (checked 10 Nov 2026)
- Google: common crawlers (Google-Extended) (checked 10 Nov 2026)
- Apple: About Applebot (checked 10 Nov 2026)
- Meta: web crawlers (checked 10 Nov 2026)
- Common Crawl: CCBot (checked 10 Nov 2026)
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)