---
title: "AI crawler user agents list 2026: GPTBot, ClaudeBot & more"
canonical: https://brandinaianswers.com/learn/ai-crawler-user-agents-list
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# AI crawler user agents list 2026: GPTBot, ClaudeBot & more

> Every major AI crawler and fetcher (OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Common Crawl), what each is for and the robots.txt token to use.

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

The AI crawlers that matter fall into three groups: search crawlers that decide whether you can be cited (OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) and training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent). Google-Extended and Applebot-Extended are not crawlers at all but robots.txt tokens that control training use.

## The list (read from each operator's page)

| Token | Operator | Type | What it does |

| --- | --- | --- | --- |

| OAI-SearchBot | OpenAI | Search | Surfaces sites in ChatGPT search answers; blocked sites are not shown there. |

| ChatGPT-User | OpenAI | User fetch | Visits pages for actions a ChatGPT user triggers; OpenAI says robots.txt may not apply. |

| GPTBot | OpenAI | Training | Crawls content that may be used to train OpenAI's models. |

| Claude-SearchBot | Anthropic | Search | Crawls to improve Claude's search results. |

| Claude-User | Anthropic | User fetch | Fetches pages when Claude users ask; honours robots.txt. |

| ClaudeBot | Anthropic | Training | Collects content that could be used for training. |

| PerplexityBot | Perplexity | Search | Surfaces and links sites in Perplexity results; not used for training. |

| Perplexity-User | Perplexity | User fetch | Fetches for a user's question; generally ignores robots.txt. |

| Google-Extended | Google | Control token | No crawler of its own; controls Gemini training and grounding use. No effect on Search. |

| Applebot-Extended | Apple | Control token | Does not crawl; opts content out of Apple model training. |

| meta-webindexer | Meta | Search | Improves Meta AI search; lets Meta AI cite your pages. |

| meta-externalagent | Meta | Training | Crawls for training AI models and indexing. |

| meta-externalfetcher | Meta | User fetch | Fetches links a user asks about; may bypass robots.txt. |

| CCBot | Common Crawl | Training | Builds the open Common Crawl archive many models are trained on. |

## How to use the list

If you want to be cited in AI answers, the search crawlers are the ones that must be allowed. Training crawlers are a business choice: allowing them may help models learn your brand over time, blocking them keeps your content out of future training sets.

User-triggered fetchers act for a person who asked about your page. Several operators say robots.txt may not apply to them, so a robots.txt block is not a reliable way to stop them; use authentication for content that must stay private.

## Example: allow everything except training

```

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: CCBot

Disallow: /

User-agent: meta-externalagent

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: Applebot-Extended

Disallow: /

User-agent: *

Allow: /

```

Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) have no group of their own here, so they follow the * group and may read everything. Check the result with the [AI crawler checker](https://brandinaianswers.com/tools/ai-crawler-checker).

## Questions

### Does blocking GPTBot remove me from ChatGPT search?

No. ChatGPT search uses OAI-SearchBot, which is controlled separately. OpenAI says each setting is independent.

### How fast does OpenAI pick up robots.txt changes?

OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update.

### Is user-agent matching case-sensitive?

No. RFC 9309 says crawlers must match the product token case-insensitively.

### Do new AI crawlers appear often?

Yes. Re-check the operators' pages a few times a year; this list was read on 10 Nov 2026.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

[Run the free audit](https://brandinaianswers.com/audit)[Check AI crawlers](https://brandinaianswers.com/tools/ai-crawler-checker)

## Sources

- [OpenAI: crawlers and user agents](https://developers.openai.com/api/docs/bots) (checked 10 Nov 2026)
- [Anthropic: how site owners can block the crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (checked 10 Nov 2026)
- [Perplexity: crawlers](https://docs.perplexity.ai/guides/bots) (checked 10 Nov 2026)
- [Google: common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (checked 10 Nov 2026)
- [Apple: About Applebot](https://support.apple.com/en-us/119829) (checked 10 Nov 2026)
- [Meta: web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) (checked 10 Nov 2026)
- [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (checked 10 Nov 2026)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (checked 10 Nov 2026)

## Related

- [GPTBot vs OAI-SearchBot vs ChatGPT-User: which one to allow](https://brandinaianswers.com/learn/gptbot-vs-oai-searchbot-vs-chatgpt-user)
- [How to allow AI search but block AI training in robots.txt](https://brandinaianswers.com/learn/allow-ai-search-block-ai-training)
- [Which AI bots ignore robots.txt, and what to do about it](https://brandinaianswers.com/learn/ai-bots-that-ignore-robots-txt)
- [All guides](https://brandinaianswers.com/learn)
