How to allow AI search but block AI training in robots.txt
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
Block the training tokens (GPTBot, ClaudeBot, CCBot, meta-externalagent, Google-Extended, Applebot-Extended) and leave the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) allowed. Each operator documents these as independent controls, so blocking training does not remove you from their search answers.
The recipe
# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including AI search crawlers
User-agent: *
Disallow: /admin
Disallow: /cart
Sitemap: https://example.com/sitemap.xmlSeveral User-agent lines in a row form one group, so the six training tokens share one Disallow rule. Search crawlers have no group of their own, so they follow the * group.
Why each line is there
| Token | Blocking it means | Search effect |
|---|---|---|
| GPTBot | Content not used to train OpenAI models | None on ChatGPT search (OAI-SearchBot is separate) |
| ClaudeBot | Future content excluded from Anthropic training | None on Claude search (Claude-SearchBot is separate) |
| Google-Extended | Not used for Gemini training or grounding | Google says no effect on Google Search inclusion or ranking |
| Applebot-Extended | Not used for Apple model training | Apple says pages can still appear in search results |
| meta-externalagent | Not crawled for Meta model training | meta-webindexer (Meta AI search) is separate |
| CCBot | Not in the Common Crawl archive | None; Common Crawl is not a search engine |
Traps that undo it
- A specific group replaces the * group for that bot. If you add "User-agent: OAI-SearchBot" with only "Disallow: /admin", it will not inherit other * rules, which is fine; but if you write "Disallow: /" there by mistake, ChatGPT search loses you.
- Google-Extended does not stop Google AI Overviews: they are part of Search and follow Googlebot and snippet controls.
- Grounding is not search: blocking Google-Extended also stops Gemini Apps grounding on your content.
- A CDN or firewall bot rule can block search crawlers even when robots.txt allows them; check both.
Before and after: what changes for each service
Before the change, a site with no AI rules lets every crawler read everything: ChatGPT search, Claude, Perplexity and Google can cite it, and OpenAI, Anthropic, Meta, Apple, Google (for Gemini) and Common Crawl can use it for training.
After the change, the same search services can still read and cite every public page. What stops is new collection for training by the six listed tokens. Content already collected is not removed by robots.txt; each operator describes the rule as applying going forward.
Test the result twice: once right after publishing (the file is valid and returns HTTP 200), and again a week later in your server logs, where GPTBot and ClaudeBot requests for pages other than robots.txt should stop while OAI-SearchBot and PerplexityBot keep coming.
Questions
Will blocking training hurt my brand in AI answers?
Answers that search the web cite pages from their search index, which this recipe keeps open. Models learn less about you from training data, which can matter for answers given without search.
Should small businesses block training at all?
It is a business choice. Many small brands allow training so models learn their name; publishers protecting paid content often block it.
How do I check the result?
Run the AI crawler checker; it shows each crawler's verdict and the exact rule line that decided it.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- OpenAI: crawlers and user agents (checked 10 Nov 2026)
- Anthropic: how site owners can block the crawler (checked 10 Nov 2026)
- Google: common crawlers (Google-Extended) (checked 10 Nov 2026)
- Apple: About Applebot (checked 10 Nov 2026)
- Meta: web crawlers (checked 10 Nov 2026)
- Common Crawl: CCBot (checked 10 Nov 2026)
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)