Which AI bots ignore robots.txt, and what to do about it
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
The bots that may not follow robots.txt are the user-triggered fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says meta-externalfetcher may bypass them. Their logic is that a person asked for the page; Anthropic says its Claude-User does honour robots.txt.
What each operator says (read 10 Nov 2026)
| Bot | Follows robots.txt? |
|---|---|
| ChatGPT-User (OpenAI) | May not, because users initiate the actions |
| Perplexity-User | Generally ignores robots.txt because a user requested the fetch |
| meta-externalfetcher (Meta) | May bypass robots.txt because fetches are user-requested |
| facebookexternalhit (Meta) | May bypass for security or integrity checks |
| Claude-User (Anthropic) | Anthropic says its bots honour robots.txt |
| Search and training crawlers (OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot) | Yes, per their operators |
What this means in practice
- robots.txt is a crawling policy, not access control. Anything public can be fetched when a user pastes the URL into an assistant.
- Private or paid content must be behind authentication or a paywall that serves no content to anonymous requests.
- For public marketing pages this is good news: buyers who ask an assistant about your pricing page get the real page.
Protecting content that must not be read
# Not enough on its own:
User-agent: ChatGPT-User
Disallow: /members/
# What actually works: /members/ returns 401 or a login page
# to any request without a valid session, whoever the user agent is.Check what an anonymous request gets with curl. If it receives the content, so can any fetcher.
A checklist for content you want to keep out
Confirm the content needs a login: open it in a private browser window; if it loads, any bot can read it.
Return 401 or 403, or a login page, for requests without a session; do not rely on hidden links or JavaScript checks.
Remove the URLs from your sitemap and from llms.txt.
Keep robots.txt rules as a signal for well-behaved crawlers, but treat authentication as the real control.
Why operators treat user fetches differently
A crawler decides by itself which pages to collect, so robots.txt is its permission slip. A user-triggered fetch happens because a person pasted a link or asked about a page, much like a browser visit, so several operators treat it as acting for that person.
Whatever you think of that reasoning, plan for it: public pages are readable by anyone with the URL, through any tool.
Questions
Is ignoring robots.txt allowed?
robots.txt is a voluntary protocol. Operators publish how their bots behave; legal questions about scraping depend on the country and the use.
Should I block user-fetch bots at the firewall?
Only for content that must stay private. For public pages, blocking them means assistants cannot read your page when buyers ask about it.
How do I see these visits?
Filter your logs for their user agents; see AI crawlers in server logs.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- OpenAI: crawlers and user agents (checked 10 Nov 2026)
- Perplexity: crawlers (checked 10 Nov 2026)
- Meta: web crawlers (checked 10 Nov 2026)
- Anthropic: how site owners can block the crawler (checked 10 Nov 2026)
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)