---
title: "Which AI bots ignore robots.txt (user-triggered fetchers)"
canonical: https://brandinaianswers.com/learn/ai-bots-that-ignore-robots-txt
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# Which AI bots ignore robots.txt (user-triggered fetchers)

> User-triggered AI fetchers like ChatGPT-User, Perplexity-User may not follow robots.txt. What each operator says and how to protect content.

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

The bots that may not follow robots.txt are the user-triggered fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says meta-externalfetcher may bypass them. Their logic is that a person asked for the page; Anthropic says its Claude-User does honour robots.txt.

## What each operator says (read 10 Nov 2026)

| Bot | Follows robots.txt? |

| --- | --- |

| ChatGPT-User (OpenAI) | May not, because users initiate the actions |

| Perplexity-User | Generally ignores robots.txt because a user requested the fetch |

| meta-externalfetcher (Meta) | May bypass robots.txt because fetches are user-requested |

| facebookexternalhit (Meta) | May bypass for security or integrity checks |

| Claude-User (Anthropic) | Anthropic says its bots honour robots.txt |

| Search and training crawlers (OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot) | Yes, per their operators |

## What this means in practice

- robots.txt is a crawling policy, not access control. Anything public can be fetched when a user pastes the URL into an assistant.
- Private or paid content must be behind authentication or a paywall that serves no content to anonymous requests.
- For public marketing pages this is good news: buyers who ask an assistant about your pricing page get the real page.

## Protecting content that must not be read

```

# Not enough on its own:

User-agent: ChatGPT-User

Disallow: /members/

# What actually works: /members/ returns 401 or a login page

# to any request without a valid session, whoever the user agent is.

```

Check what an anonymous request gets with curl. If it receives the content, so can any fetcher.

## A checklist for content you want to keep out

Confirm the content needs a login: open it in a private browser window; if it loads, any bot can read it.

Return 401 or 403, or a login page, for requests without a session; do not rely on hidden links or JavaScript checks.

Remove the URLs from your sitemap and from llms.txt.

Keep robots.txt rules as a signal for well-behaved crawlers, but treat authentication as the real control.

## Why operators treat user fetches differently

A crawler decides by itself which pages to collect, so robots.txt is its permission slip. A user-triggered fetch happens because a person pasted a link or asked about a page, much like a browser visit, so several operators treat it as acting for that person.

Whatever you think of that reasoning, plan for it: public pages are readable by anyone with the URL, through any tool.

## Questions

### Is ignoring robots.txt allowed?

robots.txt is a voluntary protocol. Operators publish how their bots behave; legal questions about scraping depend on the country and the use.

### Should I block user-fetch bots at the firewall?

Only for content that must stay private. For public pages, blocking them means assistants cannot read your page when buyers ask about it.

### How do I see these visits?

Filter your logs for their user agents; see [AI crawlers in server logs](https://brandinaianswers.com/learn/ai-crawlers-in-server-logs).

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

[Run the free audit](https://brandinaianswers.com/audit)[Check AI crawlers](https://brandinaianswers.com/tools/ai-crawler-checker)

## Sources

- [OpenAI: crawlers and user agents](https://developers.openai.com/api/docs/bots) (checked 10 Nov 2026)
- [Perplexity: crawlers](https://docs.perplexity.ai/guides/bots) (checked 10 Nov 2026)
- [Meta: web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) (checked 10 Nov 2026)
- [Anthropic: how site owners can block the crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (checked 10 Nov 2026)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (checked 10 Nov 2026)

## Related

- [AI crawler user agents: the list, what each does and how to allow it](https://brandinaianswers.com/learn/ai-crawler-user-agents-list)
- [GPTBot vs OAI-SearchBot vs ChatGPT-User: which one to allow](https://brandinaianswers.com/learn/gptbot-vs-oai-searchbot-vs-chatgpt-user)
- [How to see AI crawler visits in your server logs](https://brandinaianswers.com/learn/ai-crawlers-in-server-logs)
- [All guides](https://brandinaianswers.com/learn)
