---
title: "How AI crawlers read robots.txt (RFC 9309 rules)"
canonical: https://brandinaianswers.com/learn/robots-txt-rules-for-ai-crawlers
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# How AI crawlers read robots.txt (RFC 9309 rules)

> RFC 9309 rules every AI crawler follows: group matching, longest-match wins, allow wins ties, 5xx means disallow all. With worked examples and common mistakes.

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

Crawlers that follow RFC 9309 use only the group whose User-agent matches their token (falling back to *), pick the longest matching path rule, prefer Allow on a tie and treat a 5xx robots.txt as 'disallow everything'. Most AI-visibility accidents come from a specific group that drops rules people assumed were inherited from *.

## The rules that matter

| Rule | What RFC 9309 says |

| --- | --- |

| Matching | Crawlers must match the product token case-insensitively; if several groups match, their rules are combined; if none match, the * group applies |

| Longest match | The most specific (longest) matching rule must be used |

| Ties | If an Allow and a Disallow are equivalent, Allow should be used |

| Wildcards | * matches any characters; $ marks the end of the path |

| 4xx | robots.txt is 'unavailable'; the crawler may access anything |

| 5xx or unreachable | The crawler must assume complete disallow |

| Size | Crawlers must parse at least 500 KiB |

| Caching | Crawlers should not use a cached copy for more than 24 hours unless robots.txt is unreachable |

## Worked example: the inherited-rule trap

```

User-agent: *

Disallow: /checkout

Disallow: /search

User-agent: PerplexityBot

Disallow: /internal

```

PerplexityBot matches its own group, so it ignores the * group completely: it may crawl /checkout and /search, and only /internal is blocked for it. Copy every rule you need into specific groups.

## Worked example: longest match

```

User-agent: *

Disallow: /blog

Allow: /blog/guides

```

/blog/guides/llms-txt is allowed (the Allow rule is longer); /blog/news is disallowed.

## Mistakes we see most

- "Disallow: /" left over from a staging site.
- robots.txt served by a firewall challenge page or with a 503 during deploys, which crawlers read as 'disallow all'.
- Blocking by IP or firewall instead of robots.txt, which also stops crawlers reading the rules.
- Typos in tokens: "GPT-Bot" or "Perplexity Bot" match nothing.

## A safe template to start from

Start with one * group that holds every rule all crawlers should follow (admin, cart, search result pages), then add specific groups only for crawlers you treat differently, and copy the shared rules into each of those groups. End with a Sitemap line.

Keep the file small, serve it as text/plain with HTTP 200, and never let it return a 5xx during deploys: put it in your static files so it is served even while the app restarts. After each change, run the checker and read the verdict for the crawlers you care about most.

## Questions

### Is the 'most specific group' idea correct?

RFC 9309 says crawlers use the group(s) matching their token, combining them if more than one matches, and the * group only if none match.

### Do all AI crawlers follow RFC 9309?

Most operators say they honour robots.txt; several user-triggered fetchers say it may not apply to them. See [bots that ignore robots.txt](https://brandinaianswers.com/learn/ai-bots-that-ignore-robots-txt).

### How do I test my file?

The [AI crawler checker](https://brandinaianswers.com/tools/ai-crawler-checker) applies these rules and shows the line that decided each crawler's verdict.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

[Run the free audit](https://brandinaianswers.com/audit)[Check AI crawlers](https://brandinaianswers.com/tools/ai-crawler-checker)

## Sources

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (checked 10 Nov 2026)

## Related

- [llms.txt vs robots.txt vs sitemap.xml: what each file does](https://brandinaianswers.com/learn/llms-txt-vs-robots-txt-vs-sitemap)
- [How to allow AI search but block AI training in robots.txt](https://brandinaianswers.com/learn/allow-ai-search-block-ai-training)
- [Is Cloudflare blocking AI crawlers on your site? How to check](https://brandinaianswers.com/learn/cloudflare-ai-crawler-settings)
- [All guides](https://brandinaianswers.com/learn)
