How AI crawlers read robots.txt: matching rules that surprise people
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
Crawlers that follow RFC 9309 use only the group whose User-agent matches their token (falling back to *), pick the longest matching path rule, prefer Allow on a tie and treat a 5xx robots.txt as 'disallow everything'. Most AI-visibility accidents come from a specific group that drops rules people assumed were inherited from *.
The rules that matter
| Rule | What RFC 9309 says |
|---|---|
| Matching | Crawlers must match the product token case-insensitively; if several groups match, their rules are combined; if none match, the * group applies |
| Longest match | The most specific (longest) matching rule must be used |
| Ties | If an Allow and a Disallow are equivalent, Allow should be used |
| Wildcards | * matches any characters; $ marks the end of the path |
| 4xx | robots.txt is 'unavailable'; the crawler may access anything |
| 5xx or unreachable | The crawler must assume complete disallow |
| Size | Crawlers must parse at least 500 KiB |
| Caching | Crawlers should not use a cached copy for more than 24 hours unless robots.txt is unreachable |
Worked example: the inherited-rule trap
User-agent: *
Disallow: /checkout
Disallow: /search
User-agent: PerplexityBot
Disallow: /internalPerplexityBot matches its own group, so it ignores the * group completely: it may crawl /checkout and /search, and only /internal is blocked for it. Copy every rule you need into specific groups.
Worked example: longest match
User-agent: *
Disallow: /blog
Allow: /blog/guides/blog/guides/llms-txt is allowed (the Allow rule is longer); /blog/news is disallowed.
Mistakes we see most
- "Disallow: /" left over from a staging site.
- robots.txt served by a firewall challenge page or with a 503 during deploys, which crawlers read as 'disallow all'.
- Blocking by IP or firewall instead of robots.txt, which also stops crawlers reading the rules.
- Typos in tokens: "GPT-Bot" or "Perplexity Bot" match nothing.
A safe template to start from
Start with one * group that holds every rule all crawlers should follow (admin, cart, search result pages), then add specific groups only for crawlers you treat differently, and copy the shared rules into each of those groups. End with a Sitemap line.
Keep the file small, serve it as text/plain with HTTP 200, and never let it return a 5xx during deploys: put it in your static files so it is served even while the app restarts. After each change, run the checker and read the verdict for the crawlers you care about most.
Questions
Is the 'most specific group' idea correct?
RFC 9309 says crawlers use the group(s) matching their token, combining them if more than one matches, and the * group only if none match.
Do all AI crawlers follow RFC 9309?
Most operators say they honour robots.txt; several user-triggered fetchers say it may not apply to them. See bots that ignore robots.txt.
How do I test my file?
The AI crawler checker applies these rules and shows the line that decided each crawler's verdict.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)