How AI crawlers read robots.txt: matching rules that surprise people

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

Crawlers that follow RFC 9309 use only the group whose User-agent matches their token (falling back to *), pick the longest matching path rule, prefer Allow on a tie and treat a 5xx robots.txt as 'disallow everything'. Most AI-visibility accidents come from a specific group that drops rules people assumed were inherited from *.

The rules that matter

RuleWhat RFC 9309 says
MatchingCrawlers must match the product token case-insensitively; if several groups match, their rules are combined; if none match, the * group applies
Longest matchThe most specific (longest) matching rule must be used
TiesIf an Allow and a Disallow are equivalent, Allow should be used
Wildcards* matches any characters; $ marks the end of the path
4xxrobots.txt is 'unavailable'; the crawler may access anything
5xx or unreachableThe crawler must assume complete disallow
SizeCrawlers must parse at least 500 KiB
CachingCrawlers should not use a cached copy for more than 24 hours unless robots.txt is unreachable

Worked example: the inherited-rule trap

User-agent: *
Disallow: /checkout
Disallow: /search

User-agent: PerplexityBot
Disallow: /internal

PerplexityBot matches its own group, so it ignores the * group completely: it may crawl /checkout and /search, and only /internal is blocked for it. Copy every rule you need into specific groups.

Worked example: longest match

User-agent: *
Disallow: /blog
Allow: /blog/guides

/blog/guides/llms-txt is allowed (the Allow rule is longer); /blog/news is disallowed.

Mistakes we see most

  • "Disallow: /" left over from a staging site.
  • robots.txt served by a firewall challenge page or with a 503 during deploys, which crawlers read as 'disallow all'.
  • Blocking by IP or firewall instead of robots.txt, which also stops crawlers reading the rules.
  • Typos in tokens: "GPT-Bot" or "Perplexity Bot" match nothing.

A safe template to start from

Start with one * group that holds every rule all crawlers should follow (admin, cart, search result pages), then add specific groups only for crawlers you treat differently, and copy the shared rules into each of those groups. End with a Sitemap line.

Keep the file small, serve it as text/plain with HTTP 200, and never let it return a 5xx during deploys: put it in your static files so it is served even while the app restarts. After each change, run the checker and read the verdict for the crawlers you care about most.

Questions

Is the 'most specific group' idea correct?

RFC 9309 says crawlers use the group(s) matching their token, combining them if more than one matches, and the * group only if none match.

Do all AI crawlers follow RFC 9309?

Most operators say they honour robots.txt; several user-triggered fetchers say it may not apply to them. See bots that ignore robots.txt.

How do I test my file?

The AI crawler checker applies these rules and shows the line that decided each crawler's verdict.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

Sources

Related

How AI crawlers read robots.txt (RFC 9309 rules)