llms.txt vs robots.txt vs sitemap.xml: what each file does

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

robots.txt tells crawlers what they may fetch, sitemap.xml lists the URLs you want crawled, and llms.txt is a short Markdown reading list for AI tools. Only robots.txt controls access; the other two are hints, and you usually want all three.

Side by side

robots.txtsitemap.xmlllms.txt
JobCrawl permission per user agentList of URLs and last-modified datesCurated summary and links for AI tools
FormatPlain text rules (RFC 9309)XML (sitemaps.org)Markdown (llmstxt.org proposal)
Who reads itEvery well-behaved crawler, before crawlingSearch engine crawlersAI assistants, coding agents and tools that choose to
Controls access?YesNoNo
Typical sizeA few lines to a few KBUp to 50,000 URLs per fileOne screen; full text goes in llms-full.txt
If missingCrawlers may read everything (a 4xx means no rules)Crawlers find pages by links onlyTools fall back to your HTML pages

How they fit together

A crawler first reads robots.txt. If it is allowed, it may use the Sitemap: line in robots.txt to find your sitemap, then fetch pages. An AI tool answering a user's question may also fetch llms.txt to understand the site quickly.

So the order of importance is: robots.txt must not block the AI crawlers you want; sitemap.xml must list every page you want found; llms.txt should point to the 5-15 pages that explain who you are and what you sell.

A worked example for one site

# robots.txt
User-agent: *
Disallow: /admin

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

# Result: every crawler may read the site except /admin;
# GPTBot (OpenAI training) is blocked everywhere;
# OAI-SearchBot (ChatGPT search) has no group of its own,
# so it follows the * group and may read everything but /admin.

The sitemap then lists the public URLs, and llms.txt links to the pricing, product and docs pages. Run the AI crawler checker to see the same verdicts for your domain.

Common mistakes

  • Putting "Disallow" lines in llms.txt. It has no rules; nobody will obey them.
  • A robots.txt that returns HTTP 500. Under RFC 9309 a server error means crawlers must assume everything is disallowed.
  • An llms.txt that returns your HTML 404 page with status 200 (common with single-page apps).
  • A sitemap that lists redirects or noindex pages.

Questions

Do I need all three files?

robots.txt and a sitemap are basics for any site. llms.txt is cheap to add and useful if AI tools or developers read your site.

Can robots.txt point to llms.txt?

There is no standard directive for that. Link llms.txt from your footer or docs instead, and keep the Sitemap: line for the sitemap.

Which file fixes 'ChatGPT cannot see my site'?

robots.txt, if it blocks OAI-SearchBot, or a firewall rule. llms.txt and sitemaps cannot override a block.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

Sources

Related

llms.txt vs robots.txt vs sitemap.xml: what each does