llms.txt vs robots.txt vs sitemap.xml: what each file does
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
robots.txt tells crawlers what they may fetch, sitemap.xml lists the URLs you want crawled, and llms.txt is a short Markdown reading list for AI tools. Only robots.txt controls access; the other two are hints, and you usually want all three.
Side by side
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Job | Crawl permission per user agent | List of URLs and last-modified dates | Curated summary and links for AI tools |
| Format | Plain text rules (RFC 9309) | XML (sitemaps.org) | Markdown (llmstxt.org proposal) |
| Who reads it | Every well-behaved crawler, before crawling | Search engine crawlers | AI assistants, coding agents and tools that choose to |
| Controls access? | Yes | No | No |
| Typical size | A few lines to a few KB | Up to 50,000 URLs per file | One screen; full text goes in llms-full.txt |
| If missing | Crawlers may read everything (a 4xx means no rules) | Crawlers find pages by links only | Tools fall back to your HTML pages |
How they fit together
A crawler first reads robots.txt. If it is allowed, it may use the Sitemap: line in robots.txt to find your sitemap, then fetch pages. An AI tool answering a user's question may also fetch llms.txt to understand the site quickly.
So the order of importance is: robots.txt must not block the AI crawlers you want; sitemap.xml must list every page you want found; llms.txt should point to the 5-15 pages that explain who you are and what you sell.
A worked example for one site
# robots.txt
User-agent: *
Disallow: /admin
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
# Result: every crawler may read the site except /admin;
# GPTBot (OpenAI training) is blocked everywhere;
# OAI-SearchBot (ChatGPT search) has no group of its own,
# so it follows the * group and may read everything but /admin.The sitemap then lists the public URLs, and llms.txt links to the pricing, product and docs pages. Run the AI crawler checker to see the same verdicts for your domain.
Common mistakes
- Putting "Disallow" lines in llms.txt. It has no rules; nobody will obey them.
- A robots.txt that returns HTTP 500. Under RFC 9309 a server error means crawlers must assume everything is disallowed.
- An llms.txt that returns your HTML 404 page with status 200 (common with single-page apps).
- A sitemap that lists redirects or noindex pages.
Questions
Do I need all three files?
robots.txt and a sitemap are basics for any site. llms.txt is cheap to add and useful if AI tools or developers read your site.
Can robots.txt point to llms.txt?
There is no standard directive for that. Link llms.txt from your footer or docs instead, and keep the Sitemap: line for the sitemap.
Which file fixes 'ChatGPT cannot see my site'?
robots.txt, if it blocks OAI-SearchBot, or a firewall rule. llms.txt and sitemaps cannot override a block.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)
- llmstxt.org: the llms.txt proposal (checked 10 Nov 2026)
- OpenAI: crawlers and user agents (checked 10 Nov 2026)