---
title: "llms.txt vs robots.txt vs sitemap.xml: what each does"
canonical: https://brandinaianswers.com/learn/llms-txt-vs-robots-txt-vs-sitemap
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# llms.txt vs robots.txt vs sitemap.xml: what each does

> Three root files, three jobs: robots.txt sets crawl rules, sitemap.xml lists URLs, llms.txt is a reading list for AI tools. A side-by-side table and examples.

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

robots.txt tells crawlers what they may fetch, sitemap.xml lists the URLs you want crawled, and llms.txt is a short Markdown reading list for AI tools. Only robots.txt controls access; the other two are hints, and you usually want all three.

## Side by side

|  | robots.txt | sitemap.xml | llms.txt |

| --- | --- | --- | --- |

| Job | Crawl permission per user agent | List of URLs and last-modified dates | Curated summary and links for AI tools |

| Format | Plain text rules (RFC 9309) | XML (sitemaps.org) | Markdown (llmstxt.org proposal) |

| Who reads it | Every well-behaved crawler, before crawling | Search engine crawlers | AI assistants, coding agents and tools that choose to |

| Controls access? | Yes | No | No |

| Typical size | A few lines to a few KB | Up to 50,000 URLs per file | One screen; full text goes in llms-full.txt |

| If missing | Crawlers may read everything (a 4xx means no rules) | Crawlers find pages by links only | Tools fall back to your HTML pages |

## How they fit together

A crawler first reads robots.txt. If it is allowed, it may use the Sitemap: line in robots.txt to find your sitemap, then fetch pages. An AI tool answering a user's question may also fetch llms.txt to understand the site quickly.

So the order of importance is: robots.txt must not block the AI crawlers you want; sitemap.xml must list every page you want found; llms.txt should point to the 5-15 pages that explain who you are and what you sell.

## A worked example for one site

```

# robots.txt

User-agent: *

Disallow: /admin

User-agent: GPTBot

Disallow: /

Sitemap: https://example.com/sitemap.xml

# Result: every crawler may read the site except /admin;

# GPTBot (OpenAI training) is blocked everywhere;

# OAI-SearchBot (ChatGPT search) has no group of its own,

# so it follows the * group and may read everything but /admin.

```

The sitemap then lists the public URLs, and llms.txt links to the pricing, product and docs pages. Run the [AI crawler checker](https://brandinaianswers.com/tools/ai-crawler-checker) to see the same verdicts for your domain.

## Common mistakes

- Putting "Disallow" lines in llms.txt. It has no rules; nobody will obey them.
- A robots.txt that returns HTTP 500. Under RFC 9309 a server error means crawlers must assume everything is disallowed.
- An llms.txt that returns your HTML 404 page with status 200 (common with single-page apps).
- A sitemap that lists redirects or noindex pages.

## Questions

### Do I need all three files?

robots.txt and a sitemap are basics for any site. llms.txt is cheap to add and useful if AI tools or developers read your site.

### Can robots.txt point to llms.txt?

There is no standard directive for that. Link llms.txt from your footer or docs instead, and keep the Sitemap: line for the sitemap.

### Which file fixes 'ChatGPT cannot see my site'?

robots.txt, if it blocks OAI-SearchBot, or a firewall rule. llms.txt and sitemaps cannot override a block.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

[Run the free audit](https://brandinaianswers.com/audit)[Check AI crawlers](https://brandinaianswers.com/tools/ai-crawler-checker)

## Sources

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (checked 10 Nov 2026)
- [llmstxt.org: the llms.txt proposal](https://llmstxt.org/) (checked 10 Nov 2026)
- [OpenAI: crawlers and user agents](https://developers.openai.com/api/docs/bots) (checked 10 Nov 2026)

## Related

- [What is llms.txt? Format, a real example and whether it helps](https://brandinaianswers.com/learn/what-is-llms-txt)
- [How AI crawlers read robots.txt: matching rules that surprise people](https://brandinaianswers.com/learn/robots-txt-rules-for-ai-crawlers)
- [How to allow AI search but block AI training in robots.txt](https://brandinaianswers.com/learn/allow-ai-search-block-ai-training)
- [All guides](https://brandinaianswers.com/learn)
