---
title: "Free AI crawler checker: robots.txt and llms.txt"
canonical: https://brandinaianswers.com/tools/ai-crawler-checker
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# Free AI crawler checker: robots.txt and llms.txt

> Check whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 12 more AI crawlers may read your site, and test your llms.txt. Free, no sign-up.

Enter a domain to see which of 16 AI crawlers your robots.txt allows on the home page, with the exact rule that decided each one, and whether your llms.txt and sitemap are in place. It is free, needs no sign-up and reads only public files.

## The crawlers it checks

Each operator documents its own crawlers; the notes below summarise their pages, read 10 Nov 2026. Search crawlers decide whether you can be cited, user fetchers act for a person who asked about your page, and training crawlers collect content for future models.

| Token | Operator | Type | What it does |

| --- | --- | --- | --- |

| [OAI-SearchBot](https://developers.openai.com/api/docs/bots) | OpenAI | AI search | Shows sites in ChatGPT search answers. Blocked = not shown there. |

| [ChatGPT-User](https://developers.openai.com/api/docs/bots) | OpenAI | User fetch | Fetches a page when a ChatGPT user asks; OpenAI says robots.txt may not apply. |

| [GPTBot](https://developers.openai.com/api/docs/bots) | OpenAI | Training | Crawls for training OpenAI's models. |

| [Claude-SearchBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Anthropic | AI search | Indexes pages for Claude's search results. |

| [Claude-User](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Anthropic | User fetch | Fetches a page when a Claude user asks; Anthropic says it honours robots.txt. |

| [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Anthropic | Training | Collects content that may be used for model training. |

| [PerplexityBot](https://docs.perplexity.ai/guides/bots) | Perplexity | AI search | Surfaces and links sites in Perplexity answers; not used for model training. |

| [Perplexity-User](https://docs.perplexity.ai/guides/bots) | Perplexity | User fetch | Fetches a page for a user's question; Perplexity says it generally ignores robots.txt. |

| [Googlebot](https://developers.google.com/search/docs/appearance/ai-features) | Google | AI search | Google Search, including AI Overviews and AI Mode. |

| [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | Google | Training control (not a crawler) | Not a crawler: a token for Gemini training and grounding. No effect on Google Search. |

| [Bingbot](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) | Microsoft | AI search | Crawls for Bing, Microsoft's search index. |

| [Applebot](https://support.apple.com/en-us/119829) | Apple | AI search | Spotlight, Siri and Safari search features. |

| [Applebot-Extended](https://support.apple.com/en-us/119829) | Apple | Training control (not a crawler) | Not a crawler: opts out of Apple model training; search inclusion stays. |

| [meta-webindexer](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | Meta | AI search | Helps Meta AI cite and link to your pages. |

| [meta-externalagent](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | Meta | Training | Crawls for training Meta's AI models and product indexing. |

| [CCBot](https://commoncrawl.org/ccbot) | Common Crawl | Training | Open web archive that many AI models are trained on. |

## How the verdicts are worked out

- Rules are read the way [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) says: a crawler uses the group that names its token (case-insensitive) and falls back to the `*` group only when none does.
- The longest matching path rule wins; Allow wins a tie; `*` and `$` wildcards are supported.
- A robots.txt that returns a server error (5xx) means "disallow everything"; a 4xx means there are no rules, so everything is allowed.
- An llms.txt passes the basic format check when it has an H1 title, a blockquote summary, H2 sections and Markdown links, and is not an HTML page.

## What to do with the result

If an AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) is blocked, remove the Disallow rule for it, or move it into a group that allows your public pages; see [how AI crawlers read robots.txt](https://brandinaianswers.com/learn/robots-txt-rules-for-ai-crawlers). If only training crawlers are blocked, that is a business choice: [allow AI search but block training](https://brandinaianswers.com/learn/allow-ai-search-block-ai-training) shows a clean setup. If everything is allowed but you still do not appear in answers, check your CDN ([Cloudflare settings](https://brandinaianswers.com/learn/cloudflare-ai-crawler-settings)) and whether your pages render without JavaScript.

No llms.txt? It is optional, but cheap: [what llms.txt is, with a copy-ready example](https://brandinaianswers.com/learn/what-is-llms-txt). For the wider picture (schema, server-rendered content and the real answers AI engines give to your buyers' questions) run the [free AI visibility audit](https://brandinaianswers.com/audit).

## Questions

### What does the AI crawler checker look at?

It reads four public files on the domain you enter: robots.txt, llms.txt, llms-full.txt and sitemap.xml. It applies the robots.txt rules to 16 AI crawler tokens for your home page path and checks your llms.txt against the llmstxt.org format.

### Why does a crawler show as allowed when I blocked a folder?

The verdict is for the home page path (/). A rule that blocks only /blog leaves the home page open. Check folder rules by reading the 'Why' column, which shows the rule line that decided each verdict.

### Does 'allowed' mean ChatGPT will cite me?

No. It means robots.txt does not stop that crawler. Being cited also depends on a firewall or CDN not blocking it, on your pages answering the question and on what other sites say about you.

### Is it safe to enter any domain?

The checker only fetches those four files from public web addresses, refuses private and reserved network addresses, follows at most three redirects (checking each one), stops after 8 seconds or 512 KB, and limits how many checks one network can run.

### Why is ChatGPT-User marked as a user fetch?

OpenAI says ChatGPT-User acts for a user and robots.txt rules may not apply to it. Perplexity says the same of Perplexity-User. A Disallow line is a request, not access control, for these fetchers.

More guides: [Learn: get your brand named in AI answers](https://brandinaianswers.com/learn) · [AI crawler user agents list](https://brandinaianswers.com/learn/ai-crawler-user-agents-list).
