Free AI crawler checker: can ChatGPT, Claude and Perplexity read your site?

Enter a domain to see which of 16 AI crawlers your robots.txt allows on the home page, with the exact rule that decided each one, and whether your llms.txt and sitemap are in place. It is free, needs no sign-up and reads only public files.

The crawlers it checks

Each operator documents its own crawlers; the notes below summarise their pages, read 10 Nov 2026. Search crawlers decide whether you can be cited, user fetchers act for a person who asked about your page, and training crawlers collect content for future models.

TokenOperatorTypeWhat it does
OAI-SearchBotOpenAIAI searchShows sites in ChatGPT search answers. Blocked = not shown there.
ChatGPT-UserOpenAIUser fetchFetches a page when a ChatGPT user asks; OpenAI says robots.txt may not apply.
GPTBotOpenAITrainingCrawls for training OpenAI's models.
Claude-SearchBotAnthropicAI searchIndexes pages for Claude's search results.
Claude-UserAnthropicUser fetchFetches a page when a Claude user asks; Anthropic says it honours robots.txt.
ClaudeBotAnthropicTrainingCollects content that may be used for model training.
PerplexityBotPerplexityAI searchSurfaces and links sites in Perplexity answers; not used for model training.
Perplexity-UserPerplexityUser fetchFetches a page for a user's question; Perplexity says it generally ignores robots.txt.
GooglebotGoogleAI searchGoogle Search, including AI Overviews and AI Mode.
Google-ExtendedGoogleTraining control (not a crawler)Not a crawler: a token for Gemini training and grounding. No effect on Google Search.
BingbotMicrosoftAI searchCrawls for Bing, Microsoft's search index.
ApplebotAppleAI searchSpotlight, Siri and Safari search features.
Applebot-ExtendedAppleTraining control (not a crawler)Not a crawler: opts out of Apple model training; search inclusion stays.
meta-webindexerMetaAI searchHelps Meta AI cite and link to your pages.
meta-externalagentMetaTrainingCrawls for training Meta's AI models and product indexing.
CCBotCommon CrawlTrainingOpen web archive that many AI models are trained on.

How the verdicts are worked out

What to do with the result

If an AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) is blocked, remove the Disallow rule for it, or move it into a group that allows your public pages; see how AI crawlers read robots.txt. If only training crawlers are blocked, that is a business choice: allow AI search but block training shows a clean setup. If everything is allowed but you still do not appear in answers, check your CDN (Cloudflare settings) and whether your pages render without JavaScript.

No llms.txt? It is optional, but cheap: what llms.txt is, with a copy-ready example. For the wider picture (schema, server-rendered content and the real answers AI engines give to your buyers' questions) run the free AI visibility audit.

Questions

What does the AI crawler checker look at?

It reads four public files on the domain you enter: robots.txt, llms.txt, llms-full.txt and sitemap.xml. It applies the robots.txt rules to 16 AI crawler tokens for your home page path and checks your llms.txt against the llmstxt.org format.

Why does a crawler show as allowed when I blocked a folder?

The verdict is for the home page path (/). A rule that blocks only /blog leaves the home page open. Check folder rules by reading the 'Why' column, which shows the rule line that decided each verdict.

Does 'allowed' mean ChatGPT will cite me?

No. It means robots.txt does not stop that crawler. Being cited also depends on a firewall or CDN not blocking it, on your pages answering the question and on what other sites say about you.

Is it safe to enter any domain?

The checker only fetches those four files from public web addresses, refuses private and reserved network addresses, follows at most three redirects (checking each one), stops after 8 seconds or 512 KB, and limits how many checks one network can run.

Why is ChatGPT-User marked as a user fetch?

OpenAI says ChatGPT-User acts for a user and robots.txt rules may not apply to it. Perplexity says the same of Perplexity-User. A Disallow line is a request, not access control, for these fetchers.

More guides: Learn: get your brand named in AI answers · AI crawler user agents list.

Free AI crawler checker: robots.txt and llms.txt