Is Cloudflare blocking AI crawlers on your site? How to check

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

Cloudflare's AI crawler and agent access policies let you allow or block AI bots by behaviour (Search, Agent, Training), so a site can block AI search crawlers at the edge even when robots.txt allows them. For new domains Cloudflare's documentation says all three behaviours default to 'Allow', but settings chosen later or in other plans can differ, so check the dashboard.

The options Cloudflare documents

ActionEffect
Allow (do not block)Default for Search, Agent and Training on new domains, per the docs read 10 Nov 2026
BlockBlocks matching requests across the zone
Block on pages with adsBlocks only where Cloudflare detects ads on the page
Disallow (Training only)Adds robots.txt directives telling training crawlers not to access content; with Bot Preference Sync it also blocks them at the edge, except mixed-purpose crawlers

How to check in 5 minutes

  • Run the AI crawler checker to see what your robots.txt says.
  • In the Cloudflare dashboard, open the AI crawler/agent access policy settings for the zone and note the action for Search, Agent and Training.
  • Open AI Crawl Control (formerly AI Audit) to see which AI services actually reached the site and whether any were blocked.
  • Check custom WAF and bot rules for user-agent conditions that match AI crawlers.
  • If robots.txt says Allow but Cloudflare says Block, the block wins: crawlers get an error, not your page.

A setup that keeps you citable

For most businesses that want to be named in AI answers: Search = Allow, Agent = Allow, Training = your choice (Allow or Disallow). Keep robots.txt consistent with that choice so the policies and the file tell crawlers the same thing.

Signs that the edge is blocking AI crawlers

Your server logs show no requests from OAI-SearchBot or PerplexityBot at all, while Cloudflare's security events show challenges or blocks for those names.

A colleague pastes your URL into ChatGPT or Perplexity and the assistant says it could not access the page, while the page loads fine in a browser.

Your robots.txt allows the crawler, yet the AI Crawl Control dashboard lists it as blocked. In each case the fix is in the Cloudflare rules, not in robots.txt.

Questions

Does Cloudflare block Googlebot with these settings?

The documentation describes policies for verified AI bots by behaviour and does not name Googlebot. Check Security Events if Search Console reports crawl errors.

What is AI Crawl Control?

Cloudflare's tool to see which AI services access the site, set allow or block rules per crawler and track robots.txt compliance. Cloudflare says it is available on all plans.

My robots.txt is managed by Cloudflare. Can I still edit it?

When Cloudflare adds directives, check the final file at /robots.txt in a browser; the checker reads that final file.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

Sources

Related

Is Cloudflare blocking AI crawlers? How to check and fix