---
title: "Is Cloudflare blocking AI crawlers? How to check and fix"
canonical: https://brandinaianswers.com/learn/cloudflare-ai-crawler-settings
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# Is Cloudflare blocking AI crawlers? How to check and fix

> Cloudflare lets you block AI crawlers by behaviour (search, agent, training) and can manage robots.txt. How to check what is on and allow AI search crawlers.

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

Cloudflare's AI crawler and agent access policies let you allow or block AI bots by behaviour (Search, Agent, Training), so a site can block AI search crawlers at the edge even when robots.txt allows them. For new domains Cloudflare's documentation says all three behaviours default to 'Allow', but settings chosen later or in other plans can differ, so check the dashboard.

## The options Cloudflare documents

| Action | Effect |

| --- | --- |

| Allow (do not block) | Default for Search, Agent and Training on new domains, per the docs read 10 Nov 2026 |

| Block | Blocks matching requests across the zone |

| Block on pages with ads | Blocks only where Cloudflare detects ads on the page |

| Disallow (Training only) | Adds robots.txt directives telling training crawlers not to access content; with Bot Preference Sync it also blocks them at the edge, except mixed-purpose crawlers |

## How to check in 5 minutes

- Run the [AI crawler checker](https://brandinaianswers.com/tools/ai-crawler-checker) to see what your robots.txt says.
- In the Cloudflare dashboard, open the AI crawler/agent access policy settings for the zone and note the action for Search, Agent and Training.
- Open AI Crawl Control (formerly AI Audit) to see which AI services actually reached the site and whether any were blocked.
- Check custom WAF and bot rules for user-agent conditions that match AI crawlers.
- If robots.txt says Allow but Cloudflare says Block, the block wins: crawlers get an error, not your page.

## A setup that keeps you citable

For most businesses that want to be named in AI answers: Search = Allow, Agent = Allow, Training = your choice (Allow or Disallow). Keep robots.txt consistent with that choice so the policies and the file tell crawlers the same thing.

## Signs that the edge is blocking AI crawlers

Your server logs show no requests from OAI-SearchBot or PerplexityBot at all, while Cloudflare's security events show challenges or blocks for those names.

A colleague pastes your URL into ChatGPT or Perplexity and the assistant says it could not access the page, while the page loads fine in a browser.

Your robots.txt allows the crawler, yet the AI Crawl Control dashboard lists it as blocked. In each case the fix is in the Cloudflare rules, not in robots.txt.

## Questions

### Does Cloudflare block Googlebot with these settings?

The documentation describes policies for verified AI bots by behaviour and does not name Googlebot. Check Security Events if Search Console reports crawl errors.

### What is AI Crawl Control?

Cloudflare's tool to see which AI services access the site, set allow or block rules per crawler and track robots.txt compliance. Cloudflare says it is available on all plans.

### My robots.txt is managed by Cloudflare. Can I still edit it?

When Cloudflare adds directives, check the final file at /robots.txt in a browser; the checker reads that final file.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

[Run the free audit](https://brandinaianswers.com/audit)[Check AI crawlers](https://brandinaianswers.com/tools/ai-crawler-checker)

## Sources

- [Cloudflare: AI crawler and agent access policies](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) (checked 10 Nov 2026)
- [Cloudflare: AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) (checked 10 Nov 2026)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (checked 10 Nov 2026)

## Related

- [How AI crawlers read robots.txt: matching rules that surprise people](https://brandinaianswers.com/learn/robots-txt-rules-for-ai-crawlers)
- [How to see AI crawler visits in your server logs](https://brandinaianswers.com/learn/ai-crawlers-in-server-logs)
- [How to allow AI search but block AI training in robots.txt](https://brandinaianswers.com/learn/allow-ai-search-block-ai-training)
- [All guides](https://brandinaianswers.com/learn)
