---
title: "Verify AI crawlers: IP lists and reverse DNS checks"
canonical: https://brandinaianswers.com/learn/verify-ai-crawler-ip-addresses
last_modified: 2026-11-10
publisher: Surge Software Solutions Private Limited
---

# Verify AI crawlers: IP lists and reverse DNS checks

> Anyone can fake a user agent. Where OpenAI, Anthropic, Perplexity and Common Crawl publish their crawler IP ranges, and how to check a request in two steps.

Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026

A user-agent string proves nothing, so verify AI crawlers by checking the request's IP against the ranges each operator publishes: OpenAI, Anthropic and Perplexity publish JSON lists, and Common Crawl supports reverse-then-forward DNS. Block fakes at the firewall, but keep the real crawlers able to read robots.txt.

## Where each operator publishes its addresses

| Crawler | Verification |

| --- | --- |

| OAI-SearchBot | https://openai.com/searchbot.json |

| GPTBot | https://openai.com/gptbot.json |

| ChatGPT-User | https://openai.com/chatgpt-user.json |

| Anthropic bots | https://claude.com/crawling/bots.json |

| PerplexityBot | https://www.perplexity.ai/perplexitybot.json |

| Perplexity-User | https://www.perplexity.ai/perplexity-user.json |

| CCBot | Reverse DNS ends in crawl.commoncrawl.org, then forward DNS returns the same IP; ranges at index.commoncrawl.org/ccbot.json |

## Reverse-then-forward DNS, step by step

```

$ dig -x 18.97.14.84 +short

18-97-14-84.crawl.commoncrawl.org.

$ dig 18-97-14-84.crawl.commoncrawl.org +short

18.97.14.84

# both match and the name ends in crawl.commoncrawl.org: genuine CCBot

```

This is the example Common Crawl gives on its own page. For operators that publish JSON lists, check whether the IP falls inside one of the listed CIDR ranges instead.

## Why not block by IP instead of robots.txt

Anthropic advises against IP blocking as an opt-out method: it stops the crawler reading robots.txt, so the opt-out may not work reliably. Use robots.txt for policy and IP checks only to catch impostors.

## Automating the check

Fetch each operator's JSON list once a day and store the CIDR ranges. When a request claims an AI crawler's user agent, check its IP against the stored ranges for that operator; if it does not match, treat it as an ordinary unknown client (rate-limit or block it) and log it.

Most CDNs and web application firewalls can do this for you with a verified-bot setting. If you build it yourself, cache the lists, keep yesterday's ranges when a download fails, and never block on the user agent alone, because a genuine crawler on a new IP range would be locked out until you update.

## What to do with impostors

Requests that use an AI crawler's name from an unlisted IP are scrapers, not the crawler. Rate-limit them like any unknown client, and block them if they are heavy. Do not write robots.txt rules against them: impostors do not read robots.txt, and a rule aimed at the name would also affect the genuine crawler.

Keep a short log of what you blocked and why, so a real crawler caught by mistake can be spotted quickly.

## Questions

### Why do fake AI crawlers exist?

Scrapers borrow well-known user agents to get past simple filters. Common Crawl explicitly warns about crawlers falsely claiming to be CCBot.

### How often do the IP lists change?

They can change at any time; fetch them automatically (daily) rather than copying ranges by hand.

### Does Cloudflare verify these for me?

Cloudflare's AI Crawl Control identifies AI services and lets you set rules per crawler.

See which AI answers name you today

BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.

[Run the free audit](https://brandinaianswers.com/audit)[Check AI crawlers](https://brandinaianswers.com/tools/ai-crawler-checker)

## Sources

- [OpenAI: crawlers and user agents](https://developers.openai.com/api/docs/bots) (checked 10 Nov 2026)
- [Anthropic: how site owners can block the crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (checked 10 Nov 2026)
- [Perplexity: crawlers](https://docs.perplexity.ai/guides/bots) (checked 10 Nov 2026)
- [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (checked 10 Nov 2026)

## Related

- [How to see AI crawler visits in your server logs](https://brandinaianswers.com/learn/ai-crawlers-in-server-logs)
- [Is Cloudflare blocking AI crawlers on your site? How to check](https://brandinaianswers.com/learn/cloudflare-ai-crawler-settings)
- [AI crawler user agents: the list, what each does and how to allow it](https://brandinaianswers.com/learn/ai-crawler-user-agents-list)
- [All guides](https://brandinaianswers.com/learn)
