How Perplexity picks sources, and why it cites so many
Updated 10 Nov 2026 · by Venu Gopal Reddy Muvva, founder of BrandInAIAnswers · official sources checked 10 Nov 2026
Perplexity answers every question from live web results and shows numbered citations, so pages its crawler PerplexityBot has indexed are the pool it picks from. Perplexity says PerplexityBot is used to surface and link websites in its search results and is not used to train foundation models, so allowing it is purely about being citable.
Perplexity's user agents
| User agent | Purpose (Perplexity's docs) | robots.txt |
|---|---|---|
| PerplexityBot | Index pages to surface and link them in Perplexity's results; not for training AI models | Follows robots.txt |
| Perplexity-User | Fetch a page when a user asks Perplexity about it | User-initiated; Perplexity says it generally ignores robots.txt, since a person asked |
What gets cited
- Pages that state the answer early, often in a short paragraph or a table Perplexity can quote.
- Comparison and list pages for 'best X for Y' questions, including honest ones that name competitors.
- Recent pages for time-sensitive questions: put the date you last checked facts on the page.
- Forum threads, reviews and directories for opinion questions; your own site for facts like prices, hours and specs.
Worked example: robots.txt that allows Perplexity search only
User-agent: PerplexityBot
Allow: /
Disallow: /account/
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Disallow: /account/Remember that a named group replaces the * group for that crawler, so repeat any Disallow lines you need inside the PerplexityBot group. Run the AI crawler checker logic on your own file with the free checker.
Checklist
- PerplexityBot allowed in robots.txt and not challenged by your CDN's bot protection.
- Key pages linked from the home page and listed in the sitemap.
- Facts in HTML text; tables for numbers.
- Referral visits from perplexity.ai tracked as their own channel.
Common reasons a page is never cited
- A bot-protection challenge (CAPTCHA or JavaScript check) returned to PerplexityBot instead of the page; check your CDN's bot settings.
- The answer is buried below a long introduction, so another page states it sooner.
- Prices and specs sit in images or PDFs that the crawler can't quote.
- The page is new and not yet linked from anywhere on your site.
- A competitor publishes a comparison table and you publish only a product page.
Questions
Does Perplexity use my content to train models?
Perplexity's documentation says PerplexityBot is not used to crawl content for AI foundation models.
Why does Perplexity cite a competitor for my brand name?
Usually because their page answers the exact question and yours doesn't, or yours can't be crawled. Publish a page that answers it directly.
Can I verify PerplexityBot traffic?
Perplexity publishes the IP ranges its crawlers use; compare the IPs in your logs against them. See verifying AI crawler IPs.
See which AI answers name you today
BrandInAIAnswers's free audit checks whether AI crawlers can read your site and asks an AI with web search the questions your buyers ask, then shows who gets named. The first 14 days of the full app are free.
Sources
- Perplexity: crawlers (checked 10 Nov 2026)
- RFC 9309: Robots Exclusion Protocol (checked 10 Nov 2026)