Perplexity ranks pages in its own index, and the CDN decides access

What changed

Perplexity ranks pages from its own index even though a reader sees only one answer, Agenzy, a GEO agency, found in an explainer on how to rank in Perplexity published on 8 November 2026. Since September 2026 Perplexity has retrieved and ranked pages with Photon, its in-house engine. Each Photon shard runs retrieval and two rounds of ranking, and a broker merges the results into one ranked list.

Perplexity searches the web for every question, the explainer reports. In three projects tracked in Peec AI, an AI visibility tracking tool, over the 30 days ending 8 November 2026, Perplexity searched in every chat, while ChatGPT searched in 83.0% to 91.5% of its chats on the same prompts. The query Perplexity shows is the buyer’s question as typed.

A page reaches Perplexity’s answers through two bots. PerplexityBot crawls pages for Perplexity’s search results, and Perplexity-User fetches a page when a person asks a question. Perplexity’s bots documentation says: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” The explainer therefore finds that the block keeping a site out of Perplexity’s answers usually sits in the CDN or firewall, the services such as Cloudflare that stand between a website and its visitors.

The explainer also cites Ahrefs’ study of 17 million citations, published on 28 July 2025, in which Perplexity’s in-text references, the numbered sources inside an answer, averaged 1,166 days since publication. Ahrefs also found that Perplexity and ChatGPT order those references from newest to oldest.

Why it happened

Perplexity’s own index came first. The company launched the Perplexity Search API on 25 September 2025 on an index covering hundreds of billions of webpages, the infrastructure behind its public answer engine. Photon followed on 24 September 2026, after Perplexity judged it simpler and cheaper to build an engine from scratch than to keep adapting its fork of an open-source one.

The firewall question dates from 4 August 2025, when Cloudflare reported that Perplexity used undeclared crawlers, including a browser posing as Google Chrome on macOS, when its declared bots were blocked. Cloudflare delisted Perplexity as a verified bot and added the crawler’s signatures to a managed rule open to all customers, free ones included. Perplexity replied the same day that Cloudflare had misattributed 3 to 6 million daily requests from BrowserBase, an automated browser service.

What it changes for GEO

Brands whose CDN blocks AI bots lose Perplexity’s answers whatever their content. A site whose CDN or host firewall blocks AI crawlers drops out of the engines it blocks, often without the owner knowing, and on Perplexity a clean robots.txt file says little about the fetcher acting on a buyer’s question. Technical work opens that door, so the access check comes before any content budget.

Perplexity also cites a different web from Google’s AI answers, so a brand’s position is judged engine by engine. Google AI Overviews takes 10.8% of its citations from user-generated content against Perplexity’s 3.8%, while Perplexity takes 13.6% from news and media against AI Overviews’ 5.2%, Agenzy’s 2026 study of 2,283,923 AI citations found (n=137 and 133 business projects). The four shares side by side:

The chart shows Perplexity leaning on news and media and Google AI Overviews on user-generated content.

Brands covered by trade press and news sites gain on Perplexity, and brands seen off their own site mostly in forum threads gain less there than in Google AI Overviews. Advice that one Reddit post fixes AI visibility is wrong on every engine and weaker still on Perplexity. The websites each AI engine cites most differ enough that one engine’s source list misleads on the next.

Perplexity searches the buyer’s question as typed, so a page whose title and opening carry the buyer’s wording meets the query word for word. When the news and media sources Perplexity prefers are hard to enter, the work goes to the brand’s own site, the one source it controls. Brands that rewrite service and comparison pages in buyers’ words gain, and brands describing themselves in product vocabulary lose.

What to do

The head of marketing asks whoever manages the site’s CDN and firewall to run an access check for Perplexity’s two bots in this order, with the exact user-agent strings from the AI crawlers list:

• Allow PerplexityBot in robots.txt.
• Allow PerplexityBot and Perplexity-User by user agent in the firewall, and match each one against the IP addresses Perplexity publishes for it.
• Turn off Cloudflare’s Bot Fight Mode and the Block AI training bots setting, since Perplexity says WAF custom rules can’t skip Bot Fight Mode.
• Wait 24 hours, the most Perplexity says a robots.txt change takes to apply, then read the CDN’s security log for both bot names.

The security log should then list both bots’ requests as allowed. After that, a prompt set for AI visibility tracking rerun each week shows whether Perplexity cites the site and names the company.