---
title: How AI search engines choose sources: what is known and what is not
description: How ChatGPT, Perplexity, Claude and Google AI Overviews pick pages to cite: retrieval versus training, what vendors document, and what nobody outside knows.
canonical: https://www.postbing.com/blog/how-ai-search-engines-choose-sources
published: 2026-11-16
updated: 2026-11-16
author: The Postbing Team
---

# How AI search engines choose sources: what is known and what is not

AI search engines choose sources mostly at answer time, not from training data. When you ask ChatGPT search, Perplexity, Claude or Google AI Mode a question, the system runs one or more web searches, fetches candidate pages with a retrieval crawler, and cites the passages it uses. The vendors document which crawlers do this. They do not publish how candidates are ranked, so any precise weighting you read is a guess.

Last updated: November 16, 2026

## Two routes into an answer: retrieval and training

An AI answer can draw on two separate things: what the model learned during training, and what it fetched from the web for this question. Only the second produces a clickable citation, and each vendor uses different crawlers for the two jobs.

| Vendor | Retrieval (decides citations in answers) | Training (feeds future models) | Source |
|---|---|---|---|
| OpenAI | OAI-SearchBot (search index), ChatGPT-User (user-triggered fetches) | GPTBot | OpenAI crawler docs |
| Anthropic | Claude-SearchBot (search index), Claude-User (user-triggered fetches) | ClaudeBot | Anthropic support article |
| Perplexity | PerplexityBot (index), Perplexity-User (user-triggered fetches) | None listed; PerplexityBot "is not used to crawl content for AI foundation models" | Perplexity bots docs |
| Google | Googlebot (Search, including AI Overviews and AI Mode) | Google-Extended token (Gemini training and grounding outside Search) | Google crawler docs |

The practical consequence: blocking a training bot does not remove you from answers, but blocking a retrieval bot does. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Google says Google-Extended "does not impact a site's inclusion in Google Search", so AI Overviews follow your Googlebot rules, not your Google-Extended rule. Training data still shapes how a model phrases things and which brands it already "knows", but you cannot see or audit that route.

## Step by step: what happens when someone asks a question

The documented and observed process has four stages. The first and last are well established; the middle two are where the vendors stay quiet.

1. **The prompt becomes searches.** The engine rewrites the question into several shorter queries, a step called query fan-out. Seer Interactive measured Gemini 3 running an average of 10.7 searches per prompt (range 3 to 28, 501 prompts, November 2025). Our [query fan-out explainer](/blog/chatgpt-query-fan-out-explained) covers the shapes these take.
2. **Candidates come from an index.** Each sub-query hits a search index: Google's own for AI Overviews, and for other engines a mix of their own crawl and partner indexes. Which partners, and how they are blended, is not fully published.
3. **Pages are fetched and passages selected.** A retrieval bot reads the page. If the text is only rendered by JavaScript, or the server returns a 403 to that bot, the page cannot be used. The selection of passages within a page is not documented by any vendor.
4. **The answer cites what it used.** Citations point to the pages whose passages made it into the final text.

The broader GEO picture, including how this affects what you publish, is in the [generative engine optimization guide](/blog/generative-engine-optimization-guide).

## What the vendors actually document

The firm facts are about eligibility, not ranking. Here is what each vendor states in its own documentation (all checked 16 November 2026):

- **Google:** a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. There are "no additional technical requirements" and no special schema. Snippet controls apply: `nosnippet` "will also prevent the content from being used as a direct input for AI Overviews and AI Mode".
- **OpenAI:** to appear in ChatGPT search, it recommends "allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges". Changes to robots.txt take "~24 hours" to apply.
- **Anthropic:** "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization", and disabling Claude-User stops Claude retrieving your page when a user asks.
- **Perplexity:** recommends "allowing PerplexityBot in your site's robots.txt file". Its user-triggered fetcher "generally ignores robots.txt rules" because a person asked for the page.

None of these pages describes a ranking formula. Anyone who claims to know the exact weights is extrapolating.

## What studies have observed

Independent studies show patterns, not rules. They are correlations from vendor tools and panels, and they often disagree with each other.

| Observation | Number | Who and when |
|---|---|---|
| Recently updated pages dominate ChatGPT's most cited pages | 76.4% of dated top cited pages updated within 30 days (564 dated URLs) | Ahrefs, October 2025 |
| ChatGPT, Gemini and Copilot citations overlap little with Google's top 10 | 12%; Perplexity 28.6% | Ahrefs, August 2025 |
| AI Overview citations that also rank in the top 10 results | 37.9% (Ahrefs notes a parsing change, not like for like with its earlier 76%) | Ahrefs, March 2026 |
| AI Overview overlap with organic rankings | Grew from 32.3% to 54.5% | BrightEdge, September 2025 |
| Brand web mentions vs AI Overview brand visibility | Correlation 0.664 (backlinks 0.218) | Ahrefs, May 2025 |
| ChatGPT cited pages with zero organic keywords | 28.3% of top 1,000 cited pages | Ahrefs, October 2025 |

Two lessons survive the disagreement. Freshness and specific, quotable content show up repeatedly. And ranking in Google helps but is not the same thing as being cited, especially outside Google.

## Source mix can swing quickly

Which domains an engine favours changes over weeks, which is a warning against building on any single platform. Semrush tracked ChatGPT citing Reddit in "close to 60%" of responses in early August 2025, "collapsing to around 10%" by mid-September. A year later, cloro's vendor data showed ChatGPT answers citing Reddit falling from 25.9% to 4.9% between the weeks of 27 July and 5 October 2026. These measure different things (share of responses versus share of citations) and are not comparable, but the volatility is the point.

## What nobody outside the companies knows

Be honest with yourself and your clients about the gaps:

- How each engine weights freshness, authority, brand mentions and on-page relevance.
- Which third party search indexes non-Google engines use for which queries, and when.
- Whether structured data affects passage selection in engines other than Google (Google says it is not needed for its AI features).
- How much training data influences which brands get named when no page is cited.
- Why the same prompt cites different sources on different runs. If you track citations, run each prompt several times, as described in our [free citation tracking method](/blog/track-ai-citations-for-free).

## What this means for your site

You cannot control the ranking step, but you control eligibility and quotability. Let the retrieval bots in, serve full text in raw HTML, answer each question in the first sentences under its heading, keep prices and numbers dated and current, and earn real mentions on other sites. For Perplexity specifically, see [how to get cited by Perplexity](/blog/how-to-get-cited-by-perplexity).

## FAQ

### Do AI search engines use their training data or live web results?
Both, but citations come from live retrieval. The engine searches, fetches pages with a retrieval bot such as OAI-SearchBot or PerplexityBot, and cites the pages it used; training data shapes the model's general knowledge without producing citations.

### If I block GPTBot, will ChatGPT stop citing me?
No. GPTBot is OpenAI's training crawler. ChatGPT search citations depend on OAI-SearchBot, and OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers.

### Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode follow your Googlebot rules and snippet controls such as nosnippet, or the Search generative AI control in Search Console.

### Does ranking first on Google mean AI answers will cite me?
Not reliably. Ahrefs found in August 2025 that only 12% of links cited by ChatGPT, Gemini and Copilot appeared in Google's top 10 for the same prompt, while Perplexity's overlap was higher at 28.6%.

### Why does the same question cite different sources each time?
The engines rewrite prompts into varying sub-searches and the results shift, and the vendors do not document the selection step. Running each prompt several times gives a more reliable picture than a single check.

## Sources

- [OpenAI, overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): OAI-SearchBot, GPTBot, ChatGPT-User, robots.txt timing, accessed 16 November 2026
- [Anthropic, does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): ClaudeBot, Claude-SearchBot, Claude-User, accessed 16 November 2026
- [Perplexity, Perplexity crawlers](https://docs.perplexity.ai/guides/bots): PerplexityBot and Perplexity-User, accessed 16 November 2026
- [Google, common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): Googlebot and Google-Extended scope, accessed 16 November 2026
- [Google Search Central, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): eligibility and controls, accessed 16 November 2026
- [Google Search Central, robots meta tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag): nosnippet and max-snippet effect on AI features, accessed 16 November 2026
- Ahrefs blog, ChatGPT's most cited pages (Louise Linehan, 28 October 2025): 76.4% freshness bucket, 28.3% zero keyword pages, accessed 16 November 2026
- Ahrefs blog, AI search overlap (11 August 2025): 12% and 28.6%, accessed 16 November 2026
- Ahrefs blog, AI Overview citations and the top 10 (2 March 2026): 37.9%, accessed 16 November 2026
- Ahrefs blog, AI Overview brand correlation (26 May 2025): 0.664 and 0.218, accessed 16 November 2026
- BrightEdge weekly AI search insights, rank overlap after 16 months of AIO (18 September 2025): 32.3% to 54.5%, accessed 16 November 2026
- Seer Interactive, Gemini 3 query fan-out research (21 November 2025): 10.7 average, accessed 16 November 2026
- Semrush blog, most cited domains in AI (10 November 2025): Reddit share of ChatGPT responses in 2025, accessed 16 November 2026
- cloro blog, Reddit AI citations (updated 13 October 2026): weekly ChatGPT Reddit share in 2026, vendor data described as directional, accessed 16 November 2026

## Related reading

- [Generative engine optimization: the full guide](/blog/generative-engine-optimization-guide)
- [ChatGPT query fan-out explained](/blog/chatgpt-query-fan-out-explained)
- [Is GEO the new SEO?](/blog/is-geo-the-new-seo)
- [How to get cited by Perplexity](/blog/how-to-get-cited-by-perplexity)
- [Postbing AI visibility tracking and pricing](/pricing)
