The cited sites changed most on ChatGPT and Gemini#
Prefer asked ChatGPT, Claude, Gemini and Perplexity 80 questions three times each on 13 September 2026, through the DataForSEO API with web search enabled, not the consumer apps (the full study). Each run was a fresh call; all three answers to a question came back within 23 seconds of each other.
For each pair of runs where both answers cited sources, we divided the sites cited in both by the sites cited in either. On average, two runs of the same question shared 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. In 39 of 206 pairs, ChatGPT’s two answers had no site in common. Perplexity cited exactly the same set of sites in 199 of 240 pairs.
Whether the engine searched also changed: Gemini and Claude each searched the web in some runs and not others on 13 of the 80 questions, ChatGPT on 4, Perplexity on none. A run without search cited nothing.
| Three runs of one question | Result |
|---|---|
| Sites shared by two runs, per engine | 35% to 96% |
| Questions searched in some runs only, per engine | 0 to 13 of 80 |
| Agency names in all three runs, per question and engine | 100 of 270 (37%) |
| Tool names in all three runs, per question and engine | 288 of 739 (39%) |
| First agency named differed across the three | 11 of 16 question and engine pairs |
The brands named changed as well#
We counted names on 4 questions asking for AI SEO or GEO agencies and 14 asking for AI visibility, AEO or GEO tools, once per question and engine. Of the agencies or tools an engine named across three runs of one question, fewer than 4 in 10 appeared in all three. ChatGPT’s agency lists moved most: on average, two runs shared 34% of the agencies named in either.
Asked “What are the best LLM SEO companies?”, ChatGPT named 19 agencies across three runs and only Omniscient Digital in all three. Asked for the best AI visibility tool, its runs 1, 2 and 3 gave “Best overall” to Profound, Semrush AI Visibility Toolkit and Profound.
On all four agency questions, Perplexity’s three runs cited the same 20 pages in the same order, yet two runs shared on average 69% of the agencies named, and on each question one run in three named a different agency first.
How to measure an answer that moves#
An April 2026 arXiv paper, “Don’t Measure Once” by Schulte, Bleeker and Kaufmann, concludes that brands need repeated measurements because answers “can vary across runs, prompts, and time, making one-off observations unreliable”. ChatGPT cited it in 6 of its 240 study answers.
- Freeze the prompt set. Start with 10 to 50 buyer prompts (how many to track). Done when each prompt’s wording, engine and model are saved with a start date.
- Run every prompt three or more times per engine in each check. In an app, use a new chat per run, with memory off or in a temporary chat that skips memory. For Google’s AI Overviews, repeat the search signed out, on the same device and from the same country each time. Done when each prompt has three or more dated runs per engine.
- Record whether each run searched the web. Apps usually show cited links or a sources list when they search; the API returns a flag (web_search in the DataForSEO results we used). A citation drop can mean a skipped search, not a lost source. Done when each report shows how many runs searched, per engine.
- Report rates, not screenshots. Write “named in 2 of 3 runs”, per engine. Done when every figure has a count and a denominator.
- Call a change real only when the next check agrees (how often to check). A move is a prompt crossing from 0 or 1 of 3 runs to 2 or 3 of 3, or back. Done when the crossing holds for two checks in a row; if it flips back, log it as noise.
Disclosure: Prefer sells AI visibility tracking, so “measure more than once” is advice we have a stake in. For a first baseline, run a free AI visibility audit.
Sources
- Across 80 questions asked three times each on ChatGPT, Claude, Gemini and Perplexity (DataForSEO API with web search enabled, 13 September 2026, 960 answers), two runs of the same question shared on average 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. Measure: for every pair of runs in which both answers cited a source, sites cited in both divided by sites cited in either, averaged over 206, 200, 179 and 240 pairs. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- Perplexity cited exactly the same set of sites in 199 of its 240 pairs of runs of the same question. In 39 of 206 pairs, ChatGPT's two answers to the same question had no cited site in common. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- Whether an engine searched the web changed between runs of the same question: Gemini and Claude each searched in some runs and not others on 13 of the 80 questions, ChatGPT on 4 and Perplexity on none. Perplexity searched on all three runs of every question, ChatGPT on 70 of 80, Gemini on 63 and Claude on 58. None of the 99 answers that skipped search cited a source. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- On 4 questions asking for AI SEO or GEO agencies (16 question and engine pairs, 13 September 2026), 100 of 270 agency names, counted once per question and engine, appeared in all three runs (37%), and the first agency named was not the same in all three runs in 11 of the 16 pairs. ChatGPT's agency lists moved most: on average, two runs shared 34% of the agencies named in either. On all four questions, Perplexity's three runs cited the same 20 pages in the same order, yet on average two runs shared 69% of the agencies named, and on each question one of the three runs named a different agency first. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- On 14 questions asking for AI visibility, AEO or GEO tools (56 question and engine pairs, 13 September 2026), 288 of 739 tool names, counted once per question and engine, appeared in all three runs (39%). (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- All three answers to each question came back within 23 seconds of each other, and each run was a fresh API call with no chat history. Models: gpt-5.6-luna, claude-sonnet-5, gemini-3.8-flash and sonar-pro. Answers from the API can differ from the consumer apps. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- Asked 'What is the best AI visibility tool?', ChatGPT's runs 1, 2 and 3, all returned within 15 seconds of each other, gave 'Best overall' to Profound, Semrush AI Visibility Toolkit and Profound respectively. Asked 'What are the best LLM SEO companies?', ChatGPT named 19 agencies across three runs and only Omniscient Digital in all three. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- 'Don't Measure Once: Measuring Visibility in AI Search (GEO)' by Julius Schulte, Malte Bleeker and Philipp Kaufmann (arXiv, submitted 8 April 2026) argues that AI answers vary across runs, prompts and time, so one-off observations are unreliable, and that a brand's visibility should be measured repeatedly and described as a distribution. arXiv: Don't Measure Once (Schulte, Bleeker and Kaufmann, 2026)
- ChatGPT cited 'Don't Measure Once' in 6 of its 240 answers in the study; no other engine cited it. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- Ronald Sielinski's 'Quantifying Uncertainty in AI Visibility' (arXiv, March 2026, revised August 2026) sampled Perplexity, OpenAI's SearchGPT and Google Gemini repeatedly and found that many apparent differences between domains fall within the noise floor of the measurement, and that single-run visibility metrics give a misleadingly precise picture. arXiv: Quantifying Uncertainty in AI Visibility (Sielinski, 2026)
- SparkToro's January 2026 study with Gumshoe.ai (600 volunteers, 12 prompts, 2,961 runs in the consumer apps of ChatGPT, Claude and Google's AI, November and December 2025) found under a 1 in 100 chance that ChatGPT or Google's AI gives the same list of brands in any two responses, about 1 in 1,000 for the same order, and suggested at least 60 to 100 runs to learn an AI's set of recommendations. SparkToro: AIs are highly inconsistent when recommending brands (27 January 2026)
- Google AI Overviews, same questions five days apart (DataForSEO Google SERP API, US English, desktop, same settings both days except result depth: 10 organic results requested on 13 September, up to 100 on 18 September): 70 of our 80 questions showed an AI Overview with sources on both 13 and 18 September 2026. Of the 602 pages cited on 13 September, counting each page once per question, 289 (48%) were cited again for the same question on 18 September. The 48% is a share of the 13 September pages, not of both days. Every one of the 70 questions kept at least one page. (Prefer research note: AI Overviews questions (2026-09-19), dated research note)
- Some answers held steady: asked for the best AI visibility tool for a small marketing team on a budget, ChatGPT, Gemini and Perplexity named the same pick, Otterly.AI, in all nine of their answers. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
- Prefer sells AI visibility tracking, so we have a commercial stake in the advice to measure more than once. Prefer pricing (our product)
People also ask
- Does ChatGPT give the same answer every time?
- Do AI engines cite the same sources every time?
- Do AI recommendations change each time you ask?
- How consistent are AI search answers?
- Is one AI visibility check enough?