Done for you: our team does the AEO work for you.Learn more

Do AI answers change if you ask the same question twice?

Prefer asked 80 questions three times each on ChatGPT, Claude, Gemini and Perplexity and measured how much the cited sites and named brands changed.

Javed Khatri Javed Khatri Co-founder, Prefer

3 min read Answer

The short answer

Does ChatGPT give the same answer every time?

Yes, on most engines. Prefer asked 80 questions three times each on 13 September 2026: two runs of the same question shared 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. Only 39% of the tools named appeared in all three runs.

Key takeaways

  • Across 80 questions asked three times each on ChatGPT, Claude, Gemini and Perplexity (DataForSEO API with web search enabled, 13 September 2026, 960 answers), two runs of the same question shared on average 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. Measure: for every pair of runs in which both answers cited a source, sites cited in both divided by sites cited in either, averaged over 206, 200, 179 and 240 pairs. (Prefer research note: repeatability in the AI answer study (2026-09-19))
  • Perplexity cited exactly the same set of sites in 199 of its 240 pairs of runs of the same question. In 39 of 206 pairs, ChatGPT's two answers to the same question had no cited site in common. (Prefer research note: repeatability in the AI answer study (2026-09-19))
  • Whether an engine searched the web changed between runs of the same question: Gemini and Claude each searched in some runs and not others on 13 of the 80 questions, ChatGPT on 4 and Perplexity on none. Perplexity searched on all three runs of every question, ChatGPT on 70 of 80, Gemini on 63 and Claude on 58. None of the 99 answers that skipped search cited a source. (Prefer research note: repeatability in the AI answer study (2026-09-19))
  • On 4 questions asking for AI SEO or GEO agencies (16 question and engine pairs, 13 September 2026), 100 of 270 agency names, counted once per question and engine, appeared in all three runs (37%), and the first agency named was not the same in all three runs in 11 of the 16 pairs. ChatGPT's agency lists moved most: on average, two runs shared 34% of the agencies named in either. On all four questions, Perplexity's three runs cited the same 20 pages in the same order, yet on average two runs shared 69% of the agencies named, and on each question one of the three runs named a different agency first. (Prefer research note: repeatability in the AI answer study (2026-09-19))
  • On 14 questions asking for AI visibility, AEO or GEO tools (56 question and engine pairs, 13 September 2026), 288 of 739 tool names, counted once per question and engine, appeared in all three runs (39%). (Prefer research note: repeatability in the AI answer study (2026-09-19))
  • All three answers to each question came back within 23 seconds of each other, and each run was a fresh API call with no chat history. Models: gpt-5.6-luna, claude-sonnet-5, gemini-3.8-flash and sonar-pro. Answers from the API can differ from the consumer apps. (Prefer research note: repeatability in the AI answer study (2026-09-19))

The cited sites changed most on ChatGPT and Gemini#

Prefer asked ChatGPT, Claude, Gemini and Perplexity 80 questions three times each on 13 September 2026, through the DataForSEO API with web search enabled, not the consumer apps (the full study). Each run was a fresh call; all three answers to a question came back within 23 seconds of each other.

For each pair of runs where both answers cited sources, we divided the sites cited in both by the sites cited in either. On average, two runs of the same question shared 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. In 39 of 206 pairs, ChatGPT’s two answers had no site in common. Perplexity cited exactly the same set of sites in 199 of 240 pairs.

Whether the engine searched also changed: Gemini and Claude each searched the web in some runs and not others on 13 of the 80 questions, ChatGPT on 4, Perplexity on none. A run without search cited nothing.

Three runs of one questionResult
Sites shared by two runs, per engine35% to 96%
Questions searched in some runs only, per engine0 to 13 of 80
Agency names in all three runs, per question and engine100 of 270 (37%)
Tool names in all three runs, per question and engine288 of 739 (39%)
First agency named differed across the three11 of 16 question and engine pairs

The brands named changed as well#

We counted names on 4 questions asking for AI SEO or GEO agencies and 14 asking for AI visibility, AEO or GEO tools, once per question and engine. Of the agencies or tools an engine named across three runs of one question, fewer than 4 in 10 appeared in all three. ChatGPT’s agency lists moved most: on average, two runs shared 34% of the agencies named in either.

Asked “What are the best LLM SEO companies?”, ChatGPT named 19 agencies across three runs and only Omniscient Digital in all three. Asked for the best AI visibility tool, its runs 1, 2 and 3 gave “Best overall” to Profound, Semrush AI Visibility Toolkit and Profound.

On all four agency questions, Perplexity’s three runs cited the same 20 pages in the same order, yet two runs shared on average 69% of the agencies named, and on each question one run in three named a different agency first.

How to measure an answer that moves#

An April 2026 arXiv paper, “Don’t Measure Once” by Schulte, Bleeker and Kaufmann, concludes that brands need repeated measurements because answers “can vary across runs, prompts, and time, making one-off observations unreliable”. ChatGPT cited it in 6 of its 240 study answers.

  1. Freeze the prompt set. Start with 10 to 50 buyer prompts (how many to track). Done when each prompt’s wording, engine and model are saved with a start date.
  2. Run every prompt three or more times per engine in each check. In an app, use a new chat per run, with memory off or in a temporary chat that skips memory. For Google’s AI Overviews, repeat the search signed out, on the same device and from the same country each time. Done when each prompt has three or more dated runs per engine.
  3. Record whether each run searched the web. Apps usually show cited links or a sources list when they search; the API returns a flag (web_search in the DataForSEO results we used). A citation drop can mean a skipped search, not a lost source. Done when each report shows how many runs searched, per engine.
  4. Report rates, not screenshots. Write “named in 2 of 3 runs”, per engine. Done when every figure has a count and a denominator.
  5. Call a change real only when the next check agrees (how often to check). A move is a prompt crossing from 0 or 1 of 3 runs to 2 or 3 of 3, or back. Done when the crossing holds for two checks in a row; if it flips back, log it as noise.

Disclosure: Prefer sells AI visibility tracking, so “measure more than once” is advice we have a stake in. For a first baseline, run a free AI visibility audit.

Sources

  1. Across 80 questions asked three times each on ChatGPT, Claude, Gemini and Perplexity (DataForSEO API with web search enabled, 13 September 2026, 960 answers), two runs of the same question shared on average 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. Measure: for every pair of runs in which both answers cited a source, sites cited in both divided by sites cited in either, averaged over 206, 200, 179 and 240 pairs. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  2. Perplexity cited exactly the same set of sites in 199 of its 240 pairs of runs of the same question. In 39 of 206 pairs, ChatGPT's two answers to the same question had no cited site in common. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  3. Whether an engine searched the web changed between runs of the same question: Gemini and Claude each searched in some runs and not others on 13 of the 80 questions, ChatGPT on 4 and Perplexity on none. Perplexity searched on all three runs of every question, ChatGPT on 70 of 80, Gemini on 63 and Claude on 58. None of the 99 answers that skipped search cited a source. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  4. On 4 questions asking for AI SEO or GEO agencies (16 question and engine pairs, 13 September 2026), 100 of 270 agency names, counted once per question and engine, appeared in all three runs (37%), and the first agency named was not the same in all three runs in 11 of the 16 pairs. ChatGPT's agency lists moved most: on average, two runs shared 34% of the agencies named in either. On all four questions, Perplexity's three runs cited the same 20 pages in the same order, yet on average two runs shared 69% of the agencies named, and on each question one of the three runs named a different agency first. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  5. On 14 questions asking for AI visibility, AEO or GEO tools (56 question and engine pairs, 13 September 2026), 288 of 739 tool names, counted once per question and engine, appeared in all three runs (39%). (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  6. All three answers to each question came back within 23 seconds of each other, and each run was a fresh API call with no chat history. Models: gpt-5.6-luna, claude-sonnet-5, gemini-3.8-flash and sonar-pro. Answers from the API can differ from the consumer apps. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  7. Asked 'What is the best AI visibility tool?', ChatGPT's runs 1, 2 and 3, all returned within 15 seconds of each other, gave 'Best overall' to Profound, Semrush AI Visibility Toolkit and Profound respectively. Asked 'What are the best LLM SEO companies?', ChatGPT named 19 agencies across three runs and only Omniscient Digital in all three. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  8. 'Don't Measure Once: Measuring Visibility in AI Search (GEO)' by Julius Schulte, Malte Bleeker and Philipp Kaufmann (arXiv, submitted 8 April 2026) argues that AI answers vary across runs, prompts and time, so one-off observations are unreliable, and that a brand's visibility should be measured repeatedly and described as a distribution. arXiv: Don't Measure Once (Schulte, Bleeker and Kaufmann, 2026)
  9. ChatGPT cited 'Don't Measure Once' in 6 of its 240 answers in the study; no other engine cited it. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  10. Ronald Sielinski's 'Quantifying Uncertainty in AI Visibility' (arXiv, March 2026, revised August 2026) sampled Perplexity, OpenAI's SearchGPT and Google Gemini repeatedly and found that many apparent differences between domains fall within the noise floor of the measurement, and that single-run visibility metrics give a misleadingly precise picture. arXiv: Quantifying Uncertainty in AI Visibility (Sielinski, 2026)
  11. SparkToro's January 2026 study with Gumshoe.ai (600 volunteers, 12 prompts, 2,961 runs in the consumer apps of ChatGPT, Claude and Google's AI, November and December 2025) found under a 1 in 100 chance that ChatGPT or Google's AI gives the same list of brands in any two responses, about 1 in 1,000 for the same order, and suggested at least 60 to 100 runs to learn an AI's set of recommendations. SparkToro: AIs are highly inconsistent when recommending brands (27 January 2026)
  12. Google AI Overviews, same questions five days apart (DataForSEO Google SERP API, US English, desktop, same settings both days except result depth: 10 organic results requested on 13 September, up to 100 on 18 September): 70 of our 80 questions showed an AI Overview with sources on both 13 and 18 September 2026. Of the 602 pages cited on 13 September, counting each page once per question, 289 (48%) were cited again for the same question on 18 September. The 48% is a share of the 13 September pages, not of both days. Every one of the 70 questions kept at least one page. (Prefer research note: AI Overviews questions (2026-09-19), dated research note)
  13. Some answers held steady: asked for the best AI visibility tool for a small marketing team on a budget, ChatGPT, Gemini and Perplexity named the same pick, Otterly.AI, in all nine of their answers. (Prefer research note: repeatability in the AI answer study (2026-09-19), dated research note)
  14. Prefer sells AI visibility tracking, so we have a commercial stake in the advice to measure more than once. Prefer pricing (our product)

People also ask

  • Does ChatGPT give the same answer every time?
  • Do AI engines cite the same sources every time?
  • Do AI recommendations change each time you ask?
  • How consistent are AI search answers?
  • Is one AI visibility check enough?

Frequently asked questions.

Updated 19 September 2026

Do AI answers change if you ask the same question twice?

Yes, on most engines. Prefer asked 80 questions three times each on 13 September 2026: two runs of the same question shared 37% of the sites cited across both on ChatGPT, 35% on Gemini, 68% on Claude and 96% on Perplexity. Only 39% of the tools named appeared in all three runs.

Do Google AI Overviews change between searches too?

Yes. Prefer's main study captured each Google AI Overview once, on 13 September 2026, so we searched the same questions again on 18 September, with the same settings except result depth (10 organic results requested the first time, up to 100 the second). For the 70 questions that showed an AI Overview with sources both times, 289 (48%) of the 602 pages cited on 13 September, counting each page once per question, were cited again for the same question five days later. The 48% is a share of the 13 September pages, not of both days. Every question kept at least one page. That counts pages across five days, not sites across back-to-back runs, so it is not the same measure as our figures for the four chat engines. For brand lists, SparkToro's January 2026 panel tested Google's AI (AI Overviews, or AI Mode when no Overview showed) in the consumer app. The chance that two responses from Google's AI or ChatGPT listed the same brands was under 1 in 100, and across the three tools it tested, the same brands in the same order came up about once in 1,000.

How many runs are enough?

Three per engine is the floor Prefer's research uses for one check. It shows whether a result holds, not a precise rate. SparkToro suggested at least 60 to 100 runs to learn an AI's set of recommendations, and Ronald Sielinski's 2026 paper found many apparent gaps between sites fall inside the measurement's noise. Repeating a three-run check on a fixed schedule builds that sample over time.

Is the change caused by chat memory or personalization?

Not in Prefer's data. Each run was a fresh API call with no chat history, and all three answers to a question came back within 23 seconds, yet the cited sites and brand lists still changed. The consumer apps add memory and custom instructions on top, which our study did not test.

Get your free AI visibility report
in about 10 minutes.

See how answer engines describe your brand today, and where the openings are to outpace the competition.

Free AI visibility audit

Takes just 30 seconds. Your report in about 5 minutes.

Free. No signup, no card.