Skip to content

Why AI Answers Change Every Time You Ask (and How to Measure Anyway)

The same prompt names different brands on each run and from month to month. What causes it, how big the swings are in our data, and how to measure them.

By Citlyze Research. Published , updated . 6 min read.

Summarize withChatGPTClaudePerplexityGrokMarkdown
On this page

Ask ChatGPT the same question five times and you will get different answers, sometimes naming different brands entirely. This is normal model behavior, and it breaks the way most teams check their AI visibility. A screenshot proves nothing. The only honest measurement of AI visibility is a rate: out of N runs of a prompt, how many mentioned you.

Answers also move on a slower clock. Between two monthly runs of the same 200 prompts, the sources Perplexity cited for a given prompt overlapped by only 24% on average. Run-to-run jitter and month-to-month drift have different causes, and you measure them differently. Teams that treat visibility as a probability get trend lines they can trust; teams that collect screenshots end up arguing over anecdotes.

Four sources of volatility

Sampling in generation. Language models generate probabilistically. At standard settings, the same input legitimately produces different phrasings, orderings, and inclusions. Brand mentions near the model's internal decision boundary flicker in and out between runs.

Retrieval variance. When an engine searches the live web, small differences compound: which sub-queries the fan-out produces, which pages the index returns at that moment, which fetches succeed within time budgets. Different retrieved sets produce different answers, and different citations. How ChatGPT picks its sources walks through that retrieval step for the most used engine.

Continuous model and product change. Providers ship model updates, reroute traffic between model variants, and redesign answer surfaces without announcements. Yesterday's behavior is not a contract.

Context and personalization. Session history, location, and account state shade results. Your incognito check does not reproduce your buyer's session.

None of this is malfunction. The mistake is measuring a probabilistic system as if it gave one fixed answer.

Run to run, and month to month

The four sources above work on two timescales, and mixing them up leads to the wrong fix.

Jitter happens between identical runs on the same day. Sampling and retrieval drive it. You cannot remove it, only average over it.

Drift happens over weeks. The engine's index refreshes, providers swap models, new pages get published, and the source mix shifts underneath your category. Drift is the part that changes your strategy, because it can hand a citation slot to a competitor or take one away.

We ran the same 200 buyer prompts through ChatGPT and Perplexity on August 9 and again on September 26, 2026, one run per prompt each time. Between the two dates:

  • On Perplexity, a prompt's cited domains overlapped by 24% on average. On ChatGPT, 63% of the 101 prompts that carried citations in either month shared no cited domain at all.
  • Reddit fell from 13.5% of Perplexity answers to 1.5%, and ChatGPT went from citing Forbes in 11.5% of answers to none.
  • Perplexity's answers cited a median of 10 sources instead of 6, at the same length. That change came from the product, and it lifted almost every domain's share without anyone publishing anything.

One run per prompt per month supports statements about the whole sample, not about any single brand, so read these as the size of the tide. The full tables are in the Q3 AI citation sources study.

What single checks get wrong

Consider a brand that appears in 60% of runs for a key prompt. One check gives a 40% chance of concluding "we're invisible" and panicking, or a 60% chance of concluding "we're fine" and moving on. Both conclusions are wrong; the truth is the rate. It gets worse across time: check once this week (hit) and once next week (miss) and you will report a collapse that never happened.

Basic sampling arithmetic says how many runs you need. The margin of error on a measured mention rate shrinks with the square root of the number of runs. For a brand that truly appears half the time, the 95% margin looks like this:

Runs behind the rateMargin of error
5about ±44 points
10about ±31 points
20about ±22 points
50about ±14 points
100about ±10 points

The runs can come from one prompt repeated or from a group of related prompts read together. Either way, a rate built on five answers can't tell 30% from 70%. One run is a coin you flipped once.

How to measure a moving target

  1. Fix a stable prompt set. Trends require holding the instrument still. Build it from real buyer language, as covered in how to find buyer prompts, and resist rewording.
  2. Run every prompt multiple times per measurement window. Same prompt, same engine, repeated runs. The repetition is the methodology.
  3. Record mentions, positions, and citations per run. Whether you were named, how prominently, and which sources the answer cited. Citation patterns are steadier than mentions and often explain them, which is why we track citation rates alongside.
  4. Aggregate into rates with a window. "Mentioned in 7 of 10 runs this week" is a measurement. Weekly windows smooth daily jitter while staying responsive.
  5. Compare like with like. Rates per engine, per prompt group, per market. Averaging ChatGPT and Perplexity into one number hides more than it reveals.
  6. Alert on sustained change, not single misses. A real drop shows up across consecutive windows and usually across related prompts. That is the signal worth waking up for.

Three numbers that measure volatility itself

If you want to know how unstable an engine is for your category, and not just where you stand, track these alongside your mention rate:

  • Consistency. Of the brands that appear in any run of a prompt, the share that appears in every run. Low consistency means you need more runs before trusting a weekly number.
  • Citation overlap. How many cited domains two answers to the same prompt share, as a percentage of all the domains either one cited. Compare it run to run for jitter and window to window for drift.
  • Competitor co-movement. Whether your rate and your competitors' rates moved together. A shared move points at the engine. A move that only you made points at something that changed about you.
  • Confidence scales with volume. A rate built on 40 runs deserves more trust than one built on 4. Treat low-run readouts as directional.
  • Expect regression to the mean. Extreme weeks are usually followed by ordinary ones. Judge campaigns on multi-week windows, not the first reading after launch.
  • Segment before you celebrate. A flat overall rate can hide a win on comparison prompts cancelled by a loss on discovery prompts. Read by group.
  • Correlate with causes cautiously. Model updates move numbers with no action from you. When every brand in a category shifts at once, the engine changed, and the change explains itself.

We measured the volatility

In August 2026 we put numbers on the jitter: 30 buyer prompts, each run five times through ChatGPT with identical wording in fresh sessions. Of the brands that appeared in at least one of a prompt's five answers, only 68.8% appeared in all five; roughly one mentioned brand in three flickered in and out across identical runs. Citations were far less stable than mentions: not one prompt of the thirty returned the same source list twice, and the median overlap between any two same-prompt citation sets was under 5%. Full design and the aggregate results sit in our statistics roundup. One run really is a coin you flipped once.

Measurement rigor, built in

This methodology is Citlyze's core design: the daily engines answer every tracked prompt more than once per window, and the dashboard reports rates with the answer counts behind them, never one-off screenshots. For how the rest of the market handles sampling, see our AI rank tracker comparison. See prompt tracking and the scheduled reports that keep the trend in front of your team.

Keep reading