ChatGPT gives different answers because it generates text from likely continuations, may use web search, and responds to wording and conversation context. Model updates can also change results between checks. For a store measuring visibility, repeated runs under matching conditions help distinguish an isolated recommendation from a pattern worth investigating.
Why does ChatGPT give different answers between runs?
A language model generates tokens, which can be words or parts of words, from a distribution of likely continuations. Sampling settings can affect which continuation gets selected, although a consumer interface may not expose those settings to users, and different choices early in an answer can change the product names and ordering that follow, so matching prompts do not guarantee matching recommendations, even when the store has made no changes to its own pages.
For a buying question with one obvious answer, the variation is small and the same product tends to come first. In a crowded category, such as protein powders or air purifiers, several plausible products may compete for a place in the answer, which means a store in a crowded category should expect its position to move between runs even when nothing about its pages has changed.
Memory and web search answer from different places
An assistant answers either from what it learned in training or from pages it reads through web search when the shopper asks, and sometimes it blends the two. Here, memory means learned model knowledge, separate from saved personal preferences or information already present in the conversation. Web search can supply newer information, while changing search results and retrieved pages add another source of variation between checks.
The draft describes a Metrisque Question Finder case with no first-place mentions across 11 saved reads covering both modes, and that observation is a reason to investigate, although it cannot establish why the store was missing from first place. Separating modes helps keep differences in retrieved information visible, rather than hiding them inside a combined score that could send a store toward the wrong explanation.
Model updates move whole categories
Providers update their models on their own schedule, and an update can change answers for an entire category overnight. A new model may have learned from a newer slice of the web, weigh sources differently or follow a different default style for product lists. Stores sometimes see a sharp drop after an update and assume a competitor did something, when in fact every answer in the category moved.
The defence is to record which model and version produced each answer, wherever the provider shows it, and to read a change on all of your tracked questions before reacting, because a same-day shift across questions is consistent with a shared change, although it does not prove a model update occurred. A fall affecting one product deserves a closer look at its pages, competing products and the sources appearing in answers.
Wording and the exact name asked
Two questions that a person would treat as the same can return different answers. “Best sauna for a small apartment” and “compact home sauna recommendations” lead the model down different paths, because the words themselves shift which products look most relevant. Adding a constraint (“under 1,000 dollars”, “for beginners”) narrows the field again.
The exact name also matters when testing whether an assistant recognizes a brand, so record the string alongside every result. Test a short name, a full name and a domain separately if shoppers use all of them to find you. Different answers may reveal ambiguity, although recognition alone does not establish that the assistant would recommend the store for a buying question.
| Cause | What it changes | How to control for it when measuring |
|---|---|---|
| Sampling | Order and membership of the product list on each run | Repeat each question three or more times and record the spread |
| Memory versus web search | Which sources the answer draws on and how current they are | Run each question in both modes and report them separately |
| Model updates | Answers in a whole category at once | Record model and date for every read, compare all questions before reacting |
| Question wording | Which products look relevant to the question | Freeze the wording, test new phrasings as separate questions |
| Name asked | Whether the model recognizes the brand at all | Test each name variant separately, scope the verdict to that string |
Table 1. Fix wording and name variants, record model and search conditions, and use repeated runs to observe sampling variation.
Position spread
Because answers vary, a single position is one draw. We call the range of positions a product takes over repeated runs of the same question, under the same conditions, its position spread, so a product that comes first on four of five runs and second on the fifth has a narrow spread and a stable lead. A product that is first on one run and absent on the next has a wide spread, and a single good run tells you little about it.
Spread changes how a store reads a fix. Say a product page is rewritten and the next run puts the product first. If its spread before the fix already included first place, that run proves nothing. An improvement across three or more runs deserves another check against the baseline, while allowing for changes in search results and model behavior. Repeated improvement supports further investigation, but it cannot isolate the effect of the page edit without stronger controls over competing explanations.
Variation distorts share-of-voice reports
A share-of-voice report can summarize visibility as the percentage of sampled answers that mention a brand, making the chosen prompt set and repeat schedule part of the result. If a tracker runs each prompt once a day, the daily share is a sample of samples, and a move of a few points can come entirely from sampling and search churn. A report that shows a falling line for a week may be showing noise, a model update or a real loss, and the line alone cannot say which.
Three habits make a share report safer to read. Keep the prompt set fixed so wording does not change under the line. Report memory and web search separately, because they move for different reasons. Mark known model update dates on the chart, so readers can consider a possible explanation before rewriting a product page. With those in place, a store can treat a drop that persists over three or more runs and both modes as worth diagnosing, and leave a single bad day alone.
Repeat runs until the spread shows
No single number of runs is right, and anyone quoting one without knowing your category is guessing. For a manual check, three runs per question per mode are a starting point for spotting variation, without guaranteeing a reliable estimate, and for a decision that costs real money, such as rewriting a whole catalog’s descriptions, more runs on a smaller set of questions is the better trade than one run on each of a large set.
Record absence explicitly instead of silently dropping a run where your product disappears, because missing answers are part of the outcome. Keep the original answer text as well as its position, since a mention may be a comparison or warning rather than a recommendation. Our explanation of brand mentions versus recommendations helps separate those outcomes when recording results for a store.
Category checks answer a different question
The draft reports a Category Fit read covering 53 stores, with 13 of 923 products below a Gemini similarity threshold of 0.40. That threshold describes the check’s classification rule; it does not demonstrate that those products remain misclassified across repeated recommendation runs.
The draft also reports correct classifications for clear descriptions in 150 of 150 synthetic tests, compared with 116 of 150 for vague descriptions. These results concern classification under the reported test conditions, so they cannot establish that rewriting a description improves recommendation position. Consult the guide to measuring AI search visibility to keep classification, recall, citation and recommendation separate in your measurement plan.
How can I measure visibility when ChatGPT answers keep changing?
Measure visibility by repeating fixed buying questions under recorded conditions, then compare recommendation frequency and position across comparable batches. Keep each prompt’s wording, search mode and brand name unchanged within a batch, and start fresh chats to reduce conversational carryover.
Save the date, visible model label and full answer with each read, including answers where the product does not appear, and when search is available, retain the cited sources too, because a changed source can help explain a changed answer.
Use three repeats as an initial check, then collect more evidence before making a costly catalog-wide decision based on an apparent gain. A narrow position spread can describe consistent placement within those checks, while a wide spread warns against treating a favorable screenshot as the normal outcome for every shopper.
The figures quoted here come from the draft’s Metrisque internal-run summaries, dated 16 November 2026; underlying records were unavailable for this review. They provide context for the measurement questions above, with no independent verification of the reported results or experimental methods.
Run a free Metrisque check to establish a starting point for measuring your store’s visibility across repeated AI answers.



