How AI works: why the same question gets a different answer.

An AI chatbot such as ChatGPT, Claude or Copilot does one thing, over and over: it works out how likely every possible next word is, and then it picks one. The first half of that is deterministic — fixed arithmetic that gives the same result every time. The second half is non-deterministic — a deliberate, weighted roll of the dice. That is why you can ask the same question twice and get two different answers, even though the model finished training long ago and nothing inside it has changed since.

Two kinds of machine: deterministic and non-deterministic

A deterministic system gives the same output for the same input, every time. A calculator is the obvious one: 2 + 2 is 4 today, tomorrow and on every calculator ever made. A spreadsheet formula is deterministic. So is the checksum that tells you a file has not been damaged — the whole trick depends on the same file always producing the same fingerprint. Nearly all traditional software works this way, and when it does not, we call it a bug.

A non-deterministic system can give different outputs for the same input. Rolling dice. Shuffling a deck. Asking three colleagues to draft the same email. Nothing is broken when the results differ; the variation is part of how the thing works.

A chatbot looks like the first kind — it is software, running on a computer — but it behaves like the second. To see why, look at what it actually does with your question.

What a language model actually does

The engine inside a chatbot is called a large language model, or LLM. Its job is much narrower than it appears: given some text, predict what comes next.

Your text is first chopped into tokens — pieces about the size of a short word, or part of a longer one. (We will say “word” from here on; the principle is identical.) The model takes everything written so far and produces a score for every token it knows — typically somewhere between tens of thousands and a couple of hundred thousand of them. Those scores are turned into probabilities: how likely each one is to come next.

Then one is chosen, added to the end of the text, and the whole process runs again for the word after that. A 300-word answer is that loop, run about four hundred times.

Notice what is missing: there is no step where it looks the answer up. There is no table of questions and replies inside. Every answer is built on the spot, one piece at a time, and each piece is chosen from a list of odds.

“But it has finished training — why isn’t the answer fixed?”

This is the natural objection, and the answer to it is the heart of this page.

Training is the process that sets the model’s numbers — billions of them, called weights or parameters. The model is shown an enormous amount of text with the next word hidden, asked to guess it, and nudged a little each time it is wrong. When training ends, the numbers are frozen. They are a file. They do not change when you chat to it. (A model seems to remember your conversation only because the earlier messages are fed back in as text with every new question.)

So the trained model is fixed — and, in principle, deterministic. The same text in gives the same list of odds out.

But look at what training taught it to produce. Not an answer. A list of odds. Language almost never has a single correct next word. “My favourite colour is…” can be finished a dozen sensible ways. The model learned from millions of people finishing sentences differently, and it captured that spread faithfully. What got trained in is not “the answer” but “the range of reasonable answers, and how likely each one is”.

The simplest way to hold it in your head: training makes the dice; answering rolls them. A weighted die is a perfectly fixed object. It is the same die on every roll. The number that comes up is not.

“The capital of Australia is …”

• Canberra97%
• Sydney2%
• everything else1%

“My favourite colour is …”

• blue31%
• green17%
• red13%
• purple11%
• black7%
• everything else21%

The roll of the dice

The step that picks a word from the odds is called sampling. It does not simply take the favourite. It draws at random, weighted by the odds: a word with a 30 per cent chance gets picked about thirty times in a hundred.

Why not just take the most likely word every time? Because the result reads worse. Always choosing the favourite tends to produce flat, repetitive writing, and with earlier models it produced text that got stuck repeating itself. A measure of randomness is what makes the output read like natural language — and it is why pressing “regenerate” gives you a different draft instead of the same one again.

And one different pick does not stay one different pick. Each chosen word is added to the text and becomes part of the input for every word after it. A different word early on sends everything that follows down a different road.

Temperature: the dial on the dice

How adventurous the sampling is gets controlled by a setting called temperature. It reshapes the odds before the dice are rolled.

• Low temperature exaggerates the favourite. Likely words become even more likely; long shots almost vanish. The output is more predictable and more alike from one try to the next.
• A temperature of 1 uses the odds exactly as the model calculated them.
• High temperature flattens the odds, so unlikely words get a real chance. The output is more surprising — and, pushed far enough, incoherent.
• Zero means no dice at all: always take the single most likely word.

Two related settings, top-p and top-k, cut off the long tail of very unlikely words before the dice are rolled, so a one-in-a-million word never gets picked by accident. Our lesson on AI technical terms works through both, and the other settings, step by step.

In a chat app you never see any of this; the provider chooses the settings for you. Developers who connect to a model directly can usually adjust them, though some of the newest models no longer let anyone change the temperature.

One more piece of jargon. Computers cannot really roll dice. Their “random” numbers come from a formula that is started off by a number called a seed; the same seed gives the same run of rolls. Chat apps use a fresh seed every time. Some developer services let you fix it — which makes repeat runs match most of the time, and only most of the time.

Turn the dice off, and it still is not guaranteed

If a temperature of zero means “always take the top word”, then surely the same input must give the same output? Mostly. Not reliably. The companies that make these models say so themselves: Anthropic’s documentation notes that even at a temperature of zero the results will not be fully deterministic, and OpenAI describes its fixed-seed option as a best-effort feature, with determinism not guaranteed.

There are four reasons, and only the first is about arithmetic.

1. Computers round, and the rounding depends on the order

Computers store numbers to a limited number of digits and round off after every step. One consequence surprises most people: the order in which you add numbers up changes the answer, very slightly.

A model does billions of these operations for every word, on chips that split the work across thousands of parallel paths. Exactly how the work is split depends on the hardware, and on how many other people’s requests happen to be processed in the same batch as yours — which changes from moment to moment. So the scores come out a hair different from one run to the next.

Usually that changes nothing. But when two candidate words are almost tied, a hair is enough to swap which one is on top — and from there the snowball does the rest.

This has been measured. In 2025, Horace He and colleagues at Thinking Machines Lab asked one openly available model the same question 1,000 times with the temperature at zero. They got 80 different answers. Every one of them was identical for the first 102 tokens; at the 103rd, 992 runs wrote “Queens, New York” and 8 wrote “New York City”, and after that they drifted apart. The same work showed that the effect can be engineered away, at some cost in speed — it is a trade-off the providers make, not a law of nature.

2. The model behind the name changes

Providers replace and update their models. The name on the product stays the same; the numbers underneath do not. An answer you got in March may simply be unavailable in October, because the model that wrote it has been retired.

3. The input was not really the same

You typed the same question. The model did not receive the same text. A chat app wraps your words in a good deal of material you never see, and most of it can change between one try and the next.

4. Longer chains have more places to diverge

Newer assistants do more than write one reply. They work through a problem step by step before answering, search the web, open files and run tools. Every one of those steps is its own run of the loop, with its own chances to go a different way — and each feeds the next.

Where the variation comes from, at a glance

Generators and classifiers: not all AI rolls the dice

Everything above is about chatbots, and chatbots belong to one of two broad families of AI. Which family a model is in tells you whether to expect the same answer twice.

• Generators make something new: text, pictures, audio, video, computer code. Chatbots and image generators are the familiar ones.
• Classifiers judge something that already exists. Is this email spam? Is this payment suspicious? What is in this photo? What is this house likely to sell for?

The formal names are generative models and discriminative models; in business writing you will more often see generative AI and predictive AI. Strictly, a classifier is a model that sorts things into categories, and one that puts a number on something — a price, a risk score, next month’s sales — is a regression model. The two work the same way for our purposes, so we will call the whole family classifiers.

Generators: different every time, by design and by circumstance

A generator does not give the same output twice, and there are two sets of reasons.

By design. There is no single right email, and no single right picture of “a dog on a beach”, so a generator is built to choose among many good ones. A chatbot draws each word from the odds, as described above. Most image generators start each picture from a field of random noise, which is why the same prompt gives a different picture every time. (Tools that let you fix the seed will usually give the same picture back on the same software and hardware. On a different machine, even that is not guaranteed.)

By circumstance. Everything in the section on turning the dice off applies as well: arithmetic that rounds differently from run to run, a model that is replaced behind the same name, hidden input that changes, and long chains of steps. Take the deliberate randomness away and those remain.

Classifiers: can be the same every time

A classifier can be deterministic, and most are, because nothing in the job calls for dice. A spam filter is a trained model too. Its numbers were frozen when training ended, and it also turns its input into odds: spam 96 per cent, not spam 4 per cent. But it never draws from those odds. It takes the top answer — or checks the score against a fixed cut-off — and stops. Training made the dice, and nobody rolls them. Show it the same email a thousand times and it gives the same verdict a thousand times.

“Can”, not “always will”. It holds on these conditions.

• The same version of the model. Retrain it on new data and it is a different set of numbers. The email it let through last month may be flagged this month.
• Training has its own dice. Many kinds of model use randomness while they are being trained — random starting numbers, examples seen in a shuffled order — so training twice on the same data can give two slightly different models. Each one, once made, is fixed.
• The rounding still happens. The scores can come out a hair different on different hardware, exactly as they do for a chatbot. Here it almost never matters: there is one decision and no snowball, so only an input sitting right on the cut-off can tip.
• Nobody built variety in on purpose. Some systems do. A recommendation engine may deliberately mix in items it is less sure you will like — a technique called exploration — to keep its suggestions fresh and to find out what else you like.
• It is not a generator doing a classifier’s job. Ask a chatbot to sort emails into spam and not spam and it writes its verdict one word at a time, dice and all. It will usually agree with itself. Not always.

And deterministic is not the same as correct. A filter that wrongly flags a supplier’s invoice today will wrongly flag it again tomorrow. What determinism buys is the ability to test: check the model against examples you already know the answer to, and what you tested is what you get.

The two families side by side

A feature or a flaw?

Both. It depends on the job.

For writing, ideas and rephrasing, the variety is the point. You want a second draft to be different from the first. You want ten name suggestions, not the same one ten times. A model with no randomness would be a duller writing partner.

For anything that has to come out the same twice — a total, a date, a yes-or-no on a policy, a step in a process that runs every day — it is a liability, and you need to work around it.

What to do about it

• Treat an answer as a draft, not a record. If you are going to rely on something an AI wrote, save it. You may never get the identical wording again.
• Ask twice when it matters. Open a fresh chat and ask the same question. Agreement is not proof, but disagreement on a fact or a figure is a warning that the model is unsure.
• Give deterministic jobs to deterministic tools. Arithmetic, totals, dates and look-ups belong in a spreadsheet, a calculator or a database. The better assistants now do this themselves: rather than predicting what a sum probably comes to, they write a small program, run it, and report the result. The program is deterministic even though the thing that wrote it is not.
• Narrow the odds. The more specific your instructions — the format, the length, an example of what good looks like, the document to work from — the fewer plausible ways there are to continue, and the more alike the answers become.
• If it is built into a process, engineer for it. Fix the exact model version, keep the wording of the prompt identical, turn the temperature down and set a seed where the service allows, and check the output automatically. Then still expect the occasional odd one.
• Do not confuse consistent with correct. A model can give the same wrong answer every time. Repeatability tells you the model is confident; it does not tell you it is right.

Common questions

Sources

All read on 1 October 2026.

• Anthropic, Messages API reference — the temperature setting, and the note that results are not fully deterministic at zero
• OpenAI Cookbook, How to make your completions outputs reproducible with the new seed parameter
• Horace He and Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (10 September 2025) — the 1,000-run test and the cause of the rounding differences
• Google for Developers, Background: What is a Generative Model? — generative and discriminative models defined
• Rina Diane Caballar, IBM, Generative AI vs. predictive AI: What’s the difference? (9 August 2024)
• Google for Developers, Machine Learning Glossary: ML Fundamentals — classification models predict a class, regression models predict a number
• Google for Developers, Thresholds and the confusion matrix — how a spam model’s probability is turned into a verdict by a fixed threshold
• Hugging Face, Diffusers documentation, Reproducibility — image generation starts from random noise; a fixed seed helps, but identical output is not guaranteed
• PyTorch documentation, Reproducibility — results are not guaranteed to match across platforms, or between CPU and GPU, even with the same seed
• scikit-learn documentation, Common pitfalls and recommended practices — the “Controlling randomness” section, on models that train differently each time unless the seed is fixed
• James McInerney and colleagues, Spotify, Explore, Exploit, Explain: Personalizing Explainable Recommendations with Bandits (RecSys 2018) — exploration in recommendation engines
• Amazon Web Services, Amazon Personalize documentation, User-Personalization recipe — exploration as a setting in a commercial recommendation service