Open the developer settings of almost any AI service and you meet a row of dials with names like temperature, top-p, top-k, frequency penalty and max tokens. They sound like engine tuning. Most of them do one simple thing: they change which word the model is allowed to pick next, and how likely each one is. This lesson goes through them one at a time, using the same example throughout.
It builds on our lesson How AI works: why the same question gets a different answer. The one idea you need from it: an AI chatbot works out the odds of every possible next word, then picks one, then does it again for the word after that. Training makes the dice; answering rolls them. Almost every setting below changes the dice before they are rolled.
Where each setting does its work
Writing an answer is a loop of six steps, run once for every word. Each piece of jargon belongs to one of them.
First, the words you need
Token
A model does not read words. It reads tokens: chunks that can be whole words, parts of words, single characters or even bytes. Common short words are usually one token each; long or unusual words get split into several. As rough guides, OpenAI says one token is about four characters, or three-quarters of a word, of English, and Google says 100 tokens is about 60 to 80 English words. Anthropic puts a Claude token at about 3.5 English characters. Each company splits text its own way, so the same paragraph comes to a different count on each.
Tokens matter because nearly everything is counted in them: the price of using a model, the limit on how much it can read, and the limit on how much it can write. From here on we will say “word”, because the principle is the same.
Context window
The context window is the most a model can take into account at once. Anthropic describes it as the text a model can look back on and refer to while writing — a “working memory”, separate from what it learned in training. Everything has to fit inside it: the hidden instructions, the conversation so far, any documents you attached, your question, and — on Google’s definition, which covers input and output together — the answer being written. When a long chat seems to forget how it started, this is usually why.
System prompt
The system prompt (Google calls it the system instruction) is the standing brief that sits in front of the conversation: context and instructions such as a goal or a role. “You are a helpful assistant for an accounting firm. Answer in Australian English.” Chat apps write their own and keep it hidden; developers write their own. It is ordinary text, and it uses up context window like any other.
Parameters — two meanings
This word causes more confusion than any other, because it means two different things.
- A model’s parameters are the billions of numbers set during training, also called weights. When you read that a model has “70 billion parameters”, that is what it means. You cannot change them by chatting; they are fixed when training ends.
- Sampling parameters are the dials on this page: temperature, top-p and the rest. They are settings on each request, not part of the model. Google’s machine-learning glossary calls temperature a hyperparameter to keep the two apart.
Using the model to answer, as opposed to training it, is called inference. Every setting on this page is an inference setting.
Logits and softmax
At step 2, the model gives every word it knows a raw score. Those scores are called logits — Google’s glossary defines them as the raw, not-yet-normalised predictions a model produces. They can be any number, positive or negative, and they do not add up to anything in particular.
A function called softmax turns them into proper odds: one probability for each possible choice, adding up to exactly 1. Bigger score, bigger share. Temperature, as the next section shows, does its work right here, by adjusting the scores just before they go through softmax.
The example we will use throughout
Say the text so far is “For the team’s morning tea, we ordered …” and the model’s odds for the next word look like this. The right-hand column is the running total, adding each word’s chance to the ones above it — you will need it for top-p.
“For the team’s morning tea, we ordered …”
Temperature: reshape the odds
Temperature makes the gaps between the odds bigger or smaller before the dice are rolled. It does not remove any word; it changes how lopsided the list is. Technically, the raw scores are divided by the temperature before softmax turns them into odds, as set out in Holtzman and colleagues’ paper on text decoding.
- Below 1 the favourite pulls further ahead. In our example, scones would climb well past 30 per cent and sushi would shrink towards nothing.
- At 1 the odds are used exactly as the model calculated them.
- Above 1 the odds flatten out, and the long tail gets a real chance.
- At 0 there are no dice at all: always the single favourite.
The ranges differ by provider. OpenAI accepts 0 to 2; Claude accepts 0 to 1, defaulting to 1; Gemini accepts 0 to 2. Our How AI works lesson draws the same five odds at four temperatures, if you want to see the shape change.
Top-k: keep a fixed number of words
Top-k keeps the k most likely words and throws the rest away. The survivors’ odds are then scaled back up to add to 100 per cent, and the dice are rolled among them. Anthropic’s documentation puts it simply: only sample from the top K options, to remove “long tail” low-probability responses.
With a top-k of 3, our example keeps scones, muffins and pastries. Between them they had 65 per cent, so each is scaled up: scones become 46 per cent, muffins 31 and pastries 23. Sandwiches, which had a perfectly respectable 10 per cent, can no longer be picked.
A top-k of 1 means always take the favourite, which Google’s documentation notes is also called greedy decoding. One early and widely cited use was a 2018 story-writing paper by Angela Fan, Mike Lewis and Yann Dauphin — presented at a conference in Melbourne — which sampled from the 10 most likely candidates to cut down the risk of very unlikely words.
Top-p: keep a fixed share of the odds
Top-p, also called nucleus sampling, keeps the smallest group of top words whose combined chance reaches p. Line the words up from most to least likely, add their odds as you go down, and stop as soon as the running total reaches the target. Everything below the line is cut. It was proposed in 2019 by Ari Holtzman and colleagues, who defined the kept group as the smallest set of words whose probabilities add up to at least p.
With a top-p of 0.7, look at the running total in our example: 30, 50, 65 — not yet 70 — then 75. So four words survive: scones, muffins, pastries and sandwiches, scaled up to 40, 27, 20 and 13 per cent.
Google gives a tidy example in its own documentation: if the top three words have chances of 0.3, 0.2 and 0.1 and top-p is 0.5, the first two are kept and the third is excluded. A top-p of 1 keeps everything, which switches it off.
Top-k or top-p? Why a fixed number goes wrong
On one example the two look interchangeable. The difference shows when the shape of the odds changes — which it does with every word. Here are the two prompts from our earlier lesson, one where a single answer dominates and one where many are reasonable, with a top-k of 5 and a top-p of 0.9 applied to each.
“The capital of Australia is …” — one clear favourite
“My favourite colour is …” — many good answers
That is exactly the argument Holtzman and colleagues made for top-p. When the odds are spread out, a small k cuts off good options; when one word dominates, a large k lets in poor ones, so a single fixed k is a poor fit across different situations. Top-p adjusts the size of the list to the shape of the odds. In fairness to top-k, the same paper found it produced considerably better text than sampling from the full, untrimmed list. It is the fixed number that is the weakness, not the idea of trimming.
Min-p: keep anything close enough to the favourite
Min-p is a newer trim, mostly found in tools for running open models yourself. Instead of a count or a share, it sets a bar relative to the favourite: a word is kept if its chance is at least p times the favourite’s. Its authors describe it as using the top word’s probability as a scaling factor for the cut-off, so the bar rises when the model is confident and falls when it is not.
With a min-p of 0.25 in our example, the favourite has 30 per cent, so the bar is a quarter of that: 7.5 per cent. Five words clear it — scones, muffins, pastries, sandwiches and doughnuts — and become 36, 24, 18, 12 and 10 per cent.
The paper’s authors (first published in 2024, presented at ICLR 2025) suggest values between 0.05 and 0.1, and report that it improves both quality and variety compared with top-p, especially at high temperatures. Those are the authors’ own findings. The Hugging Face library describes typical values of 0.01 to 0.2.
The three trims side by side
| Word (original chance) | Top-k 3 | Top-p 0.7 | Min-p 0.25 |
|---|---|---|---|
| scones (30%) | 46% | 40% | 36% |
| muffins (20%) | 31% | 27% | 24% |
| pastries (15%) | 23% | 20% | 18% |
| sandwiches (10%) | cut | 13% | 12% |
| doughnuts (8%) | cut | cut | 10% |
| fruit, sushi and the rest | cut | cut | cut |
| The rule | the top 3 words | top words until the total reaches 70% | words with at least a quarter of the favourite’s chance |
After trimming, the dice are rolled among the survivors using the scaled-up odds. Every trim makes the long shots impossible rather than merely unlikely, which is the whole point: a one-in-ten-thousand word cannot sneak in and send the answer somewhere strange.
How the settings stack
The settings are applied one after another, and the order depends on the system. Google’s documentation describes top-k first, then top-p, with the final word chosen using temperature. The min-p paper applies its cut-off after the temperature has reshaped the odds. Because each setting changes what the next one sees, turning two dials at once makes it hard to tell which did what.
The providers say so. OpenAI’s API reference says “We generally recommend altering this or top_p but not both”, and some Claude models go further and reject a request that sets both temperature and top-p. Change one, see what happens, then decide.
Greedy decoding and beam search: no dice at all
Greedy decoding always takes the single most likely word. It is the same as a temperature of 0 or a top-k of 1, and in the popular Hugging Face library it is the default unless you switch sampling on. That library’s own guide notes that greedy output tends to start repeating itself on longer passages.
Beam search is a more thorough version. Rather than committing to one word at a time, it keeps several candidate sentences going at once and returns the one with the highest overall probability. It suits jobs with one right-ish answer, such as translation or describing an image. For open-ended writing it does badly: Holtzman and colleagues found that aiming for the most probable text produces bland, repetitive output, which is a large part of why chatbots roll dice at all.
Penalties: discourage repeats
Penalties lower the odds of words that have already been used, so the model is less likely to go round in circles. They act at step 3, before any trimming.
- Presence penalty — a one-off deduction for any word that has appeared at all, however many times. OpenAI describes it as increasing the model’s likelihood to talk about new topics.
- Frequency penalty — a deduction that grows with every repeat, so a word used five times is pushed down harder than one used once. OpenAI describes it as decreasing the likelihood of repeating the same line word for word.
- Repetition penalty — the version found in open-source tools. It was proposed by Salesforce researchers in 2019, who discounted the scores of words already generated and found a value of about 1.2 worked well. A value of 1 means no penalty.
OpenAI accepts values from −2 to 2 for both of its penalties and suggests around 0.1 to 1 as reasonable, warning that values near 2 can noticeably degrade the output. Negative values do the opposite and encourage repetition. Use penalties gently: words like “the” and your customer’s name need to repeat.
Logit bias: push or ban particular words
Logit bias lets a developer adjust the score of specific tokens before the odds are worked out. OpenAI accepts values from −100 to 100, where the extremes should in effect ban the token or make it the only choice. It works on tokens, not words, so banning a word reliably means banning every way it can be spelled out in tokens — which is why it is a specialist tool.
Max tokens and stop sequences: when the answer ends
Max tokens is the most the model may write in one reply. It is a ceiling, not a target; Anthropic notes its models may stop before reaching it. Set it too low and the answer simply stops when the count runs out, often mid-sentence. On services that let the model think before answering, the limit can include that hidden reasoning too: OpenAI’s limit covers visible output tokens and reasoning tokens.
A stop sequence is a piece of text that ends the answer the moment it appears — handy when you want exactly one item, one line or one section. OpenAI allows up to four, and leaves the stop sequence itself out of the returned text; Gemini allows up to five. Google’s sensible advice: avoid a sequence that might turn up in the answer itself, or the reply ends early.
Seed: the starting point for the dice
Computers make “random” numbers from a formula started by a number called the seed; the same seed gives the same run of rolls. Some services let you fix it so repeat requests come out the same — but only on a best-effort basis. OpenAI’s reference now marks its seed setting as deprecated and says determinism is not guaranteed; Google’s says the same of its own: deterministic output isn’t guaranteed. Our How AI works lesson explains why even a fixed seed and a temperature of 0 still let the odd difference through.
Which services let you change what
In the everyday ChatGPT, Claude and Copilot apps you will not find any of these settings: the provider chooses them for you. They are for developers who connect to a model directly. Even there, the trend is towards fewer dials, not more.
| Service | What you can set | On the newest models |
|---|---|---|
| Anthropic (Claude API) | Temperature (0–1), top-p, top-k, max tokens, stop sequences, system prompt | Claude Opus 4.7 and later reject any non-default temperature, top-p or top-k; Anthropic recommends prompting instead |
| OpenAI (API) | Temperature (0–2), top-p, frequency and presence penalties, logit bias, stop sequences, max tokens, seed (deprecated). No top-k. | OpenAI’s guide says to remove temperature and top-p when the model is set to reason |
| Google (Gemini API) | Temperature (0–2), top-p, top-k (not on every model), presence and frequency penalties, stop sequences, max output tokens, seed | Google strongly recommends leaving Gemini 3 at its default temperature of 1.0, warning that lower values can cause looping or worse answers |
| Open models you run yourself (for example with Hugging Face Transformers) | All of the above, plus min-p, repetition penalty and beam search | Library defaults are temperature 1.0, top-k 50 and top-p 1.0 unless the model sets its own |
The details in this table were checked on 2 October 2026 and change often — check the provider’s documentation before you build on them. The principle underneath does not change: these are all ways of reshaping and trimming the same list of odds.
Why are the dials disappearing? The newest models are tuned to work best at their defaults, and the providers would rather you steer them with words. Anthropic’s migration guide is blunt: prompting is the recommended way to guide model behaviour, and temperature 0 never guaranteed identical output anyway.
Sensible starting points
- Start from the defaults. They are what the provider tested. Change something only when you can say what is wrong with the output.
- Change one setting at a time — temperature or top-p, not both.
- For sorting, extracting and yes-or-no answers, go low, where the model allows it. Anthropic suggests closer to 0 for analytical and multiple-choice work, closer to 1 for creative work. Low settings make answers more alike; they do not make them more correct.
- On a model that will not let you change them, use words. A clear brief, a fixed format and an example of a good answer narrow the odds more reliably than any dial.
- Set max tokens generously and handle the cut-off case. An answer that stops mid-sentence is worse than a long one.
- Do not rely on a seed for identical output. If you need the same answer every time, save it.
The terms at a glance
| Term | In one line |
|---|---|
| Beam search | Keeps several candidate sentences going and returns the most probable one overall. Good for translation, poor for open writing. |
| Context window | The most text a model can take into account at once, in tokens. |
| Frequency penalty | Lowers a word’s odds more each time it is repeated. |
| Greedy decoding | Always take the most likely word. No dice. |
| Inference | Using a trained model to produce an answer, as opposed to training it. |
| Logit bias | Push particular tokens up or down, or ban them outright. |
| Logits | The raw scores the model gives every possible next token. |
| Max tokens | The most the model may write in one reply. A ceiling, not a target. |
| Min-p | Keep words with at least a set fraction of the favourite’s chance. |
| Parameters (weights) | The billions of numbers fixed during training. Not the same as the settings on this page. |
| Presence penalty | A one-off deduction for any word already used, nudging towards new topics. |
| Repetition penalty | The open-source version: discounts every word already generated. 1 means off. |
| Sampling | Picking the next word by a weighted random draw from the odds. |
| Seed | The number that starts the random draws. Fixing it helps repeatability; it does not guarantee it. |
| Softmax | Turns raw scores into odds that add up to 100 per cent. |
| Stop sequence | Text that ends the answer the moment it appears. |
| System prompt | The standing instructions placed in front of the conversation. |
| Temperature | Makes the odds more lopsided (low) or more even (high). Zero means always the favourite. |
| Token | The chunk of text a model reads and writes — roughly three-quarters of an English word. |
| Top-k | Keep only the k most likely words. |
| Top-p (nucleus sampling) | Keep the fewest top words whose chances add up to p. |
The practical lesson
For most people at work, these dials are not the lever — the providers have set them, and on the newest models they cannot be moved. The lever is the instruction: the clearer the brief, the narrower the odds. That is a skill a team can learn in an afternoon.
We run hands-on AI workshops for teams, at your office, in Microsoft 365 Copilot, Claude or ChatGPT, built around your real work.
Common questions
What is top-p in AI?
What is top-k in AI?
What is the difference between top-p and top-k?
Should I change temperature and top-p at the same time?
What temperature should I use?
Can I change temperature or top-p in ChatGPT, Claude or Copilot?
What is the difference between frequency penalty and presence penalty?
What does max tokens mean?
What is a context window?
Why don’t the newest AI models let you change the temperature?
Sources
All read on 2 October 2026.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi, The Curious Case of Neural Text Degeneration (ICLR 2020) — nucleus (top-p) sampling defined; the case against a fixed k; temperature applied to the scores before softmax; most-probable decoding producing bland, repetitive text
- Angela Fan, Mike Lewis and Yann Dauphin, Hierarchical Neural Story Generation (ACL 2018, Melbourne) — top-k random sampling from the 10 most likely candidates
- Minh Nhat Nguyen and colleagues, Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025) — min-p defined, applied after temperature, suggested values of 0.05 to 0.1
- Nitish Shirish Keskar and colleagues, Salesforce Research, CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) — the repetition penalty, and a value of about 1.2
- Hugging Face, Generation strategies — greedy search as the default, its tendency to repeat, and beam search
- Hugging Face, Generation (GenerationConfig reference) — library defaults for temperature, top-k and top-p; min-p’s typical range; repetition penalty of 1 meaning none
- Google for Developers, Machine Learning Glossary — the entries for logits, softmax and temperature
- Anthropic, Messages API reference — temperature (0 to 1, default 1, and when to go low or high), top-k, top-p, max tokens, stop sequences and the system prompt
- Anthropic, Glossary — tokens (about 3.5 English characters for Claude) and the context window as working memory
- Anthropic, Migrating to Claude Opus 5.5 — non-default temperature, top-p and top-k rejected on Claude Opus 4.7 and later; prompting recommended instead
- Anthropic, Migrating to Claude Haiku 4.5 — temperature or top-p, not both
- OpenAI, Create chat completion (API reference) — temperature range, top-p, “not both”, frequency and presence penalties, logit bias, stop sequences, max completion tokens and the deprecated seed
- OpenAI, Advanced usage — about 4 characters or 0.75 words per token; how the two penalties differ, and reasonable values
- OpenAI, Using the latest model — remove temperature and top-p when reasoning is on
- Google, Prompt design strategies (Gemini API) — top-k, top-p, the order they are applied in, greedy decoding, the top-p worked example and advice on stop sequences
- Google, Understand and count tokens (Gemini API) — about 4 characters per token, 100 tokens to 60–80 words; the context window covers input and output
- Google, Gemini 3 developer guide — keep temperature at the default 1.0
- Google, Generating content (API reference) — temperature range, top-k availability, stop sequences (up to 5) and penalties
- Google Cloud, Experiment with parameter values — the seed setting as best effort