What Are Top-P and Top-K in LLMs? How Models Pick Tokens

Top-P and Top-K are truncation parameters that control how a large language model (LLM) picks its next token from a probability distribution. They are filters: they decide which candidates survive before the model makes its final draw.

Generation is auto-regressive, one token at a time, each conditioned on everything before it, and at every step the model scores its entire vocabulary, often more than 100,000 tokens. Without truncation you get one of two failure modes: repetitive, degenerate loops from greedy decoding, or incoherent output from sampling freely across the whole vocabulary.

Neither parameter teaches the model anything new. They only decide how far it may reach into what it already knew.

A language model narrowing hundreds of thousands of candidate tokens down to the one it picks

From logits to probabilities: what happens before the model picks a token

Before any token is selected, your transformer's final linear layer produces a raw score for every token in the vocabulary. These are logits. They can be negative, they can exceed 1.0, and they do not sum to anything meaningful, which makes them unusable for sampling as they stand.

Raw logits passing through Softmax to become a probability distribution summing to 1

The Softmax function converts them into a real probability distribution. It exponentiates each logit, making every value positive and stretching the gaps between them, then divides by the total so everything sums to 1.0:

text

P(x_i) = exp(l_i) / Σ(j=1..V) exp(l_j)

Here V is the vocabulary size and l_i is the logit for token i. This distribution is the landscape every sampling decision runs on. But picking the single highest-probability token at every step, greedy decoding, performs worse than intuition suggests. Holtzman and colleagues showed that human text does not consistently choose the highest-probability word; real language needs some surprise in it, and a model optimised purely for likelihood produces bland, looping prose.

Sampling freely from the full distribution fails in the opposite direction. The "long tail" holds tens of thousands of very low-probability tokens. Individually negligible, their combined probability mass is large enough that one eventually gets drawn, and a single bad draw is enough to derail a sentence. Top-P and Top-K exist to cut that tail off.

What is Top-K, and where does it cut?

Top-K (Fan and colleagues, 2018) is a hard cutoff based on count. The model sorts every token by probability in descending order, keeps the top K, and sets the probability of everything else to zero. After the cut, the surviving probabilities are renormalised so they sum to 1.0 again.

Top-K sorting tokens by descending probability and cutting hard at position K

Set K = 1 and you have greedy decoding. Set K = 50 and the model draws from the fifty strongest candidates. In pseudocode:

python

# Top-K filter, then renormalise
probs = softmax(logits)
top_k_indices = np.argsort(probs)[-K:]
 
filtered_probs = np.zeros_like(probs)
filtered_probs[top_k_indices] = probs[top_k_indices]
 
# The step that matters: renormalise so the total is 1
normalized_probs = filtered_probs / np.sum(filtered_probs)

The core weakness is the static K problem. In a high-certainty context such as "The capital of France is...", nearly all the probability mass sits on one or two tokens, and forcing the model to consider fifty candidates simply admits noise. In an open-ended context — "The future of this company is..." — there may be hundreds of reasonable continuations, and K = 50 cuts good options off. One fixed number cannot be right for both, because it has no way of knowing whether the model is confident or uncertain at this particular step.

What is Top-P (nucleus sampling), and why does it stretch and shrink?

Holtzman and colleagues (2020) introduced Top-P, or nucleus sampling, to fix exactly that rigidity. Instead of a fixed count, Top-P sorts tokens by probability and accumulates them into a "nucleus" until their cumulative probability reaches a threshold p of, say, 0.95. Everything outside the nucleus is dropped, and what remains is renormalised.

The Top-P nucleus shrinking when the model is confident and expanding when it is uncertain

The point is that the candidate pool adapts. When the model is confident, the nucleus shrinks: if the leading token already holds 96% and your threshold is 0.95, exactly one token survives. When the model is uncertain, the nucleus expands, potentially to hundreds of tokens, until it has gathered 95% of the mass. This is precisely what Top-K cannot do, because Top-P measures confidence, while Top-K only counts cardinality.

That adaptiveness is why Top-P became the default across most commercial APIs. It lets the model range widely when several continuations are genuinely plausible, and tightens automatically when the next token is obvious. It has a failure mode of its own, though, and it shows up exactly when you would least want it to, and the section below gets to it.

How is temperature different from Top-P and Top-K?

Temperature (T) is not a filter. Where Top-P and Top-K are scissors, temperature is a reshaping dial that divides the logits by T before Softmax. What is LLM temperature works through that formula and through what to set it to for each kind of task; here we only need to see how it differs from the two truncation knobs.

Think of it as contrast. With T < 1 the gaps between probabilities are exaggerated, strong tokens get stronger, and the model becomes close to deterministic. With T > 1 the distribution flattens, giving weaker tokens more room and trading coherence for variety. Concretely, a token sitting at 68.6% at T = 1.0 climbs past 99% at T = 0.2, while a token at 0.5% rises to around 4% at T = 2.0.

One distribution shown at three temperatures, with the token ranking unchanged

Two properties matter more than the numbers. First, temperature removes nothing: even at very low settings, every token in the vocabulary retains a non-zero probability, which is why you still need Top-P or Top-K to actually eliminate candidates. Second, because l_i / T is applied uniformly to every logit, the ranking never changes, so the most likely token at T = 0.2 is the most likely token at T = 2.0. That is the honest answer to the "turn temperature up for creativity" advice: it does not add anything the model did not already have, it only widens how far down its own ranked list the model is willing to reach.

Order of application: why the same settings give different output

The sequence in which samplers run changes the output, and it is a common source of production bugs. A standard chain runs like this:

Penalties (repetition, frequency, presence) run first.
Truncation (Top-K, Top-P, Min-P) removes weak candidates.
Temperature reshapes whatever survived.
Sampling draws the final token.

The sampler chain: penalties, truncation, temperature, then sampling, and the llama.cpp versus HuggingFace split

Not every framework agrees on that order. llama.cpp applies temperature last; HuggingFace Transformers typically applies it before truncation. That difference is not cosmetic. Because temperature changes the relative sizes of the probabilities, running it first changes which tokens fall inside the Top-P nucleus. The same top_p = 0.95 on two frameworks can therefore yield two different candidate sets, which is exactly the kind of discrepancy that surfaces when you port a working configuration from one runtime to another.

Google documents the order explicitly for Gemini: the top-K tokens with the highest probabilities are sampled first, then filtered further by top-P, with the final token selected using temperature sampling. If you are tuning against a default without knowing how your own stack sequences these stages, you are adjusting a parameter without knowing what it acts on.

When Top-P breaks, and what min-p was made for

Top-P fails at the worst possible moment: when the model is genuinely confused. On a flat distribution, where no token stands out, reaching a 0.95 cumulative threshold can require pulling in hundreds or thousands of low-quality tokens. The perverse result is that the less certain the model is, the more noise Top-P admits — the opposite of what you want.

On a flat distribution Top-P admits noise tokens while min-p uses a threshold scaled to the strongest token

Min-P (Nguyen and colleagues, 2025) fixes this by making the threshold relative rather than cumulative. It is derived from the strongest token:

text

threshold = min_p × P(top token)

If the leading token holds 90% and min_p is 0.1, the threshold is 0.09, which is strict. If the leading token holds only 5%, the threshold drops to 0.005, which is permissive. The filter tightens when the model is confident and loosens when it is not, which is the inverse of Top-P's behaviour on a flat distribution.

Top-n-sigma (Tang and colleagues, 2025) attacks a different coupling. It cuts using the standard deviation of the logits: threshold = max(l) - n × σ(l). Because that runs on raw logits rather than post-Softmax probabilities, the truncation boundary is temperature-invariant. You can turn randomness up or down without inadvertently widening the candidate pool, something neither Top-P nor Top-K guarantees.

Every provider does it differently: Anthropic deprecates, Google keeps

The two largest providers are moving in opposite directions, and that matters more than most tuning guides you will find.

Anthropic marking the sampling parameters deprecated while Google keeps topK and topP

At Anthropic, all three parameters are now marked deprecated in the official API reference. Models released after Claude Opus 4.6 do not support setting temperature (only 1.0 is accepted, for backwards compatibility), do not support top_p (only values of 0.99 or above are accepted), and reject any top_k value outright. Anything else returns a 400 error. Even for older models the guidance was already conservative: you usually only need temperature, top_k is for advanced use cases only, and you should alter temperature or top_p but not both.

Google has gone the other way. Gemini keeps topK, topP and temperature as first-class fields in generationConfig, and documents how they compose. Two details are worth knowing before you copy a value from a blog post: the defaults vary by model rather than being fixed platform-wide, and some models run nucleus-only and will not accept a topK setting at all.

I found Anthropic's direction irritating at first, because it takes away a dial I was used to reaching for, but it is the more honest position: if the provider already knows which parameter range produces good output, letting you hunt around inside that range mostly sells you the feeling of control. The pattern underneath is straightforward. These knobs are being withdrawn at the frontier-model layer, where the provider wants to own sampling behaviour, while remaining fully alive in open infrastructure like llama.cpp and vLLM, where you run the model and own the consequences. Before spending an afternoon searching for the perfect values, check whether the model you are calling still accepts them.

So what should you actually set for your work?

If you are calling a recent frontier model over an API, the answer may be nothing at all, because the parameters are no longer accepted, and the lever that remains is your prompt rather than your sampler configuration.

If you run the model yourself and genuinely have the controls, keep it minimal. For code and maths, hold temperature low (0.0–0.2) with a high top_p (0.95). For chat and open-ended writing, temperature between 0.7 and 1.0 paired with min_p at 0.05–0.1 is where most llama.cpp and vLLM users have converged. What you should not do is enable all four filters at once and then wonder why the output looks strange: each one cuts into the same distribution, and stacking them leaves you unable to tell which is actually making the decision.