What Is LLM Temperature, and What Should You Set It To?

LLM temperature is a hyperparameter applied during the sampling stage of inference to control the variance of token selection

— when you prompt a model, it does not output a single string directly. It generates a probability distribution across its entire vocabulary for the next potential token, and temperature scales that distribution before the final selection is made.

What you are adjusting, when you touch this number, is the entropy of the output. Turn it down and the probability mass concentrates on the most likely candidates, so the model behaves more deterministically. Turn it up and the mass spreads more evenly across the tail of the distribution, which lets the sampler reach for lower-probability tokens. You get more variety out of that, but the risk you take on does not grow in a straight line: past a point you are buying logical inconsistency and neural text degeneration, the failure mode where output decays into repetition and filler.

Temperature is not a training parameter, and the distinction is worth holding onto, because it is an inference-time setting that belongs to the decoding strategy. It leaves the underlying weights of the transformer, and the knowledge encoded in them, exactly where they were. Temperature does not change what the model knows — only how boldly it picks from what it already believes.

A dial reshaping the probability distribution curve of a large language model

What is temperature?

Underneath the metaphors, temperature is a scaling factor on the probability mass function for the next token in a sequence. During the forward pass of inference, a large language model produces a raw score for every possible token in its vocabulary, and those raw scores are the logits (z_i). Temperature (T) is applied as a divisor before the logits are converted into probabilities, so the probability P of a token i falls out of a modified softmax function: P(w_i) = exp(z_i / T) / Σ_j exp(z_j / T). That one division before the exponentiation is the whole mechanism, because it changes the relative differences between the scores in logit-space.

Because the division happens at the point of sampling, temperature is an inference-time parameter: it decides how the model moves through the possibilities it has already worked out from its pre-trained weights, not what those possibilities are. When T = 1.0, the logits pass through the softmax function without modification, so the distribution learned during training survives intact. That is the neutral setting, and it is the one to keep in your head as the reference point, because it is where the model's internal confidence shows up directly in the output probabilities.

Everything you do to T after that is a trade between typicality and novelty. A lower T raises the typicality of the response and holds the model to its high-confidence patterns, while a higher T raises entropy and lets it step off the most statistically probable path. If you build on these APIs, this is the one control that governs how far two identical calls are allowed to diverge, which is why I have come to read it less as a style setting and more as a variance budget for whatever a given production workload can tolerate.

What does temperature do to the probability distribution?

Temperature does its work by steepening the probability mass function or flattening it, and that is the entire repertoire. A lower temperature steepens it, which widens the gap between the most likely tokens and the rest of the vocabulary until the probability mass piles onto the argmax candidate, the single most likely token. A higher temperature flattens the distribution, so the relative probability distance between the top candidates and the long tail of less likely tokens shrinks, and the sampler stands a better statistical chance of picking a surprising token.

The same probability distribution of horses 0.7, zebras 0.2 and unicorns 0.1, steepened at low temperature and flattened at high temperature

Numbers show this faster than any amount of description. Take a model predicting the next token with a default distribution of "horses" (0.7), "zebras" (0.2), and "unicorns" (0.1). At a low temperature the distribution steepens and "horses" pulls further ahead, while at a high temperature it flattens and the tail closes much of the gap. Rescaling that same distribution gives:

json

{
  "default_distribution_T1.0": {
    "horses": 0.7,
    "zebras": 0.2,
    "unicorns": 0.1
  },
  "low_temperature_T0.7": {
    "horses": 0.814,
    "zebras": 0.136,
    "unicorns": 0.05
  },
  "high_temperature_T1.5": {
    "horses": 0.586,
    "zebras": 0.254,
    "unicorns": 0.16
  }
}

At T = 1.5, "unicorns" has gone from a one-in-ten shot to roughly one in six, which is the appeal of this parameter and the danger of it in a single line. Flattening the distribution does introduce variety, but it is a high-risk operation, because if T is set too high the distribution becomes flat enough that the model begins selecting tokens with no logical relation to the context in front of them. Coherence goes with it: the model loses its grip on the linguistic and logical constraints it learned during training, and what comes back is vacuous filler.

What does temperature 0 actually mean?

In most inference engines, setting temperature to 0 is a flag that triggers greedy decoding. You cannot divide by zero in the softmax function (z_i / 0), so the system bypasses the sampling logic entirely. In this mode the model consistently selects the token with the highest probability (argmax) at every timestep. Zero is therefore an API convention rather than a temperature value — the distribution approaches an argmax distribution as T approaches zero from above, but the engine never actually computes it at zero.

At temperature 0 the model always takes the highest peak of the distribution, which leads to repetition loops

Greedy decoding gives you the most stability on offer, and it will return the same response to the same prompt across repeated calls provided the underlying hardware and software state stays consistent. That proviso does more work than it looks like it does, because on shared inference infrastructure the same configuration can still drift between runs. Greedy decoding also runs straight into the repetition problem. When a model is forced to maximize likelihood at every step, it often falls into local optima where a sequence of high-confidence tokens reinforces itself — a model repeating a phrase such as "I'm sorry, but..." indefinitely. Lacking the entropy to break the pattern, it cannot transition to a lower-probability token that might have led to a more useful completion.

There are likelihood traps waiting further down the sequence as well. Selecting the most probable token at step N might lead to a path where only low-quality or repetitive tokens are available at step N+10, while a slightly less probable token at step N would have opened a much better overall sequence. With T = 0 you prevent the model from ever exploring those alternative paths, and on complex or open-ended tasks the text that comes back reads as rigid and robotic as that constraint sounds.

How do temperature, Top-P and Top-K differ?

Temperature, Top-K, and Top-P (nucleus) sampling all shape the probability distribution, but they are three separate controls working through different mechanisms, and they get conflated constantly. Top-K sampling is a truncation strategy that limits the model to a fixed number (K) of the most likely next tokens. If you set K = 50, the model ignores the entire vocabulary outside the top 50 candidates, regardless of their probabilities. This prevents the model from ever picking the gibberish tokens in the extreme tail.

Pipeline order: temperature rescales the whole distribution first, then Top-K truncates by count, then Top-P by cumulative probability, and only then sampling

Top-P sampling, or nucleus sampling, is a dynamic truncation method. Instead of a fixed number of tokens, it selects the smallest set of tokens whose cumulative probability exceeds a threshold p (for example 0.9). The sample pool expands when the model is uncertain and contracts when the next token is highly predictable. Temperature rescales the entire distribution, whereas Top-K and Top-P work by pruning it, and that is the distinction worth carrying around.

Order matters, and it is the opposite of what most people assume. Temperature is applied first, rescaling the logits; Top-K then filters to a fixed candidate count, Top-P narrows further to the nucleus, and only then does the sampler draw a token. Because rescaling and pruning do different jobs, turning both dials at once leaves you unable to say which one produced what you are looking at, and the common guidance is to adjust one and leave the other at its default. A typical configuration sets Top-P to 0.9 to remove the long tail while holding temperature around 0.7 to preserve variety among the surviving candidates.

Is temperature really a creativity dial?

Temperature gets sold as the creativity dial, and that framing does not survive contact with the measurements. Creativity requires a balance between novelty (dissimilarity from existing patterns) and typicality (adherence to the expected logic and form of a category). Raising temperature does increase novelty by allowing the selection of surprising tokens, but it does so at the direct expense of typicality, which shows up as a loss of adherence to the prompt's constraints.

As temperature rises, novelty increases only slightly while coherence falls clearly

When generated stories are scored against creativity criteria, temperature turns out to correlate only weakly with novelty, more clearly with incoherence, and not at all with cohesion or typicality. Higher temperatures do not meaningfully widen the model's access to its own distribution, and what they mostly do is raise the odds of variety inside a narrow sampling band. So what you buy by turning the dial up is largely risk rather than ideas.

The failure mode compounds, too, which is what makes it worse than an odd bad sentence here and there. During training a model is conditioned on gold-standard text, while at inference it has to condition on its own previously generated tokens. If a high temperature causes the model to pick a logically weak token early, that error propagates through everything that follows — the exposure-bias problem behind neural text degeneration, where output turns repetitive, vacuous, or self-contradictory. Reliable creative output comes from grounding and structured prompting, with temperature used for final adjustment rather than as the source of the ideas.

What temperature should you use for which task?

The right temperature is not a matter of taste, and it is set by how much variance your task can absorb. For work requiring precision and strict adherence to a schema, use a low temperature (0 to 0.3). This is the standard for coding, mathematical reasoning, and data extraction, where any deviation from the most likely token usually surfaces as a syntax error or a logical failure. Generating structured JSON is the clearest case: a low temperature is what keeps the model from inventing invalid keys or dropping a closing brace.

A map of temperature values by task: coding and data extraction at 0 to 0.3, general chat at 0.7 to 1.0, creative brainstorming at 1.2 and above

For general assistance, standard chat, and balanced writing, a neutral temperature (0.7 to 1.0) is the usual starting point. This range introduces enough entropy to avoid a robotic, repetitive tone while holding the logical thread of the prompt. Reserve a high temperature (1.2+) for unstructured brainstorming or creative fiction, where variety is the goal and logical consistency is secondary.

Treat these numbers as provider-relative rather than absolute, because the valid range and the default differ between vendors — 1.0 is a common default, and some APIs accept values up to 2.0 — so 0.7 on one platform does not necessarily produce the same amount of randomness as 0.7 on another. So re-tune when you switch models instead of carrying a configuration across, however tempting it is to paste the old value in and move on. The snippet below sets a low temperature for a high-precision extraction task on a non-reasoning model:

python

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {"role": "system", "content": "You are a senior systems engineer."},
    {"role": "user", "content": "Extract every entity from the text below as JSON."}
  ],
  temperature=0.1  # minimal variance for schema reliability
)

Why do reasoning models lock temperature?

Modern reasoning models — the o1, o3, and GPT-5 series among them — do not accept traditional sampling parameters at all. On Azure OpenAI the unsupported list for these models covers temperature, top_p, presence_penalty, frequency_penalty, logprobs, top_logprobs, logit_bias, and max_tokens. That list is long because these models spend reasoning tokens internally exploring multiple logical paths, breaking a problem apart and abandoning the approaches that fail. Those tokens never appear in the message content, but they occupy space in the context window and are billed as output tokens.

On reasoning models the temperature dial is locked and replaced by a reasoning_effort parameter

That internal process is why the knob is gone, and I think taking it away was the right call. Variance introduced early in a reasoning chain does not average out — it compounds, and a model that takes a weak turn in its first few reasoning tokens cannot be rescued by sampling settings applied to the final answer. Locking the sampling parameters keeps the calibration the provider tuned the reasoning process against.

In these architectures you tune computational depth instead of distribution variance. The reasoning_effort parameter tells the model how much to think before answering, with supported values varying by model across none, minimal, low, medium, high, xhigh, and max. Higher effort generally improves quality on complex debugging and planning, though you pay for it in latency and in spend. Adjacent controls handle output shape rather than randomness: a phase field distinguishes commentary (intermediate updates, such as the preamble before a tool call) from final_answer, and a reasoning summary parameter returns a condensed view of the model's chain of thought.

When is tuning temperature actually worth it?

Temperature is a late optimization, and it is the one people reach for first. It manages distribution variance, and that is all it manages, so it will not repair an ambiguous prompt. If a model is failing to follow your instructions, lowering the temperature may well mask the symptom by forcing a safer response, but the lack of clarity in your input is still sitting there, and it will resurface on the next edge case.

Tune your prompt first, pick the right reasoning effort second, and reach for temperature last — to shift a system that already works toward more reliability or more variety. If changing this number is fixing your output, the number was probably not the problem.