Skip to content

LLM Overthinking: Why Thinking More Can Make an LLM Dumb

Allocating excess test-time compute to simple prompts triggers LLM overthinking, degrading model accuracy through premise drift and redundant reasoning.

Tuan Tran Van
8 min read
Contents (7 sections)
  1. The Promise of Extended Thinking: How Reasoning Models Evolved
  2. The Overthinking Trap on Simple Prompts
  3. Compounding Autoregressive Errors and the Drifting Premise
  4. Diminishing Returns and the "Negative Flip" Phenomenon
  5. How Modern Systems Mitigate Overthinking: From Dynamic Budgets to Hybrid Reasoning
  6. When to Let an LLM Think — and When to Cut It Short
  7. References

Large language models generate text autoregressively, computing output probabilities by token based on previously emitted sequence history. Allocating additional test-time compute through extended reasoning trajectories allows models to solve multi-step logic and complex mathematics. However, forcing additional inference compute on simple or noisy prompts triggers LLM overthinking, causing accuracy to degrade as extended reasoning re-evaluates and overturns verified conclusions.

Instead of improving the output, generating surplus thinking tokens on trivial inputs leads the system to reconsider valid assumptions, drift away from core prompt instructions, or introduce false premises into its scratchpad.

Understanding why surplus test-time compute degrades model accuracy requires analyzing autoregressive attention dynamics, reinforcement learning reward structures, and production latency tradeoffs.

LLM Overthinking: Why Extended Thinking Can Degrade Model Accuracy

The Promise of Extended Thinking: How Reasoning Models Evolved

The Evolution of Reasoning Models: Hidden Scratchpads and Autonomous Verification

Frontier architectures have evolved beyond single-pass next-token prediction toward test-time compute scaling. Rather than forcing a model to generate an immediate output token in a single forward pass, architectures like OpenAI o1, DeepSeek-R1, and Claude 3.7 Sonnet allocate test-time compute to generate intermediate "thinking tokens." These tokens exist in a hidden scratchpad where the model can decompose complex problems, test intermediate hypotheses, and execute self-correction prior to finalizing its response.

Reinforcement learning algorithms like Group Relative Policy Optimization (GRPO) train models to use these intermediate scratchpads without requiring step-by-step human annotations.

Unlike traditional Proximal Policy Optimization (PPO), GRPO calculates baseline performance metrics directly from group output statistics rather than maintaining a separate, memory-heavy critic network, significantly reducing VRAM overhead during RL training scaling. This self-directed exploration yields autonomous verification mechanisms, such as DeepSeek R1-Zero's emergent "aha moments" in math problems, where the model squared complex algebraic expressions, recognized unnecessary expansion, halted its execution, and backtracked to the original equation.

On high-uncertainty tasks, extended test-time compute delivers measurable empirical gains.

On SWE-bench Verified, Claude 3.7 Sonnet achieves a 63.7% pass@1 score in standard execution, which rises to 70.3% under high compute scaffolding using parallel attempts and rejection sampling against visible regression tests. Similarly, expanding multi-turn trajectory allocations on TAU-bench from 30 to 100 maximum steps enables agents to execute deeper tool-use chains and complete complex environment interactions.

The Overthinking Trap on Simple Prompts

The Overthinking Trap on Simple Prompts: Redundant Verification and Task Deviation

While test-time compute scaling enhances multi-step problem solving, applying unconstrained reasoning workloads to trivial prompts causes systematic resource allocation failures. Research demonstrates that reasoning models often generate thousands of redundant tokens when executing basic operations. When presented with elementary arithmetic like 2 + 3, models such as QwQ-32B-Preview can emit up to 13 distinct solution attempts in their internal scratchpad before outputting the final answer.

This overthinking stems from outcome-based reward modeling in reinforcement learning. Standard reward functions score attempts based solely on final output correctness, granting equal reward to a direct one-step answer of 5 and a 2,000-token trace that recalculates the addition through multiple redundant pathways. To eliminate this inefficiency, researchers performed supervised (SFT) on trajectory-pruned trajectories that cut reasoning after the first verified correct answer plus a single verification check. On the MATH500 benchmark, this SFT approach reduced average token usage from 2,408 to 1,331 tokens per problem while preserving accuracy (93.0% versus 92.8%).

In production environments, unpruned reasoning tokens introduce severe economic and SLA penalties. Output tokens—including hidden thinking tokens—are typically billed at premium rates, such as 15 USD per million output tokens versus 3 USD per million input tokens. Beyond raw financial expenditure, redundant reasoning tokens directly inflate Total Generation Latency and Time Between Tokens (TBT), penalizing real-time system performance and breaching operational SLA targets for zero marginal gain in output accuracy.

Compounding Autoregressive Errors and the Drifting Premise

Compounding Autoregressive Errors: Attention Drift Triggered by Distractor Clauses

Beyond generating financial and operational latency penalties, extended reasoning trajectories on simple prompts can actively corrupt final outputs through premise drift.

architectures compute self-attention vectors over all previously generated tokens in the .

Once a model writes an incorrect intermediate assumption, flawed edge case, or alternate interpretation into its scratchpad, subsequent tokens attend directly to that erroneous context, steering generation off-course.

Consider a prompt requesting a scheduled train departure from a timetable that lists 9:00, accompanied by a side note stating trains on the line average a 10-minute delay. A standard extracts 9:00 instantly. An overthinking reasoning model, however, may spend multiple thinking cycles analyzing the operational impact of the delay. As the scratchpad accumulates intermediate calculations regarding actual predicted arrival times (9:10), self-attention weights shift toward these recent reasoning tokens. When generating the final user-facing response, the model outputs 9:10, failing the explicit instruction of the prompt.

Empirical research using the GSM-Symbolic benchmark—specifically under the GSM-NoOp test setting—demonstrates this vulnerability. By inserting a single logically irrelevant clause into grade-school math problems, model accuracy dropped by up to 65% across state-of-the-art architectures. Because language models replicate logic templates from training data rather than performing pure symbolic deduction, intermediate reasoning noise triggers attention drift, compounding minor scratchpad hallucinations into final output failures.

Diminishing Returns and the "Negative Flip" Phenomenon

Diminishing Returns and the Negative Flip Phenomenon in Test-Time Compute Scaling

Continuous test-time compute scaling exhibits clear non-linear boundaries. Investigations confirm that the marginal utility of additional reasoning tokens follows a steep curve of diminishing returns. Beyond a task-specific compute threshold, generating extra thinking tokens yields zero performance improvement and increases the risk of solution corruption.

text
Model Accuracy
  ^
  |        /--- Maximum Utility Threshold
  |       /   |      /   \  <-- Negative Flip Zone (Abandoning Correct Answers)
  |     /       |    /       \--- Diminishing Returns
  +-------------------------------------> Test-Time Compute (Tokens)

Scaling compute past this optimal threshold triggers the "negative flip" phenomenon, where a model abandons an initially correct solution. In these instances, the model identifies the correct response within its initial reasoning steps but lacks a deterministic stopping condition. As it continues generating tokens to re-evaluate its work, it introduces unnecessary edge cases or hyper-analyzes noise, eventually overwriting its valid intermediate answer with an incorrect final conclusion.

In parallel, empirical evaluations establish that optimal thinking token distributions vary significantly by domain. Forcing uniform, extended token budgets across heterogeneous workloads degrades performance on creative generation, direct lookup, and structured output tasks, proving that unconstrained reasoning actively impairs accuracy outside complex mathematical and symbolic domains.

How Modern Systems Mitigate Overthinking: From Dynamic Budgets to Hybrid Reasoning

Mitigating LLM Overthinking: Early Truncation, Thinking Budgets, and Hybrid Reasoning

To prevent overthinking while retaining extended reasoning capabilities, modern infrastructure incorporates architectural controls, API parameters, and specialized training pipelines. At the architecture tier, hybrid reasoning models like Claude 3.7 Sonnet decouple standard generation from extended thinking. System designers can route low-latency requests through standard execution or explicitly enable extended thinking mode when handling high-complexity workloads.

At the inference API layer, providers offer granular budget management, allowing infrastructure engineers to set an explicit maximum cap on thinking tokens (N tokens up to 128K).

This token budget enforces strict cost and latency boundaries, terminating scratchpad generation before compounding errors or redundant verification loops impair response quality.

At the model training layer, systems apply dynamic step reduction and input-prompt difficulty routing. Through Thinking-Optimal Scaling strategies, models like Qwen2.5-32B-Instruct are fine-tuned on seed datasets with varied reasoning length distributions. By scoring attempts on both correctness and token conciseness, the model learns to evaluate prompt difficulty at runtime and autonomously select the shortest valid reasoning path.

When to Let an LLM Think — and When to Cut It Short

Optimizing test-time compute requires matching inference allocation directly to task complexity. Disable or strictly cap extended thinking tokens when processing deterministic lookups, basic arithmetic, string extractions, schema-constrained JSON outputs, or real-time interactive user routes where added latency violates SLAs and raises premise drift risk.

text
                                  [ Incoming Request ]
                                           |
                                 Is the task complex?
                                 /                                               ( No )                ( Yes )
                               /                            [ Disable / Cap Extended Thinking ]     [ Enable Extended Thinking ]
      • Deterministic lookups                 • Repository-level coding
      • Basic arithmetic                      • Multi-step agent workflows
      • String & schema extraction            • Advanced math & physics
      • Low-latency API routes                • Ambiguity-heavy logic puzzles

Enable extended thinking modes when deploying systems for repository-level software refactoring, multi-turn agentic tool integration, advanced mathematical derivations, and ambiguity-heavy logic problems.

Aligning your test-time compute budget strictly with prompt difficulty eliminates redundant token expenditure, guarantees predictable SLA latency, and prevents overthinking from corrupting model outputs.

References

Check yourself

Three quick questions to see whether the main ideas stuck.

  1. Why can excessive test-time compute degrade an LLM's accuracy on trivial math prompts like 2 + 3?
  2. What does the 'negative flip' phenomenon signify in test-time compute scaling?
  3. In which production scenario should engineers disable or strictly limit an LLM's thinking budget?

Share this article

AI glossary