PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models

Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, Amrit Singh Bedi

arXiv:2506.04210v3cs.AIcs.CL

TL;DR

The paper asks whether extended test-time thinking genuinely improves reasoning, given limited evidence about its effects. It studies extended thinking and proposes parallel thinking; performance can decline after initial gains, while parallel thinking yields consistent gains.

  • Problem

    The paper examines whether prompting reasoning models to think longer genuinely improves reasoning, while noting that extending a single trace is brittle because there is no reliable stopping criterion.

    Source: §1 · pp. 1–2

  • Method

    The authors study extended thinking empirically and with a probabilistic model, then split the inference budget across independent reasoning paths and select by majority vote.

    Source: §1 · p. 2, §3 · p. 6

  • Results

    Extended thinking initially improves accuracy but later degrades through overthinking, while parallel thinking yields consistent gains.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Under a fixed budget, extending one trace is brittle because performance is non-monotonic and there is no reliable stopping criterion.

    Source: §1 · p. 2

    The study is limited to mid-sized models, and a formal theory of how variance and entropy affect reasoning performance remains future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent trends in test-time scaling for reasoning models (e.g., OpenAI o1, DeepSeek R1) have led to a popular belief that extending thinking traces using prompts like "Wait" or "Let me rethink" can improve performance. This raises a natural question: Does thinking more at test-time truly lead to better reasoning? To answer this question, we perform a detailed empirical study across models and benchmarks, which reveals a consistent pattern of initial performance improvements from additional thinking followed by a decline, due to "overthinking". To understand this non-monotonic trend, we consider a simple probabilistic model, which reveals that additional thinking increases output variance-creating an illusion of improved reasoning while ultimately undermining precision. Thus, observed gains from "more thinking" are not true indicators of improved reasoning, but artifacts stemming from the connection between model uncertainty and evaluation metric. This suggests that test-time scaling through extended thinking is not an effective way to utilize the inference thinking budget. Recognizing these limitations, we introduce an alternative test-time scaling approach, parallel thinking, inspired by Best-of-N sampling. Our method generates multiple independent reasoning paths within the same inference budget and selects the most consistent response via majority vote, achieving up to 20% higher accuracy compared to extended thinking. This provides a simple yet effective mechanism for test-time scaling of reasoning models.

1 Test-time Scaling Capability in LLMs for Reasoning

The section finds that extended thinking can first improve accuracy and then degrade it, a pattern the authors call overthinking. It proposes parallel thinking as a same-budget alternative and attributes overthinking to increased output variance.

1 Test-time Scaling Capability in LLMs for Reasoning

  • Extended thinking initially boosts accuracy but later degrades it across tasks and datasets, a non-monotonic pattern termed overthinking.The decline begins beyond a critical thinking-trace length.Source: §1 · p. 2
  • Under a fixed budget, extending one reasoning trace is brittle because performance is non-monotonic and offers no reliable stopping criterion.Source: §1 · p. 2
  • At a 16K token budget, parallel thinking yields up to 22% higher accuracy than sequential scaling and outperforms overthinking across all benchmarks.It divides the budget among independent reasoning paths and selects the final answer by majority vote.Source: §1 · p. 2
  • The authors attribute the non-monotonic pattern to increased output variance, which initially aligns with reward but eventually disrupts reward alignment.Their analysis measures response-distribution entropy under extended thinking.Source: §1 · p. 2

2 Analysis of Test-Time Scaling in SoTA Reasoning LLMs

The section formalizes test-time reasoning as a thinking trace followed by an answer, then examines budget controls across open-source models and math benchmarks. Accuracy initially improves with more thinking, but eventually declines.

2 Analysis of Test-Time Scaling in SoTA Reasoning LLMs

  • The paper represents reasoning as a prompt-conditioned trace z followed by an answer y, and models test-time scaling as continuing the trace through repeated prompted steps.In standard thinking, the model generates a trace and then the final answer.Source: §2 · p. 3

2.1 Test-Time Budget Control (TTBC)

  • Two budget controls extend thinking: Wait & Think More repeatedly suppresses the end delimiter, while Exact Thinking Tokens stops at a specified token count.The exact-token setup varies the target budget across 256, 512, 1,024, 2,048, 4,096, 8,192, and 16,384 tokens.Source: §2.1 · p. 3

2.2 Experimental Setup

  • The evaluation measures answer accuracy on GSM-8K, MATH-500, and AIME 2024 using three distilled DeepSeek-R1 models of different sizes and architectures.Accuracy is the expected indicator that the generated answer matches the correct answer across test prompts.Source: §2.2 · pp. 4–5

2.3 Key Observations: Failure of Overthinking

  • 82.2% to 87.3% as average thinking tokens rise from 385 to 1,100, then 70.3% at 15,980; exact- and minimum-token controls also show non-monotonic accuracy.The Wait & Think More results rise before falling; the exact-token setup and minimum-token analysis report the same broader pattern.Source: Figure 2 · p. 4, §2.3 · pp. 5–6, Figure 3 · p. 5

3 Mirage of Test-Time Scaling: An Alternative Explanation

A simple probabilistic framework explains the non-monotonic performance pattern as a possible variance-driven mirage rather than policy refinement. In experiments, longer thinking increased answer entropy, with accuracy first improving and then declining; repeating the same reasoning step did not increase entropy.

3 Mirage of Test-Time Scaling: An Alternative Explanation

  • The illustration models answers with a Gaussian policy and reward with a bell-shaped Gaussian, then examines expected reward as policy variance changes.The framework is presented as a simplified probabilistic explanation of the observed non-monotonic pattern.Source: §3 · p. 6
  • Increasing variance can first improve expected reward by broadening policy overlap with the reward, but excessive spread reduces reward and creates a critical point.The proposed explanation balances greater coverage against dilution of probability mass away from the reward peak.Source: §3 · pp. 6–7
  • The paper hypothesizes that longer thinking similarly increases output-distribution variance, so early gains can reflect greater overlap rather than genuine policy refinement.Further thinking is described as increasing entropy and broadening sampling until the distribution becomes too diffuse and performance deteriorates.Source: §3 · p. 7

3.1 Role of Variance in test-time scaling of reasoning models

  • On MATH-500, extending thinking from 2657 to 4983 tokens raised entropy from 0.02 to 0.12 and corresponded to a 1.5% accuracy gain; extending it to 16109 tokens raised entropy to 3.0 as accuracy fell from 83.2% to 78.3%.Figure 5 reports these results for DeepSeek-R1-Distill-Qwen-1.5B; across both benchmarks, entropy increased with additional thinking.Source: Figure 5 · p. 7, §3.1 · p. 8
  • The authors conclude that Wait & Think More is not an optimal use of the thinking budget because its initial accuracy gains are largely a mirage.The paper frames fixed-budget allocation as the next question for test-time scaling.Source: §3.1 · p. 8
  • Repeatedly appending the same reasoning step to lengthen the input left response entropy nearly unchanged, suggesting context length alone does not explain the uncertainty increase.The experiment used GSM-8K and DeepSeek-R1-Distill-Qwen-1.5B.Source: §3.1 · p. 8, Figure 6 · p. 8

4 Effective Test-Time Budget Utilization via Parallel Thinking

Parallel thinking divides a fixed inference budget among independent reasoning paths and selects the most consistent answer. Across the reported comparisons, it outperforms extended sequential thinking, which can degrade with overthinking.

Parallel thinking

  • The method generates multiple independent reasoning traces within budget B, rather than spending the entire budget on one extended chain of thought.The total length of the parallel traces is constrained to no more than B tokens.Source: §4 · pp. 8–9
  • It samples one output per trace and selects the response with the greatest frequency among the candidates.The paper defines this frequency-based selection as a self-reward mechanism for favoring high-likelihood responses.Source: §4 · p. 9

Empirical evaluations

  • At a fixed 16K-token inference budget, parallel thinking yields around 22% higher accuracy than Wait & Think More and 47% higher accuracy than Exact Thinking TTBC.Figure 8 reports these two comparisons.Source: Figure 8 · p. 9
  • On GSM-8K with DeepSeek-R1-Distill-Qwen-1.5B, 16,000 tokens of “Wait & Think More” drop performance 11.8%, while parallel thinking improves accuracy 10.1% over standard thinking.Figure 7 compares accuracy against average thinking tokens across three benchmarks and three models.Source: Figure 7 · p. 9, §4 · p. 10

5 Related Works

Related work covers reasoning models and inference-time scaling alongside training-based approaches. The paper situates its study among work on test-time scaling and trace-length effects.

Large reasoning models

  • Prior reasoning-model approaches include Monte-Carlo Tree Search and distillation from o1-like reasoning traces; DeepSeek-R1 used sparse-reward-based RL training.The paper focuses on test-time scaling behavior in reasoning models.Source: §5 · p. 10

Test-time scaling in reasoning language models

6 Conclusions and Limitations

The paper concludes that extended thinking can first improve accuracy and then degrade it, while parallel thinking distributes the inference budget across paths and selects an answer by majority vote. The study is limited to mid-sized models and leaves formal variance-and-entropy theory for future work.

Conclusions

  • Extended thinking initially improves accuracy, but continued reasoning beyond a threshold degrades performance; the paper attributes early gains often to increased response variance rather than genuine reasoning improvements.The authors call the later degradation overthinking and report that parallel thinking yields consistent gains.Source: §6 · p. 10

Limitations

  • The study covers mid-sized models because of resource constraints; evaluating 32B and 70B models and developing a formal account of variance and entropy remain future directions.The authors describe the current explanation as supported by empirical evidence and a simple probabilistic model.Source: §6 · p. 10

B Broader Impact

The paper reports no anticipated direct harmful consequences and notes that sequential reasoning can degrade accuracy, whereas parallel paths perform better. It also places its work among test-time scaling research and related analyses of reasoning-trace length.

B Broader Impact

  • The authors report no anticipated direct harmful consequences and say sequential reasoning can degrade accuracy, whereas multiple parallel paths consistently yield better results.They hope the findings inspire methods that guarantee reliable performance gains from test-time compute scaling.Source: B Broader Impact · p. 16

C Additional Context of Related Works

  • Test-time scaling is presented alongside continued pretraining and targeted fine-tuning as an approach to reasoning tasks; prior work also studied trace length, while this study extends analysis across datasets and models.The paper says it also offers an intuitive explanation and evaluates parallel thinking as a test-time budget strategy.Source: C Additional Context of Related Works · p. 16

D Software and Hardware

Experiments used Python 3.11.11, PyTorch 2.6.0, Transformers 4.49.0, and Nvidia RTX A6000 GPUs.

D Software and Hardware

E.1 Minimum Thinking setup

Minimum thinking tokens impose a lower bound on each reasoning trace by appending “Wait” until a chosen token threshold is reached. Across three models and three math benchmarks, accuracy varies non-monotonically with that threshold.

E.1 Minimum Thinking setup

  • The setup suppresses the end-of-thinking delimiter and appends “Wait” until a trajectory reaches at least tmin tokens, then appends the delimiter.Source: §E.1 · pp. 16–17
  • Figure 9 plots accuracy against thinking-token budget for Qwen-1.5B, Qwen-7B, and Llama-8B across GSM-8K, MATH-500, and AIME.Source: Figure 9 · p. 17
  • In this analysis, tmin varies across minimum budgets of 256, 512, 1024, 2048, 4096, 8192, and 16384 tokens to study scaling effects, with </think> appended once each threshold is met.Source: §E.1 · p. 17
  • Accuracy has a non-monotonic relationship with the minimum thinking-token threshold across three models and three mathematical benchmarks.The reported benchmarks are GSM-8K, MATH-500, and AIME 2024.Source: §E.1 · p. 17

E.2 Extended analysis on role of variance in test-time scaling

The analysis links longer thinking traces with rising policy entropy and a non-monotonic performance pattern. It extends the uncertainty analysis to additional open-source reasoning models.

E.2 Extended analysis on role of variance in test-time scaling

  • The extension examines additional open-source models after earlier analysis found initial test-time-scaling gains on GSM-8K and MATH-500 were largely a mirage driven by randomness.Source: §E.2 · p. 17
  • Across the reported models, policy-distribution entropy increases with thinking-token length.Source: §E.2 · p. 18
  • After an initial performance improvement, further increases in thinking tokens produce a sharp entropy rise that negatively affects performance.Figure 10 uses the “Wait & Think more” setup with DeepSeek-R1-Distill-Qwen-7B on GSM-8K and MATH-500.Source: Figure 10 · p. 18

F Generation Examples

The examples show incorrect answers in extended-thinking generations, including cases where longer traces end differently from shorter ones. The section includes math and word-problem generations under several budget-control setups.

F Generation Examples

  • The examples across Wait & Think More, Exact Thinking Token, and Minimum Thinking Token include incorrect answers after extended thinking.Source: F Generation Examples · p. 18
  • Figure 11 examines entropy against additional thinking tokens on GSM-8K and MATH-500 with DeepSeek-R1-Distill-Llama-8B; it reports initial gains followed by decline.Source: Figure 11 · p. 19
  • For the house-profit problem, the 5k-token Wait & Think more example concludes $70,000, while the 12k-token example concludes −10000.Source: F Generation Examples · p. 20
  • In the Exact Thinking (2048 thinking tokens) example simplifying √242, the response derives 11√2, expresses confusion, and tries factoring again before ending with “Final Answer: 2”.Source: F Generation Examples · p. 21
  • For the kitten problem, the 256-token minimum-thinking generation concludes 40, while the 2048-token generation concludes 33.Source: F Generation Examples · p. 22
Loading 2506.04210v3…