Source-linked AI summary
Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu
TL;DR
The paper studies underthinking, in which o1-like LLMs prematurely switch away from promising reasoning paths, and introduces a metric to quantify this inefficiency. It proposes the thought switching penalty TIP, which encourages deeper exploration during decoding. TIP improves accuracy across challenging test sets without additional model fine-tuning.
Problem
The paper addresses the underexplored question of whether o1-like LLMs think deeply enough, focusing on premature abandonment of promising reasoning paths.
Method
The paper quantifies underthinking using token efficiency in incorrect responses and proposes TIP, a decoding strategy that penalizes premature thought switching.
Results
TIP consistently improves accuracy across three challenging test sets without requiring additional model fine-tuning.
Takeaways & Limitations
The findings provide a practical approach for reducing underthinking and a framework for assessing reasoning inefficiencies in o1-like LLMs.
Abstract
from arXiv · showhide
Large language models (LLMs) such as OpenAI's o1 have demonstrated remarkable abilities in complex reasoning tasks by scaling test-time compute and exhibiting human-like deep thinking. However, we identify a phenomenon we term underthinking, where o1-like LLMs frequently switch between different reasoning thoughts without sufficiently exploring promising paths to reach a correct solution. This behavior leads to inadequate depth of reasoning and decreased performance, particularly on challenging mathematical problems. To systematically analyze this issue, we conduct experiments on three challenging test sets and two representative open-source o1-like models, revealing that frequent thought switching correlates with incorrect responses. We introduce a novel metric to quantify underthinking by measuring token efficiency in incorrect answers. To address underthinking, we propose a decoding strategy with thought switching penalty TIP that discourages premature transitions between thoughts, encouraging deeper exploration of each reasoning path. Experimental results demonstrate that our approach improves accuracy across challenging datasets without requiring model fine-tuning. Our findings contribute to understanding reasoning inefficiencies in o1-like LLMs and offer a practical solution to enhance their problem-solving capabilities.
1 Introduction
The introduction identifies underthinking in o1-like LLMs as premature abandonment of promising reasoning paths and presents metrics and decoding-based mitigation. The study analyzes this issue across challenging benchmarks and proposes TIP to encourage deeper exploration.
- The study examines underthinking across three challenging test sets and two open-source o1-like models with visible long chains of thought.
- 225% more tokens are consumed by incorrect responses on average because of 418% more frequent thought-switching behaviors.
- The proposed underthinking metric measures token efficiency in incorrect responses to quantify reasoning inefficiencies.
- TIP penalizes premature thought transitions during decoding, encouraging models to explore each reasoning path more thoroughly without additional fine-tuning.
- Underthinking is the premature abandonment of promising reasoning paths, producing inadequate depth of reasoning on challenging problems.
2 Observing Underthinking Issues
The paper examines underthinking in o1-like LLMs: frequent switching between reasoning thoughts can cause promising paths to be abandoned before completion. Across challenging datasets and models, the authors quantify this inefficiency and analyze how it relates to incorrect responses.
- 2 Observing Underthinking Issues: The study analyzes underthinking in o1-like models using MATH500, GPQA Diamond, and AIME datasets, with visible long-chain-of-thought models.The evaluated models include QwQ-32B-Preview, DeepSeek-R1-671B, and DeepSeek-R1-Preview.
- 2.1 Frequent Thinking Switch of o1-Like LLMs: As MATH500 difficulty increases, models generate more thoughts and tokens, indicating more frequent thought switching on harder problems.Figure 3 reports average thoughts and tokens across difficulty levels.
- 2.1 Frequent Thinking Switch of o1-Like LLMs: Incorrect responses contain more frequent thought switching, causing models to spend additional tokens without improving accuracy.The paper presents this as a recurring pattern across all test sets.
- 2.3 Empirical Underthinking Results: For the Figure 2 example, approximately 7,270 tokens followed a promising first thought without reaching a correct answer, yielding ξUT = 0.946.The example treats the response as extremely inefficient because the first correct thought occurred after 411 tokens within a 7,681-token response.
- 2.2 Existence of Underthinking: Early reasoning thoughts are often correct but abandoned, leaving potentially valid solution paths insufficiently explored.More than 70% of incorrect responses contain at least one correct thought, and over 50% contain more than 10% correct thoughts.
- 2.3 Empirical Underthinking Results: The underthinking score measures token inefficiency in incorrect responses by comparing total response tokens with tokens through the first correct thought.If no correct thought exists, all response tokens are counted, distinguishing lack of understanding from underthinking.
- 2.3 Empirical Underthinking Results: Underthinking varies by dataset: DeepSeek-R1-671B has higher accuracy but higher UT scores on MATH500-Hard and GPQA Diamond, while AIME2024 shows higher accuracy and lower UT.The paper notes that greater model capability does not uniformly reduce underthinking.
3 Mitigating Underthinking Issues
The paper introduces TIP, a decoding-time penalty that discourages premature thought switching and encourages deeper development of individual reasoning paths. Across challenging test sets, TIP improves accuracy and reduces underthinking without fine-tuning, though penalty settings require calibration.
- 3.1 Decoding with Thought Switching Penalty: TIP reduces the probability of thought-switching tokens during a specified window, encouraging the model to continue expanding its current reasoning thought.The method modifies logits for switching tokens using penalty strength α and duration β; setting either parameter to zero disables the penalty.
- 3.1 Decoding with Thought Switching Penalty: 35.2% to 39.8% accuracy was obtained when β increased from 300 to 600 at α = 3, whereas overly long penalties could reduce accuracy at higher α.The authors selected α = 3 and β = 600 for subsequent experiments after grid-search tuning on AIME2022-23.
- 3.2 Experimental Results: TIP consistently improves accuracy over vanilla QwQ-32B-Preview across the three challenging test sets by mitigating premature thought switching.The approach operates during decoding and does not require model fine-tuning or architectural modification.
- 3.2 Experimental Results: TIP reduces the number of thought-switching tokens and increases the average interval between them across all test sets.These changes indicate deeper commitment to individual reasoning lines before considering alternatives.
- 3.2 Experimental Results: With Self-Consistency, TIP raises QwQ-32B-Preview’s Pass@4 accuracy on AIME2024 from 43.7% to 51.4% while lowering the Underthinking Score.TIP also improves R1-Distill-Qwen-32B with Laconic Decoding, increasing Pass@4 from 74.4% to 78.0% and reducing the Underthinking Score from 8.7 to 7.3.
- 3.2 Experimental Results: TIP’s consistent improvements across models, sampling methods, and test sets support its generalizability despite hyperparameters being tuned on AIME2022-23 with QwQ-32B-Preview.The method complements sampling-based strategies by promoting thorough exploration within individual reasoning paths.
4 Related Work
Prior work frames efficient reasoning as a complement to deep thinking, distinguishes underthinking from overthinking, and uses decoding penalties to control generation behavior. This paper applies decoding-penalty ideas specifically to underthinking by discouraging frequent strategy shifts.
- Efficient Thinking: Test-time compute research expands candidate-solution search or seeks human-like deep thinking through iterative reasoning.The latter direction includes Chain-of-Thought and reinforcement-learning-based o1 replicas.
- Efficient Thinking: Overthinking wastes computation by revisiting trivial or self-evident paths, whereas underthinking prematurely abandons promising reasoning paths.The paper positions these as contrasting inefficiencies in o1-like models.
- Decoding Penalties: Prior decoding methods use penalties to address shortcomings of likelihood-focused greedy and beam search.Examples include length normalization and coverage penalties for controlling verbosity, repetition, and source coverage.
- Decoding Penalties: This paper adjusts decoding penalties to preserve the model’s current reasoning line and encourage deeper thought instead of frequent strategy shifts.The authors describe this as the first investigation of decoding penalties for mitigating underthinking.
5 Conclusion
The paper identifies underthinking as a performance-limiting inefficiency in o1-like LLMs, quantifies it through token efficiency, and proposes TIP to mitigate it. TIP reduces underthinking and improves accuracy across difficult mathematical and scientific problem sets without additional model training.
- 5 Conclusion: Underthinking limits o1-like LLM performance when models prematurely abandon promising reasoning paths on challenging tasks.The conclusion links this behavior to inefficient problem-solving and lower accuracy.
- 5 Conclusion: The proposed metric quantifies underthinking by assessing token efficiency in incorrect responses.It measures how much of an incorrect response contributes to reaching correct thoughts.
- 5 Conclusion: TIP encourages models to thoroughly explore each reasoning thought before considering alternatives.The strategy uses a thought switching penalty to mitigate premature transitions.
- 5 Conclusion: TIP provides a practical approach for improving problem-solving capabilities and understanding reasoning processes in o1-like LLMs.The paper also identifies adaptive self-regulation of thought transitions as a future direction.