Source-linked AI summary
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu
TL;DR
o1-like models can overallocate test-time compute to simple problems, raising questions about whether scaling is efficient and intelligent. The paper analyzes this overthinking, introduces outcome- and process-based efficiency metrics, and uses self-training to streamline reasoning. Across benchmarks of varying difficulty, the approach reduces unnecessary computation while maintaining model performance.
Problem
o1-like models often expend excessive tokens or thinking rounds on simple questions, with overthinking contributing minimally to accuracy and lacking diverse strategies.
Method
The paper introduces outcome- and process-based efficiency metrics and uses self-training to remove redundant solutions while preserving basic reflexivity.
Results
The approach reduces unnecessary computation while maintaining performance across reasoning benchmarks with varying difficulty levels.
Takeaways & Limitations
The findings support more efficient test-time resource use without compromising model performance across the evaluated reasoning benchmarks.
Takeaways & Limitations
Future work is needed to explore adaptive compute strategies and refine efficiency metrics for broader model generalization.
Abstract
from arXiv · showhide
The remarkable performance of models like the OpenAI o1 can be attributed to their ability to emulate human-like long-time thinking during inference. These models employ extended chain-of-thought (CoT) processes, exploring multiple strategies to enhance problem-solving capabilities. However, a critical question remains: How to intelligently and efficiently scale computational resources during testing. This paper presents the first comprehensive study on the prevalent issue of overthinking in these models, where excessive computational resources are allocated for simple problems with minimal benefit. We introduce novel efficiency metrics from both outcome and process perspectives to evaluate the rational use of computational resources by o1-like models. Using a self-training paradigm, we propose strategies to mitigate overthinking, streamlining reasoning processes without compromising accuracy. Experimental results show that our approach successfully reduces computational overhead while preserving model performance across a range of testsets with varying difficulty levels, such as GSM8K, MATH500, GPQA, and AIME.
1 Introduction
o1-like models improve reasoning by scaling test-time compute, but often overthink simple problems with little accuracy benefit. This study introduces efficiency metrics and self-training strategies to reduce redundant computation while preserving performance.
- Motivation: Scaling test-time compute lets o1-like models use extended reasoning, multiple strategies, decomposition, and double-checking to improve responses.The approach allocates more computational resources during inference to generally yield more accurate answers.
- Overthinking Problem: 1,953% more tokens were consumed by o1-like models than conventional models for the same answer to “what is the answer of 2 plus 3?”The paper also reports 13 generated solutions for this trivially simple question.
- Efficiency Evaluation: Two efficiency metrics assess o1-like models from outcome and process perspectives, extending evaluation beyond commonly used effectiveness metrics.The metrics are motivated by inefficient inference and the need to match reasoning complexity to problem requirements.
- Mitigation: 48.6% fewer tokens were produced on MATH500 while maintaining accuracy through self-training that removes redundant solutions and preserves basic reflexivity.The approach was evaluated across testsets with varying difficulty levels, including GSM8K, MATH500, GPQA, and AIME.
- Contributions: The study analyzes overthinking, proposes outcome- and process-based efficiency metrics, and explores strategies that reduce token generation while maintaining performance.These contributions are reported across testsets of varying difficulty.
2 Observing Overthinking Issues
The analysis finds that o1-like models often overthink, generating multiple redundant solutions and allocating more rounds to easier math problems despite increasing token use with difficulty. Outcome- and process-oriented metrics reveal limited gains in accuracy and reasoning diversity from later solutions.
- 2.1 Solution Distribution of o1-Like Models: o1-like models typically generate 2 to 4 solution rounds, covering 76% to 80% of QwQ responses and 59% to 63% of DeepSeek-R1 responses across test sets.The study examines QwQ-32B-Preview and DeepSeek-R1 on ASDIV, GSM8K, and MATH500, ordered by increasing difficulty.
- 2.1 Solution Distribution of o1-Like Models: 3.7 and 4.6 average solution rounds occur on MATH500 levels 1-2, compared with 3.0 and 3.9 on levels 4-5 for QwQ and DeepSeek-R1, respectively.Token counts consistently increase with difficulty, so easier problems receive more solution rounds despite requiring fewer tokens.
- 2.2 Efficiency on Accuracy Improvements: More than 92% of correct responses reach the correct answer in the initial solution round, while that round uses less than 60% of total generated tokens.For QwQ on ASDIV, the first round averages 287 tokens, or 38.7% of the full response, indicating limited accuracy gains from later solutions.
- 2.2 Efficiency on Accuracy Improvements / 2.3 Efficiency on Diverse Thinking: The outcome efficiency metric ξO measures efficient tokens contributing to correct answers, while the process efficiency metric ξP measures later solutions’ contribution to diversity.The outcome metric uses total tokens, efficient tokens, and whether an instance receives a correct answer; the process metric uses distinctness ratios.
- 2.3 Efficiency on Diverse Thinking: Later solutions often repeat earlier reasoning strategies: the average distinctness ratio for Solution#≥4 decreases by 11.5% compared with Solution#3.The example groups thirteen solutions into seven strategy clusters, including repeated basic arithmetic, stepwise, and counting-analogy solutions.
- 2.4 Empirical Efficiency Results: Both models achieve less than 50% outcome efficiency on MATH500 Level 1, showing that overthinking is especially pronounced on the simplest math problems.This pattern corresponds with the easy ASDIV results and supports the broader finding that efficiency is hindered by redundant generated tokens.
3 Mitigating Overthinking Issues
The paper uses self-training and response simplification to reduce overthinking while preserving reasoning performance. It evaluates preference optimization and simplification strategies across testsets with different difficulty levels.
- Self-Training: Self-training generates training data for improving the efficiency of o1-like model responses.The QwQ-32B-Preview model is used as the testing platform, with PRM12K supplying training data for self-training.
- Preference Optimization: The study compares SFT, DPO, RPO, and SimPO as self-improvement methods for efficient reasoning.SFT uses positive synthetic data, while the preference optimization methods train on preferred and unpreferred response pairs.
- Response Simplification: First-Correct Solutions retains the earliest correct solutions, while FCS+Reflection adds a second correct solution to preserve reflective reasoning.Greedily Diverse Solutions instead expands responses when they introduce new reasoning strategies rather than repeating earlier reasoning.
- Response Simplification: FCS is most efficient, FCS+Reflection uses approximately one additional solution round, and Greedily Diverse Solutions provides a compromise between solution count and token use.For each instance, the shortest result of each simplification type is selected from 10 samples.
- Preference Optimization: 22.3% fewer generated tokens makes SimPO the strongest length-preference method on MATH500 and the default post-training method.SFT only slightly reduces solution rounds and tokens relative to vanilla QwQ-32B-Preview, whereas SimPO achieves the best reduction among the compared methods.
- Experimental Results: First-Correction Solutions achieves the greatest length reduction but lowers performance on difficult MATH500, while FCS+Reflection surpasses it by 1.4%.The paper adopts FCS+Reflection as the default response simplification method because difficult problems may require additional reflection.
- Experimental Results: The approach improves performance on easier ASDIV and GSM8K testsets with fewer tokens and maintains performance on challenging GPQA and AIME.These experiments are presented as evidence that the method addresses overthinking across reasoning tasks of varying difficulty.
4 Related Work
Prior work scales test-time compute through broader search and human-like reasoning patterns, but efficient thinking in o1-like models remains underexplored. This paper distinguishes its focus by training models to think efficiently rather than imposing user-specified limits.
- Expanding Search Space: Test-time compute scaling expands the search space through self-consistency, best-of-n, minimum Bayes risk, and structured search methods.These approaches generate or organize multiple candidate reasoning paths before selecting an answer.
- Human-Like Thinking Patterns: Human-like thinking methods encourage step-by-step reasoning through chain-of-thought, debate, self-correction, self-critique, and planning.These methods model extended natural-language reasoning rather than relying only on direct answers.
- Efficient Thinking: Few studies focus on efficient thinking, and prior efficiency methods mainly address conventional models or user-controlled token budgets.In contrast, this work targets overthinking in o1-like models and trains them to learn efficient reasoning.
5 Conclusion
The study identifies inefficient test-time computation as a key challenge in o1-like LLMs and analyzes overthinking through efficiency metrics. Its self-training approach reduces unnecessary computation while maintaining performance across reasoning benchmarks.
- Conclusion: The paper identifies efficient and intelligent scaling of test-time computational resources as a key challenge in o1-like LLMs.Its analysis centers on how these models use computational resources during reasoning.
- Conclusion: The study presents a comprehensive analysis of overthinking and proposes efficiency metrics to improve understanding of resource utilization.The metrics are intended to assess efficiency in o1-like models from the study’s analysis of overthinking.
- Conclusion: The self-training approach mitigates overthinking by reducing unnecessary computation while maintaining performance across reasoning benchmarks of varying difficulty.The paper identifies adaptive compute strategies and broader efficiency-metric generalization as future directions.
A.1 Case Overview for Deepseek-R1-Preview
Figure 8 presents a Deepseek-R1-Preview response to the simple query “What is the answer of 2 plus 3?” The figure serves as a case overview of the model’s response to elementary arithmetic.
- Case Overview: The case overview concerns the Deepseek-R1-Preview model.
- Case Overview: The queried task is determining the answer to 2 plus 3.
- Case Overview: Figure 8 displays the model response for this arithmetic query.
A.2 Prompts for Clustering Solutions
The clustering prompt groups mathematical solutions according to differences in approach, intermediate reasoning, or assumptions, using a strict output format with named clusters and reasons.
- Solutions are clustered when their fundamental approaches differ, such as algebraic manipulation versus geometric reasoning.
- Solutions may form separate clusters when their intermediate steps or reasoning processes vary significantly, even if their final results match.
- Solutions relying on different assumptions or conditions are treated as distinct according to the clustering criteria.
- The prompt supplies a mathematical problem followed by multiple candidate solutions for clustering.
- The required output lists each cluster, its solution names, and the reason for grouping, without additional explanation.
- The example distinguishes an algebraic volume-solving cluster from a cluster focused on verification, consistency, units, and logical reasoning.