Source-linked AI summary
Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs
Jinyan Su, Jennifer Healey, Preslav Nakov, Claire Cardie
TL;DR
The paper addresses whether longer reasoning reliably improves LLM correctness and studies this question through sample-level and question-level empirical analyses. It finds that models overthink easier problems, underthink harder ones, and can shorten generation substantially while retaining relatively strong accuracy.
Problem
Existing evidence indicates that the relationship between reasoning length and correctness is complex, but when longer reasoning helps or merely adds computational and latency overhead remains insufficiently studied.
Method
The study empirically analyzes generation length and correctness at sample and question levels across reasoning models, mathematics benchmarks, and difficulty patterns, then evaluates preference optimization for shorter responses.
Results
Models underthink questions they answer incorrectly and overthink easier questions, while preferring shorter generations reduces token length without significantly harming performance.
Takeaways & Limitations
Generation length is a meaningful signal of reasoning behavior and supports further work on models’ self-awareness in adapting length to problem difficulty.
Takeaways & Limitations
The analysis uses N = 10 samples per question and focuses on two models and two reasoning datasets, limiting statistical reliability and generality.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly optimized for long reasoning, under the assumption that more reasoning leads to better performance. However, emerging evidence suggests that longer responses can sometimes degrade accuracy rather than improve it. In this paper, we conduct a systematic empirical study of the relationship between reasoning length and answer correctness. We find that LLMs tend to overthink simple problems, generating unnecessarily long outputs, and underthink harder ones, failing to extend their reasoning when it is most needed. This indicates that models might misjudge problem difficulty and fail to calibrate their response length appropriately. Furthermore, we investigate the effects of length reduction with a preference optimization algorithm when simply preferring the shorter responses regardless of answer correctness. Experiments show that the generation length can be significantly reduced while maintaining acceptable accuracy. Our findings highlight generation length as a meaningful signal for reasoning behavior and motivate further exploration into LLMs' self-awareness in reasoning length adaptation.
1 Introduction
The paper examines when longer reasoning improves correctness and when it instead causes overthinking, underthinking, or unnecessary computational cost. It systematically analyzes reasoning length across models, benchmarks, sample-level variation, and question difficulty.
- Longer reasoning can introduce unnecessary steps and compounding errors, with accuracy sometimes declining beyond a length threshold.
- The study systematically analyzes generation length and correctness across two reasoning-capable models and the GSM8K and MATH benchmarks.The analysis considers both sample-level and question-level relationships.
- For a fixed question, accuracy generally rises with reasoning length initially but declines when responses become excessively long.
- Models often underthink difficult questions, while easier questions can receive unnecessarily long responses despite being answered correctly.
- Preference optimization toward shorter responses can reduce token length while preserving relatively strong accuracy without ground-truth supervision.Correct-response lengths also decrease by 10%–25%.
2 Related Work
Related work studies overthinking, adaptive reasoning budgets, inference-time compute allocation, and the existence of an optimal chain-of-thought length. These directions motivate calibrating reasoning length to problem difficulty and model capability.
- Overthinking: Prior work identifies overthinking as excessively lengthy reasoning and explores length rewards, concise reasoning paths, and other mitigation strategies.
- Adaptive Thinking: Adaptive-thinking research adjusts reasoning tokens, prompt complexity, or inference-time computation according to question difficulty or stopping conditions.
- Optimal Thinking: Theoretical work suggests performance initially improves with more reasoning steps but eventually decreases, implying an optimal CoT length.
3 Experimental Setting
The experiments use two reasoning models and two mathematics benchmarks, generating ten diverse sampled reasoning paths per question under fixed sampling settings. Each sample is represented by its output, token length, and correctness.
- Models and Dataset: The study evaluates R1-Distill and R1-Preview on the GSM8K and MATH mathematics benchmarks.
- Models and Dataset: For each question, the experiment generates N = 10 diverse reasoning paths using temperature T = 1.0, top-p = 1, and an 8k maximum token length.
- The sample-level setup records each output’s reasoning length as its number of output tokens and correctness according to ground truth.
4 Sample-Level Analysis
Sample-level analysis holds the question and model fixed while comparing independently sampled responses of different lengths. Accuracy follows a non-monotonic pattern, with correct responses often concentrated among shorter or moderately long samples.
- Holding a question and model fixed isolates generation-length variation from differences in problem difficulty.
- Accuracy Across Length-Ranked Samples: Accuracy initially improves as length-ranked samples become longer, then declines after an excessive-length threshold across models and datasets.
- Accuracy Across Length-Ranked Samples: Peak accuracy occurs at r∗= 1 on MATH and r∗= 3 on GSM8K for R1-Distill, while R1-Preview peaks at r∗= 1 on GSM8K and r∗= 0 on MATH.
- Overly brief responses may lack sufficient reasoning, whereas excessively long outputs can harm both accuracy and efficiency.
- Shortest Correct Response: For over 60% of questions across all model–dataset combinations, the shortest sampled response is already correct.
- Correct responses are often concentrated among shorter or moderately long samples, making further generation a potential source of overthinking errors.
5 Question-Level Analysis
Question-level analysis shows that response length is negatively associated with correctness overall, while models adapt length to perceived difficulty inconsistently across easy and hard questions.
- Incorrect responses are significantly longer than correct responses across all four model–dataset combinations, especially on MATH.On MATH, incorrect responses average more than 6,000 tokens while correct responses average fewer than 3,000 tokens for both models.
- The analysis compares shared-easy and model-specific advantage sets using response-length averages and t-tests, with significance determined at p < 0.05.The hard-question analysis similarly compares shared-hard and model-specific advantage subsets.
- Average response length increases on each model’s advantage set relative to the shared-easy set, indicating length sensitivity to finer-grained difficulty among easy questions.For R1-Preview, GSM8K lengths rise from 1015.37 to 1348.60, while MATH lengths rise from 1690.62 to 2832.86.
- For hard questions, shared-hard responses are sometimes shorter than model-specific advantage-set responses, showing inconsistent adaptation to increased difficulty.The models may underestimate difficulty or lack the capacity to generate reasoning whose length reflects the problem’s complexity.
- Across questions with different accuracy, response length generally rises and then falls as accuracy improves rather than decreasing monotonically.For R1-Preview on GSM8K, perplexity also rises initially before declining, unlike the steady perplexity decrease observed for the other model–dataset pairs.
6 Effect of Length Preference Optimization
The paper tests whether preference optimization that favors shorter responses, regardless of correctness, can reduce generation length without substantially harming accuracy. It finds that length falls mainly because incorrect responses become shorter.
- Figure 4 tracks testing accuracy and average token length across SimPO training steps.
- Preference pairs favoring shorter responses can reduce generation length without relying on correctness labels.The approach uses SimPO and unlabeled data, addressing the limitations of correctness-dependent preference-pair construction.
- Accuracy remains within an acceptable range while average token length decreases by approximately 30% to 60% during training.Accuracy fluctuates across training steps but does not show significant degradation.
- Incorrect responses are generally much longer than correct responses, making them the main source of the overall length reduction.On GSM8K, incorrect-response length drops significantly; the reduction is less prominent for incorrect responses on MATH.
- Figure 5 tracks average token length separately for correct and incorrect responses across training steps.
7 Conclusion
The paper concludes that reasoning-oriented LLMs miscalibrate response length: they underthink questions they answer incorrectly and overthink easier questions they solve correctly. Shorter-generation preferences can reduce token use without significantly harming performance.
- Models struggle to assess problem difficulty and adjust generation length appropriately for questions they ultimately answer incorrectly, suggesting underthinking.
- For easier problems they consistently solve correctly, models estimate difficulty better but still generate unnecessarily long responses, indicating overthinking.
- Simply preferring shorter generations may reduce token length without access to correctness signals and without significantly harming performance.
8 Limitations and Future Work
The paper identifies limits in sample size and experimental scope, while proposing inference-time refinements and further study of calibration and self-correction. These directions aim to test generality and clarify how models adapt reasoning length.
- Limitations: The sample-level analysis uses N = 10 samples per question, and larger N could improve statistical reliability and enable finer-grained insights.
- Limitations: The study covers two datasets and two representative reasoning-capable LLMs, so broader models and difficulty levels are needed to assess generality.
- Future Work: Deprioritizing excessively long generations could refine multiple-chain inference strategies such as self-consistency for accuracy and efficiency.
- Future Work: The differing behaviors on easy and challenging questions suggest a connection between model calibration and the success of self-critique or self-correction.
- Implementation: The prompt template used to generate the data is given in Figure 6.
A.2 Metrics
The metrics evaluate how reasoning length relates to correctness at both the sample and question levels. Sample-level metrics compare ranked reasoning paths for the same question, while question-level metrics aggregate length and correctness across samples and question sets.
- Sample-Level Metrics: Sample-level analysis sorts each question’s N samples by reasoning length to create a length-based ranking.
- Sample-Level Metrics: The sample-level metrics test whether shorter or longer reasoning paths for the same question tend to yield higher accuracy.
- Implementation: Figure 6 presents the prompt template used to generate responses.
- Question-Level Metrics: Question-level analysis relates each question’s average reasoning length to its correctness averaged over N samples.
- Question-Level Metrics: Overall response-length analysis compares average token lengths for correctly and incorrectly answered questions using L✓ and L×.
B.1 Distribution of Extremes of Rank Indices.
The rank-based analysis identifies where correct answers first appear and where additional reasoning begins to undermine correctness. Correct answers often occur among the shortest responses, while excessive reasoning can introduce errors.
- The rank r✓max(q) marks the longest reasoning length at which correctness remains preserved; lower values indicate that further length introduces noise or errors.
- For R1-Preview, over 80% of GSM8K and MATH questions have a correct shortest sample; the proportions exceed 60% for GSM8K and 70% for MATH with R1-Distill.
- For most settings, the cumulative proportion of questions solved by the shortest responses rises rapidly and plateaus after i > 1.Thus, the top two shortest generations are sufficient for nearly all questions the model can solve, except R1-Distill–GSM8K.
- Approximately 5% of R1-Preview questions correct at rank N−2 become incorrect at the next longer response, compared with around 10% for R1-Distill.The reported degradation occurs near the longest few samples and may result from exceeding the maximum token length or introducing errors.
- For a fixed question and model, token length and correctness show strong rank-level correlation, while Pearson correlation exhibits the same trend as Spearman correlation.The analysis evaluates all questions and a medium subset whose sampled responses vary in correctness.
D Pair-wise Heatmap of Accuracy, Token Length and Perplexity
The heatmaps show that easier questions generally receive shorter responses, whereas extremely difficult questions have dispersed generation lengths and more extreme perplexity values. Most questions cluster at both short length and low perplexity.
- For high-accuracy questions, generation lengths cluster around shorter responses, while accuracy-0 questions disperse across the full length range.
- Perplexity remains mostly within a similar range across accuracy levels, but low-accuracy questions contain more prominent high-perplexity outliers.Easier questions have tighter perplexity distributions with fewer extreme values.
- Most questions occupy the joint region of low perplexity and short generation length.
- During SimPO training, GSM8K shows significantly greater length reduction for incorrect than correct responses, whereas MATH shows similar reduction rates for both.