Source-linked AI summary
Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
Wenkai Yang, Shuming Ma, Yankai Lin, Furu Wei
TL;DR
The paper asks whether increasing CoT length can harm reasoning rather than consistently improve it. It proposes TOPS, which teaches varied reasoning efforts and selects shortest correct responses for self-improvement. On mathematical benchmarks, Qwen2.5-32B-based models outperform distillation-based 32B o1-like models and achieve performance comparable to QwQ-32B-Preview.
Problem
The paper investigates whether excessively scaling CoT length can impair LLM reasoning performance, beyond the known efficiency problem of overthinking.
Method
TOPS uses seed responses with varied reasoning efforts, then selects the shortest correct response across efforts to self-improve the model.
Results
Qwen2.5-32B-based self-improved models outperform distillation-based 32B o1-like models across various math benchmarks and are comparable to QwQ-32B-Preview.
Takeaways & Limitations
Effective test-time scaling requires matching reasoning effort to problem difficulty rather than uniformly extending CoTs.
Takeaways & Limitations
The analysis mainly focuses on mathematical reasoning and the method primarily uses supervised fine-tuning, leaving other domains and reinforcement-learning settings for future study.
Abstract
from arXiv · showhide
Recent studies have shown that making a model spend more time thinking through longer Chain of Thoughts (CoTs) enables it to gain significant improvements in complex reasoning tasks. While current researches continue to explore the benefits of increasing test-time compute by extending the CoT lengths of Large Language Models (LLMs), we are concerned about a potential issue hidden behind the current pursuit of test-time scaling: Would excessively scaling the CoT length actually bring adverse effects to a model's reasoning performance? Our explorations on mathematical reasoning tasks reveal an unexpected finding that scaling with longer CoTs can indeed impair the reasoning performance of LLMs in certain domains. Moreover, we discover that there exists an optimal scaled length distribution that differs across different domains. Based on these insights, we propose a Thinking-Optimal Scaling strategy. Our method first uses a small set of seed data with varying response length distributions to teach the model to adopt different reasoning efforts for deep thinking. Then, the model selects its shortest correct response under different reasoning efforts on additional problems for self-improvement. Our self-improved models built upon Qwen2.5-32B-Instruct outperform other distillation-based 32B o1-like models across various math benchmarks, and achieve performance on par with the teacher model QwQ-32B-Preview that produces the seed data.
1 Introduction
The paper examines whether extending LLM reasoning traces always improves mathematical reasoning, finding that excessive scaling can hurt performance and motivating Thinking-Optimal Scaling. TOPS teaches varied reasoning efforts and selects shortest correct responses for self-improvement.
- 1 Introduction: Longer Chain-of-Thought reasoning increases test-time compute and has improved complex reasoning in o1-like systems.Follow-up studies use distillation or reinforcement learning to scale reasoning tokens.
- 1 Introduction: Excessively scaling CoT length can impair LLM reasoning performance in certain mathematical domains, beyond causing inefficient overthinking.The paper frames this as a central question under current test-time scaling research.
- 1 Introduction: Preliminary comparisons find that QwQ-32B-Preview generates many more tokens than corresponding System-1 models while yielding only limited performance improvements.The authors use MATH500 and AIME2024 to compare accuracy and reasoning-token usage.
- 1 Introduction: TOPS teaches a tag model to generate responses under varied reasoning efforts, then selects the shortest correct response across efforts for self-improvement.The strategy is designed to let models choose how many tokens each problem requires.
- 1 Introduction: TOPS models based on Qwen2.5-32B-Instruct outperform existing distillation-based 32B o1-like models across varied math benchmarks and approach QwQ-32B-Preview performance.The supplied passage states this as the paper’s reported outcome without specifying benchmark values.
2 Related work
Related work develops LLM reasoning through Chain-of-Thought and test-time scaling, including sampling, tree search, and in-context search. This paper distinguishes itself by studying performance degradation from excessive token scaling, not only overthinking inefficiency.
- LLM Reasoning: Chain-of-Thought enables LLMs to decompose problems into sub-goals and reason step by step toward answers.Mathematical reasoning is identified as a major application area for LLM reasoning research.
- Test-Time Scaling: Sampling-based test-time scaling generates multiple reasoning paths and selects among them using voting or best-of-N mechanisms.This is one of several categories of test-time scaling methods reviewed in the paper.
- Test-Time Scaling: Tree search-based scaling expands reasoning paths through tree-like trajectories, while in-context search supports search, backtracking, and re-exploration within one CoT.Tree-of-Thoughts, Monte Carlo Tree Search, and in-context search are cited examples.
- Novelty: Existing work addresses overthinking by shortening CoTs while preserving performance, whereas this paper studies cases where more tokens can degrade performance.The distinction concerns both reasoning effectiveness and efficiency.
3 The impact of scaling efforts on the effectiveness of test-time scaling
The authors examine whether increasing reasoning-token budgets consistently improves test-time scaling, first comparing existing o1-like models and then controlling for base-model differences. Their analyses find that excessive CoT length can hurt performance, especially on easier tasks, and motivate selecting reasoning effort adaptively.
- 3.2 Deeper explorations on the scaling process of CoT length: The controlled study compares LLaMA3.1-8B and Qwen2.5-32B models under Low, Medium, and High reasoning efforts across GSM8K, MATH500, and AIME2024.Responses are generated with effort-specific prompts and evaluated under the settings described by the authors.
- 3.2 Deeper explorations on the scaling process of CoT length: Longer CoTs can reduce reasoning performance in certain domains, particularly on easier tasks, despite the effectiveness of a small number of o1-like responses.The authors report this as a conclusion from their benchmark comparisons.
- 3.2 Deeper explorations on the scaling process of CoT length: The best-performing reasoning effort on a benchmark generally also produces the fewest distinct answers across five samples per prompt.Table 2 reports distinct-answer distributions together with average benchmark accuracy.
- 3.3 Analysis on the adverse effects of excessive length scaling: Higher reasoning effort increases both reasoning rounds and erroneous reasoning rounds, so excess wrong steps can harm learning even when final answers are correct.Controlled experiments further find better performance when loss is masked on identified erroneous steps.
- 3.3 Analysis on the adverse effects of excessive length scaling: TOPS defines an optimal response as the shortest correct System-2 response and combines format imitation, effort-conditioned generation, and self-improvement.Self-improvement selects the shortest correct response among outputs produced at different reasoning efforts.
4 Thinking-optimal test-time scaling
TOPS treats test-time scaling as a problem of choosing reasoning effort: shorter correct responses are preferred for easier problems, while harder problems can receive more tokens. It implements this through varied-effort format imitation, model-selected shortest-correct responses, and self-improvement.
- Thinking-Optimal Scaling: TOPS defines a thinking-optimal response as the shortest correct System-2 response, balancing insufficient reasoning against overthinking.The method treats fewer tokens as potentially incorrect and additional tokens as potentially error-inducing.
- Format Imitation: A small seed set of o1-like responses under different reasoning efforts teaches the model System-2 thinking formats and variable effort levels.The seed responses cover patterns such as searching, reflecting, verification, and backtracking.
- Reasoning Effort-Conditioned Generation: The tag model generates solutions to additional math problems under multiple reasoning efforts, and TOPS selects the shortest correct solution for each problem.These selected responses form the thinking-optimal self-improvement dataset.
- Self-Improvement: The thinking-optimal dataset is used to fine-tune the base model so it can improve its System-2 reasoning through self-improvement.The described self-improvement stage performs supervised fine-tuning on the selected dataset.
5 Experiments and analysis
Experiments use Qwen2.5-32B-Instruct as the primary base model, evaluate TOPS across three math benchmarks, and test generalizability on LLaMA3.1-8B-Instruct. TOPS consistently outperforms its random-sample counterpart, improves efficiency on easier tasks while allocating more reasoning to harder ones, and iterative preference optimization reaches performance comparable to QwQ-32B-Preview.
- Experimental settings: Qwen2.5-32B-Instruct is the primary base model, with additional experiments on LLaMA3.1-8B-Instruct testing generalizability across architectures.The main evaluation uses GSM8K, MATH500, and AIME2024.
- Experimental settings: The evaluation compares TOPS and iterative TOPS models with QwQ-32B-Preview, STILL-2-32B, Sky-T1-32B-Preview, and the same-base-model baseline.Table 3 reports results on GSM8K, MATH500, and AIME2024; Table 4 reports LLaMA3.1-8B-Instruct results.
- Experimental settings: The reported results include both accuracy and the number of CoT tokens used on each dataset.For STILL-2-32B and Sky-T1-32B-Preview, token counts include only thought-part tokens, excluding summary tokens.
- Main results: Qwen2.5-32B-TOPS consistently outperforms Qwen2.5-32B-Random, supporting selection of model-determined shortest correct responses for test-time scaling.TOPS also achieves better results than STILL-2-32B and Sky-T1-32B-Preview across the reported benchmarks except AIME2024, where it slightly underperforms STILL-2-32B.
- Main results: Qwen2.5-32B-TOPS uses fewer reasoning tokens on easier GSM8K tasks while spending more time on harder AIME2024 problems.The reported token pattern reflects adaptive reasoning depths and mitigates overthinking on easier tasks.
- Results of iterative self-improvement: Iterative self-improvement samples eight responses per additional problem and selects the shortest correct response, with incorrect responses used for preference optimization.The procedure uses additional MATH problems and AIME1983-2023 problems.
- Results of iterative self-improvement: Iterative SFT mainly shortens CoTs without necessarily improving reasoning performance, whereas preference optimization improves both efficiency and effectiveness.The resulting preference-optimized model is comparable to QwQ-32B-Preview.
6 Conclusion
The paper shows that overly long CoTs can harm mathematical reasoning in some domains and introduces Thinking-Optimal Scaling to adapt reasoning effort. Its self-improved Qwen2.5-32B-Instruct models outperform existing distillation-based o1-like models and match QwQ-32B-Preview across math benchmarks.
- Overly long CoTs can negatively impact reasoning performance in certain mathematical domains.
- Thinking-Optimal Scaling teaches varying reasoning efforts from a small seed set, then selects the shortest correct response for self-improvement.
- Qwen2.5-32B-Instruct models trained with this strategy outperform existing distillation-based o1-like models across various math benchmarks.
- The self-improved models achieve comparable performance with QwQ-32B-Preview across various math benchmarks.
7 Limitations
The work’s limitations concern its domain and training setting: the analysis mainly covers mathematical reasoning and focuses on supervised fine-tuning. The authors identify broader-domain and reinforcement-learning evaluation as future directions.
- The analysis mainly focuses on mathematical reasoning because math tasks provide relatively accurate and reliable performance verification.The authors note that general reasoning results are only preliminary and appear in Appendix J.
- The work primarily studies the supervised fine-tuning setting rather than reinforcement learning.The authors describe CoT-length effects in reinforcement learning as a compelling direction for future work.
- The authors suggest that reinforcement learning may prefer shorter correct solutions because it assigns positive rewards to solutions with correct final answers.
E.5 Evaluation settings
Evaluation uses standardized decoding and hardware settings, with results averaged across random seeds. The supplied passages also identify tables covering difficulty breakdowns and response-length distributions.
- All o1-like models use decoding temperature 1.0, average results over 5 random seeds, and have a maximum generation length of 16,384.
- All evaluations are conducted on 4 × NVIDIA A100 80G.
- Table 6 reports tag-model results on MATH500 categorized by problem difficulty levels.
- Table 7 reports raw response-length rankings for QwQ-32B-Preview under different reasoning-effort prompts.
F Breakdown results of tag models on MATH500
The breakdown results show that increasing reasoning effort can reduce performance on lower-level problems, while controlled response-length ordering is needed to study CoT effects reliably.
- 92.09 (Low) -> 87.44 (Medium/High) and 97.67 (Low) -> 96.74 (Medium/High) on Level 1 problems show adverse effects from increased reasoning effort.These figures correspond to LLaMA3.1-8B-Tag and Qwen2.5-32B-Tag, respectively.
- In nearly 15% of cases, low-effort prompts produced the longest response among the three reasoning-effort conditions.
- Training responses were reordered so lengths increased consistently from low to medium to high reasoning effort.This controlled setting supports fairer observation of CoT-length effects on reasoning performance.
- Direct-prompting results for QwQ-32B-Preview and QwQ-32B across three benchmarks re-validate that longer CoTs do not necessarily improve performance.
- Table 9 reports performance after filtering out solutions identified as containing erroneous steps.
I Results of fine-tuning on samples after filtering out solutions with erroneous steps
Filtering out solutions with erroneous steps shortens generated CoTs but has domain-dependent effects: it improves GSM8K while degrading harder benchmarks.
- Filtering solutions with erroneous steps produces much shorter CoTs.Removed solutions are typically lengthy because they contain more self-reflections and self-corrections.
- GSM8K performance improves significantly after removing solutions with erroneous steps.GSM8K is relatively simple and does not require excessive reflection and correction.
- MATH500 and AIME2024 performance degrades after removing solutions with erroneous steps.Harder benchmarks benefit from deeper thinking, including correcting errors made in previous reasoning rounds.
- Longer reasoning chains with reflections and corrections are more beneficial on challenging datasets than on GSM8K.The comparison suggests that removing these behaviors helps simpler tasks but removes useful deeper-thinking behavior on harder tasks.
J Results on general reasoning tasks
General reasoning experiments show that excessive longer-CoT scaling can hurt performance, while reasoning effort conditioning and Thinking-Optimal Scaling remain effective on general tasks.
- Excessive scaling with longer CoTs can negatively affect performance on general reasoning tasks.This finding is reported as consistent with the mathematical reasoning results.
- More reasoning effort is needed for stronger performance on general reasoning tasks.The passage reports this pattern on MMLU-Pro and GPQA-Diamond under reasoning-effort-conditioned prompting.
- The general-reasoning evaluation uses one sample per prompt because of evaluation cost.Results are reported for MMLU-Pro and GPQA-Diamond in Table 10.
- Qwen2.5-7B-TOPS-General performs well on general reasoning tasks.It is evaluated on a held-out WebInstruct-verified set using the described setup.
- Qwen2.5-7B-Random-General is evaluated alongside Qwen2.5-7B-TOPS-General after random scaling on a held-out WebInstruct-verified set.Table 10 reports the evaluation results for both scaling strategies.
K Standard deviation results
The paper provides detailed standard deviation results in supplementary tables for the reported evaluations.
- Detailed standard deviation results are provided in Table 11 and Table 12.The tables are supplied for reference.
- Table 12 contains detailed standard deviation results on Qwen2.5-32B-Instruct.