Source-linked AI summary
Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, Xipeng Qiu
TL;DR
The paper asks whether o1-like models truly scale at test time as their reasoning chains grow. It analyzes CoT length, self-revision, and sequential versus parallel scaling, then proposes Shortest Majority Vote. Longer CoTs do not consistently improve accuracy, while parallel scaling and the proposed length-aware voting method substantially improve scalability.
Problem
Whether QwQ, R1 and LIMO truly possess test-time scaling capabilities that consistently improve performance with longer CoTs remains unverified.
Method
The study compares CoT length, self-revision, sequential scaling and parallel scaling across QwQ, R1 and LIMO, and proposes Shortest Majority Vote.
Results
Longer CoTs do not consistently improve accuracy; parallel scaling has better coverage and scalability, and Shortest Majority Vote significantly outperforms traditional majority voting.
Takeaways & Limitations
For these o1-like models, test-time scaling is limited sequentially but strong in parallel, and solution length can improve majority-vote scaling.
Takeaways & Limitations
The experiments use static model checkpoints, leaving test-time scaling with dynamic reinforcement-learning checkpoints for future research.
Abstract
from arXiv · showhide
The advent of test-time scaling in large language models (LLMs), exemplified by OpenAI's o1 series, has advanced reasoning capabilities by scaling computational resource allocation during inference. While successors like QwQ, Deepseek-R1 (R1) and LIMO replicate these advancements, whether these models truly possess test-time scaling capabilities remains underexplored. This study found that longer CoTs of these o1-like models do not consistently enhance accuracy; in fact, correct solutions are often shorter than incorrect ones for the same questions. Further investigation shows this phenomenon is closely related to models' self-revision capabilities - longer CoTs contain more self-revisions, which often lead to performance degradation. We then compare sequential and parallel scaling strategies on QwQ, R1 and LIMO, finding that parallel scaling achieves better coverage and scalability. Based on these insights, we propose Shortest Majority Vote, a method that combines parallel scaling strategies with CoT length characteristics, significantly improving models' test-time scalability compared to conventional majority voting approaches.
1 Introduction
The paper tests whether longer reasoning chains provide reliable test-time scaling in o1-like models and finds that sequential scaling is limited, while parallel scaling and length-aware voting are more effective.
- QwQ, R1 and LIMO do not consistently improve accuracy as CoT length increases.
- Correct solutions are shorter on average than incorrect solutions for the same questions.
- Longer CoTs contain more self-revisions, while increased reflection degrades some models and produces oscillatory behavior in others.QwQ and R1-Distill-1.5b degrade with longer reflection; R1-Distill-14b, R1-Distill-32b and LIMO initially improve before oscillating.
- QwQ, R1 and LIMO have limited ability to revise incorrect answers into correct ones, and some models more often damage correct answers.
- Parallel scaling achieves better coverage and scalability than sequential scaling for QwQ and R1.
- Shortest Majority Vote prioritizes solution clusters with more solutions and shorter lengths, substantially outperforming conventional Majority Vote.
2 Related Work
The related work frames test-time computation as parallel sampling or sequentially lengthened reasoning, while prior studies disagree about whether self-revision works without external feedback.
- Test-time compute scaling allocates additional inference computation through parallel or sequential scaling.
- Parallel Scaling: Parallel scaling samples multiple solutions and selects one using signals such as rewards or model uncertainty.
- Sequential Scaling: Sequential scaling generates progressively longer solutions, commonly through iterative self-revision.
- Sequential Scaling: Prior work disputes whether models can self-refine effectively without external feedback, while other studies suggest self-evaluation can be learned or is feasible.
- o1-like Models: QwQ, R1 and LIMO are prominent open-source attempts to reproduce o1-like reasoning, but extending their solutions does not necessarily improve performance.
3 Experiment Setting
The experiments evaluate QwQ, LIMO and several R1 variants across mathematical and scientific benchmarks, analyzing solution-length groups and their accuracy.
- Models: The study evaluates QwQ, LIMO, R1-671b, R1-Distill-32b, R1-Distill-14b and R1-Distill-1.5b.
- Benchmark: Evaluations cover MATH-500, AIME, Omini-MATH and GPQA, spanning mathematical reasoning and broader scientific domains.
- Evaluation Analysis: Figure 2 groups QwQ and R1 solutions by length for each question, then reports group-level solution length and accuracy.
4 The Failure of Sequential Scaling
Across QwQ, R1 and LIMO, increasing CoT length does not reliably improve accuracy and can even produce inverse scaling. The evidence links this failure to self-revision: longer solutions contain more revisions, but models rarely correct wrong answers and may damage correct ones.
- 4.1 Invalid Scaling of CoT Length: Longer CoTs Do not Improve Performance: The longest sampled solutions are approximately twice the length of the shortest, revealing substantial within-model solution-length diversity.This pattern appears across QwQ, R1 and LIMO.
- 4.1 Invalid Scaling of CoT Length: Longer CoTs Do not Improve Performance: Solution length does not track model size and is shorter on Math than on harder datasets such as AIME.The comparison suggests solution length adapts to problem difficulty rather than simply to parameter count.
- 4.1 Invalid Scaling of CoT Length: Longer CoTs Do not Improve Performance: Accuracy does not consistently improve with solution length across model variants and datasets, with inverse scaling especially evident on AIME and Omini-MATH.The result challenges the assumption that longer reasoning chains inherently yield better performance.
- 4.1 Invalid Scaling of CoT Length: Longer CoTs Do not Improve Performance: For the same questions, correct solutions are consistently shorter than incorrect ones across QwQ, R1 and LIMO.The length gap is larger for weaker models such as QwQ and R1-Distill-1.5B than for stronger models such as R1-671b.
- 4.2 Explaining Invalid Scaling: The Key Factor is the Failure of Self-Revision: Longer solutions are generated mainly through self-revision markers such as “Wait” and “Alternatively,” whose frequency is strongly linearly correlated with solution length.This connects longer CoTs to increased self-correction activity rather than necessarily to better reasoning.
- 4.2 Explaining Invalid Scaling: The Key Factor is the Failure of Self-Revision: Sequential revision decreases QwQ and R1-Distill-1.5b accuracy, while R1-Distill-32b, R1-Distill-14b and LIMO improve initially before oscillating.For the latter models, successful-revision gains plateau after approximately 10 steps; incorrect-answer corrections remain below 10% overall.
5 Sequential Scaling vs. Parallel Scaling
Because sequential self-revision scales poorly, the paper compares it with parallel sampling and finds parallel scaling more effective and computationally efficient.
- Scaling strategies: Parallel scaling generates multiple solutions in parallel, while sequential scaling repeatedly revises a solution and selects the final revision.The comparison evaluates coverage and accuracy using 40 revision steps for sequential scaling and 10 sampled solutions for parallel scaling.
- Coverage: For the same number of generated tokens, parallel scaling produces a significantly larger coverage improvement than sequential scaling for R1-Distill-32b and QwQ.Coverage is measured using pass@k, based on whether candidate answers include a correct one.
- Scaling limits: Sequential revision benefits R1-Distill-32b up to 10k tokens, but further scaling yields no additional benefit.Majority voting provides no benefit when increasing sampled solutions from one to two because it requires at least three solutions to be effective.
- Computational cost: Parallel scaling has lower computational cost than sequential scaling when generating the same number of tokens because sequential revision attends over a longer context.The passage also reports stronger coverage and scalability for parallel scaling in the comparison.
6 Application of Our Findings: Shortest Majority Vote
The paper introduces Shortest Majority Vote, which combines parallel candidate sampling with solution length to improve answer selection over conventional majority voting.
- Method: Shortest Majority Vote groups solutions by answer, counts each category, and also computes its average solution length.Unlike Majority Vote, it uses both category size and length information when selecting the final answer.
- Scoring: The category score is designed to favor answers supported by more solutions and shorter average solution lengths.The final answer comes from the category with the highest score.
- Motivation: Shortest Majority Vote is especially intended for o1-like models whose performance deteriorates as solution length increases.Solution length also provides a selection signal when only a limited number of candidate solutions are available.
- Evaluation: On AIME and GPQA with 16 sampled solutions, Shortest Majority Vote significantly outperforms Majority Vote and the shortest-solution baseline, particularly on AIME.The experiments compare Majority Vote, Shortest, and Shortest Majority Vote using QwQ, R1, and LIMO.
7 Conclusion
The paper finds that longer solutions often underperform shorter ones and that sequential self-revision has limited effectiveness, motivating a length-aware parallel method.
- Conclusion: The study finds that shorter solutions often outperform longer ones and that sequential scaling through self-revision has limited effectiveness.These findings challenge the assumption that o1-like models automatically improve with increased test-time computation.
- Conclusion: The authors develop Shortest Majority Vote, a parallel scaling method that considers solution length and significantly outperforms traditional majority vote.The method is based on the paper’s observations about solution length and sequential scaling.
Limitations
The study’s scope is constrained by model cost, static checkpoints, and the possibility that shortest-solution selection is unsuitable for models with strong sequential scaling.
- Experimental scope: Evaluation of R1-671b was limited to Figures 1 and 2 because of its considerable cost; distilled R1 was used thereafter.This limits later experimental coverage for the full R1-671b model.
- Experimental scope: The experimental framework used static model checkpoints rather than dynamic checkpoints in reinforcement learning settings.The authors identify dynamic-checkpoint evaluation as a direction for future research.
- Method boundary: Shortest Majority Vote may have limited applicability to models with strong sequential-scaling capabilities.The authors suggest adapting it to a Longest Majority Vote variant in such cases.
- Length analysis: Correct solutions predominantly occur in shorter-length groups in AIME, although their tokens are concentrated in longer-solution groups.The apparent contradiction reflects both the number of solutions and the average tokens per solution; the longest group is approximately twice as long as the shortest.
- Interpretation: The authors hypothesize that reinforcement learning may favor longer solutions because they contribute more tokens to the gradient, even when less accurate.This is presented as a hypothesis explaining the observed length pattern, not as an established causal result.
B Further analysis on Sequential Scaling on R1-Distill-14b, R1-Distill-32b and LIMO
Self-revision improves short solutions more reliably than long ones, helping explain why correct solutions are generally shorter and why sequential scaling gains can stabilize.
- R1-Distill-14b, R1-Distill-32b and LIMO showed performance improvements after multiple self-revision rounds, followed by stabilization.
- Correct solutions generated by R1-Distill-14b, R1-Distill-32b and LIMO were generally shorter than incorrect solutions.
- Figure 10 compares Majority Vote with Shortest Majority Vote on GPQA.
- Short solutions showed significant performance improvements after self-revision, whereas improvements were less pronounced for long solutions.
- Improvements from self-revision primarily came from revising short solutions in R1-Distill-14b, R1-Distill-32b and LIMO.
C Parallel Scaling of Shortest Majority Vote on GPQA
On GPQA, Shortest Majority Vote consistently outperforms the Shortest method, but does not substantially surpass Majority Vote in parallel scaling performance.
- Shortest Majority Vote consistently outperforms the Shortest method on GPQA.
- On GPQA, Shortest Majority Vote does not show significantly better parallel scaling performance than Majority Vote.
- The paper attributes this GPQA phenomenon potentially to the small size of the benchmark.
- The evaluation instructions for MATH-500, AIME and Omini-MATH request final answers enclosed in boxed{} format.
- Multiple-choice responses must output only the correct option letter enclosed in boxed{} format.
E Examples of self-revision
The examples show self-revision through repeated verification, alternative approaches and reconsideration of intermediate assumptions or answers.
- The examples use explicit verification prompts such as “let me verify that again” to initiate revision.
- Self-revision examples repeatedly recheck constants, consistency and possible mistakes after an initial solution.
- One revision explicitly questions whether the answer is 756 and then checks whether it is consistent.
- Another example revisits a geometric assumption by questioning whether centers in 3D space are coplanar.
- A further example reformulates a divisibility condition by considering m dividing k for k from 1 to 999.
- The examples include switching to alternative reasoning strategies, such as roots of unity, general formulas and modular constraints.