Source-linked AI summary
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, Matthias Bethge
TL;DR
The paper addresses weak transparency, robustness, and statistical grounding in mathematical reasoning evaluations. It systematically reassesses benchmarks and recent methods under standardized practices, finding unstable measurements, modest RL gains with overfitting, and stronger SFT generalization. Code, prompts, and outputs are released to support reproducibility.
Problem
Mathematical reasoning evaluations often rely on heterogeneous and insufficiently rigorous practices, leaving reported progress vulnerable to unclear comparisons and unreported variance.
Method
The paper conducts an empirical study of mathematical reasoning benchmarks, proposes standardized evaluation best practices, and reevaluates recent methods with a reproducible evaluation stack.
Results
Across the studied settings, RL generally yields modest or non-significant gains and is prone to overfitting, while SFT provides consistent, generalizable improvements.
Takeaways & Limitations
Reliable reasoning progress requires standardized, transparent evaluation protocols and explicit control of benchmark and implementation variability.
Takeaways & Limitations
OpenReasoner-Zero is identified as a consistent exception to the reported pattern of limited RL-related performance.
Abstract
from arXiv · showhide
Reasoning has emerged as the next major frontier for language models (LMs), with rapid advances from both academic and industrial labs. However, this progress often outpaces methodological rigor, with many evaluations relying on benchmarking practices that lack transparency, robustness, or statistical grounding. In this work, we conduct a comprehensive empirical study and find that current mathematical reasoning benchmarks are highly sensitive to subtle implementation choices--including decoding parameters, random seeds, prompt formatting, and even hardware and software configurations. Performance gains reported in recent studies frequently hinge on unclear comparisons or unreported sources of variance. To address these issues, we propose a standardized evaluation framework with clearly defined best practices and reporting standards. Using this framework, we reassess recent methods and find that most reinforcement learning (RL) approaches yield only modest improvements--far below prior claims--and are prone to overfitting, especially on small-scale benchmarks like AIME'24. In contrast, supervised finetuning (SFT) methods show consistently stronger generalization in the settings we study. To foster reproducibility, we release all code, prompts, and model outputs, for reasoning benchmarks, establishing more rigorous foundations for future work.
1 Introduction
The paper argues that rapid progress in LLM reasoning has outpaced methodological rigor, motivating a careful study of mathematical reasoning benchmarks and standardized evaluation practices. It finds that evaluation choices can substantially affect reported results, while releasing tools to support reproducibility.
- Reasoning research has accelerated rapidly, but evaluation practices remain vulnerable to non-reproducible or inconclusive conclusions.The paper links these concerns to potential distortion of scientific understanding, adoption decisions, and research priorities.
- The study investigates inconsistent empirical claims by rigorously examining mathematical reasoning benchmarks, a widely used testbed for algorithmic advances.
- Minor choices involving decoding, prompts, random seeds, hardware, and software can significantly distort reasoning-evaluation conclusions.
- AIME’24 and AMC’23 contain only 30–40 examples, so one question can shift Pass@1 by over 3 percentage points and seeds can produce double-digit swings.
- Under standardized reevaluation, RL methods generally provide limited or non-significant gains and show overfitting, whereas SFT delivers more stable, generalizable improvements.OpenReasoner-Zero is noted as a consistent exception among the discussed RL-related results.
- The paper releases code, prompts, and outputs to make reasoning evaluation more transparent, robust, and reproducible.
2 Related Works
Related work places this study within broader concerns that rapid machine-learning progress is often evaluated without sufficient rigor. It emphasizes the need for statistically careful, standardized comparisons in reasoning research, especially for RL-based methods.
- The paper reevaluates recent reasoning claims under a standardized lens against well-tuned baselines, finding that many reported gains do not hold strongly.
- Sobering Studies on ML Progress: Rapid publication across machine learning has produced cases where simple, well-tuned baselines outperform claimed progress in several subfields.
- Sobering Studies on ML Progress: Reasoning research requires statistically rigorous model comparisons because evaluation standardization across variability factors is rarely reported for RL algorithms.
3 Exploring the Design Space of Reasoning: What Matters Most?
Reasoning benchmark results are highly sensitive to evaluation design choices, including seeds, sampling parameters, generation limits, prompts, and inference systems. The study therefore finds that reliable comparisons require standardized settings, multi-seed evaluation, and explicit systems reporting.
- 3.2 Seed Variance in Evaluation: Pass@1 standard deviation ranges from 5 to 15 percentage points across seeds, with small benchmarks producing especially unstable results.AIME’24 and AMC’23 contain only 30 and 40 test samples, respectively; one question shifts Pass@1 by 2.5–3.3 percentage points.
- 3.2 Seed Variance in Evaluation: Single-seed evaluations on small datasets are highly unstable and require averaging over multiple seeds for accurate reporting.The analysis evaluates 20 independent runs for each of nine models.
- 3.2.1 Can Bootstrapping Improve Mean Estimates?: K ≥30 bootstrap runs sharply reduce variance in AIME’24 Mean Pass@1 estimates, whereas variance remains extreme for small K.The study examines 1 to 60 evaluation runs and identifies K = 30 as the point where variance is significantly reduced.
- 3.2.2 Variance from Sampling Parameters: Temperature and top-p: Higher temperatures improve peak accuracy but reduce stability, while higher top_p values generally improve performance without increasing variance.The paper recommends tuning these parameters per model and comparing the best parameter setting for each model.
- 3.3 Effect of Prompt Format and Context Length: Reducing max_new_tokens, omitting native chat templates, and changing hardware or software configurations can measurably lower or alter benchmark accuracy.Hardware and software differences persist at 2–5%, while OpenRS-1.5B and DeepSeek-R1-Distill-7B vary by up to 8% and 6% on AIME’24; consecutive runs on identical A100 hardware range from 19.3% to 23.0% for Bespoke-Stratos-7B.
4 Way Forward: Standardization in Evaluations
The paper standardizes evaluation and re-evaluates reasoning methods, finding stronger generalization for SFT than for most RL approaches, which often overfit.
- The study standardizes evaluation frameworks, proposes best practices, and comprehensively evaluates existing methods.
- RL methods applied to DeepSeek R1-Distill generally yield little or no statistically significant improvement over the baseline.OpenRS, Still-3, and Light-R1 failed to replicate reported gains, while some newer methods showed significant improvements.
- RL training on Qwen base models produces significant gains, but these are often comparable to instruction-tuning improvements and do not generalize from AIME’24 to AIME’25.
- SFT methods consistently outperform instruction-tuned baselines across benchmarks and generalize comparatively well to AIME’25.The results highlight reasoning-trace data curation and larger training datasets as important contributors.
- At 32B scale, SFT remains the most reliable approach, while larger models achieve substantially higher absolute performance than smaller counterparts.OpenThinker reaches 71.3% and 68.5% on AIME’24, while QwQ-32B reaches 76.3%.
- RL-trained models are susceptible to overfitting, whereas SFT models show stronger generalization and resilience.
5 Do Discovered Phenomena Replicate? A Detailed Analysis.
The study tests whether reported reasoning-model phenomena replicate and finds consistent length-related errors, diversity loss after RL, and unexpectedly strong spurious-prompt effects.
- The analysis examines response length, output diversity after reasoning-focused training, and spurious-prompt effects.
- 5.1 Are Incorrect Responses Longer?: Shorter responses are more likely to be correct, while longer responses become progressively more error-prone across six benchmarks.
- 5.1 Are Incorrect Responses Longer?: The length–error relationship holds for both RL- and SFT-trained models and is more pronounced in RL-trained models.
- 5.1 Are Incorrect Responses Longer?: Excessive length remains associated with reasoning failures even in complete responses well below the 32,000-token limit.
- 5.2 Is There Diversity Collapse in Reasoning Training?: RL-trained models show a minor diversity collapse: gains in Pass@1 generally coincide with lower Pass@k performance.The reported decay varies in magnitude.
- 5.3 Do Spurious Prompts Improve Performance?: The spurious \lipsum prompt produces unexpectedly high MATH500 accuracy and surpasses the dedicated Math prompt for Qwen2.5-Math-7B-Instruct.
6 Conclusion
The conclusion argues that mathematical reasoning progress is often unstable and non-reproducible, and presents standardized protocols and open resources as a foundation for more rigorous evaluation.
- Perceived progress in LLM mathematical reasoning often rests on unstable and non-reproducible foundations.
- RL methods provide at best modest gains and are prone to overfitting, while SFT delivers more consistent, generalizable improvements across benchmarks and model sizes.
- The released framework includes Dockerized environments, seed-averaged metrics, and robust answer matching to support reproducible research.
A Bootstrapping Results on Additional Datasets
Additional bootstrapping analyses show that small-benchmark estimates remain volatile with few runs, while larger datasets have smaller variance but can still vary across seeds.
- High Pass@1 variance persists on AMC’23 and AIME’25 even with K = 5 evaluation runs.
- MATH500, Minerva, and Olympiad Bench remain volatile across seeds, but their larger sizes produce much smaller variance than AIME’24.
- The evaluation uses Math, Default, and No Template prompt formats adapted to each model’s chat template.
C Hardware Differences
Hardware and environment differences produce non-trivial performance variation on MATH500, extending discrepancies previously observed on AIME and AMC.
- Hardware-induced performance variance persists on MATH500, including on this larger test set.The figure attributes these shifts to differences in GPU type and environment.
D Effect of Output Length Limits
Model accuracy is sensitive to max_new_tokens across models with both long and shorter context limits. Restricting output length can truncate reasoning chains and reduce accuracy.
- Shortening max_new_tokens causes clear accuracy drops for OpenRS models with 131,072-token context windows.The reported mechanism is premature truncation of incomplete reasoning chains.
- OpenThinker-7B and S1.1-7B also degrade when output length is constrained despite 32,768-token context limits.The same sensitivity appears across models with shorter context windows.
- Appropriate generation limits are important because premature truncation produces incomplete reasoning chains and incorrect answers.
E Response Length vs. Accuracy — Per-Model Breakdown
Across nearly all analyzed models, incorrect responses tend to be longer than correct responses. This relationship persists below maximum sequence length, making response length a useful low-confidence signal.
- Incorrect responses tend to be longer than correct responses across nearly all models.Per-model histograms average response-length distributions over random seeds.
- Longer responses consistently mark incorrect outputs across the remaining model-wise analyses.The appendix splits the per-model histograms into two figures for clarity.
- The correlation persists below maximum sequence length, ruling out truncation as the sole cause.The paper associates excessively long outputs with hallucinated reasoning, verbose overthinking, or degenerate loops.
- Response length can serve as a signal for detecting low-confidence or erroneous reasoning chains.
F Diversity Collapse
The diversity-collapse results differ by training approach: RL-trained models show collapse, whereas the reported SFT-trained models do not. Additional datasets support this distinction for the evaluated models.
- SFT-trained DeepSeek-R1-Distill-1.5B could not be replicated as exhibiting a diversity collapse.
- RL-trained DeepScaleR-1.5B and FastCuRL-1.5B show diversity collapse, with ΔPass@k below zero relative to DeepSeek-R1-Distill-1.5B.All models used the decoding parameters listed in Table 8.
- SFT-trained DeepSeek-R1-Distill-1.5B and OpenThinker3-1.5B outperform their respective baselines in ΔPass@k across benchmarks.The corresponding baselines are Qwen2.5-Math-1.5B and Qwen2.5-1.5B-Instruct.
G Optimal Decoding Parameters
The paper identifies optimal temperature and top-p decoding settings for the evaluated models, as reported in Table 8.
- The listed decoding parameters consistently produced optimal performance for the evaluated models.
- Table 8 reports optimal temperature and top-p settings for various evaluated models.