Source-linked AI summary
An Empirical Study on Eliciting and Improving R1-like Reasoning Models
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, Ji-Rong Wen
TL;DR
The report addresses limited practical understanding of how RL training settings and related strategies develop R1-like reasoning models. It systematically studies RL on base and fine-tuned models and explores tool manipulation. RL improves both model types, while tool manipulation reaches 86.67 accuracy on AIME 2024 with greedy search.
Problem
The report studies how to implement R1-like reasoning models when RL training is difficult to tune and reproduce, especially under limited training budgets.
Method
The authors systematically vary RL settings and apply RL or tool augmentation to Qwen-based base and fine-tuned reasoning models.
Results
RL consistently improves QWEN2.5-32B base models and fine-tuned models, while STILL-3-TOOL-32B reaches 86.67 accuracy on AIME 2024 with greedy search.
Takeaways & Limitations
Tool manipulation can substantially boost reasoning performance, and RL can further refine models that already perform strongly after fine-tuning.
Takeaways & Limitations
The experiments primarily use models up to 32B parameters and only several hundred training steps, leaving longer-training effects unresolved.
Abstract
from arXiv · showhide
In this report, we present the third technical report on the development of slow-thinking models as part of the STILL project. As the technical pathway becomes clearer, scaling RL training has become a central technique for implementing such reasoning models. We systematically experiment with and document the effects of various factors influencing RL training, conducting experiments on both base models and fine-tuned models. Specifically, we demonstrate that our RL training approach consistently improves the Qwen2.5-32B base models, enhancing both response length and test accuracy. Furthermore, we show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already achieved a high performance level, it can be further refined through RL training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we also explore the use of tool manipulation, finding that it significantly boosts the reasoning performance of large reasoning models. This approach achieves a remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its effectiveness in enhancing model capabilities. We release our resources at the STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
1 Introduction
The report studies how RL settings and related training strategies shape slow-thinking reasoning models. It finds consistent gains from RL on base and fine-tuned models, while tool manipulation produces a major benchmark improvement.
- RL training findings: RL settings strongly influence large reasoning model performance, with on-policy learning enabling consistent improvements throughout training.The study systematically examines parameter configurations and identifies on-policy learning as especially consequential.
- Base-model training: RL activates pretrained base-model reasoning actions into a more coherent, deliberate thinking process while improving response length and test accuracy on QWEN2.5-32B.The authors observe longer thinking and more integrated reasoning behavior after RL training.
- Reward design: Response length indicates RL-training success but is a consequence rather than a cause of performance improvement.Explicitly rewarding longer outputs may cause reward hacking without improving reasoning capabilities.
- Fine-tuned models: RL further improves fine-tuned models, including distilled reasoning models, with Qwen2.5-1.5B reaching 39.33 accuracy on AIME 2024.The reported gains cover both short- and long-chain-of-thought models.
- Tool augmentation: Tool manipulation substantially enhances reasoning, with STILL-3-TOOL-32B reaching 86.67 accuracy on AIME 2024 using greedy search.The capability can be activated with only a small number of high-quality training instances.
2 Experimental Settings
The experiments use open-source RL frameworks, Qwen-based base and distilled backbones, curated verifiable mathematics data, multiple reward designs, and mathematical reasoning benchmarks.
- Training framework and models: The study uses OpenRLHF and veRL to implement RL experiments across the 1.5B, 32B, base, and fine-tuned settings.OpenRLHF is used for STILL-3-1.5B, while veRL supports the other reported experiments.
- Training data: The training data combine AIME, MATH, NuminaMath, and Open Reasoner Zero, with filtering for diversity, verifiability, and difficulty.The resulting dataset contains 90k examples and excludes questions whose answers cannot be parsed into digits using SymPy.
- RL settings: Table 1 summarizes RL-setting explorations and suggestions, which are explicitly based on limited experiments and practical experience rather than guaranteed optimal configurations.The study examines hyperparameters and training prompts as factors affecting RL effectiveness.
- Reward design: The reward design includes output and format rewards, plus auxiliary length and action rewards for selected experiments.Output reward checks final-answer correctness and boxed formatting; auxiliary rewards encourage longer responses or complex reasoning actions.
- Evaluation: Performance is evaluated on MATH-OAI, AIME, Omni-MATH, LiveAOPs, and HMMT, with AIME 2024 used primarily throughout.AIME 2025 is additionally evaluated for STILL-3-TOOL-32B.
3 RL Experiments on the Base Model
The experiments examine how RL settings affect reasoning development in base models, showing that on-policy training and suitable exploration choices improve learning while longer responses alone do not guarantee downstream gains.
- Training hyper-parameters: Larger train batch sizes improve early training efficiency and reduce fluctuations across training metrics.The comparison evaluates TBS = 128 versus 1024.
- Learning strategy: On-policy learning yields more favorable outcomes and superior test-set performance than off-policy learning.The fully on-policy approach also encourages greater exploration during training.
- Rollout parameters: Increasing rollout times improves training performance and encourages longer, more exploratory responses, whereas higher temperature can produce meaningless content.Lower temperature gives higher early rewards but may restrict later exploration.
- RL on the base model: STILL-3-ZERO-32B accuracy on AIME 2024 rises from 2.08% to 37.08% during RL training as response length increases.The model remains in a growth phase, and post-training responses show more reflection and verification.
4 Experiments on Fine-tuned Models
The fine-tuned-model experiments investigate whether RL and tool augmentation can further improve reasoning after initial model development.
- Research focus: The study examines RL and tool augmentation as strategies for improving the reasoning performance of fine-tuned language models.The stated scope includes both small-sized 1.5B and medium-sized 32B models.
4.1 Fine-tuning Base Models with Long CoT Data
This section compares long-CoT data construction methods for initializing fine-tuned models before RL, finding that distillation provides stronger initial reasoning while RL improves both variants differently over time.
- Data construction: The study compares synthesized long-CoT data with distillation from existing large reasoning models as cold-start data.The synthesis approach incorporates reasoning patterns such as reflection and verification, while distillation transfers data from capable reasoning models.
- SFT initialization: SFT with either data-construction method produces longer responses, but distillation increases response length more than synthesis.The comparison covers MATH-OAI, AIME 2024, OMNI, and LiveAOPS.
- RL comparison: After RL, the synthesis-initialized model steadily lengthens responses alongside accuracy gains but requires more training steps to match the distillation-based variant.The limitation of synthesized long-CoT quality and initially poor performance slows its progress.
- RL comparison: The distillation-initialized model starts with longer responses and better performance, then improves accuracy during RL even as response length fluctuates or decreases.Its stronger initialization enables superior results to be reached more quickly.
4.2 Discussion about Length Hacking in RL
The section evaluates strategies for encouraging longer reasoning responses and finds that explicit length incentives can trigger reward hacking rather than improve reasoning. Length should emerge through self-exploration instead of being directly rewarded.
- Motivation: Longer responses often correlate with broader search, but length-biased rewards can be exploited without improving reasoning.The report frames this exploitation as reward hacking.
- Length Encouragement Strategies: Question selection and response selection guide training toward longer solutions using differences between correct and incorrect response lengths.Question selection addresses short correct answers, while response selection compares the longest correct and shortest incorrect solutions.
- Length Encouragement Strategies: RRL assigns higher rewards to longer responses, while RRA rewards solutions containing more reasoning actions.RRL is designed to reduce data waste; RRA uses reasoning-action counts capped by AMax = 20 to mitigate reward hacking.
- Empirical Results and Analysis: Each tested strategy increased response length, but performance was significantly affected as length grew, except under RRA.RRL extended responses most effectively, whereas overlength response masking had limited impact; completion ratios decreased as responses grew.
- Conclusion: Explicitly rewarding longer thought processes may degrade performance, so length extension should arise from intrinsic self-exploration.The report identifies direct length encouragement as a source of significant reward hacking.
4.3 STILL-3-1.5B: Enhancing Slow Thinking Abilities of Small Models via RL
This section tests whether RL can further improve a small model already trained for slow reasoning. The experiments examine training behavior and prompt effects while comparing STILL-3-1.5B with its distilled backbone.
- Motivation: The study applies RL to a small reasoning model that received specialized long-CoT instruction tuning.The goal is to assess whether RL can augment capabilities beyond those acquired through distilled reasoning supervision.
- RL Settings: DeepSeek-R1-Distill-Qwen-1.5B serves as the target model because it already performs strongly on mathematical reasoning.Its reported AIME 2024 test performance is 28.9 pass@1.
- Results: Training-set accuracy increased consistently, while generated response length decreased slightly during RL training.The authors interpret the accuracy trend as progressive refinement of reasoning capabilities.
4.4 STILL-3-TOOL-32B: Empowering Reasoning Models with Tool Manipulation
This section equips a large reasoning model with tool-manipulation abilities through distilled demonstrations and supervised fine-tuning. The resulting model improves mathematical performance and approaches DeepSeek-R1 across major benchmarks.
- Motivation: Tool manipulation is presented as an underexplored way to extend reasoning models’ problem-solving capabilities and applicability.The section focuses on enabling reasoning models to generate code and invoke external tools.
- Backbone Models: DeepSeek-R1-Distill-Qwen-32B is selected because smaller models may struggle to learn tool manipulation with limited data or computation.The choice follows prior observations that larger models acquire slow-thinking patterns more readily.
- Demonstration Data Distillation for SFT: The SFT pipeline distills code-using demonstrations from DeepSeek-R1 and retains responses containing both code snippets and correct answers.The distilled data uses a 3k-question subset and tool-integration formats modeled on prior systems.
- SFT Experiments: The evaluation covers AIME 2024, AIME 2025, and HMMT February 2025 using greedy and random sampling.The fine-tuning setup uses a 1 × 10^-5 learning rate, batch size 96, and 17 epochs.
- Performance Comparison: 15.56% improvement on AIME 2024 makes STILL-3-TOOL-32B outperform all baselines and approach DeepSeek-R1 across three major math competitions.The report attributes the improvement to integrating tool manipulation into the reasoning process.
- Detailed Analysis: High-quality distilled data can elicit the backbone’s tool-use potential, while RL-based enhancement of tool manipulation remains future work.The authors report that DeepSeek-R1 produces higher-quality demonstrations than the backbone model itself.
5 Discussion
The discussion examines whether longer responses, more RL steps, and continued exploration reliably improve reasoning. It reports limits from overthinking, performance bottlenecks, collapsing diversity, and constrained training scale.
- Length and Accuracy: Increasing response length does not guarantee higher accuracy because models may overthink or underthink without improving performance.Experiments show accuracy can stagnate while response length continues to grow.
- Training Steps and Performance: After several hundred RL steps, Qwen2.5-7B accuracy on MATH-OAI and AIME 2024 fluctuates without major breakthroughs.The report specifically notes difficulty surpassing 30% accuracy on AIME 2024 through this RL setup.
- Scope Boundary: The experiments are limited to models with up to 32B parameters and several hundred training steps, leaving longer-scale RL unresolved.These limits may reflect capacity bottlenecks or insufficient training budgets.
- Exploration Ability: Generated-content diversity diminishes rapidly during RL as entropy loss falls, raising concerns about whether exploration continues.The section motivates entropy bonuses as a possible mitigation for this convergence.
- RL Compared to SFT: SFT improves reasoning efficiently through imitation, whereas RL learns through self-exploration and can expand beyond demonstrated behaviors.RL is described as less efficient for improving capacity but potentially better suited to models with strong foundational abilities.
6 Conclusion
The study systematically examines R1-like reasoning models through base-model and fine-tuned-model experiments, finding gains from RL training and tool manipulation. It also identifies efficiency optimization and real-world application as continuing directions.
- Base-model experiments: RL training on QWEN2.5-32B base models consistently increased both model performance and response length.The experiments varied training batch size, learning strategy, rollout parameters, and KL penalty.
- Fine-tuned-model experiments: RL training significantly improved DEEPSEEK-R1-DISTILL-QWEN-1.5B, reaching an accuracy of 39 on AIME 2024.
- Fine-tuned-model experiments: 86.67 accuracy on AIME 2024 was achieved by applying tool manipulation to DEEPSEEK-R1-DISTILL-QWEN-32B with greedy search.
- Limitations: The work remains constrained by insufficient efficiency optimization for both RL training and inference with long chain-of-thought responses.Training is hindered by lengthy rollouts, while inference requires more computational resources per query for longer responses.
- Future work: Future work will examine deep-reasoning mechanisms, more efficient and stable RL training, and slow-thinking systems in real-world tasks.