PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum

arXiv:2503.24290v2cs.LGcs.CL

TL;DR

The paper addresses how to scale reasoning-oriented reinforcement learning directly on base language models with a simple and accessible open-source approach. It uses minimalist PPO-based training with verifiable rewards and analyzes the resulting training behavior. ORZ achieves strong benchmark performance with substantially fewer training steps while providing resources and practical insights for further study.

  • Problem

    Large-scale RL has shown scaling of reasoning performance and response length, but accessible open-source training directly on base models remains a research focus.

  • Method

    ORZ trains base LLMs with vanilla PPO using GAE (λ=1, γ=1), straightforward terminal rule-based rewards, and no KL regularization, while analyzing training dynamics and advantage estimation.

  • Results

    ORZ achieves superior performance across AIME24, MATH500, and GPQA Diamond versus DeepSeek-R1-Zero-Qwen-32B while requiring 1/10 of the training steps.

  • Takeaways & Limitations

    A minimalist recipe can scale reasoning capabilities and response length, while the learned critic identifies repetitive patterns and supports more robust advantage estimation.

  • Takeaways & Limitations

    The authors identify data, model, multimodal, sequence-length, and test-time scaling as areas for future investigation.

Abstract

from arXiv · show

We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training on the base model focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE ($λ=1$, $γ=1$) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both benchmark performance and response length, replicating the scaling phenomenon observed in DeepSeek-R1-Zero. Using the same base model, Qwen2.5-32B base, as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance across AIME2024, MATH500, and GPQA Diamond, while demonstrating remarkable efficiency, requiring only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline. Moreover, our analysis not only covers training dynamics and ablation for critical design choices, but also quantitatively shows how the learned critic in Reasoner-Zero training effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stability. Embracing the principles of open-source, we release our source code, training data, and various model weights, fostering reproducibility and encouraging further exploration of the properties of related models.

1. Introduction

Open-Reasoner-Zero introduces an open-source approach to scaling reasoning-oriented RL directly on base language models. The work studies a simple, robust training recipe and reports scalable reward and response-length growth across model sizes.

  • The study examines large-scale RL on reasoning tasks because prior systems showed benchmark performance and response length increasing with training computation.The authors extend this exploration to base models through the Reasoner-Zero training paradigm.
  • Open-Reasoner-Zero is an open-source implementation of large-scale reasoning-oriented RL training directly on base LLMs.It is designed to be robust, scalable, and simple to follow.
  • Vanilla PPO with GAE (λ=1, γ=1), simple rule-based rewards, and no KL regularization are reported as sufficient for steady scalability.The result holds across varying model sizes when training uses large-scale, carefully curated datasets.
  • ORZ-32B achieves superior performance on AIME24, MATH500, and GPQA Diamond compared with DeepSeek-R1-Zero-Qwen-32B while requiring 1/10 of the training steps.
  • The project releases source code, training data, model weights, and critic weights to support reproducibility and further research.
  • Training reward and response length increase consistently across ORZ models from 0.5B to 32B, although ORZ-32B response length fluctuates without harming training stability.

2. Scale-up Reinforcement Learning from a Base Model

ORZ scales reasoning-oriented reinforcement learning directly from base models using PPO, learned value estimation, diverse data, and minimal reward and regularization choices. Its design emphasizes stable, scalable training while avoiding preliminary fine-tuning.

  • RL algorithm: ORZ trains base language models directly with PPO, generating response trajectories and assigning each a terminal binary reward based on answer correctness.For each prompt, the policy samples multiple responses; rewards are 1 for exact matches and 0 otherwise.
  • RL algorithm: A learned critic supports token-level value estimation and more robust advantage estimation by identifying and devaluing repetitive behaviors.This motivates selecting PPO over GRPO, which lacks a dedicated value network.
  • Design principles: Setting GAE γ=1 and λ=1 captures long-term dependencies and provides stable training, while vanilla PPO requires no additional algorithmic modifications.The paper reports that these GAE choices substantially affect reasoning-task performance.
  • Design principles: Removing KL loss and penalty improves training stability, performance, and response-length scaling while reducing computation, tuning burden, and implementation complexity.The comparison evaluates configurations with and without KL-related regularization.
  • Data and implementation: The minimalist rule-based reward checks answer correctness without format rewards, and unaligned base models quickly learn the desired response format.The approach directly launches RL from Qwen2.5-7B and 32B base models without preliminary SFT or distillation.

3. Experiments

Experiments show that Open-Reasoner-Zero scales reasoning performance, response length, and training reward across model sizes while remaining efficient and stable. Ablations identify GAE λ=1, no KL regularization, larger data scale, and PPO’s learned critic as important design choices.

  • 3.1. Training Results: Training reward and average response length consistently improve for ORZ-32B, ORZ-7B, ORZ-1.5B, and ORZ-0.5B.The authors interpret these trends as evidence that models learn desired reasoning behaviors.
  • 3.1. Training Results: Correct responses containing reflection steps remain longer than average responses, and both length metrics increase throughout training.Representative reflection patterns include “wait,” “recheck,” “retry,” “alternatively,” and “however.”
  • 3.2. Ablation Study: GAE λ=1.0 yields faster reward growth and stable response-length increases, whereas λ=0.95 produces slower rewards and collapsed length dynamics.The experiments associate λ=1.0 with better training stability and generation quality.
  • 3.2. Ablation Study: Omitting KL loss and penalty achieves the best reported training stability, performance, and response-length scaling while reducing computation and tuning complexity.The comparison is conducted on ORZ-7B.
  • 3.2. Ablation Study: The ORZ 57k dataset sustains reward and response-length improvements, while MATH train 7.5k reaches early plateaus.This comparison highlights the role of dataset scale in training performance.
  • 3.3. Analysis for Critic and Advantage Estimation: PPO assigns more negative advantages to repetitive tokens than GRPO, while its critic assigns lower values to repetitive states than coherent states.These value and advantage patterns provide a more robust signal against undesirable generation patterns.
  • 3.4. Evaluation Results: ORZ-32B outperforms DeepSeek-R1-Zero-Qwen2.5-32B across reported benchmarks while requiring an order of magnitude fewer training steps.ORZ also matches DAPO on AIME, outperforms it on MATH500 and GPQA Diamond, and uses fewer training iterations.
  • 3.4. Evaluation Results: Pure scaled-up RL on reasoning tasks enables ORZ-32B to significantly outperform Qwen2.5-Instruct-32B on MMLU and MMLU_PRO without additional instruction tuning.The results are presented as evidence of generalization capability.

4. Related Work

Related work distinguishes RL applied directly to base models from RL applied to reasoning-enhanced models. ORZ positions itself as a fully open, scalable Reasoner-Zero framework with broad configurations, verified data, and strong training efficiency.

  • Scaling RL on Base Models for Reasoning: Reasoner-Zero training applies RL directly to base language models to develop complex reasoning skills.Earlier work established this direction, while several subsequent studies proposed detailed training recipes.
  • Scaling RL on Base Models for Reasoning: ORZ is presented as the first fully open-source implementation of large-scale RL applied directly to base language models for reasoning.The framework is contrasted with pilot studies that provided more limited training recipes.
  • Scaling RL on Base Models for Reasoning: DAPO matches ORZ’s AIME performance with roughly fivefold more training iterations, while underperforming on other benchmarks; VAPO reaches about 60% of ORZ’s score at the same iteration budget.The passage attributes possible differences partly to data-processing strategies and reports that ORZ uses a simpler algorithm.
  • Scaling RL on Reasoning-Enhanced Models: Reasoning-enhanced models commonly receive SFT distillation or other cold-start preparation before RL, which is described as improving subsequent training stability.This line of work differs from direct base-model Reasoner-Zero training.
  • Our Contributions: ORZ contributes an open framework spanning 0.5B–32B models, extensive configurations, a large verified reasoning dataset, and state-of-the-art training efficiency.The stated contributions also include a simple scalable algorithm implementation for future research.

5. Conclusion and Future Directions

ORZ is presented as an open-source, minimalist approach to scaling reasoning-oriented RL, with released resources supporting reproducibility. Future work targets scaling data, models, test-time computation, and reasoning scenarios.

  • ORZ is introduced as a comprehensive open-source implementation of large-scale reasoning-oriented RL.
  • Vanilla PPO with GAE and simple rule-based rewards scales reasoning capabilities without KL regularization.
  • The released code, data, and model resources are intended to support reproducibility and further community research.
  • Future directions include scaling training data, model capacity, test-time computation, and scenario complexity.

A. More Evaluation Results

Evaluation across ORZ model sizes covers multiple reasoning benchmarks and reports consistent reasoning improvements as parameter counts increase. Table 4 summarizes benchmark performance across the model sizes.

  • ORZ evaluation covers 0.5B, 1.5B, 7B, and 32B models on AIME 2024, AIME 2025, MATH500, and GPQA Diamond.
  • Consistent reasoning improvements with increased model size demonstrate strong scaling properties for the minimalist RL setup.
  • Table 4 reports reasoning-oriented benchmark performance across Open-Reasoner-Zero model sizes.

B. Detailed Setting for Training

The training setup initializes separate policy and critic networks from Qwen-2.5 base models and uses packed, largely on-policy optimization. Evaluation and training materials include a generation template, multi-response sampling, and an additional difficult-prompt stage for the 32B model.

  • Policy and critic networks are initialized from Qwen-2.5 base models, with separate weights and a randomly initialized critic value head.
  • The training and evaluation materials include a prompt template in which the prompt is replaced by a specific reasoning question.
  • The policy uses strict on-policy optimization, while the critic processes experiences in 12 mini-batches per iteration.
  • Each generation step samples 128 unique prompts and produces 64 responses per prompt with temperature and top-p set to 1.0.
  • The 32B variant adds an annealing stage based on 13k prompts where the model produced fewer than 4 correct answers in 64 attempts during early training.

C.1. More Analysis for Critic and Advantage Estimation

The analysis contrasts PPO and GRPO stability in the ORZ-7B setting. GRPO develops severe mid-training degeneration, whereas PPO maintains stable rewards and low truncation and repetition scores.

  • GRPO instability manifests as deteriorating generation quality and repetitive or incoherent text midway through training.
  • GRPO becomes severely unstable around 240 training steps, with reward destabilization and Truncate Rate and Average Repeat Score reaching 1.0.
  • PPO maintains stable rewards and low Truncate and Repeat Scores throughout the ORZ-7B training comparison.

C.2. Ablation on Data Curation

The ablation compares English-only data with combined English-and-Chinese data, finding that English-only curation improves training stability and final performance.

  • The study evaluates English-only versus combined English-and-Chinese data curation strategies.
  • The comparison is motivated by reported degradation on Chinese subsets associated with simpler question patterns.
  • English-only data yields superior training stability and final model performance.

D. Derivation and Code for PPO with GAE(1, 1)

With γ=1 and λ=1, GAE simplifies by summing reward and value differences across the trajectory; with terminal-only reward, it reduces to return minus the current value estimate.

  • Substituting γ=1 and λ=1 makes each temporal-difference term r_t+k + V_ϕ(s_t+k+1) − V_ϕ(s_t+k).
  • The resulting GAE sums temporal-difference terms from timestep t through the trajectory end.
  • Intermediate rewards are zero, so the summed rewards equal the trajectory’s single terminal reward R.
  • The value-difference terms telescope to V_ϕ(s_T) − V_ϕ(s_t), with terminal value V_ϕ(s_T)=0.
  • Therefore, the simplified advantage estimate is R − V_ϕ(s_t).
  • The appendix also provides the algorithm’s initialization requirements and final policy output.
Loading 2503.24290v2…