PaperlayerView exact revision on arXiv

Source-linked AI summary

SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild

Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, Junxian He

arXiv:2503.18892v3cs.LGcs.AIcs.CL

TL;DR

The paper addresses limited evidence about zero RL training beyond Qwen2.5 and studies its behavior across diverse open base models. Using direct RL from base models with simple rule-based rewards and design adjustments, it finds broad accuracy gains and distinct training dynamics, while response-length growth can be misleading.

  • Problem

    Prior zero RL studies largely focus on Qwen2.5 models that already possess instruction-following and reflective behaviors, leaving broader base-model behavior unclear.

  • Method

    The paper applies GRPO-based zero RL directly to diverse open base models, using GSM8K and MATH data while varying reward design, data difficulty, and prompting.

  • Results

    Zero RL significantly improves accuracy across all tested base models and increases response length for 9 of 10 models, while verification emerges in Llama3-8B and DeepSeek-Math-7B.

  • Takeaways & Limitations

    Reward design, query difficulty, and inherent model abilities shape zero RL success and the emergence of advanced reasoning behaviors.

  • Takeaways & Limitations

    Response-length increases can reflect repetitive, incoherent outputs rather than genuine reasoning growth, as observed for Mistral-7B-v0.1.

Abstract

from arXiv · show

DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can naturally emerge through a simple reinforcement learning (RL) framework with rule-based rewards, where the training may directly start from the base models-a paradigm referred to as zero RL training. Most recent efforts to reproduce zero RL training have primarily focused on the Qwen2.5 model series, which may not be representative as we find the base models already exhibit strong instruction-following and self-reflection abilities. In this work, we investigate zero RL training across 10 diverse base models, spanning different families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B, Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. Leveraging several key design strategies-such as adjusting format reward and controlling query difficulty-we achieve substantial improvements in both reasoning accuracy and response length across most settings. However, by carefully monitoring the training dynamics, we observe that different base models exhibit distinct patterns during training. For instance, the increased response length does not always correlate with the emergence of certain cognitive behaviors such as verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for the first time in small models not from the Qwen family. We share the key designs that enable successful zero RL training, along with our findings and practices. To facilitate further research, we open-source the code, models, and analysis tools.

1 Introduction

The paper investigates whether zero RL training generalizes beyond Qwen2.5 and how reasoning behaviors develop across diverse base models. It identifies design factors that improve training while showing that longer responses do not necessarily indicate cognitive advances.

  • Research questions: The study asks how reasoning develops, whether the “aha moment” occurs in models initially lacking strong instruction-following and self-verification, and which factors enable successful zero RL.
  • Scope and motivation: Zero RL training is examined across 10 base models spanning multiple families and sizes, rather than primarily Qwen2.5 models.The study includes Llama3-8B, Mistral-7B/24B, DeepSeek-Math-7B, Qwen2.5-Math-7B, and Qwen2.5 models from 0.5B to 32B.
  • Findings: Significant accuracy improvements occur across all base models, while response length increases for 9 of 10 models except Qwen2.5-Math-7B.
  • Findings: Longer responses do not always accompany increased self-reflection frequency, particularly among most Qwen2.5 models.
  • Findings: Verification frequency increases significantly for the first time in small non-Qwen models, notably Llama3-8B and DeepSeek-Math-7B.
  • Design factors: Successful zero RL depends on avoiding overly rigid format rewards, matching training-data difficulty to model exploration capabilities, and preserving exploration during RL.Rigid formatting can penalize exploration and induce overthinking, while SFT cold starts can suppress post-RL performance and advanced reasoning.
  • Findings: Zero RL raises pass@k accuracy by 10-30 absolute points, supporting the conclusion that it genuinely enhances models rather than merely reranking responses.

2 On Emerging Reasoning in Zero RL Training

The paper studies zero RL training across diverse open base models, finding broad gains in mathematical reasoning while showing that response length alone does not reliably indicate genuine reasoning changes. Training dynamics and emerging behaviors differ substantially by model, with verification increasing in several non-Qwen models.

  • 2.1 Experimental Setup: Zero RL training applies GRPO directly to open base models without prior supervised fine-tuning, using GSM8K and MATH data with correctness-based rewards.The experiments vary data difficulty across Easy, Medium, and Hard subsets, while avoiding format-based reward.
  • 2.3 Main Results: Zero RL training steadily improves response length and average accuracy across mathematical benchmarks, including competition-level AIME24 and AMC23.After 80 iterations, DeepSeek-Math-7B improves more than threefold while its response length grows from around 300 to over 1200 tokens.
  • 2.3 Main Results: Zero RL training also improves instruction-following, general knowledge, and domain-specific expertise on IFEVAL, MMLU, and GPQA-Diamond after training on 8K math examples.The evaluated generalization tasks cover instruction following, general knowledge, and chemistry, physics, and biology.
  • 2.3 Main Results: For Mistral-Small-24B, pass@8 grows robustly, while pass@1 eventually exceeds the base model’s initial pass@8 and the pass@1–pass@8 gap widens during training.The trained model retains a pass@k advantage over the base model from k=1 through k=128, supporting changes beyond reranking.
  • 2.4 The “Aha Moment” – Quantifying Emergence of Reasoning Behaviors: Response length can increase without genuine reasoning growth: unstable Mistral-7B training produces repetitive mixed-language gibberish and a high clip ratio.Reasoning-behavior ratios better account for stopping-length variation, and different models follow distinct behavioral trajectories.
  • 2.4 The “Aha Moment” – Quantifying Emergence of Reasoning Behaviors: Small Qwen models primarily increase subgoal setting, whereas DeepSeek-Math-7B, Llama-3.1-8B, and Mistral-Small-24B show roughly 3–4x growth in enumeration and verification.In Mistral-Small-24B, verification and backtracking rise from nearly 0% to approximately 50%, while Qwen-2.5-7B and 32B begin with strong reasoning behaviors and change little.

3 Key Factors Shaping Zero RL Training

Zero RL training is shaped by reward design, data difficulty, and starting conditions, which affect exploration, accuracy, response length, and reasoning behavior differently across models.

  • 3 Key Factors Shaping Zero RL Training: The paper investigates format rewards, data difficulty, and different SFT starting points as factors influencing zero RL stability and performance.SFT-start analysis is presented as a separate factor affecting exploration and reasoning behavior.
  • 3 Key Factors Shaping Zero RL Training: Reasoning-behavior ratios on OlympiadBench track variations in average stopped length, but their training trends differ substantially across models.The tracked behaviors are backtracking, verification, subgoal setting, and enumeration.
  • 3.1 Over-Reliance on Format Rewards: Strict format rewards can penalize correct exploration when base models initially struggle to follow required formats.The paper compares correctness-only rewards with rewards that also require boxed answers.
  • 3.1 Over-Reliance on Format Rewards: Format constraints can cause weaker models to lengthen responses without improving performance and can ultimately induce collapse.Llama-3.1-8B shows this pattern early in training, while Qwen-2.5-7B can degrade later despite initially following the format.
  • 3.2 Data Difficulty on Exploratory Behavior: Data difficulty changes exploratory behavior: harder data can trigger Mistral-7B training collapse, whereas Qwen-2.5-7B shows lower accuracy and response length on easier data.The comparison uses Easy, Medium, and Hard datasets, each containing approximately 8,000 problems.

4 Revisiting Traditional SFT for RL-Driven Reasoning Emergence

Starting RL from conventionally SFT-trained models initially improves performance but limits later accuracy, response length, and reasoning-behavior development relative to starting from the base model.

  • 4 Revisiting Traditional SFT for RL-Driven Reasoning Emergence: SFT cold starts use short-CoT NuminaMath responses, with Mistral 24B models trained for 100 or 500 steps before RL.The subset contains approximately 15K high-quality short-CoT responses derived from GSM8K and MATH.
  • 4 Revisiting Traditional SFT for RL-Driven Reasoning Emergence: SFT starting points initially boost performance but impose lower maximum accuracy and response length than base-model starts during RL.These limits become more pronounced as the number of initial SFT steps increases.
  • 4 Revisiting Traditional SFT for RL-Driven Reasoning Emergence: Initial SFT reduces the upper limits of enumeration, verification, and backtracking during RL compared with starting from the base model.The passage specifically reports this reduction for models initialized with 100 SFT steps.

5 Conclusion

SimpleRL-Zoo reports effective zero RL training across diverse base models, while showing that reward design, data difficulty, and starting conditions shape both performance and reasoning behavior.

  • 5 Conclusion: Zero RL training improves accuracy and response length across a diverse range of base models and provides evidence beyond response reranking.The paper frames these findings as evidence of genuine enhancement through training.
  • 5 Conclusion: Reward design, data difficulty, and inherent model abilities influence the emergence of advanced reasoning behaviors during zero RL training.The conclusion identifies these as key factors shaping training outcomes.
  • 5 Conclusion: Traditional SFT initialization may limit the development of advanced reasoning behaviors during subsequent RL training.This conclusion is consistent with the paper’s SFT comparison.

A Detailed Background: “Zero RL Training”

Zero RL training applies RL directly to a base model, while GRPO uses group-normalized rewards and a clipped policy objective to update the model without a separate value model.

  • A Detailed Background: “Zero RL Training”: Zero RL training means applying reinforcement learning directly from the base model without prior supervised fine-tuning.The experiments use the GRPO algorithm.
  • A Detailed Background: “Zero RL Training”: GRPO samples a group of responses from the old policy and uses their rewards to estimate relative advantages for policy optimization.The advantage compares each response with the average reward within the sampled group.
  • A Detailed Background: “Zero RL Training”: The objective uses a clipped policy update and a KL-divergence constraint to limit deviation from a reference model.The reference model is represented by πref.

B.1 Dataset

The study uses GSM8K and MATH data with correctness-based rewards, adapting prompts to model instruction-following ability across a diverse base-model zoo.

  • Dataset: Training data comes exclusively from GSM8K and MATH, with MATH500 reserved for testing and 500 additional problems used for validation.The remaining 4,000 MATH test problems are combined with 7,500 original training problems.
  • Reward: The reward function assigns +1 for a correct final answer and 0 for an incorrect answer.The setup uses correctness-only rewards rather than format-based reward rules.
  • Reward: Format-based rewards can hinder exploration and harm performance for base models that initially struggle with instruction following.Rigid output requirements may be especially restrictive during early training.
  • Models: Experiments cover Llama-3.1-8B, DeepSeekMath-7B, two Mistral models, and Qwen-2.5 models from 0.5B to 32B.The paper calls these models SimpleRL-Zoo because they use a simple recipe across a model zoo.
  • Prompts: Simpler step-by-step prompts are used for weaker instruction-following models, while stronger models receive prompts requiring boxed final answers.Complex prompts caused irrelevant or nonsensical early outputs for weaker models, leading to instability.
  • Evaluation: Evaluation spans GSM8K, MATH500, Minerva Math, OlympiadBench, AIME 2024, and AMC 2023.These include standard mathematical reasoning and competition-level benchmarks.

C Detailed Evaluation Metrics

The paper supplements accuracy and response length with metrics designed to track cognitive behavior, truncation, valid stopping, and exploration during training.

  • Reasoning Behavior Ratio: Reasoning Behavior Ratio measures the fraction of responses containing Backtracking, Verification, Subgoal Setting, or Enumeration.GPT-4o identifies these behaviors using a cognitive behavior framework.
  • Prompt Comparison: Figure 10 compares simple prompts with more complex prompts.The comparison concerns prompt design rather than a stated numerical outcome.
  • Clip Ratio: Clip Ratio is the proportion of outputs truncated because generated responses exceed the fixed maximum context length.Truncation can arise from irrelevant, repetitive, or excessively long outputs.
  • Average Stopped Length: Average Stopped Length tracks the mean length of responses that stop normally, excluding truncated generations.This avoids interference from repetitive or incomplete unstopped outputs.
  • Pass@k Accuracy: Pass@k Accuracy is the percentage of questions with at least one correct response among k sampled responses.It serves as an indicator of exploration because a sampled response must achieve positive reward.

D Detailed Result of SimpleRL

The trained base models are evaluated across six mathematical benchmarks, while training dynamics are examined through accuracy, response length, clip ratio, and stopped length.

  • Evaluation: Zero-trained models are evaluated on GSM8K, MATH 500, Minerva Math, OlympiadBench, AIME2024, and AMC2023.Average results across these benchmarks are presented in Figures 1 and 4.
  • Training Dynamics: Figure 11 shows accuracy and response-length trends across training steps, while Figure 12 shows clip-ratio and stopped-length trends.These figures provide detailed views of training dynamics for various models.

E Quantitative Behavior Validation

The paper validates GPT-4o behavior labels against human annotations and examines how supervised fine-tuning and sampling choices relate to subsequent reinforcement-learning outcomes.

  • Behavior Validation: Human experts annotate 105 model outputs to assess consistency with GPT-4o reasoning-behavior labels.Prediction rate measures behavior-identification frequency, while agreement rate measures matching human and GPT-4o labels.
  • Training Dynamics: Figure 11 evaluates accuracy and response length over training steps, while Figure 12 evaluates clip ratio and average stopped length.The plotted quantities are identified by their respective colored lines.
  • Behavior Validation: Table 3 presents the consistency between GPT-4o-labeled reasoning behaviors and human annotations.The table summarizes the validation described using prediction and agreement rates.
  • SFT and RL: Table 4 reports the best benchmark performance after up to 100 RL iterations for Mistral-Small-24B models receiving different SFT-step counts.The table varies the number of SFT steps before RL and reports multiple evaluation benchmarks.
  • Behavior Validation: GPT-4o generally agrees well with human annotations but is more conservative for Verification and Subgoal Setting.It may favor obvious Enumeration labels while overlooking subtler behaviors in long CoT responses containing multiple behaviors.
  • SFT and RL: More SFT steps are associated with diminished subsequent RL performance in Mistral-Small-24B models.Performance after 10 SFT steps remained comparable to the base model but still showed some negative effects; over 20 steps caused substantial degradation.
  • Sampling: Figure 13 compares accuracy and response length for sampling numbers N = 1, 4, 8, 32 on Hard-part MATH levels 3–5.The setting matches the main results configuration described in Section 2.1.

G Impact of Exploration-Related Hyperparameters

Zero RL training benefits from stronger exploration through larger sampling sizes and higher training temperatures, while response-length growth depends on task difficulty and can diverge from performance.

  • Sampling Size: Sampling size N=32 achieves approximately 6 points higher average accuracy than N=8 after 100 training steps.Larger sampling sizes also stabilize response-length variability.
  • Sampling Temperature: Higher training temperatures generally improve average performance on Qwen-2.5-0.5B, with temperatures 1.0 and 1.2 outperforming 0.8 and 0.6.The optimal evaluation temperature depends on the training temperature.
  • Algorithm Comparison: GRPO slightly outperforms PPO on Qwen2.5-Math-7B, while both algorithms converge to similar final performance and response length.Both methods use 8K MATH lv3-5 examples under the stated hard-part setting.
  • Task Difficulty: Qwen2.5-Math-7B response length remains relatively unchanged as performance improves, suggesting the 8K training examples may not be challenging enough to extend responses.The authors hypothesize that more challenging data may be needed to push this capable model further.
  • Reasoning Behavior Analysis: The reasoning-behavior analysis uses GPT4-o and a cognitive framework covering backtracking, verification, subgoal setting, and enumeration.The framework replaces backward chaining with enumeration for this task.

I.2 Comparison of Different Reasoning Behavior Tracking Methods

The paper compares keyword tracking with a cognitive-behavior framework and finds that the framework captures reasoning changes that keyword frequencies miss across model case studies.

  • Tracking Methods: Keyword frequencies in DeepSeek Math responses remain consistently low during training, whereas cognitive behaviors identified by the framework increase significantly.The comparison is illustrated in Figure 15.
  • Tracking Methods: Cognitive-behavior analysis identifies verification and backtracking even when responses contain none of the tracked keywords.The authors conclude that keywords alone cannot capture nuanced reasoning behaviors effectively.
Loading 2503.18892v3…