PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding R1-Zero-Like Training: A Critical Perspective

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin

arXiv:2503.20783v2cs.LGcs.AIcs.CL

TL;DR

The paper asks how base-model properties and RL optimization affect R1-Zero-like training without supervised fine-tuning. It studies model templates, pretraining, and GRPO, proposing Dr. GRPO to remove optimization bias. The resulting minimalist recipe achieves state-of-the-art performance with a 7B base model using 27 hours on 8× A100 GPUs.

  • Problem

    The paper addresses limited understanding of how base models and RL separately influence R1-Zero-like training.

  • Method

    The authors evaluate diverse base models and templates, analyze GRPO’s optimization bias, and propose the unbiased Dr. GRPO modification.

  • Results

    The minimalist recipe RL-tunes Qwen2.5-Math-7B with Dr. GRPO on MATH level 3-5 questions and achieves state-of-the-art performance with 27 hours on 8× A100 GPUs.

  • Takeaways & Limitations

    Base-model pretraining and template choice materially shape apparent reasoning ability, while Dr. GRPO provides better token efficiency.

  • Takeaways & Limitations

    Removing KL regularization relies on rule-based verifiers, which eliminate the distributional-shift concern motivating KL regularization in reward-model RL.

Abstract

from arXiv · show

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.

1 Introduction

The paper analyzes how base-model properties and RL optimization shape R1-Zero-like training. It finds that templates and pretraining strongly affect apparent reasoning ability, while Dr. GRPO reduces optimization bias and improves token efficiency.

  • Scope: The study examines R1-Zero-like training through two components: base models and reinforcement learning.It investigates Qwen2.5 models and DeepSeek-V3-Base, then analyzes GRPO optimization.
  • Minimalist recipe: The minimalist recipe RL-tunes Qwen2.5-Math-7B with unbiased Dr. GRPO on MATH level 3-5 questions using the Qwen-Math template.It uses 27 hours of compute on 8× A100 GPUs and achieves state-of-the-art performance.
  • Base models and templates: Template choice can determine whether base models answer questions or complete sentences before RL.Model-template mismatch can destroy reasoning capabilities before RL reconstructs them.
  • Base models and templates: Qwen2.5 base models gain an immediate ∼60% improvement when no template is used, suggesting possible question-answer pretraining bias.The authors hypothesize that these models may have been pretrained on concatenated question-answer texts.
  • Base-model capabilities: Nearly all tested base models already exhibit the “Aha moment”, including DeepSeek-V3-Base.This finding bears on interpretations of self-reflection emerging through RL alone.
  • RL optimization: Dr. GRPO fixes GRPO’s optimization bias and achieves better token efficiency.The paper presents it as a simple modification to GRPO.

2 Analysis on Base Models

The analysis examines how base-model attributes and prompting templates shape R1-Zero-like trainability, finding that templates affect answer formatting while tested models already support exploration. Qwen2.5 models perform unusually well without templates, and DeepSeek-V3-Base already exhibits self-reflection behaviors.

  • The study evaluates Qwen2.5, Llama-3.1, and DeepSeek base models on 500 MATH training questions, comparing answer format, exploration, and self-reflection.Exploration is assessed with pass@8, while self-reflection counts use keyword- and LLM-based detection cross-validation.
  • 2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies: Templates are used to elicit question-answering behavior from sentence-completion base models, forming question-answering base policies for RL.The analysis compares the R1 template, Qwen-Math template, and no template.
  • 2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies: 100% answering rate is achieved by Qwen2.5 models without templates, whereas Llama and DeepSeek models improve with the R1 template.DeepSeek-V3-Base's lowest no-template answering rate suggests it is a nearly pure base model.
  • 2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies: All tested base models are exploratory according to pass@8, with Qwen2.5 models performing best across sampling temperatures.Exploration matters because RL requires sampled trajectories that can reach correct final answers and provide reward signals.
  • 2.2 Qwen2.5 Base Models’ Pretraining Bias: ∼60% improvement is obtained by omitting templates versus traditional 4-shot prompting on Qwen2.5-Math benchmarks.The authors hypothesize that question-answer pretraining may make these models effectively SFT-like without templates.
  • 2.3 Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base: DeepSeek-V3-Base already generates a decent amount of self-reflection, including keywords such as “Aha” and “wait,” before RL tuning.The result extends the examination to the base model used for the real R1-Zero model.
  • 2.3 Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base: Self-reflection behaviors occur more frequently in R1-Zero but are not positively correlated with higher accuracy.This finding addresses whether the behavior itself tracks post-RL performance.

3 Analysis on Reinforcement Learning

The analysis examines how RL objectives and training choices shape reasoning-model optimization. It identifies biases in GRPO and shows that Dr. GRPO improves token efficiency while base-model, template, and question-set choices strongly affect RL dynamics.

  • RL optimization: GRPO estimates token advantages from grouped response returns without a learned value model.For each question, it samples multiple responses and assigns every token in a response its return minus the group mean.
  • GRPO biases: GRPO introduces response-length and question-difficulty biases through response-length and within-group standard-deviation normalization.These terms reweight responses and questions differently during policy updates.
  • Dr. GRPO: Removing both normalization terms yields Dr. GRPO, which computes unbiased policy gradients and improves token efficiency.The method removes 1/|oi| and the within-group reward standard-deviation normalization.
  • Training dynamics: GRPO continues lengthening responses after reward gains slow, whereas Dr. GRPO prevents uncontrolled growth and reduces incorrect-response length during evaluation.The latter pattern suggests reduced overthinking relative to the baseline.
  • Template and question-set effects: Templates determine initial policy performance, while suitable question sets can bring different policies to about 40% performance; narrower coverage lowers the plateau under the R1 template.With the Qwen-Math template, GSM-8K training nearly doubles accuracy on harder test questions.
  • Template and question-set effects: Qwen2.5-Math-1.5B already has strong math-solving ability, but applying mismatched templates can destroy it before RL reconstructs the capability.When template mismatch is large, question-set coverage becomes important for policy improvement.

4 Closing Remarks

The paper critically examines how base-model pretraining and RL optimization shape R1-Zero-like training. Dr. GRPO addresses an optimization bias by improving token efficiency while preserving reasoning performance.

  • The analysis examines how pretraining biases influence RL outcomes and how optimization choices such as GRPO shape model behavior.
  • Dr. GRPO improves token efficiency while preserving reasoning performance.
  • Scaling RL can be effective and efficient when optimization avoids unnecessary response-length growth.

A Policy Gradient Derivations

This section derives the policy-gradient formulation for token-level language-model RL and examines how variance reduction and GRPO normalization affect optimization. The derivation concludes that response standard deviation and length should not appear in the RL objective.

  • Language-model RL maximizes expected return over questions and generated responses.
  • The return sums token-level rewards across a response trajectory.
  • A baseline invariant to the selected token reduces variance without changing the expected policy-gradient contribution.
  • In outcome-reward settings, GRPO estimates advantages using group response rewards and their mean and standard deviation.
  • The derivation concludes that both std and |o| should not appear in the RL objective.
  • The resulting advantage estimator is equivalent to REINFORCE Leave-One-Out up to a learning-rate scaling factor.

B Detailed Benchmark Results

The detailed benchmark results compare the minimalist recipe against same-scale instruct and R1-Distill models across 1.5B, 3B, and 7B settings. Generation budgets differ according to model context length.

  • The benchmarks cover 1.5B, 3B, and 7B model scales, with same-scale instruct and R1-Distill models included for comparison.
  • Qwen2.5-Math baselines use a 3k-token generation budget because their context length is 4k.
  • OpenReasoner-Zero and R1-Distill-Qwen are also evaluated with an 8k generation budget because they support longer contexts.
  • The table marks the best-template setting, including no template, when reporting base-model test scores.

C Extended Empirical Results

Extended experiments ablate GRPO bias terms and test Dr. GRPO across independent runs. Dr. GRPO consistently improves token efficiency and final accuracy with statistical significance across random seeds.

  • The experiments ablate GRPO bias terms and evaluate the statistical significance of Dr. GRPO results.
  • The ablation uses Qwen2.5-1.5B with 3K diverse math questions from ASDiv, MATH, and pre-2023 AIME.
  • Figure 8 presents ablation results for the two bias terms in GRPO.
  • Figure 9 reports mean curves as solid lines and standard deviation in shaded areas across three independent RL runs.
  • Dr. GRPO consistently achieves statistically significant improvements in token efficiency and final accuracy across random seeds.

D Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors

The paper combines selective keyword matching with LLM-based analysis to detect explicit and implicit self-reflection while reducing false positives. Cross-validation further improves robustness, though each detector has distinct failure modes.

  • Detection methods: A selective keyword pool targets explicit self-reflection while avoiding ambiguous terms that generate false positives.The pool includes terms such as “recheck,” “reevaluate,” and “double-check,” while excluding less specific examples like “wait” and “try again.”
  • Observed model differences: Keyword frequencies differ across model families, with Qwen2.5 favoring verification phrases and DeepSeek omitting several of them.The paper hypothesizes that these differences may result from pretraining-data variation related to reasoning and mathematics.
  • Detection methods: LLM-based detection identifies implicit self-reflection patterns that keyword matching may miss.The LLM evaluates sophisticated behaviors such as revisiting solutions or questioning assumptions without explicit keywords.
  • Limitations: LLM-based detection can misclassify lengthy, complex responses, while keyword-based detection can produce false positives.Figure 11 gives examples of both error types.
  • Cross-validation: Cross-validation combines keyword-based and LLM-based detection to improve robustness against their complementary errors.The combined results are reported alongside each individual detector in Figure 12.

E Examples of Aha Moment in DeepSeek-V3-Base

The paper presents examples showing that DeepSeek-V3-Base exhibits the so-called “aha moment” before reinforcement-learning tuning. This challenges interpretations that self-reflection necessarily emerges only through RL.

  • Pre-RL behavior: DeepSeek-V3-Base already exhibits the so-called “aha moment” before RL tuning.Figure 13 presents two example cases demonstrating this behavior.
  • Examples: Figure 13 uses two examples to demonstrate pre-existing self-reflection behavior in DeepSeek-V3-Base.The figure focuses on qualitative cases rather than a reported aggregate metric.
  • Interpretation: The observed behavior shows that DeepSeek-V3-Base does not require RL tuning for the paper’s examples of an “aha moment.”The claim is limited to the examples presented in Figure 13.

F Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero

The comparison examines how DeepSeek-V3-Base and DeepSeek-R1-Zero differ across MATH response categories, lengths, and self-reflection behavior. RL training corrects many incorrect responses, but self-reflection is not positively correlated with higher accuracy.

  • Response categories: RL training corrects most incorrect responses across difficulty levels in the 500-question MATH evaluation.The comparison reports substantial performance gains for DeepSeek-R1-Zero relative to DeepSeek-V3-Base.
  • Response lengths: Response lengths increase substantially across correct and incorrect categories after training.Truncated responses are excluded because they could belong to another category with a larger context size.
  • Self-reflection: DeepSeek-V3-Base already exhibits the “Aha moment” before RL tuning.Figure 13 provides cases illustrating this pre-training behavior.
  • Response lengths: Incorrect responses are longer on average than correct responses.The paper hypothesizes that this reflects harder questions requiring longer reasoning and producing more incorrect outputs.
  • Self-reflection and accuracy: Self-reflection behaviors occur more frequently in DeepSeek-R1-Zero but are not positively correlated with higher accuracy.The analysis samples responses across eight trials for questions eliciting at least one self-reflective response.

G Detailed Experimental Settings

The experiments use a standardized configuration and prompts to evaluate question answering and self-reflection across base-model responses. Responses are assigned categories based on relevance, self-reflection, and whether they consist solely of Python code.

  • Hardware and training: All experiments run on 8 × A100 GPUs and finish in about one day with actor-learner collocation enabled.The configurations are listed in Table 6.
  • Question-answering classification: The question-answering evaluation prompts determine whether a response is off-topic, answers directly, self-reflects, or consists solely of Python code.The output format requires a brief explanation followed by a final category number.
  • LLM-based detection: The LLM-based detector first separates irrelevant responses from responses attempting to answer the question.It then assesses self-reflection according to the category rules.
  • Self-reflection classification: Category 2 includes explicit keywords and implicit behaviors such as revisiting solutions, questioning assumptions, and considering alternatives.Any self-reflection is assigned Category 2 regardless of correctness or answer quality.
  • Prompt inputs: The evaluation passes each question and generated response into the classification prompts.The prompts use the placeholders “{question}” and “{response}”.
Loading 2503.20783v2…