PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding R1-Zero-Like Training: A Critical Perspective

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin

arXiv:2503.20783v2cs.LGcs.AIcs.CL

TL;DR

The paper asks how base-model properties and RL optimization affect R1-Zero-like training without supervised fine-tuning. It studies model templates, pretraining, and GRPO, proposing Dr. GRPO to remove optimization bias. The resulting minimalist recipe achieves state-of-the-art performance with a 7B base model using 27 hours on 8× A100 GPUs.

  • Problem

    The paper addresses limited understanding of how base models and RL separately influence R1-Zero-like training.

    Source: §1 · p. 2, §3.4 · p. 9

  • Method

    The authors evaluate diverse base models and templates, analyze GRPO’s optimization bias, and propose the unbiased Dr. GRPO modification.

    Source: §1 · p. 2, §2.1 · p. 3, §3 · p. 5

  • Results

    The minimalist recipe RL-tunes Qwen2.5-Math-7B with Dr. GRPO on MATH level 3-5 questions and achieves state-of-the-art performance with 27 hours on 8× A100 GPUs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Base-model pretraining and template choice materially shape apparent reasoning ability, while Dr. GRPO provides better token efficiency.

    Source: §1 · p. 2

    Removing KL regularization relies on rule-based verifiers, which eliminate the distributional-shift concern motivating KL regularization in reward-model RL.

    Source: §3 · p. 5

Abstract

from arXiv · show

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.

1 Introduction

The paper analyzes how base-model properties and RL optimization shape R1-Zero-like training. It finds that templates and pretraining strongly affect apparent reasoning ability, while Dr. GRPO reduces optimization bias and improves token efficiency.

Scope

  • The study examines R1-Zero-like training through two components: base models and reinforcement learning.It investigates Qwen2.5 models and DeepSeek-V3-Base, then analyzes GRPO optimization.Source: §1 · p. 2

Minimalist recipe

  • The minimalist recipe RL-tunes Qwen2.5-Math-7B with unbiased Dr. GRPO on MATH level 3-5 questions using the Qwen-Math template.It uses 27 hours of compute on 8× A100 GPUs and achieves state-of-the-art performance.Source: §1 · p. 2

Base models and templates

  • Template choice can determine whether base models answer questions or complete sentences before RL.Model-template mismatch can destroy reasoning capabilities before RL reconstructs them.Source: §1 · p. 2
  • Qwen2.5 base models gain an immediate ∼60% improvement when no template is used, suggesting possible question-answer pretraining bias.The authors hypothesize that these models may have been pretrained on concatenated question-answer texts.Source: §1 · p. 2

Base-model capabilities

  • Nearly all tested base models already exhibit the “Aha moment”, including DeepSeek-V3-Base.This finding bears on interpretations of self-reflection emerging through RL alone.Source: §1 · p. 2

RL optimization

  • Dr. GRPO fixes GRPO’s optimization bias and achieves better token efficiency.The paper presents it as a simple modification to GRPO.Source: §1 · p. 2

2 Analysis on Base Models

The analysis examines how base-model attributes and prompting templates shape R1-Zero-like trainability, finding that templates affect answer formatting while tested models already support exploration. Qwen2.5 models perform unusually well without templates, and DeepSeek-V3-Base already exhibits self-reflection behaviors.

  • The study evaluates Qwen2.5, Llama-3.1, and DeepSeek base models on 500 MATH training questions, comparing answer format, exploration, and self-reflection.Exploration is assessed with pass@8, while self-reflection counts use keyword- and LLM-based detection cross-validation.Source: §2 · p. 2, Figure 3 · p. 3

2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies

  • Templates are used to elicit question-answering behavior from sentence-completion base models, forming question-answering base policies for RL.The analysis compares the R1 template, Qwen-Math template, and no template.Source: §2.1 · p. 3
  • 100% answering rate is achieved by Qwen2.5 models without templates, whereas Llama and DeepSeek models improve with the R1 template.DeepSeek-V3-Base's lowest no-template answering rate suggests it is a nearly pure base model.Source: §2.1 · p. 3
  • All tested base models are exploratory according to pass@8, with Qwen2.5 models performing best across sampling temperatures.Exploration matters because RL requires sampled trajectories that can reach correct final answers and provide reward signals.Source: §2.1 · p. 4

2.2 Qwen2.5 Base Models’ Pretraining Bias

  • ∼60% improvement is obtained by omitting templates versus traditional 4-shot prompting on Qwen2.5-Math benchmarks.The authors hypothesize that question-answer pretraining may make these models effectively SFT-like without templates.Source: §2.2 · p. 4, Table 1 · p. 4

2.3 Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base

  • DeepSeek-V3-Base already generates a decent amount of self-reflection, including keywords such as “Aha” and “wait,” before RL tuning.The result extends the examination to the base model used for the real R1-Zero model.Source: §2.3 · p. 4
  • Self-reflection behaviors occur more frequently in R1-Zero but are not positively correlated with higher accuracy.This finding addresses whether the behavior itself tracks post-RL performance.Source: §2.3 · p. 4

3 Analysis on Reinforcement Learning

The analysis examines how RL objectives and training choices shape reasoning-model optimization. It identifies biases in GRPO and shows that Dr. GRPO improves token efficiency while base-model, template, and question-set choices strongly affect RL dynamics.

RL optimization

  • GRPO estimates token advantages from grouped response returns without a learned value model.For each question, it samples multiple responses and assigns every token in a response its return minus the group mean.Source: §3 · p. 5

GRPO biases

  • GRPO introduces response-length and question-difficulty biases through response-length and within-group standard-deviation normalization.These terms reweight responses and questions differently during policy updates.Source: §3.1 · p. 6

Dr. GRPO

  • Removing both normalization terms yields Dr. GRPO, which computes unbiased policy gradients and improves token efficiency.The method removes 1/|oi| and the within-group reward standard-deviation normalization.Source: §3.2 · p. 6

Training dynamics

  • GRPO continues lengthening responses after reward gains slow, whereas Dr. GRPO prevents uncontrolled growth and reduces incorrect-response length during evaluation.The latter pattern suggests reduced overthinking relative to the baseline.Source: §3.2 · pp. 7–8

Template and question-set effects

  • Templates determine initial policy performance, while suitable question sets can bring different policies to about 40% performance; narrower coverage lowers the plateau under the R1 template.With the Qwen-Math template, GSM-8K training nearly doubles accuracy on harder test questions.Source: §3.3 · p. 8
  • Qwen2.5-Math-1.5B already has strong math-solving ability, but applying mismatched templates can destroy it before RL reconstructs the capability.When template mismatch is large, question-set coverage becomes important for policy improvement.Source: §3.3 · pp. 8–9

4 Closing Remarks

The paper critically examines how base-model pretraining and RL optimization shape R1-Zero-like training. Dr. GRPO addresses an optimization bias by improving token efficiency while preserving reasoning performance.

  • The analysis examines how pretraining biases influence RL outcomes and how optimization choices such as GRPO shape model behavior.Source: §4 · pp. 9–10
  • Dr. GRPO improves token efficiency while preserving reasoning performance.Source: §4 · p. 10
  • Scaling RL can be effective and efficient when optimization avoids unnecessary response-length growth.Source: §4 · p. 10

A Policy Gradient Derivations

This section derives the policy-gradient formulation for token-level language-model RL and examines how variance reduction and GRPO normalization affect optimization. The derivation concludes that response standard deviation and length should not appear in the RL objective.

B Detailed Benchmark Results

The detailed benchmark results compare the minimalist recipe against same-scale instruct and R1-Distill models across 1.5B, 3B, and 7B settings. Generation budgets differ according to model context length.

C Extended Empirical Results

Extended experiments ablate GRPO bias terms and test Dr. GRPO across independent runs. Dr. GRPO consistently improves token efficiency and final accuracy with statistical significance across random seeds.

D Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors

The paper combines selective keyword matching with LLM-based analysis to detect explicit and implicit self-reflection while reducing false positives. Cross-validation further improves robustness, though each detector has distinct failure modes.

Detection methods

  • A selective keyword pool targets explicit self-reflection while avoiding ambiguous terms that generate false positives.The pool includes terms such as “recheck,” “reevaluate,” and “double-check,” while excluding less specific examples like “wait” and “try again.”Source: D Keyword-based Detection and… · pp. 15–16

Observed model differences

  • Keyword frequencies differ across model families, with Qwen2.5 favoring verification phrases and DeepSeek omitting several of them.The paper hypothesizes that these differences may result from pretraining-data variation related to reasoning and mathematics.Source: D Keyword-based Detection and… · p. 16

Detection methods

  • LLM-based detection identifies implicit self-reflection patterns that keyword matching may miss.The LLM evaluates sophisticated behaviors such as revisiting solutions or questioning assumptions without explicit keywords.Source: D Keyword-based Detection and… · p. 16

Limitations

Cross-validation

E Examples of Aha Moment in DeepSeek-V3-Base

The paper presents examples showing that DeepSeek-V3-Base exhibits the so-called “aha moment” before reinforcement-learning tuning. This challenges interpretations that self-reflection necessarily emerges only through RL.

Pre-RL behavior

Examples

  • Figure 13 uses two examples to demonstrate pre-existing self-reflection behavior in DeepSeek-V3-Base.The figure focuses on qualitative cases rather than a reported aggregate metric.Source: E Examples of Aha Moment in… · p. 17

Interpretation

  • The observed behavior shows that DeepSeek-V3-Base does not require RL tuning for the paper’s examples of an “aha moment.”The claim is limited to the examples presented in Figure 13.Source: E Examples of Aha Moment in… · p. 17

F Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero

The comparison examines how DeepSeek-V3-Base and DeepSeek-R1-Zero differ across MATH response categories, lengths, and self-reflection behavior. RL training corrects many incorrect responses, but self-reflection is not positively correlated with higher accuracy.

Response categories

Response lengths

Self-reflection

  • DeepSeek-V3-Base already exhibits the “Aha moment” before RL tuning.Figure 13 provides cases illustrating this pre-training behavior.Source: Figure 13 · p. 18

Response lengths

  • Incorrect responses are longer on average than correct responses.The paper hypothesizes that this reflects harder questions requiring longer reasoning and producing more incorrect outputs.Source: F Comparison Between DeepSeek-V3-Base… · p. 18

Self-reflection and accuracy

G Detailed Experimental Settings

The experiments use a standardized configuration and prompts to evaluate question answering and self-reflection across base-model responses. Responses are assigned categories based on relevance, self-reflection, and whether they consist solely of Python code.

Hardware and training

Question-answering classification

  • The question-answering evaluation prompts determine whether a response is off-topic, answers directly, self-reflects, or consists solely of Python code.The output format requires a brief explanation followed by a final category number.Source: H Prompts Used for GPT-As-A-Judge · pp. 19–20

LLM-based detection

Self-reflection classification

  • Category 2 includes explicit keywords and implicit behaviors such as revisiting solutions, questioning assumptions, and considering alternatives.Any self-reflection is assigned Category 2 regardless of correctness or answer quality.Source: H Prompts Used for GPT-As-A-Judge · p. 20

Prompt inputs

Related papers

Loading 2503.20783v2…