PaperlayerView exact revision on arXiv

Source-linked AI summary

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, Mingxuan Wang

arXiv:2503.14476v2cs.LGcs.CL

TL;DR

Key algorithms and recipes for scalable reinforcement learning remain concealed in reasoning-model reports, leaving the community struggling to reproduce training results. DAPO is an open-source large-scale RL system that achieves 50 points on AIME 2024 with Qwen2.5-32B, outperforming DeepSeek-R1-Zero-Qwen-32B (47 points) using 50% training steps.

  • Problem

    Key algorithms and recipes for scalable RL remain concealed in reasoning-model reports, leaving the community struggling to reproduce training results.

    Source: §1 · p. 2

  • Method

    DAPO combines Clip-Higher, Dynamic Sampling, token-level policy-gradient loss, and overlong reward shaping for long-CoT RL.

    Source: §1 · p. 2

  • Results

    50 points on AIME 2024 with Qwen2.5-32B, outperforming DeepSeek-R1-Zero-Qwen-32B (47 points) using 50% training steps.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The open-source code and dataset provide practical access to a scalable RL solution.

    Source: §5 · p. 11

    The algorithm excludes the KL penalty because long-CoT training can diverge significantly from the initial model, making that restriction unnecessary.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

1 Introduction

DAPO is an open-source large-scale LLM reinforcement learning system that achieves 50 points on AIME 2024 with Qwen2.5-32B, and introduces four techniques for long-CoT RL. The paper releases the algorithm, training code, and dataset to support reproducibility.

1 Introduction

  • Test-time scaling, exemplified by OpenAI’s o1 and DeepSeek’s R1 [2], enables longer chain-of-thought reasoning and stronger performance on competitive math and coding tasks.Source: §1 · p. 2
  • The authors release the algorithm, training code, and dataset to address the scarcity of disclosed scalable RL recipes and support research in large-scale LLM RL.Their implementation is based on verl [20].Source: §1 · p. 2
  • The initial GRPO run reached 30 AIME points, below DeepSeek’s 47, while analysis identified entropy collapse, reward noise, and training instability as obstacles.The paper notes that other researchers faced similar difficulties reproducing DeepSeek’s results [13] [15] [16] [17] [18] [19].Source: §1 · p. 2
  • 50 points on AIME 2024 with Qwen2.5-32B outperforms DeepSeek-R1-Zero-Qwen-32B’s 47 points using 50% of its training steps.The paper presents this as the result of its open-source large-scale RL system.Source: §1 · p. 2
  • DAPO combines Clip-Higher to avoid entropy collapse, Dynamic Sampling for efficiency and stability, token-level policy gradient loss for long-CoT RL, and Overlong Reward Shaping to reduce reward noise and stabilize training.Source: §1 · p. 2

2 Preliminary

The preliminary section reviews PPO and GRPO, then motivates DAPO’s removal of KL regularization and use of rule-based outcome rewards. Together, these choices frame its approach to training long-CoT reasoning models.

2.1 Proximal Policy Optimization (PPO)

  • PPO [21] stabilizes policy updates and improves sample efficiency by clipping updates around the previous policy, with advantages estimated using GAE.The clipping range constrains the importance-sampling ratio; GAE uses the value and reward functions.Source: §2.1 · p. 2

2.2 Group Relative Policy Optimization (GRPO)

  • GRPO removes the value function, estimates advantages by normalizing rewards across sampled responses, and uses a clipped objective with a KL penalty.It averages loss within each generated sequence before averaging across samples, a distinction the paper says may affect performance.Source: §2.2 · p. 3

2.3 Removing KL Divergence

  • DAPO excludes the KL penalty because long-CoT training can diverge substantially from the initial model, unlike RLHF alignment [23], where limiting divergence is a goal.Source: §2.3 · p. 3

2.4 Rule-based Reward Modeling

  • To avoid reward hacking associated with reward models, DAPO uses final-answer accuracy on verifiable tasks as the outcome reward.The rule compares prediction ˆy with ground truth y; the approach has activated base-model reasoning in theorem proving, programming [34], and mathematics competition [2].Source: §2.4 · p. 4

3 DAPO

DAPO defines a group-sampled policy-optimization objective and develops four training techniques alongside an integer-answer math dataset.

3 DAPO

  • DAPO samples groups of outputs for each question and computes token advantages from each reward relative to the group mean.The section introduces the DAPO objective and points to the full algorithm in Algorithm 1.Source: §3 · p. 4

3.1 Raise the Ceiling: Clip-Higher

  • Clip-Higher addresses entropy collapse by widening the upper clip to let low-probability exploration tokens gain probability, while retaining the lower clip to avoid collapsing the sampling space.The authors report that the adjustment enhances policy entropy and sample diversity; up-clipped tokens have mean probability below 0.2.Source: §3.1 · pp. 4–5

3.2 The More the Merrier: Dynamic Sampling

  • Dynamic Sampling oversamples and filters prompts with accuracy 0 or 1, keeping batches filled with prompts that have effective gradients and achieving the same performance faster.Prompts with all-correct outputs can yield zero advantages and gradients, reducing the effective prompt count and increasing gradient noise.Source: §3.2 · pp. 5–6

3.3 Rebalancing Act: Token-Level Policy Gradient Loss

  • Token-level policy-gradient loss addresses sample-level loss reduction, which underweights tokens in long responses and can impede learning from good samples while failing to penalize gibberish and repetition.With token-level loss, longer sequences can exert more influence on the gradient, and reward-affecting patterns are prompted or suppressed regardless of response length.Source: §3.3 · p. 6

3.4 Hide and Seek: Overlong Reward Shaping

  • Overlong Filtering masks truncated-sample loss to stabilize training and enhance performance, while Soft Overlong Punishment adds an increasing length-aware penalty to the correctness reward within a defined interval.The authors warn that blanket punitive rewards can penalize sound reasoning solely because it exceeds the generation limit.Source: §3.4 · pp. 6–7

3.5 Dataset Transformation

  • DAPO-Math-17K contains 17K prompts paired with integer answers, which the authors use to make rule-based answer parsing more reliable.The dataset is sourced from the web and official competition homepages, with answers selected and transformed into integers.Source: §3.5 · p. 8

4 Experiments

DAPO trains Qwen2.5-32B Base to 50% accuracy on AIME 2024 in half the training steps used by DeepSeek-R1-Zero-Qwen-32B. The experiments also examine progressive training techniques, training dynamics, and the emergence of new reasoning behaviors.

4.1 Training Details

  • Training used verl with naive GRPO, 512 prompts per rollout, 16 responses per prompt, and 16 gradient updates per rollout step.The setup used AdamW at a constant learning rate of 1 × 10−6, a 20-step linear warm-up, and a 20,480-token generation limit; AIME evaluation repeated the set 32 times and reported avg@32.Source: §4.1 · p. 8

4.2 Main Results

  • 50% AIME 2024 accuracy was reached from near 0% using half the training steps required by DeepSeek-R1-Zero-Qwen-32B.The experiments report that DAPO’s Qwen-32B Base performance surpassed DeepSeek’s Qwen2.5-32B experiments using the R1 approach.Source: §4.2 · p. 8
  • Vanilla GRPO reached 30% AIME 2024 accuracy, while the progressive techniques each contributed several accuracy points.Token-level loss brought less performance improvement but enhanced training stability and supported healthier response-length growth; Dynamic Sampling increased sampling instances without significantly affecting overall training time, while reducing convergence time through fewer training steps.Source: §4.2 · pp. 8–9

4.3 Training Dynamics

  • Response length can stagnate or decline, so it is monitored with validation accuracy; training reward was generally stable but often poorly correlated with validation accuracy, indicating overfitting.The paper notes that length provides room for exploration and cites prior evidence for non-monotonic length trends [2].Source: §4.3 · p. 9
  • Excessively low entropy risks lost exploration, while excessive entropy can yield gibberish or repetition; Clip-Higher addresses entropy collapse, and slowly rising entropy supports performance.Generation probability behaves oppositely to entropy.Source: §4.3 · pp. 9–10

4.4 Case Study

  • As RL training progressed, the actor developed reflection and backtracking behaviors that were virtually absent early in training.The authors describe these as new reasoning modes emerging alongside reinforcement of existing patterns that support correct problem-solving.Source: §4.4 · p. 10

5 Conclusion

DAPO is an open-source large-scale LLM reinforcement learning system that achieves AIME 50 with a Qwen-32B pretrained model. It combines the algorithm, code infrastructure, and dataset with four techniques for effective and efficient long-CoT reinforcement learning.

5 Conclusion

  • AIME 50 with a Qwen-32B pretrained model is the system’s reported state-of-the-art large-scale LLM RL result.The released system includes the algorithm, code infrastructure, and dataset.Source: §5 · p. 10
  • DAPO introduces four techniques to make reinforcement learning effective and efficient in long-CoT scenarios.The paper identifies DAPO as Decoupled Clip and Dynamic sAmpling Policy Optimization.Source: §5 · p. 11
  • Open-sourcing the training code and dataset provides the research community and society practical access to a scalable reinforcement learning solution.Source: §5 · p. 11

Appendix · A Dataset Transformation

The appendix illustrates a dataset transformation process that reformulates problems into answer-format tasks. It uses guided chain-of-thought reasoning to encourage thorough reasoning while avoiding hallucinations.

A Dataset Transformation

  • The transformation example presents an original problem, its transformed version, and model outputs during the process.Source: A Dataset Transformation · p. 15
  • The framework uses CoT reasoning and four steps: extract the answer format, rewrite the problem, solve it, and provide an integer answer.Each step is guided by few-shot solutions or detailed guidelines.Source: A Dataset Transformation · p. 15
  • In most cases, the LLM generates reformulations with satisfactory format and quality.This is the authors’ reported observation about the approach.Source: A Dataset Transformation · p. 15
  • In the example, the original smallest-value problem is reframed to ask for k + m + n when its answer has the form k − m√n.The transformed prompt retains the original problem and specifies that k, m, and n are integers.Source: A Dataset Transformation · p. 15
  • The example’s transformation reasoning identifies the original answer as an algebraic expression and maps it to the required integer form.Source: A Dataset Transformation · p. 15
  • For the example, k = 11, m = 2, and n = 6, yielding k + m + n = 19.Source: A Dataset Transformation · p. 15

B Supplementary Case

The supplementary case presents an example of reflective behavior emergence.

B Supplementary Case

  • Table 3 presents a case of reflective behavior emergence.Source: Table 3 · p. 16

Related papers

Loading 2503.14476v2…