Source-linked AI summary
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, Mingxuan Wang
TL;DR
Large-scale LLM reinforcement learning remains difficult to reproduce because key training algorithms and recipes are concealed. DAPO addresses this gap with an open-source scalable RL system and four techniques, achieving 50 points on AIME 2024 with Qwen2.5-32B.
Problem
Key algorithms and training recipes for scalable large-scale LLM reinforcement learning remain concealed, limiting reproducibility of reported reasoning-model results.
Method
DAPO is an open-source large-scale LLM RL system combining Decoupled Clip and Dynamic sAmpling Policy Optimization with four techniques for long-CoT reinforcement learning.
Results
50 points on AIME 2024 were achieved using the Qwen2.5-32B model, outperforming DeepSeek-R1-Zero-Qwen-32B's 47 points with 50% training steps.
Takeaways & Limitations
Open-sourcing the algorithm, training code, and dataset provides practical access to a scalable reinforcement learning solution and supports reproducibility and future research.
Takeaways & Limitations
DAPO excludes the KL penalty term because the authors judge its restriction unnecessary when long-CoT training diverges substantially from the initial model.
Abstract
from arXiv · showhide
Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.
1 Introduction
The paper identifies missing scalable-RL details as a barrier to reproducing strong reasoning-model results and introduces an open-source system to address it. DAPO combines four techniques and reaches 50 AIME 2024 points with Qwen2.5-32B.
- Large-scale RL elicits complex reasoning behaviors, but existing reasoning-model reports conceal key algorithms and training recipes.
- 30 points on AIME from an initial GRPO run lagged DeepSeek’s 47 points, with entropy collapse, reward noise, and training instability identified as obstacles.
- 50 points on AIME 2024 surpassed DeepSeek-R1-Zero-Qwen-32B’s 47 points using 50% of its training steps.
- DAPO introduces Clip-Higher to promote diversity and avoid entropy collapse, and Dynamic Sampling to improve training efficiency and stability.
- The system also uses Token-Level Policy Gradient Loss for long-CoT RL and Overlong Reward Shaping to reduce reward noise and stabilize training.
- The authors release the training code and dataset, with implementation based on verl.
2 Preliminary
This section reviews PPO and GRPO objectives, advantage estimation, reward construction, and the decision to remove KL regularization for long-CoT RL. It also highlights sample-level loss calculation as a potential performance issue.
- PPO stabilizes policy optimization by constraining updates within a proximal region using a clipped surrogate objective.
- Generalized Advantage Estimation computes the advantage estimator from the value function and reward function.
- GRPO removes the value function and estimates advantages relative to rewards from a group of sampled responses.
- GRPO retains a clipped objective and directly applies a KL penalty, while computing the objective at the sample level.
- The proposed algorithm excludes KL regularization because long-CoT training can diverge substantially from the initial model, making that restriction unnecessary in this setting.
- Rule-based outcome rewards directly use verifiable-task accuracy rather than a reward model, comparing predicted and ground-truth answers.
3 DAPO
DAPO combines decoupled clipping, dynamic sampling, token-level policy gradients, and overlong reward shaping to address exploration, gradient, long-response, and reward-noise problems in long-CoT RL.
- 3 DAPO: DAPO introduces a policy-optimization algorithm designed for large-scale reinforcement learning in long-CoT scenarios.Its full procedure samples outputs, computes rewards and advantages, dynamically filters samples, and optimizes the policy objective.
- 3.1 Raise the Ceiling: Clip-Higher: Entropy collapse limits exploration because sampled responses become nearly identical as training progresses.DAPO addresses this with Clip-Higher, which increases the upper clipping range while preserving the lower range.
- 3.2 The More the Merrier: Dynamic Sampling: Prompts whose outputs are all correct produce zero advantages and zero policy gradients, reducing effective batch size and increasing gradient noise.Dynamic sampling oversamples and filters prompts with accuracy 0 or 1 until the batch contains samples with intermediate accuracy.
- 3.2 The More the Merrier: Dynamic Sampling: Dynamic sampling achieves the same performance faster despite increasing the number of sampling instances.The reported convergence-time reduction is attributed to requiring fewer training steps.
- 3.3 Rebalancing Act: Token-Level Policy Gradient Loss: Token-level policy-gradient loss gives longer sequences more influence and applies reward-driven token updates independently of response length.This addresses sample-level averaging, which can underweight useful long responses and insufficiently penalize undesirable patterns in excessively long samples.
- 3.4 Hide and Seek: Overlong Reward Shaping: Reward penalties for truncated samples can introduce noise and disrupt training when sound reasoning is penalized solely for excessive length.Overlong Filtering masks truncated-sample losses, while Soft Overlong Punishment increases the penalty across a defined length interval.
4 Experiments
Experiments evaluate DAPO on mathematical tasks using verl and compare progressive techniques, training behavior, and monitoring metrics. DAPO improves AIME 2024 performance while token-level loss and dynamic sampling affect stability and convergence.
- Main Results: DAPO trains the Qwen-32B Base model to 50% accuracy on AIME 2024, improving from near 0% with 50% of DeepSeek-R1-Zero-Qwen-32B’s training steps.
- Main Results: 30% accuracy is reached under vanilla GRPO, while the progressive techniques each contribute several accuracy points on AIME 2024.
- Ablations: Token-level loss improves training stability and produces a healthier increase in response length, despite yielding less performance improvement.
- Ablations: Dynamic Sampling filters zero-gradient data; although it increases sampling instances, convergence time is reduced because fewer training steps are required.
- Training Dynamics: Response length, reward, entropy, and generation probability are monitored because they reflect training dynamics, exploration, stability, and potential deterioration.
- Case Study: RL training produces evolving reasoning patterns, including reflection and backtracking that emerge as training progresses.
5 Conclusion
The paper releases an open-source large-scale LLM RL system comprising its algorithm, infrastructure, and dataset. It presents DAPO as effective and efficient for long-CoT reinforcement learning and reports AIME 50 with Qwen-32B.
- The released system includes the DAPO algorithm, code infrastructure, and dataset for large-scale LLM reinforcement learning.
- The system reports AIME 50 using a Qwen-32B pretrained model.
- DAPO introduces four techniques intended to make long-CoT reinforcement learning effective and efficient.
Contributions
The contributions section lists the paper’s authors and their institutional affiliations.
- The paper lists its authors in the contribution record.
- The listed affiliations include Tsinghua University’s Institute for AI Industry Research.
- The affiliations also include the University of Hong Kong and SIA-Lab of Tsinghua AIR and ByteDance Seed.
A Dataset Transformation
The dataset transformation process reformulates competition problems into integer-answer tasks while preserving the original mathematical content. It uses structured reasoning steps and examples to guide the transformation and verify the resulting answer.
- Dataset transformation reformulates answers expressed as algebraic forms into integer-answer problems.
- The transformation prompts the model to extract the answer format, rewrite the problem, solve it, and provide an integer final answer.
- The authors report that most generated reformulations have satisfactory format and quality.
- An example converts an answer of 11 − 2√6 into the integer task k + m + n, whose answer is 19.
B Supplementary Case
Table 3 presents a supplementary case of reflective behavior emergence.
- Table 3 reports a case of reflective behavior emergence.