Source-linked AI summary
VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, Lin Yan
TL;DR
Value-based reinforcement learning for long-CoT reasoning is limited by value-model bias, heterogeneous sequence lengths, and sparse verifier rewards. VAPO integrates seven techniques to address these issues and achieves state-of-the-art AIME24 performance with efficient, stable training. The framework reaches 60.4 within 5,000 steps and reports no crashes across repeated runs.
Problem
Value-based reinforcement learning for long-CoT reasoning must address biased value estimates, variable response lengths, and sparse verifier rewards.
Method
VAPO is a value-model-based PPO framework integrating techniques for value learning, length-adaptive advantage estimation, exploration, and policy optimization.
Results
60.4 on AIME24 is achieved by VAPO within 5,000 steps, while peak scores of 60-61 recur across three experiments.
Takeaways & Limitations
VAPO provides a robust value-model-based framework for reasoning-intensive large language model tasks.
Abstract
from arXiv · showhide
We present VAPO, Value-based Augmented Proximal Policy Optimization framework for reasoning models., a novel framework tailored for reasoning models within the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the Qwen 32B pre-trained model, attains a state-of-the-art score of $\mathbf{60.4}$. In direct comparison under identical experimental settings, VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B and DAPO by more than 10 points. The training process of VAPO stands out for its stability and efficiency. It reaches state-of-the-art performance within a mere 5,000 steps. Moreover, across multiple independent runs, no training crashes occur, underscoring its reliability. This research delves into long chain-of-thought (long-CoT) reasoning using a value-based reinforcement learning framework. We pinpoint three key challenges that plague value-based methods: value model bias, the presence of heterogeneous sequence lengths, and the sparsity of reward signals. Through systematic design, VAPO offers an integrated solution that effectively alleviates these challenges, enabling enhanced performance in long-CoT reasoning tasks.
1 Introduction
VAPO is a value-model-based reinforcement learning framework for long-CoT reasoning that addresses value-model bias, heterogeneous sequence lengths, and sparse rewards. It integrates several design improvements and achieves strong, efficient, and stable performance on AIME24.
- Motivation and framework: Value models offer finer-grained credit assignment and lower-variance token-level estimates than value-model-free approaches when trained reliably.These advantages are especially relevant when individual reasoning-step errors can cause failure.
- Motivation and framework: VAPO addresses value-model bias, heterogeneous response lengths, and sparse verifier rewards in long-CoT reinforcement learning.The framework is designed to improve value-based training for reasoning tasks.
- Method: Length-adaptive GAE adjusts λ according to response length to accommodate different bias-variance trade-offs.This targets the distinct estimation needs of short and long responses.
- Method: VAPO integrates techniques from DAPO, VC-PPO, SIL, and GRPO, with ablation studies validating their necessity.The integrated system combines policy-loss, value-pretraining, GAE, self-imitation, and sampling modifications.
- Results: VAPO combines these improvements in experiments using the Qwen2.5-32B pre-trained model without introducing SFT data.The authors describe the combined result as better than the sum of individual parts.
- Results: 60 points are achieved by VAPO versus 5 for vanilla PPO, while surpassing DAPO by 10 points under comparable experiments.Training is reported as crash-free across multiple runs with consistently similar results.
2 Preliminaries
The paper models language generation as a token-level Markov decision process and formulates learning as KL-regularized reinforcement learning. PPO provides clipped policy updates, while GAE estimates token-level advantages from rewards and value predictions.
- Token-level MDP: Language generation is cast as a Markov decision process in which prompts initialize states and generated tokens form sequential actions.The environment transitions deterministically as each token is appended to the response.
- Token-level MDP: The token-level MDP is defined as M = (S, A, P, R, d0, ω), covering states, vocabulary actions, dynamics, rewards, initial prompts, and termination.Termination occurs when the end-of-sentence action ω is executed.
- Rewards and objective: The reward function supplies scalar feedback, while verifier-style and RLHF settings commonly provide reward sparsely at the terminal token.The terminal action is typically the end-of-sentence token.
- Rewards and objective: The optimization objective is KL-regularized, balancing cumulative token-level rewards against divergence from an initialization reference policy.β controls the strength of KL regularization, and πref is the initialization policy.
- PPO and GAE: PPO limits policy changes through a clipped surrogate objective based on the probability ratio, estimated advantage, and clipping hyperparameter.Clipping is intended to prevent large updates that could destabilize training.
- PPO and GAE: GAE combines multi-step bootstrapping to estimate token-level advantages using the discount factor, λ, rewards, and temporal-difference errors.The paper notes that γ = 1.0 is commonly used in RLHF and is omitted later.
3 Challenges in Long-CoT RL for Reasoning Tasks
Long-CoT reinforcement learning makes value-based training difficult because value estimates can be biased, sequence lengths vary widely, and verifier rewards are sparse. These issues affect credit assignment, advantage estimation, and the exploration-exploitation balance.
- Overview: Long-CoT reinforcement learning faces technical problems from sequence-length dynamics, value-function instability, and reward sparsity.These challenges are central to applying value models to reasoning tasks.
- Value-model bias: Initializing a value model from a reward model creates positive initialization bias because the models optimize different targets.Reward models score terminal outcomes, whereas value models estimate expected cumulative rewards for preceding tokens.
- Value-model bias: λ = 0.95 can attenuate terminal rewards toward near zero on long sequences, leaving value updates heavily dependent on biased bootstrapping.This undermines the value model as a variance-reduction baseline.
- Heterogeneous lengths: Highly variable response lengths make fixed-λ GAE difficult to tune across short and long reasoning trajectories.Short responses tend toward high-variance estimates, whereas long responses tend toward high-bias estimates.
- Reward sparsity: Verifier rewards are typically binary rather than dense, and long CoT further reduces reward frequency while increasing computation.This sparse feedback creates a distinct exploration-exploitation dilemma.
4 VAPO: Addressing the Challenges in Long-CoT RL
VAPO combines value-model pretraining, decoupled GAE, length adaptation, token-level loss, and sparse-reward techniques to stabilize long-CoT reinforcement learning. These designs target value bias, heterogeneous sequence lengths, and inefficient exploration.
- 4.1 Mitigating Value Model Bias over Long Sequences: Value-Pretraining and decoupled-GAE address value-model bias over long sequences by improving value initialization and separating value and policy advantage computation.Value-Pretraining mitigates initialization bias, while decoupled-GAE uses λcritic = 1.0 for value updates and a smaller λpolicy for policy updates.
- 4.2 Managing Heterogeneous Sequence Lengths during Training: Length-Adaptive GAE dynamically adjusts λpolicy with sequence length, addressing inconsistent bias-variance requirements across short and long responses.With fixed λpolicy = 0.95, the reward coefficient becomes 0.95100 ≈ 0.006 for sequences longer than 100 tokens, leaving bootstrapping TD-errors dominant.
- 4.2 Managing Heterogeneous Sequence Lengths during Training: Token-level policy gradient loss assigns uniform weights across batch tokens, preventing long-sequence tokens from being underweighted during optimization.The prior sequence-level averaging reduced the contribution of longer sequences and could contribute to instability or collapse when long-sequence processing failed.
- 4.3 Dealing with Sparsity of Reward Signal in Verifier-based Tasks: VAPO addresses sparse verifier rewards with Clip-Higher, Positive Example LM Loss, and Group-Sampling to improve exploration-exploitation efficiency.Clip-Higher mitigates entropy collapse, Positive Example LM Loss improves use of correct samples, and Group-Sampling obtains discriminative positive and negative samples within prompts.
5 Experiments
VAPO is evaluated against PPO and prior methods on AIME24, with ablations examining its training modifications and stability-related behavior.
- Main Results: 60.4 AIME24 points were achieved by VAPO within 5,000 steps, while matching DAPO using only 60% of DAPO’s steps.The evaluation averages AIME24 pass rate over 32 samples.
- Ablation Results: 60 points were achieved by VAPO, compared with 5 points from vanilla PPO after value-model learning collapse.Vanilla PPO also showed drastically reduced response lengths and directly answered questions without reasoning.
- Ablation Results: 27 points were lost when decoupled GAE was removed, because reward signals exponentially decayed during backpropagation.
- Ablation Results: 15 points were gained from adaptive GAE, while removing Clip-Higher limited maximum convergence to 46 points.Adaptive GAE balances optimization for short and long responses.
- Ablation Results: Token-level loss, positive-example LM loss, and Group-Sampling contributed gains of 7, nearly 6, and 5 points, respectively.Group-Sampling used fewer prompts with more repetitions.
- Training Dynamics: VAPO’s training curve was smoother than DAPO’s, its score grew faster, and its later-stage entropy was lower.The lower entropy may hinder exploration but was reported to have minimal negative impact on final performance while improving stability and reproducibility.
6 Related Work
Recent reasoning models use extended chain-of-thought, while DeepSeek R1 and DAPO represent influential open approaches based on value-model-free reinforcement learning.
- OpenAI O1 introduced extended reasoning before delivering a final response, and DeepSeek R1 open-sourced its training algorithm and model weights.
- DeepSeek R1 uses value-model-free GRPO, while DAPO addresses entropy collapse during value-model-free language-model reinforcement-learning scaling.
- DeepSeek R1’s open-source model weights were reported as comparable in performance to O1.
7 Conclusion
The paper concludes that VAPO combines seven PPO-based techniques to improve value learning and exploration for reasoning-intensive tasks, achieving state-of-the-art AIME24 performance.
- VAPO achieves state-of-the-art performance on the AIME24 benchmark using the Qwen2.5-32B model.
- Seven techniques built atop PPO refine value learning and balance exploration in VAPO.
- VAPO outperforms contemporary value-model-free methods such as GRPO and DAPO.
- The work provides a robust framework for advancing language models on reasoning-intensive tasks.
Contributions
The supplied contribution passages list the paper’s authors and their affiliations or ordering notation.
- The first listed author group includes Yu Yue, Yufeng Yuan, Qiying Yu, and Xiaochen Zuo.
- The next listed author group includes Gaohong Liu, Juncai Liu, Lingjun Liu, and Haibin Lin.
- The author list marks the last name as being in alphabetical order.