VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

1 Introduction

• Value models can offer finer-grained credit assignment, lower-variance token values, and better sample use, but long-CoT training faces value bias, variable lengths, and sparse rewards.VAPO is introduced to address these challenges and realize the potential of value-model-based methods for reasoning tasks.Source: §1 · p. 2
• Length-adaptive GAE adjusts λ according to response length to accommodate differing bias-variance trade-offs and improve advantage-estimation accuracy and stability.The method targets settings where sequence lengths vary widely.Source: §1 · p. 2
• VAPO integrates Clip-Higher and Token-level Loss from DAPO [29], Value-Pretraining and Decoupled-GAE from VC-PPO [30], self-imitation learning from SIL [14], and Group-Sampling from GRPO [22].The paper reports validating the techniques’ necessity through ablation studies.Source: §1 · p. 3
• VAPO improves from vanilla PPO’s score of 5 to 60, surpasses DAPO [29] by 10 points, and shows no training crashes across multiple runs.Experiments used the Qwen2.5-32B pre-trained model without SFT data, enabling comparison with DAPO and DeepSeek-R1-Zero-Qwen-32B; results across runs were consistently similar.Source: §1 · p. 3

2.1 Modeling Language Generation as Token-Level MDP

• Language generation is modeled as a token-level MDP in which prompt and response tokens form sequences and each action deterministically appends a token.The action space is a fixed discrete vocabulary, and a reward function provides scalar feedback.Source: §2.1 · p. 3

2.2 RLHF Learning Objective

• The RLHF objective seeks an optimal KL-regularized policy, with β controlling regularization strength relative to the initialization policy πref.Traditional RLHF and most LLM tasks assign reward sparsely, only at the terminal <eos> action.Source: §2.2 · pp. 3–4

2.3 Proximal Policy Optimization

• PPO [21] limits policy updates with a clipped surrogate objective, while GAE [20] uses multi-step bootstrapping to reduce advantage-estimate variance.Because γ = 1.0 is common in RLHF, the paper omits γ in later notation.Source: §2.3 · p. 4

3.1 Value Model Bias over Long Sequences

• Reward-model initialization creates positive value bias, while GAE with λ = 0.95 attenuates terminal rewards on long trajectories and leaves updates reliant on biased bootstrapping [30].The bias stems from the objective mismatch between reward models scoring at <EOS> and value models estimating cumulative reward at preceding tokens.Source: §3.1 · p. 4

3.2 Heterogeneous Sequence Lengths during Training

• A fixed λ struggles with variable response lengths: GAE estimates are variance-dominated for short responses and increasingly biased for long responses, even with a perfect value model.The bias in long responses accumulates through GAE’s recursive reliance on future state values.Source: §3.2 · p. 5

3.3 Sparsity of Reward Signal in Verifier-based Tasks

• Verifier rewards are typically binary rather than dense, and long CoT further lowers nonzero-reward frequency, making correct sampled responses scarce [6].Longer outputs also increase computational time.Source: §3.3 · p. 5
• Sparse rewards create an exploration-exploitation dilemma: models must preserve uncertainty for diverse sampling while effectively using correct responses to improve learning efficiency.Excessive exploitation can trap the model in suboptimal solutions, while unproductive exploration wastes computational resources.Source: §3.3 · p. 5

4.1 Mitigating Value Model Bias over Long Sequences

• Value-Pretraining addresses value initialization bias from a reward model with a mismatched objective; PPO applied naively to long-CoT can collapse output lengths and degrade performance [30].The value model is trained on Monte-Carlo returns from a fixed policy until value loss and explained variance are sufficiently low, then checkpointed for later experiments.Source: §4.1 · p. 5
• Decoupled-GAE uses λ = 1.0 for value updates to yield unbiased gradient descent and address long-CoT reward decay, while policy updates use a smaller λ for faster convergence [30].VC-PPO uses λcritic = 1.0 and λpolicy = 0.95; VAPO adopts the core idea of decoupling GAE computation.Source: §4.1 · pp. 5–6

4.2 Managing Heterogeneous Sequence Lengths during Training

• For sequences longer than 100 tokens, fixed λpolicy = 0.95 makes the reward TD-error coefficient 0.95^100 ≈ 0.006, so biased bootstrapping can dominate GAE.Length-Adaptive GAE adjusts λpolicy with output length to distribute TD-errors more uniformly across short and long sequences; α controls the bias-variance trade-off.Source: §4.2 · p. 6
• Token-level policy-gradient loss gives every batch token uniform weight, addressing how sequence-level averaging underweights longer trajectories and can leave long-sequence failures insufficiently suppressed.This weighting change is intended to improve training stability when long-sequence processing issues arise during RL exploration; it follows DAPO [29].Source: §4.2 · p. 6

4.3 Dealing with Sparsity of Reward Signal in Verifier-based Tasks

• To address sparse verifier rewards, VAPO adopts Clip-Higher, Positive Example LM Loss, and Group-Sampling; Clip-Higher mitigates entropy collapse by decoupling lower and higher clipping ranges [29].VAPO increases εhigh to allow low-probability tokens more room to increase, while keeping εlow relatively small to avoid collapsing the sampling space.Source: §4.3 · p. 7
• Positive Example LM Loss adds an imitation-learning NLL objective for correct answers sampled during RL, increasing the utility of scarce positive samples in low-accuracy reasoning tasks.The NLL loss is combined with policy-gradient loss using weighting coefficient µ.Source: §4.3 · p. 7
• Group-Sampling repeats generations for fewer prompts within a fixed compute budget, yielding marginally better performance through richer within-prompt positive-negative contrastive signals.The alternative samples as many prompts as possible once each.Source: §4.3 · p. 7

5.1 Training Details

• AIME24 evaluation used average pass rate over 32 runs with topp=0.7 and temperature=1.0; VAPO also used a 50-step value-network warmup before policy training.The warmup was based on the reward model.Source: §5.1 · p. 8

5.2 Ablation Results

• 60.4 AIME24 points within 5,000 steps gives VAPO a new SOTA; DeepSeek R1 reaches 47 points and DAPO 50 points, while VAPO peaks at 60–61 across three runs.VAPO matches DAPO’s performance using 60% of DAPO’s update steps.Source: §5.2 · p. 8
• Removing value pretraining led to approximately 11 points; removing decoupled GAE caused a 27-point drop, while adaptive GAE yielded a 15-point improvement.Without value pretraining, training collapsed as in Vanilla PPO; without decoupled GAE, reward signals exponentially decayed during backpropagation.Source: §5.2 · p. 8
• Removing clip higher limited convergence to 46 points; token-level loss contributed a 7-point gain, positive-example LM loss nearly 6 points, and Group-Sampling 5 points.Token-level loss implicitly increased the weight of long responses; Group-Sampling used fewer prompts with more repetitions.Source: §5.2 · p. 9

5.3 Training Dynamics

• VAPO’s training curve was smoother than DAPO’s, with superior length scaling and faster score growth attributed to granular value-model signals.The paper identifies smoother changes and faster growth as desirable training-curve characteristics.Source: §5.3 · p. 9, Figure 2 · p. 9
• VAPO’s entropy dropped below DAPO’s in later training; the paper notes this may hinder exploration but had minimal negative impact on final performance.The lower entropy was associated with improved stability and reproducibility.Source: §5.3 · p. 10

6 Related Work

• O1 popularizes extended reasoning before a final response, while DeepSeek R1 [6] open-sources the value-model-free GRPO algorithm [22] and model weights.The passage says DeepSeek R1’s weights are comparable in performance to O1.Source: §6 · p. 10
• DAPO [29] addresses entropy collapse with four techniques; Dr. GRPO [12] removes GRPO’s length and standard-deviation normalization terms.Source: §6 · p. 10
• ORZ uses PPO with a value model and Monte Carlo advantage estimation instead of GAE, but achieves performance comparable to GRPO and DAPO.VAPO also uses a value model and is reported to outperform DAPO.Source: §6 · p. 10

• VAPO achieves state-of-the-art performance on the AIME24 benchmark using Qwen2.5-32B.Source: §7 · p. 10
• Seven novel techniques atop PPO refine value learning and balance exploration in VAPO.Source: §7 · p. 10
• VAPO outperforms value-model-free methods including GRPO and DAPO, providing a robust framework for reasoning-intensive tasks.Source: §7 · p. 10