Source-linked AI summary
What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, Lin Yan
TL;DR
PPO can fail on long-CoT tasks because value initialization bias and GAE reward-signal decay impair value estimation. VC-PPO addresses these issues through value pretraining and decoupled GAE, and substantially improves PPO performance on AIME, with both techniques essential in ablations.
Problem
PPO fails in long-CoT tasks as value estimation becomes difficult, while value initialization bias and decaying GAE signals impair advantage estimation.
Method
VC-PPO pretrains the value model on fixed-SFT-policy responses and decouples GAE computation for policy and value models.
Results
5.6 to 49.0: VC-PPO significantly boosts baseline PPO performance on AIME and exceeds the score previously reported in [4].
Takeaways & Limitations
Value pretraining and decoupled GAE are essential components for improving PPO on long-CoT tasks.
Takeaways & Limitations
Direct comparison with DeepSeek-R1 is not entirely feasible because its RL-training dataset is unavailable.
Abstract
from arXiv · showhide
Reinforcement learning (RL) is pivotal for enabling large language models (LLMs) to generate long chains of thought (CoT) for complex tasks like math and reasoning. However, Proximal Policy Optimization (PPO), effective in many RL scenarios, fails in long CoT tasks. This paper identifies that value initialization bias and reward signal decay are the root causes of PPO's failure. We propose Value-Calibrated PPO (VC-PPO) to address these issues. In VC-PPO, the value model is pretrained to tackle initialization bias, and the Generalized Advantage Estimation (GAE) computation is decoupled between the actor and critic to mitigate reward signal decay. Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance. Ablation studies show that techniques in VC-PPO are essential in enhancing PPO for long CoT tasks.
1 Introduction
Long-CoT reasoning depends on reinforcement learning, yet directly applying PPO can fail as responses lengthen because value estimation becomes difficult. VC-PPO addresses value initialization bias and reward-signal decay through value pretraining and decoupled GAE, substantially improving PPO on AIME.
- Long-CoT generation enables LLMs to solve Olympiad-level math and reasoning problems by developing and verifying hypotheses before answering.
- PPO can fail on long-CoT tasks because accurate value modeling becomes increasingly challenging as response length increases.
- PPO offers token-level feedback, whereas GRPO replaces the value model with a Leave-One-Out estimate and uses response-level feedback.
- Value initialization bias produces inaccurate early advantage estimates, while in-training bias arises from the decaying GAE reward signal.
- VC-PPO pretrains its value model on fixed-SFT-policy responses and decouples policy and value GAE computations, using a larger λ for value estimation.
- 5.6 to 49.0: On AIME, the two VC-PPO techniques significantly improve baseline PPO and exceed the score previously reported in [4].Ablations identify both techniques as essential for superior AIME performance.
2 Preliminaries
The paper models language generation as a token-level Markov decision process and formulates learning as KL-regularized reinforcement learning. PPO then uses clipped policy updates, with GAE providing advantage estimates from multi-step temporal-difference errors.
- Language generation is cast as a Markov decision process in which a policy maximizes cumulative reward through interaction with an environment.
- Prompts and responses are token sequences, and the token-level MDP represents generated prefixes as states.
- The token-level MDP is defined by state space, vocabulary-based actions, deterministic token transitions, termination action ω, and a reward function.
- The KL-regularized objective maximizes token-level rewards while penalizing divergence from an initialization reference policy.
- RLHF commonly assigns sparse reward only at the terminal end-of-sentence action.
- PPO limits policy-update changes with a clipped surrogate objective based on the probability ratio and estimated advantage.
- GAE estimates advantages using multi-step bootstrapping, combining discounting, λ-weighting, rewards, and value-function temporal-difference errors.
3 Identifying and Addressing PPO’s Failure Modes in Long CoT Tasks
PPO fails on long-CoT tasks because value initialization bias and GAE reward-signal decay distort advantage and value estimates. Value pretraining and Decoupled-GAE address these issues by calibrating the value model and separating policy and value λ choices.
- Failure modes: PPO commonly initializes its value model from a reward model and uses default GAE, practices that can fail on long-CoT tasks.These practices arise from reward-model/value-model similarity and variance-reduction conventions in traditional RL.
- Failure modes: Validation performance can degrade alongside a substantial output-length decrease, which is associated with poorer complex-reasoning performance.The paper identifies output-length collapse as the root cause of this performance degradation.
- Value initialization bias: Reward-model objective mismatch creates position-dependent positive advantage bias because reward-model scores are trained at <EOS>, while value prediction estimates expected token rewards.The bias accumulates along the trajectory and favors shorter responses, disrupting long chains of thought.
- Value initialization bias: Value-Pretraining offline-trains the value model to convergence on responses sampled from a fixed policy, using λ = 1.0 for stable Monte-Carlo-return optimization.The resulting checkpoint is used in subsequent experiments as a more accurate initial value estimate.
- In-training value bias: For long trajectories, λ < 1.0 rapidly attenuates the <EOS> reward before it reaches preceding tokens, making unbiased value optimization preferable for trajectory-level rewards.The paper contrasts this with policy optimization, where variance reduction may still be necessary.
- VC-PPO: VC-PPO combines Value-Pretraining and Decoupled-GAE to enhance PPO for long-CoT tasks.The method is presented as a simple modification of baseline PPO targeting both initialization and in-training value bias.
4 Experiments
Experiments evaluate VC-PPO on AIME using controlled value pretraining and decoupled-GAE settings, with comparisons against GRPO and component ablations. VC-PPO improves PPO substantially, while both proposed components and calibrated hyperparameters materially affect performance.
- 4.2 Experimental Results: VC-PPO significantly outperforms GRPO under the same 16K-context experimental setting.
- 4.2 Experimental Results: A pass@1 score of 48.8 on AIME is reported as the highest result for Qwen-32B-Base without distillation, exceeding the comparable reported score of 47.0.The paper cautions that direct comparison is not entirely feasible because DeepSeek-R1's RL-training dataset is unavailable.
- 4.2 Experimental Results: Removing either Value-Pretraining or Decoupled-GAE causes a notable performance drop, indicating that both components contribute to VC-PPO's effectiveness.
- 4.2 Experimental Results: Pretraining the value model for 100 steps is optimal in the reported ablation, whereas additional training might induce overfitting and harm generalization.
- 4.2 Experimental Results: For AIME, λactor = 0.99 outperforms lower tested values, while performance decreases only slightly from 0.95 to 1.0; the recommended range is λactor ∈[0.95, 1.0).The value-side setting uses λcritic = 1.0, while the policy-side ablation varies λactor.
- 4.3 Discussion: Value-Pretraining aligns the value model with the initial policy and prevents the observed collapse in output length.The method is described as a value warm-up that injects knowledge into the value model.
- 4.3 Discussion: Value loss shows rapid range alignment followed by slower knowledge injection, and the latter stage substantially affects final performance.
5 Conclusion
VC-PPO addresses PPO’s failure on long-CoT tasks by correcting value initialization bias and reward signal decay. Experiments and ablations show that value pretraining and decoupled GAE are central to improved performance.
- VC-PPO targets value initialization bias and reward signal decay, the identified main problems underlying PPO’s failure on long-CoT tasks.
- Value pretraining aligns the value model with the initial policy, preventing loss of the CoT pattern and improving performance.
- Decoupled GAE separates policy and value computations to provide better bias-variance trade-offs for both components.
- VC-PPO significantly outperforms baseline PPO on AIME, CodeForces, and GPQA, while ablations emphasize both techniques’ crucial roles.
- The study identifies differing variance-bias preferences between value and policy models as a promising direction for future RL and RLHF research.