PaperlayerView exact revision on arXiv

Source-linked AI summary

EasyPPO: Stabilizing the Critic Is Key

Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao, Dacheng Li, Wenhao Chai, Mayank Mishra, Yichuan Wang, Karthik Narasimhan, Alvin Cheung, Joseph E. Gonzalez

arXiv:2609.36802v1cs.LG

TL;DR

PPO’s learned critic can destabilize LLM reinforcement learning, including through completion-conditioned objectives and heterogeneous return noise. EasyPPO changes critic training and mini-batching to address these problems, and remains stable while outperforming PPO and recent variants across three tasks.

  • Problem

    Critic learning can destabilize PPO for LLMs: joint filtering makes the policy objective completion-conditioned, while heterogeneous return noise can let high-variance prompts dominate finite-batch updates.

    Source: §1 · pp. 1–2

  • Method

    EasyPPO uses actor-only overlong filtering, inverse-return-standard-deviation weighting for critic regression, and moderately smaller critic mini-batches under gradient clipping.

    Source: §1 · p. 3

  • Results

    Across FrontierCS, AIME24, and Search-R1, EasyPPO stayed stable and outperformed PPO, VAPO, and HL-Gauss PPO, with best-score gains over PPO of 14.89%, 2.28%, and 9.47%, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across coding, mathematical reasoning, and multi-turn search, the results support stabilizing the critic as key to reliable PPO training for LLMs.

    Source: §7 · p. 12

    Using the same rollout group to estimate weights and train the critic can introduce bias, though it works well in practice.

    Source: §4 · p. 6

Abstract

from arXiv · show

A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, allowing truncation to increase even as conditional reward improves. Second, heterogeneous return noise can cause high-variance prompts to dominate critic updates in finite batches. We introduce EasyPPO to address these failures. Actor-only overlong filtering trains the critic on returns from both completed and truncated rollouts. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts. Moderately smaller critic mini-batches confine outlier influence to fewer rollouts during gradient clipping. Across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1, EasyPPO remains stable throughout the full training horizon and consistently outperforms vanilla PPO, VAPO, and HL-Gauss PPO. Its best validation scores show relative gains of 14.89%, 2.28%, and 9.47% over PPO, respectively.

1 Introduction

PPO’s learned critic supports token-level return estimates and lower-variance policy updates, but critic learning can destabilize LLM training. EasyPPO addresses two critic failure modes, and remains stable while outperforming PPO and two recent variants across three tasks.

1 Introduction

  • Jointly filtering truncated rollouts shifts the critic target to completed responses, so conditional reward can improve while truncation grows.EasyPPO retains truncated rollouts for critic learning while filtering them only from actor updates.Source: §1 · pp. 2–3
  • Heterogeneous return noise can let high-variance prompts disproportionately influence critic updates in finite batches, especially as predictions improve.EasyPPO weights each prompt inversely by the empirical standard deviation of its rollout returns.Source: §1 · pp. 2–3
  • Moderately smaller critic mini-batches confine outlier influence to fewer rollouts under gradient clipping, balancing noise averaging against outlier effects.PPO’s critic is appealing for long-horizon reasoning because it estimates token-level returns from historical rollouts and supports lower-variance policy updates.Source: §1 · pp. 1, 3
  • EasyPPO stayed stable across 200–300 updates and outperformed PPO, VAPO, and HL-Gauss PPO on FrontierCS, AIME24, and Search-R1.Its best validation scores gained 14.89%, 2.28%, and 9.47% over PPO, respectively; it was the only compared method stable across all three tasks.Source: §1 · p. 3

2 Preliminaries

The preliminaries describe PPO’s terminal-reward actor–critic setup and the implementation choices used in the study. They also show that actor-only filtering improves stability over the alternatives but leaves residual instability.

2 Preliminaries

  • In the terminal-reward setting with γ = 1, the critic estimates expected final reward from each token-prefix state, and its values form token-level advantages with GAE.The response receives a scalar reward at its end; the implementation retains token-level GAE, PPO clipping, and KL regularization.Source: §2 · p. 3
  • The study’s implementation uses the full clipped PPO objective and KL regularization, with each rollout batch sampled from the latest policy snapshot.The simplified exposition omits PPO clipping and KL regularization.Source: §2 · p. 3
  • On FrontierCS, actor-only filtering achieves higher reward and lower truncation overall than joint filtering or no filtering, but instability remains.A spike and reward drop near step 160 mark residual instability; runs use 200 problems, batches of 512, a 30-step critic warm-up, and a 32,768-token response limit.Source: Figure 2 · p. 4

3 Overlong Filtering in PPO

Actor-only overlong filtering retains all rollouts for critic training while excluding truncated responses from actor updates. Joint actor-and-critic filtering instead learns a completion-conditioned target, which can raise completed-response reward without reducing truncation.

3 Overlong Filtering in PPO

  • The FrontierCS comparison uses Qwen3.5-9B, 200 generated problems, batches of 512 rollouts, a 30-step critic warm-up, and a 32,768-token response limit.The section notes that joint filtering also produces rising truncation and deteriorating overall reward in the AIME setting.Source: §3 · p. 4
  • Among the three filtering strategies, actor-only filtering is most stable and achieves the highest overall score.Joint filtering lets truncation approach 1 while overall reward remains low; without filtering, PPO exhibits repeated reward collapses.Source: §3 · p. 4
  • Filtering critic updates to completed responses changes the population target from E[R | s] to E[R | s, C], where C denotes non-truncation.Under joint filtering, the policy objective becomes reward conditioned on completion, without directly encouraging avoidance of truncation.Source: §3 · pp. 4–5
  • Actor-only filtering preserves the critic target E[R | s] by retaining all rollouts, while truncated rollouts remain excluded from actor updates.The remaining truncation spike and reward drop show that this filtering change alone is not sufficient for stability.Source: §3 · p. 5

4 Stabilizing the Critic

EasyPPO addresses heterogeneous return noise by normalizing critic regression across prompts and limiting outlier influence with smaller critic mini-batches. These changes balance gradient contributions while trading off outlier control against averaging return noise.

Noise-Normalized Critic Regression

  • As critic prediction error decreases, return noise can dominate gradient second moments, giving high-variability prompts disproportionate influence in finite batches.This heterogeneity is especially pronounced in continuous-score tasks with different reward scales.Source: §4 · pp. 5–6
  • Dividing each prefix’s critic loss by its conditional return standard deviation removes return-noise scale differences from the gradient second moment.Positive state weighting preserves the pointwise population optimum E[R | s].Source: §4 · p. 6
  • In practice, EasyPPO estimates prompt weights from rollout groups including truncated responses and normalizes them to mean one across the prompt batch.The proposed floor for discrete rewards is ε = ∆/(2√n).Source: §4 · p. 6
  • Using the same rollout group to estimate weights and train the critic can introduce bias, although the authors report that it works well in practice.Offline profiling with online updates is suggested as an alternative when fewer rollouts per prompt are desired.Source: §4 · p. 6

Return-Variability Diagnostics

  • Noise normalization reduces the dependence of critic gradient contributions on prompt return variability, with a similar overall trend observed on AIME.Some outliers remain, motivating smaller critic mini-batches.Source: Figure 3 · p. 7, §4 · p. 7

Critic Mini-Batch Updates

  • Smaller mini-batches confine outlier-driven gradient rescaling to fewer rollouts but average out less return noise.At fixed B, the analysis bounds a single outlier’s influence by O(m) and mini-batch gradient variance by O(1/m); the default uses four critic mini-batches per rollout batch.Source: §4 · pp. 7–8

5 Experiments

Across continuous-score coding, mathematical reasoning, and multi-turn search, EasyPPO achieves the best validation performance among the compared methods and avoids training collapse in evaluated settings. The experiments also examine critic behavior, seed sensitivity, and the effects of normalization and mini-batch size.

EasyPPO Stabilizes Critic Learning and Policy Training

  • EasyPPO improves over PPO by 1.92, 1.46, and 3.74 validation points on FrontierCS, AIME24, and Search-R1, respectively, and has the best validation performance on all three tasks.Every baseline collapses in at least one setting, while EasyPPO has no training collapse in the evaluated settings.Source: §5 · p. 9
  • With the same actor-only filtering rule, PPO fluctuates on FrontierCS and collapses on AIME, while EasyPPO remains stable.The authors attribute the difference to EasyPPO’s critic modifications.Source: §5 · p. 9
  • On FrontierCS, EasyPPO’s post-warmup critic gradient norm declines and explained variance rises steadily, while baselines show sharp explained-variance swings.HL-Gauss approaches explained variance of 1 only after its actor collapses.Source: Figure 5 · p. 9, §5 · p. 10

Sensitivity to Random Seeds

  • None of three EasyPPO runs collapses on FrontierCS, compared with two of three runs using PPO with actor-only filtering.EasyPPO also sustains validation gains with a narrow min–max band across runs.Source: §5 · p. 10, Figure 6 · p. 10

Noise Normalization Improves Stability Across Critic Mini-Batch Sizes

  • Noise normalization stabilizes FrontierCS training with both tested critic mini-batch configurations, whereas without normalization the four-mini-batch setting collapses.With normalization, both configurations preserve their training and validation gains.Source: §5 · pp. 10–11, Figure 7 · p. 10

The Trade-off in Critic Mini-Batch Size

  • On FrontierCS, the first raw explained-variance crossing of zero occurs earlier as mini-batch count increases, while K = 4 and 8 achieve higher explained variance later.The observed trade-off motivates the default K = 4.Source: §5 · p. 11, Figure 8 · p. 11

6 Related Work

Related work spans critic-free and actor–critic approaches to LLM post-training, as well as methods for continuous rewards and variance-aware regression. EasyPPO focuses on prompt-level return noise in critic training and on how filtering actor and critic updates affects the policy objective.

Policy Optimization for LLM Post-Training

  • PPO is described as a standard actor–critic method for LLM post-training [3], while RLVR uses task-specific verifiers instead of learned reward models [2] [14].Critic-free methods commonly estimate advantages from response groups and filter truncated rollouts [10] [14] [23] [24].Source: §6 · p. 11
  • Prior actor–critic work improves value learning through pretraining, decoupled GAE, categorical prediction, and hybrid designs [12] [13] [25] [26] [27].EasyPPO instead targets prompt-level return noise and shows how filtering both networks shifts the policy objective.Source: §6 · pp. 11–12

Continuous Verifiable Rewards and Variance-Weighted Critic Regression

  • Continuous-reward benchmarks expose differences in return scale and variability across prompts, motivating focus on critic stability.Prior approaches include upper-tail optimization with GRPO [20], variance-weighted critic objectives [36], and heteroscedastic regression [40]; EasyPPO uses group sampling to estimate prompt-level return variance without an auxiliary variance model.Source: §6 · p. 12

7 Conclusion

The conclusion identifies overlong-rollout handling and heterogeneous return noise as sources of PPO critic instability, motivating three EasyPPO changes. Across coding, math reasoning, and multi-turn search, the authors conclude that stabilizing the critic is key to reliable PPO training for LLMs.

7 Conclusion

  • The paper identifies overlong-rollout handling and heterogeneous return noise as two sources of critic instability in PPO.Its analysis links joint actor-and-critic filtering to a changed policy objective and return noise to critic-gradient second moments.Source: §7 · p. 12
  • EasyPPO combines actor-only overlong filtering, noise-normalized critic regression, and moderately smaller critic mini-batches with gradient clipping while retaining the standard PPO actor update.The conclusion presents these changes as responses to the two identified sources of critic instability.Source: §7 · p. 12
  • Across continuous-score coding, mathematical reasoning, and multi-turn search, the authors report stability and performance gains from EasyPPO.They conclude that stabilizing the critic is key to reliable PPO training for LLMs.Source: §7 · p. 12

Appendix Contents

  • The appendices list training configurations, theoretical analyses, and additional experimental results, including critic diagnostics and validation scores.The listed appendices also cover initialization, filtering, return-variability diagnostics, and individual Search-R1 validation sets.Source: Appendix Contents · p. 16

A Training Configurations and Implementation Details

This appendix gives configurations for Section 5 and the Section 3 filtering study. For FrontierCS, it describes the data and supervised-fine-tuning initialization used to create the starting model.

A Training Configurations and Implementation Details

A.1 Data and FrontierCS Initialization

  • FrontierCS experiments use a distilled Qwen3.5-9B model initialized by one SFT epoch on 347 retained responses covering 179 problems.The responses were retained after removing 253 zero-score responses from 600 samples drawn for 200 FrontierSmith-generated problems.Source: §A.1 · p. 16

A.2 Training and Evaluation Settings

This appendix summarizes task, training, and optimizer settings, then describes critic configurations for the main comparisons. It also identifies the HL-Gauss PPO baseline settings and the actor KL estimator.

A.2 Training and Evaluation Settings

  • AIME and Search-R1 start from Qwen3.5-9B-Base, while FrontierCS uses the SFT initialization described in the appendix.The tables summarize task settings for Figure 4 and PPO and optimizer settings; learning rates remain constant after any optimizer warmup.Source: §A.2 · p. 16
  • The actor KL loss uses the low var kl estimator.Source: §A.2 · p. 16

A.3 Critic and Baseline Configurations

  • EasyPPO uses four critic mini-batches per rollout batch, with 128 responses per mini-batch on FrontierCS and AIME and 256 on Search-R1.All methods retain truncated responses for critic training; VAPO keeps its value-learning and advantage-estimation recipe but disables its auxiliary language-modeling loss.Source: §A.3 · p. 17
  • Table A.3 summarizes critic configurations in the main comparisons, and Table A.5 lists the task-specific categorical-critic settings for HL-Gauss PPO.Source: §A.3 · p. 17

A.4 Diagnostic and Ablation Configurations

The appendix documents the diagnostic and ablation configurations, then analyzes how return variability affects critic updates and how filtering truncated rollouts changes the policy objective. Its analysis distinguishes actor-only filtering from filtering both actor and critic updates.

A.4 Diagnostic and Ablation Configurations

  • The filtering study uses 512-response rollout batches and 30 critic warmup steps to compare no overlong filtering, filtering both actor and critic updates, and actor-only filtering using training-rollout statistics.The offline diagnostic uses terminal-return MSE gradients before clipping and takes no optimizer step.Source: §A.4 · pp. 17, 19, Table A.6 · p. 19
  • The ablation configurations vary noise normalization and critic mini-batch settings, with Figure 7 holding the critic learning rate fixed to isolate normalization.Figure 8 instead scales the learning rate in proportion to the mini-batch configuration; Table A.7 lists plotted configurations.Source: §A.4 · p. 19, Table A.7 · p. 19

B.1 Effect of Return Heterogeneity on Critic Updates

  • Even when the expected value-head gradient vanishes at an accurate prediction of expected return, individual sampled returns can still produce stochastic updates.The analysis parameterizes the critic with a hidden representation and linear value head, and considers the squared value loss and gradients over sampled returns.Source: §B.1 · p. 20

B.2 Actor-only Overlong Filtering

  • Filtering truncated rollouts from both actor and critic makes the critic target completion-conditioned and shifts the objective to expected reward among trajectories that avoid truncation.This is the population-level conclusion for the conditional critic analyzed in the appendix.Source: §B.2 · p. 21
  • When completed trajectories have higher expected reward, the filtered actor update includes a signal favoring avoidance of truncation, with truncated returns affecting that signal through the critic baseline.Source: §B.2 · p. 22
  • The discrepancy between actor filtering and the unfiltered policy gradient vanishes as the probability of truncation approaches zero.At token level, the full-distribution critic retains the target E[R | st], so truncated returns still affect advantages through the baseline.Source: §B.2 · p. 22

B.3 Noise-Normalized Critic Regression

The appendix derives prompt-level noise normalization and analyzes critic mini-batch clipping, then reports validation-score scope and critic diagnostics across tasks and initializations.

B.3 Noise-Normalized Critic Regression

  • Prompt-level return variance upper-bounds average prefix-level conditional return variance; the law of total variance also includes variation in true state values across prefixes.The analysis assumes a locally bounded critic Jacobian when bounding the prefix value-gradient second moment.Source: §B.3 · pp. 22–23

B.3 Noise-Normalized Critic Regression; B.4 Critic Mini-Batch Clipping

  • Inverse-return-STD prompt weighting makes the noise bound invariant to prompt-level return variability, while smaller critic mini-batches trade lower outlier sensitivity for larger per-clipping noise.The outlier-sensitivity bound scales as O(m), while the noise entering each clipping operation follows O(1/m); the latter is not the variance of the full-batch average.Source: §B.3 · p. 23, §B.4 · p. 24

B.4 Critic Mini-Batch Clipping

  • The scaling analysis is idealized at fixed critic parameters: sequential Adam updates are not bounded, and correlated rollouts or jointly estimated weights may change the variance scaling.The variance identity assumes independent rollout contributions.Source: §B.4 · p. 24

C.1 Best Validation Scores

  • Table C.1 reports each method’s best unsmoothed validation score from Figure 4’s runs and horizons, using Search-R1’s seven-dataset mean at a single checkpoint.Scores use a 0–100 scale, and the second-best method is selected separately for each task.Source: §C.1 · p. 24, Table C.1 · p. 25

C.2 Critic Diagnostics on AIME and Search-R1

  • On AIME and Search-R1, EasyPPO avoids the large negative explained-variance excursions seen for VAPO on AIME and for PPO and VAPO on Search-R1.HL-Gauss retains positive explained variance during its Search-R1 performance decline, so these diagnostics should be read alongside task scores.Source: §C.2 · p. 24, Figure C.1 · p. 25

C.3 Different Critic Initializations on FrontierCS

  • Across three critic initializations, all EasyPPO FrontierCS runs maintain training gains; validation summaries cover three runs through step 170 and two from step 175 onward.Run 3 ends at step 172, so the later summary does not extrapolate its validation results.Source: §C.3 · p. 25

C.4 Offline Return-Variability and Gradient Diagnostics

Offline diagnostics indicate that noise normalization can reduce return-scale imbalance without removing all gradient variation; the appendix also compares filtering strategies on AIME and individual Search-R1 validation sets.

C.4 Offline Return-Variability and Gradient Diagnostics

  • Noise normalization weakens FrontierCS gradient–return-variability trends at each checkpoint, but large individual contributions remain and AIME’s checkpoint trends are less uniform.In pooled AIME diagnostics, gradient norms depend less on return variability after normalization, though the effect varies across checkpoints; the diagnostic supports reducing return-scale imbalance, not eliminating all gradient variation.Source: Figure C.3 · p. 26, §C.4 · pp. 27–28, Figure C.4 · p. 27

C.5 Overlong Filtering on AIME

  • On AIME, joint filtering increases truncation and lowers training and validation scores; actor-only filtering later collapses, so filtering alone is insufficient for stability.The compared runs share core settings, with filtering strategy as the only recorded training-setting difference.Source: §C.5 · p. 28, Figure C.6 · p. 28

C.6 Search-R1 Results on Individual Validation Sets

  • EasyPPO achieves the highest final validation score on all seven Search-R1 datasets, while PPO, VAPO, and PPO with actor-only filtering fall to nearly zero across them.HL-Gauss PPO also loses substantial performance, particularly on MuSiQue and Bamboogle.Source: §C.6 · p. 29, Figure C.7 · p. 29
Loading 2609.36802v1…