Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Yizhuo Li; Jianhao Yan; Yun Luo; Zhi Wang; Futing Wang; Rong-Xi Tan; Kanghui Tian; Ganqu Cui; Ning Ding; Peilin Zhao; Yafu Li; Yu Cheng

Source-linked AI summary

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, Ning Ding, Peilin Zhao, Yafu Li, Yu Cheng

arXiv:2609.18708v1Sep 16, 2026cs.LGcs.AI

TL;DR

The paper identifies Value Flattening, where PPO critic predictions stay comparatively flat despite sharp within-response changes in Monte Carlo state values. It analyzes the critic loss and temporal redundancy, then introduces sparse, well-separated supervision through SP3O; experiments show improved policy performance across model scales and evaluation suites.

Problem
Standard PPO critics can fail to capture substantial within-response changes in policy-conditioned state values, limiting their resolution of intermediate states.
Method
SP3O changes only critic supervision, applying the value loss to a small set of well-separated states while leaving the actor objective, rollout procedure, and return targets unchanged.
Results
SP3O mitigates Value Flattening and consistently improves actor performance across Qwen3 model sizes and mathematical and out-of-distribution reasoning benchmarks.
Takeaways & Limitations
Sparse critic supervision better preserves within-response value variation and improves within-response value resolution.
Takeaways & Limitations
The experiments use terminal-only task rewards with zero KL coefficient, so the reported setting is narrower than general PPO training configurations.

Abstract

from arXiv · show

In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat. We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows. Our theoretical and empirical analyses relate Value Flattening to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients. Motivated by these findings, we introduce SParse Proximal Policy Optimization (SP$^3$O), which applies the value loss to only a few well-separated states in each response to mitigate both effects. Experiments on Qwen3-Base show that SP$^3$O with only three states supervised per response can mitigate Value Flattening and consistently improve the learned policy across model sizes and evaluation suites. Together, our results identify Value Flattening as an important yet overlooked failure mode of critic learning in standard PPO and show that a simple sparse supervision strategy can mitigate it.

1 Introduction

The paper identifies Value Flattening as a systematic PPO critic failure: Monte Carlo state values can change sharply within responses while critic predictions remain comparatively flat. It attributes this mismatch to the critic loss and temporal redundancy, motivating sparse supervision with SP3O.

1 Introduction: Value Flattening occurs when Monte Carlo state values change sharply within responses while PPO critic predictions remain comparatively flat.The mismatch appears across training checkpoints and in both correct and incorrect responses, indicating a learned-critic property rather than a trajectory-specific effect.
1 Introduction: As FrozenLake mazes grow, critic predictions become progressively smoother and less accurate, suggesting Value Flattening may worsen with state-space size.The controlled environment varies maze size while keeping the training configuration fixed.
1 Introduction: The critic loss can impose an implicit variance penalty, while temporally correlated states with similar gradients can receive redundant updates.These two factors are presented as mechanisms contributing to Value Flattening.
1 Introduction: SP3O applies critic supervision to a few well-separated states to mitigate both the implicit variance penalty and redundant neighboring updates.The method is designed to preserve within-response value variation while reducing redundant updates.
1 Introduction: SP3O mitigates Value Flattening and improves policy performance across Qwen3 model scales and evaluation suites.The reported scope includes Qwen3-4B-Base and Qwen3-8B-Base on mathematical and out-of-distribution reasoning benchmarks.

2 Related Work

Prior methods obtain intermediate credit through response organization, process supervision, or auxiliary continuations, but they differ in whether they directly estimate policy-conditioned state values. PPO uses a separate critic for token-level advantages, whereas critic-free methods rely on complete-response signals.

2 Related Work: PPO learns a value function and uses predictions at each generation step to construct token-level advantages, unlike critic-free methods such as RLOO, GRPO, and DAPO.Critic-free methods derive advantages from complete responses and offer little direct distinction among states within one response.
2 Related Work: Process supervision and segment- or tree-organized outcomes provide finer feedback but do not directly estimate policy-conditioned state values.The policy-conditioned state value is the expected terminal return when the current policy continues from an intermediate state.
2 Related Work: VinePPO estimates policy-conditioned values with auxiliary continuations, but fine-grained estimation requires additional rollouts from each evaluated state.

3 Preliminaries

The preliminaries define PPO trajectories, returns, critic-based advantages, and the Monte Carlo diagnostic used to assess within-response state values. In the terminal-reward setting, standard PPO repeats one sampled response-level return at every state even though the policy-conditioned value need not be constant.

3 Preliminaries: Given prompt x and response y, PPO forms trajectory states from the prompt and preceding tokens, then computes returns from per-step rewards.The action at step t is the generated token y_t, and the return aggregates discounted future rewards.
3 Preliminaries: PPO uses critic predictions Vϕ(s_t) to approximate the policy-conditioned value and constructs estimated advantages for the clipped actor objective.The probability ratio compares the current and rollout policies, while the critic supplies value estimates for advantage construction.
3 Preliminaries: With terminal-only rewards, zero KL coefficient, and γ = λ = 1, every state along a sampled response receives the same realized terminal return as its critic target.This repeated target is the supervision structure studied in the paper and does not imply that the underlying policy-conditioned value is constant.
3 Preliminaries: The paper estimates a state value diagnostically by averaging realized terminal rewards from independent continuations sampled from the same policy after fixing an intermediate state.In the binary terminal-reward setting, this average is the empirical success rate of the sampled continuations.

4 Understanding and Mitigating Value Flattening

The paper identifies Value Flattening as a critic failure mode that worsens with state-space size and attributes it to variance penalization and redundant updates. SP3O mitigates these effects by supervising fewer, separated states, improving critic value resolution.

4.1 Value Flattening in Critic Learning: Value Flattening appears in LLM reasoning and FrozenLake, where critic predictions remain smoother than sharply changing state-value references and worsen as the state space grows.In FrozenLake, larger mazes produce smoother predictions and weaker agreement with ground truth.
4.2 Why Does Value Flattening Occur?: Shared terminal-return targets make the dense critic MSE penalize variation among predictions across all token positions within a response.The loss decomposes into fitting the mean prediction to the outcome plus an empirical within-response variance penalty.
4.2 Why Does Value Flattening Occur?: Adjacent LLM states overlap heavily, yielding similar representations, aligned gradients, and redundant updates that accumulate under dense supervision.Gradient similarity decreases as token-position distance increases, while alignment and update energy remain high across training.
4.3 SP3O: Sparse Critic Supervision: SP3O selects fewer, well-separated supervised states to restrict the variance penalty and reduce accumulated updates from temporally correlated states.The actor objective, rollout procedure, return targets, and value predictions at unsupervised states remain unchanged.
4.3 SP3O: Sparse Critic Supervision: SP3O reduces response-centered critic profile MSE by 36%, 11%, and 21% at 30%, 60%, and 90% response progress, respectively.Compared with PPO, its profiles more closely follow the direction and magnitude of policy-conditioned MC values.
4.3 SP3O: Sparse Critic Supervision: Optimization diagnostics compare response-level discrimination, within-response variation, hidden-state effective rank, and gradient discrepancies across PPO training stages.These measures are used to examine how sparse supervision changes critic optimization.

M L PPO SP³O

The paper evaluates SP3O against PPO through actor-optimization diagnostics and critic-optimization analyses. These experiments examine update changes, outcome discrimination, prediction variation, representation diversity, and gradient discrepancies.

M L PPO SP³O: Figure 5 evaluates critic optimization using response AUC versus prediction variation, effective-rank distributions, and gradient discrepancies across training stages.The panels measure outcome discrimination, dimensional diversity of value-head inputs, and differences between terminal-return and policy-conditioned MC gradients.
M L PPO SP³O: SP3O has smaller within-iteration actor update changes than PPO over most of training, consistent with smoother actor optimization.The comparison concerns online runs and retains the benefits of a learned critic.

5 Experiments

SP3O consistently improves policy performance over PPO and GRPO across model sizes and evaluation suites while producing more stable actor updates. Ablations show that three well-spaced supervised states, fixed placement, and explicit tail coverage are especially effective.

5.2 Main Results: SP3O consistently outperforms standard PPO and GRPO across both model sizes and mathematical and out-of-distribution evaluation suites.Gains over PPO reach 7.97 percentage points on in-domain mathematics and 7.33 percentage points on out-of-distribution reasoning.
5.2 Main Results: SP3O maintains higher validation accuracy and rollout reward after early training while producing smaller, less variable within-iteration actor updates than PPO.SP3O also generates longer responses in the Qwen3-4B-Base online-learning comparison.
5.3 Sparse-Supervision Ablations: Using three supervised states gives the highest mean training reward, while increasing supervision to 16 or 64 states substantially reduces performance toward dense PPO.The overall trend suggests that a small set of well-spaced anchors limits redundant critic updates while providing sufficient trajectory coverage.
5.3 Sparse-Supervision Ablations: At K = 3, fixed-position anchor schemes outperform random placement, which performs worst, consistent with well-spaced coverage reducing repeated updates on nearby states.The placement comparison is reported in Table 3 and Figure 9.
5.3 Sparse-Supervision Ablations: Adding the final tail anchor improves performance from 44.10 to 45.57 and reduces repetition from 18.33 to 1.12.The result highlights explicit response-tail coverage for policy performance and generation stability.

6 Conclusion

The paper identifies Value Flattening as a systematic critic failure in language-model reasoning and links it to dense supervision and temporal correlation. SP3O mitigates these effects through sparse, well-separated critic supervision and improves actor performance across evaluated settings.

6 Conclusion: Value Flattening occurs when critic predictions fail to capture policy-conditioned state-value changes within individual responses, a pattern also observed in FrozenLake and amplified by larger state spaces.The paper characterizes this as an important yet overlooked problem in critic learning.
6 Conclusion: The analyses relate Value Flattening to an implicit variance penalty from dense token-level supervision and redundant updates from temporally correlated states with similar gradients.These factors motivate changing which states receive the critic loss rather than changing the actor objective or rollout procedure.
6 Conclusion: SP3O applies the critic loss at a few well-separated states and mitigates Value Flattening while improving actor performance across mathematical and out-of-distribution reasoning benchmarks.Experiments cover Qwen3-4B-Base and Qwen3-8B-Base.

A.1 Experimental Details and Diagnostic Metrics

The appendix specifies the experimental configurations and diagnostic procedures used to evaluate sparse critic supervision. It details anchor placement, FrozenLake termination, Monte Carlo value estimation, and metrics for critic variation and gradient behavior.

Experimental Details: The experiments use the hyperparameter configurations summarized in Table 4 for Qwen3-4B-Base and Qwen3-8B-Base.
Experimental Details: Figure 7 uses mean performance with variation bands, and its K = 4 configuration places anchors at relative positions 0.3, 0.5, 0.7, and 0.9.
Experimental Details: FrozenLake rollouts terminate at the goal, a hole, or 8192 steps, with returns of 1 for reaching the goal and 0 otherwise.
Diagnostic Metrics: Policy-conditioned Monte Carlo values average terminal rewards from independently sampled continuations at intermediate response anchors, using adaptive sample counts of 128, 192, or 256.The terminal anchor uses its observed terminal reward directly.
Diagnostic Metrics: Critic diagnostics quantify update energy, response-level discrimination, and within-response prediction flatness across rollout batches and training stages.Lower within-response variation ratios indicate flatter predictions.
Diagnostic Metrics: The appendix also analyzes empirical gradient differences between terminal-return and Monte Carlo targets under the squared-error value loss across evaluated positions and responses.

A.2 Theoretical Analysis of Value Flattening

The analysis decomposes critic error into response-level and within-response components, showing how dense supervision penalizes within-response variation and can affect finite-batch actor updates. It also explains why expected baseline invariance does not guarantee identical practical PPO updates, motivating sparse supervision.

From Critic Error to Actor Updates: Within-response critic errors can change finite-batch actor updates through their interaction with centered log-policy gradients.The critic-gradient error decomposes into response-mean and within-response components, each bounded by critic RMSE times the corresponding RMS log-policy-gradient norm.
Implicit Variance Penalty in Dense Critic Supervision: Dense token-level supervision reuses one terminal return across states, creating an implicit penalty on within-response critic variance and flattening predictions toward the response mean.The shared return provides no direct information about state differences, yet the loss penalizes deviations from the response mean.
Decomposing Critic Error: The exact-value regression reference cancels shared response-level Jacobian components, so correcting within-response error requires state-level Jacobians to differ across states.This connects value-profile correction to the availability of state-varying critic representations and gradients.
Expected-Gradient Invariance and Practical Scope: Expected policy-gradient invariance to action-independent baselines does not imply identical practical PPO updates on finite batches or across multiple clipped-update epochs.Finite-batch centered errors may correlate with centered policy gradients, while value errors can also alter advantage signs, magnitudes, and the active clipping branch.
Implications for Critic Supervision: These findings motivate evaluating within-response value profiles alongside response-level accuracy and applying shared terminal returns to fewer, well-separated states.The sparse-supervision motivation combines the variance-penalty analysis with evidence about temporal correlation and redundant supervision.

A.3 Additional Comparisons of Critic Predictions and MC Values

Additional comparisons examine critic profiles against Monte Carlo value profiles and vary critic-supervision anchor placement. The reported evidence indicates that SP3O can reduce centered profile error, while later-state coverage and conditional tail coverage are important placement considerations.

Additional Prompt-Matched Profiles: For four additional prompt-matched examples, SP3O has lower centered profile MSE than PPO.Figure 8 extends the comparison beyond the two prompt-matched examples shown earlier using prompts from the same checkpoint evaluation panel.
Critic-Supervision Anchor Placement: Later-state coverage is generally beneficial, whereas adding anchors alone does not reliably improve performance.The placement study evaluates how the number and locations of critic-supervision anchors affect Qwen3-4B-Base.
Critic-Supervision Anchor Placement: Conditional tail coverage further improves the matched late-tail comparison and reduces repetitive behavior.The matched comparison is reported in Table 5, whose scope is the late-tail ablation for SP3O.