How to Train a Critic Stably and Efficiently

Penghui Qi; Xiangxin Zhou; Wee Sun Lee

Source-linked AI summary

How to Train a Critic Stably and Efficiently

Penghui Qi, Xiangxin Zhou, Wee Sun Lee

arXiv:2608.23566v1Aug 24, 2026cs.LGcs.AIcs.CL

TL;DR

Critic-based reinforcement learning for language models can be unstable, motivating a reliable single-rollout alternative to group-based estimation. BPCO combines several critic-optimization choices and consistently improves standard critic-based training, matching or exceeding group-based optimization while using one response per prompt.

Problem
Critic-based language-model reinforcement learning remains fragile because common clipping, value targets, GAE, value-range, and advantage-normalization choices can destabilize training.
Method
BPCO combines DPPO, reward-range-bounded value predictions, Monte Carlo critic targets, unnormalized advantages, length-adaptive GAE, and optional training-only privileged critic information.
Results
BPCO consistently improves the standard critic-based recipe and matches or exceeds group-based optimization while sampling one response per prompt.
Takeaways & Limitations
A coherently designed critic can provide a stable and efficient single-rollout alternative to group-based advantage estimation.
Takeaways & Limitations
Evidence is limited to mathematical and rubric rewards; BPCO assumes a known reward range, and privileged variants require evaluator information.

Abstract

from arXiv · show

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop \textbf{Best-Practice Critic Optimization (BPCO)}, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic

1 INTRODUCTION

The introduction motivates BPCO as a stable, single-rollout critic alternative to group-relative estimation and combines several design choices to align critic training with policy optimization. It evaluates this recipe across mathematical reasoning, mixture-of-experts models, and rubric-based rewards.

Group-based methods such as GRPO estimate token credit by sampling several responses per prompt and comparing their rewards, whereas a critic can derive token-level advantages from one rollout.
Critic-based LLM training is fragile because PPO clipping treats tokens unevenly, bootstrapped targets inherit critic error, and fixed GAE may be unsuitable across response lengths.
A discarded training-time critic can receive reward-defining information hidden from the policy, including reference answers, official solutions, or prompt-specific rubrics.
BPCO combines DPPO, reward-range-bounded value predictions, observed-outcome targets, raw policy advantages, length-adaptive GAE, and optional privileged critic inputs.
Controlled experiments evaluate BPCO on a 40.3K-problem mathematical dataset, two 30B-A3B mixture-of-experts models, and a rubric-reward task.

2 BACKGROUND

This section defines outcome-reward language-model generation as token-level reinforcement learning and introduces PPO, DPPO, critic-based advantage estimation, and group-based alternatives. It highlights how DPPO constrains absolute token-probability shifts, while GAE trades variance against critic sensitivity and group methods avoid critics through multiple sampled responses.

Outcome-reward formulation: A language model generates tokens autoregressively from prefix states, with a scalar reward assigned only after the response is complete.Intermediate rewards are zero.
PPO and DPPO: PPO uses a clipped surrogate objective whose clipping boundary limits incentive to move sampled-token probability ratios farther in the favored direction.This forms a trust region intended to stabilize training.
PPO and DPPO: DPPO replaces a common ratio threshold with a common absolute-probability threshold, constraining each sampled token’s shift as |πθ(yt | st) − µ(yt | st)| ≤ ϵ.This avoids clipping small absolute changes to low-probability tokens while allowing much larger absolute changes to high-probability tokens.
Critic-based estimation: Critic-based methods estimate expected prefix returns, and GAE forms advantages from temporal-difference residuals using exponentially weighted bootstrapping.Smaller λ can reduce variance but increases sensitivity to critic error; γ = 1 and λ = 1 yields R(x, y) − Vϕold(st).
Group-based alternatives: Group-based methods avoid a critic by sampling multiple responses for each prompt and deriving token advantages from the group’s reward statistics.Dr. GRPO removes standard-deviation normalization, which can otherwise reweight prompts by within-group reward variance.

3 BUILDING BPCO: A CONTROLLED STUDY

The controlled sanity study identifies critic and policy-optimization choices that determine stability: DPPO, bounded values, unbiased critic targets, unnormalized advantages, privileged critic inputs, and length-adaptive GAE each address distinct failure modes. Together, these changes make small-data training more stable while exposing trade-offs between optimization speed, validation performance, and overfitting.

Optimization objective and GAE: PPO training reward collapses, whereas DPPO remains stable with λ = 1; setting λ = 0.99 makes DPPO unstable by exposing sensitivity to critic error.With λ < 1, bootstrapped critic predictions bias policy advantages unless the critic is exact.
Bounded critic predictions: Bounding value predictions to the reward range removes extreme linear-head outputs and allows binary-reward training reward to approach one.The bounded parameterization keeps predictions within [0, 1] in the sanity test.
Critic targets: Decoupling policy and critic estimators—λπ = 0.99 and λV = 1—removes bootstrapping from critic targets, improving reward stability and convergence speed.Explained variance against the bootstrapped target can approach one while prediction remains inaccurate because the target contains the old critic.
Policy advantages: Removing batch advantage normalization keeps advantages small and stable, preserves comparable reward, and improves validation performance by preventing noise amplification near optimality.Normalization can rescale small estimation noise into a large update when advantage variance is small.
Privileged information and length adaptation: Privileged critic inputs accelerate optimization but increase overfitting risk, while LA-GAE with α = 0.4 provides the best trade-off between optimization speed and validation decline.Fixed λπ = 0.99 fits fastest but declines on validation; λπ = 1 avoids that decline more slowly.

4 BROADER EVALUATION

Across larger datasets, mixture-of-experts models, and rubric-based rewards, BPCO consistently improves critic-based training and matches or exceeds group-based methods with single-rollout advantages.

Method setup: BPCO differs from the critic baseline only by bounding value predictions and removing batch-wise advantage normalization.Privileged-input variants provide the critic with ground-truth answers or official solutions while leaving policy inputs unchanged.
Larger dataset: On DeepScaleR, BPCO consistently outperforms both group-based and critic-based baselines in training and validation performance.It also achieves consistently higher explained variance during training, indicating a better-learned critic at larger dataset scale.
Larger MoE models: On Qwen3-30B-A3B and Qwen3-30B-A3B-Base, BPCO substantially improves AIME 2025 accuracy over the critic baseline.The critic baseline on Qwen3-30B-A3B stopped improving after 100 training steps because of unstable optimization.
Larger MoE models: Against the group baseline, BPCO performs better on Qwen3-30B-A3B and comparably on Qwen3-30B-A3B-Base without group sampling.These results establish BPCO as an alternative to group-based advantage estimation.
Rubric-based rewards: With rubric-based rewards, both BPCO variants improve faster than group and critic baselines, although the group baseline eventually reaches similar performance.Privileged rubric information increases explained variance but provides no performance benefit in this relatively trivial task.

5 CONCLUSION

BPCO combines several critic-training stabilizations and optional training-only privileged information to improve critic-based RL for LLMs. Across tested settings, it consistently improves the standard recipe and matches or exceeds group-based optimization, while its assumptions and computational costs remain limitations.

Method: BPCO combines DPPO, reward-range-bounded value predictions, unbiased Monte Carlo critic targets, unnormalized policy advantages, and length-adaptive GAE.The training-only critic may also receive privileged information such as a ground-truth answer, solution, or grading rubric without changing the policy’s inputs.
Results: BPCO consistently improves the standard critic-based recipe across controlled tests, dataset scales, 1.5B and 30B-A3B models, and rubric-based rewards.It also matches or exceeds group-based optimization while using one response per prompt.
Limitations: Privileged information improves critic learning when it provides useful reward context, but gains are task dependent and can be offset by overfitting in small-data regimes.BPCO is evaluated only with mathematical and rubric rewards, assumes a known reward range, and adds computation and memory from critic training.