PaperlayerView exact revision on arXiv

Source-linked AI summary

Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR

Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Zeyuan Liu, Qingyang Zhang, Saiyong Yang, Yunfang Wu

arXiv:2609.37825v1cs.LGcs.AI

TL;DR

Sparse terminal rewards make reliable values for intermediate reasoning states difficult, yet those values shape token-level policy advantages. πPPO conditions the critic on verified same-prompt rollouts while leaving the actor’s conditioning unchanged, and improves critic quality and end-to-end accuracy across mathematical reasoning benchmarks.

  • Problem

    With sparse terminal rewards, critics must estimate uncertain intermediate-state values while assessing solution correctness and predicting future policy behavior.

    Source: §1 · p. 1

  • Method

    πPPO reuses verified same-prompt rollouts as leave-target-out critic context while preserving the actor’s original conditioning and policy objective.

    Source: §1 · p. 2

  • Results

    πPPO improves critic quality over state-only actor–critic counterparts and achieves the strongest end-to-end accuracy across evaluated mathematical reasoning benchmarks, outperforming representative actor–critic and critic-free baselines.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings support using verified same-prompt rollout evidence for value estimation without changing the actor’s conditioning or requiring additional rollout generation.

    Source: §1 · p. 2

    The formal value-estimation proposition assumes binary terminal returns and finite squared error.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $\pi$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $\pi$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $\pi$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.

1 INTRODUCTION

Sparse terminal rewards make intermediate value estimation difficult, while errors can distort credit assignment and destabilize policy updates. πPPO uses verified same-prompt rollouts as critic-side references and improves value estimation and policy performance on mathematical reasoning tasks.

1 INTRODUCTION

  • Accurate value estimation is difficult because critics must understand correct solutions and predict future policy behavior, while sparse terminal rewards leave intermediate targets weakly informative and uncertain.Inaccurate estimates can distort token-level advantages and misguide updates, undermining training stability and final performance.Source: §1 · p. 1
  • Existing approaches strengthen critics or enrich their targets, but still place both demands on the critic using only information from its prescribed vantage point.The paper questions whether that premise must hold.Source: §1 · p. 1
  • πPPO reuses verified trajectories as online references for state-level value prediction, with each rollout also serving as a potential reference for evaluating sibling partial responses.The leave-target-out references evolve with the policy and require no additional generation beyond policy-update rollout groups.Source: §1 · pp. 1–2
  • πPPO improves value-estimation quality and policy performance on mathematical reasoning tasks, including against representative actor–critic and critic-free baselines, while remaining effective with substantially smaller asymmetric critics.The introduction reports consistent improvements across two model backbones.Source: §1 · p. 2

2 PRELIMINARY

The preliminaries define the policy and verifier reward, then describe PPO’s value-based token advantages and GRPO’s group-relative alternative.

2 PRELIMINARY

  • A query-response pair receives a binary rule-based reward indicating whether the response aligns with the query’s ground truth.The response likelihood is expressed under policy πθ given query x.Source: §2 · p. 2
  • PPO updates an actor using a clipped policy surrogate, with token-level probability ratios and a learned critic’s advantages informing the update.The critic supports dense token-level credit assignment even when the environment supplies only a terminal reward.Source: §2 · p. 2
  • GRPO bypasses the value model by computing each response’s relative advantage within a same-query group, with all tokens in a response sharing that advantage.Its clipped surrogate is averaged over the group.Source: §2 · p. 2
  • Privileged information augments a standard input with task-relevant information available during training but absent from the standard inference input.For LLMs, it can be represented as token sequences and may encode hints, verified answers, or reference rationales.Source: §2 · p. 2

3 METHODOLOGY

The methodology treats value estimation as an information problem and introduces πPPO, which conditions the critic on verified same-prompt evidence while preserving the actor’s original policy objective and conditioning.

3.1 INTERPRETING A STATE: THE “MISSING” CONTEXT IN VALUE ESTIMATION

  • State-only value prediction leaves residual return uncertainty that additional evidence can reduce when it explains variation beyond the target state.Under binary verifiable rewards, the paper relates prediction error to the critic’s contrast between successful and failed trajectories; outperforming the optimal constant predictor requires positive success–failure contrast.Source: §3.1 · p. 3
  • πPPO constructs a leave-target-out context from verified same-prompt rollouts, allowing the critic to assess a target prefix against successful and failed attempts.If only one outcome appears among the remaining rollouts, two siblings of that outcome are used; the ground-truth answer is additionally supplied in the all-negative case.Source: §3.1 · p. 4, Figure 1 · p. 4
  • The context-conditioned critic values enter standard TD residuals and GAE to produce advantages, which replace the PPO advantages while the actor retains its original conditioning.The privileged context is available to the critic, not the actor.Source: §3.1 · p. 4
  • Because the privileged value estimate is independent of the current action, it remains a valid policy-gradient baseline without adding bias to the expected actor gradient or changing the policy objective.The actor uses the same conditioning during training and inference.Source: §3.1 · p. 5

3 METHODOLOGY

  • The rollout groups collected for PPO supply the critic’s contrastive evidence, which is refreshed as the policy evolves without additional rollout generation.The methodology uses information available within training, without external expert knowledge.Source: §3 · p. 3, §3.1 · p. 5

4 EXPERIMENTS

Experiments evaluate πPPO against actor–critic and value-free baselines on mathematical reasoning benchmarks, finding stronger policy performance and value estimates, including with smaller critics.

4.1 EXPERIMENTAL SETTINGS

  • The experiments use Qwen3-4B and Qwen3-8B, compare PPO, VAPO, GRPO, and DAPO, and report Pass@1 on five mathematical reasoning benchmarks.Critic assessment focuses on explained variance and value-function loss.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • 50.3% on Qwen3-4B and 51.6% on Qwen3-8B overall, exceeding DAPO by 4.1 and 2.3 percentage points, respectively; πPPO leads all five benchmarks at both scales.Overall is the mean across the five benchmarks.Source: Table 1 · p. 6, §4.2 · p. 6
  • πPPO achieves higher explained variance and lower value-function loss than PPO and VAPO across both backbones, with improvements evident from early training.Source: Figure 2 · p. 6

4.3 πPPO ADMITS SMALLER ASYMMETRIC CRITICS

  • With asymmetric critics, πPPO reaches 47.8% and 49.9% overall accuracy, exceeding the strongest baseline on both backbones and actor–critic baselines with actor-sized critics.Source: Figure 3 · p. 7, §4.3 · p. 7
  • Reducing critics to 0.6B and 1.7B cuts parameters by approximately 85% and 79%, while πPPO EV falls by only 0.037 and 0.027.The resulting EVs, 0.655 and 0.632, exceed PPO’s 0.446 and 0.564 with actor-sized critics; accuracy decreases by 2.5 and 1.7 percentage points versus symmetric configurations.Source: §4.3 · p. 7

5 ANALYSIS

The analysis compares PPO and πPPO critics against Monte Carlo references, finding improvements in value accuracy, state ranking, and tracking local value changes. Ablations indicate that correctness labels and opposite-outcome pairing are important to the gains.

5.1 CHARACTERIZING PRIVILEGED VALUE ESTIMATION

  • The accuracy analysis uses 256 Monte Carlo samples per prefix state as an empirical reference because ground-truth state values are not directly observable.The compared critics provide advantage estimates for an otherwise unchanged PPO update.Source: §5.1 · p. 7
  • At the final evaluated position, πPPO’s MSE is approximately 0.05 versus 0.15 for PPO, and its error remains comparatively stable as generation progresses.Across evaluated prefix positions, πPPO also has lower MSE and predictions that more closely follow the MC reference.Source: §5.1 · p. 7, Figure 4 · p. 8
  • At the final position, πPPO reaches approximately 94% pairwise accuracy versus 78% for PPO, and approximately 50% versus 15% chance-normalized top-1 accuracy.πPPO’s pairwise-ranking advantage widens as generation progresses; its top-1 accuracy is higher at most evaluated positions.Source: Figure 5 · p. 8, §5.1 · p. 8
  • πPPO’s predicted value changes correlate more strongly with MC-estimated changes overall and within Analyze and Implement, where PPO’s correlations are near zero and πPPO’s are approximately 0.2.Analyze and Implement together account for approximately 90% of evaluated nodes.Source: Figure 6 · p. 8, §5.1 · p. 8

5.2 THE ROLE OF CONTRASTIVE EVIDENCE

  • Removing correctness labels or opposite-outcome pairing erases most gains over baselines; the offline ground-truth-only variant trails full πPPO by 4.7 and 3.1 points on Qwen3-4B and Qwen3-8B, respectively.The results link gains to the contrastive organization of online trajectory evidence.Source: §5.2 · p. 9

6 RELATED WORK

The related work frames value estimation as relevant to both actor–critic and critic-free policy-gradient methods. It distinguishes πPPO’s critic-side privileged information from approaches that alter policy inputs or use model-induced supervision.

6 RELATED WORK

  • Table 2 compares effects of correctness labels, reference polarity, and ground-truth-only context on average accuracy across five benchmarks.Source: Table 2 · p. 9
  • Actor–critic and critic-free post-training methods can both be examined through value estimation, with reliability central to their relative effectiveness in the bias–variance trade-off.πPPO introduces training-time privileged information to improve critic value estimates for policy optimization.Source: §6 · p. 9
  • Hint-based RL adds external expert information to policy inputs or response prefixes, whereas πPPO places privileged information in critic-side value estimation while preserving actor conditioning and the environmental reward objective.The passage notes hint-based RL is off-policy and causes a distributional mismatch between training and inference.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents πPPO as a critic-context extension that outperforms state-only counterparts in critic quality and end-to-end accuracy across mathematical reasoning benchmarks. Its results also hold with a smaller critic, while retaining the actor’s conditioning and the environment-defined reward objective.

7 CONCLUSION

  • Across mathematical reasoning benchmarks with Qwen3-4B and Qwen3-8B, πPPO achieves higher explained variance and lower value loss than its state-only actor–critic counterparts.Source: §7 · p. 9
  • πPPO delivers the strongest end-to-end accuracy among evaluated actor–critic and critic-free baselines, including when paired with a smaller critic model.Source: §7 · p. 9
  • The method conditions the critic on verified same-prompt siblings while preserving the actor’s original conditioning and environment-defined reward objective, without additional generation.Source: §7 · p. 9

A THEORETICAL PROOF

The appendix proof develops a risk comparison for value estimation under privileged conditioning and a necessary contrast condition under binary outcomes. It then lists implementation and training settings for the experiments.

A.1 VALUE ESTIMATION UNDER PRIVILEGED CONDITIONING

  • The privileged-conditioning proof takes expectations under a fixed policy-induced joint distribution of state, context, and return, assuming binary returns are square-integrable and critics have finite squared error.Source: §A.1 · p. 13
  • The proof compares optimal contextual and state-only risks by assuming each state-only predictor can be represented by a contextual critic that ignores the added context.The derivation uses the conditional-expectation decomposition and takes infima over shared critic parameters.Source: §A.1 · p. 13

A.2 SUCCESS–FAILURE CONTRAST IN VALUE ESTIMATION

  • Under binary returns, any critic that improves on optimal constant-prediction risk must have a positive contrast between its predictions on successful and failed trajectories.Source: §A.2 · pp. 13–14

B.1 TRAINING DETAILS

  • Training uses 250 steps, batch size 256, and 50 frozen-actor value-pretraining steps; policy rollouts use eight responses per prompt, and πPPO references are selected leave-target-out.The appendix also specifies ablation context variants, including removing labels, using same-polarity references, or providing only the verified ground-truth answer.Source: §B.1 · p. 14

B.2 EVALUATION AND ANALYSIS DETAILS

Evaluation uses fixed sampling settings and sampled-response accuracy to estimate pass@1, while critic diagnostics test value estimates and rankings at selected reasoning prefixes. The prompt example illustrates how privileged references expose a process-level contrast between otherwise matching attempts.

B.2 EVALUATION AND ANALYSIS DETAILS

  • Evaluation samples 16 responses per problem on four AIME/HMMT benchmarks and eight on BeyondAIME, averaging correctness to estimate pass@1.Decoding uses temperature 0.6, top-p 0.95, top-k 20, and a 32768-token maximum; Overall is the mean across five benchmarks.Source: §B.2 · p. 14
  • Critic diagnostics use step-250 Qwen3-8B checkpoints, 128 prompts per method, and 256 Monte Carlo continuations at seven semantic prefixes.Pairwise ranking uses 20 prefix positions; top-1 identification keeps candidate sets whose MC-value range exceeds twice the estimated uncertainty.Source: §B.2 · p. 14

C PROMPT TEMPLATE FOR πPPO

  • The πPPO prompt example contrasts two reference attempts that match through step (2) but differ in their expansion of (x − y)^2 at step (3).The labeled block appended to the user turn supplies the privileged critic with the additional reference information.Source: Table 3 · p. 16
  • Table 4 defines Avg@k as mean accuracy over k random generations and Overall as the mean across two benchmarks; Table 5 averages five benchmark scores for Overall.Table 4 identifies smaller-critic variants as Asym, using 0.6B critics at 4B and 1.7B critics at 8B.Source: Table 4 · p. 17, Table 5 · p. 17

D.1 EVALUATION ON OOD BENCHMARKS

On out-of-domain science, humanities, and professional-knowledge benchmarks, πPPO policies achieve the best reported Overall scores at both model sizes. The asymmetric 8B variant attains the highest score while using a 1.7B critic.

D.1 EVALUATION ON OOD BENCHMARKS

  • 60.5 is the best 4B Overall score, while the asymmetric 8B variant scores 65.2 Overall with a 1.7B critic.πPPO reaches 64.7 Overall at 8B; Table 4 reports accuracy (%) and defines Overall as the mean across two benchmarks.Source: §D.1 · p. 17
  • All RL-trained policies outperform their initial counterparts on the evaluated out-of-domain benchmarks.The benchmarks are GPQA-Diamond and MMLU-Pro, used to examine out-of-distribution generalization.Source: §D.1 · p. 17

D.2 ADDITIONAL COMPARISON WITH SELF-DISTILLATION METHODS

Under the tested experimental setting, OPSD and RLCSD oscillate and collapse without effective gains, while RLSD achieves reasonable downstream performance. These methods are compared as approaches that leverage privileged information for supervision or token-level reward shaping.

D.2 ADDITIONAL COMPARISON WITH SELF-DISTILLATION METHODS

  • The comparison treats OPSD, RLSD, and RLCSD as methods leveraging privileged information to provide additional supervision or shape token-level rewards.The methods were adapted from their original implementations to the main-experiment setup, with multiple configurations explored around a reference of 256 batch size, 10^-6 actor learning rate, and 250 policy updates.Source: §D.2 · pp. 17–18
  • OPSD and RLCSD consistently show training oscillations and collapse across tested configurations, with no effective gain over the initial model.Only RLSD among the three baselines achieves reasonable downstream performance under the setting following their original setups.Source: §D.2 · p. 18

Related papers

Loading 2609.37825v1…