PaperlayerView exact revision on arXiv

Source-linked AI summary

VASPO: Self-Distillation with Privileged Value Estimation

Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du

arXiv:2610.20516v1cs.CLcs.AIcs.LG

TL;DR

On-policy self-distillation supplies privileged token guidance, but those local preferences do not estimate the expected return of an unguided student continuation. VASPO shares privileged information between a self-teacher and critic at student-generated prefixes, and improves mean accuracy over PPO on mathematical benchmarks while leading evaluated methods on Knights and Knaves.

  • Problem

    Privileged token preferences from on-policy self-distillation do not measure whether the student can successfully complete its unguided continuation.

    Source: §1 · pp. 1–2

  • Method

    VASPO shares privileged information between a self-teacher and critic at student-generated prefixes, using critic-derived PPO advantages and bounded teacher modulation while keeping the actor prompt-only.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    2.59 percentage points on Qwen3 and 1.81 points on OLMo3: VASPO improves mean score over PPO across six mathematical benchmarks and achieves the highest average accuracy among evaluated methods on Knights and Knaves.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Analyses indicate that privileged conditioning trades higher early-prefix error and worse calibration for improved late-prefix value prediction; ablations support full solution derivations and selective, moderate teacher modulation.

    Source: §5 · p. 9

    Flawed intermediate reasoning in privileged reference solutions may introduce noisy supervision, and conditioning both readers on full solutions increases training costs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.

1 Introduction

The introduction argues that token-level teacher preferences do not measure whether a student can complete an unguided continuation, leaving a distinct role for privileged value estimation. VASPO shares privileged information between a self-teacher and critic while the actor remains prompt-only, and improves reported reasoning accuracy over PPO.

1 Introduction

  • Privileged teacher preferences distinguish tokens but are not verifier-grounded measures of the student’s eventual success, and may favor style or yield unreliable targets at some prefixes.The introduction also notes reported risks of suppressed deliberation in reasoning models.Source: §1 · p. 1
  • Prior combinations use verifier outcomes to anchor or route teacher guidance, but the cited methods do not use the same privileged information to train a PPO-style prefix-value estimator for clean student returns.The paper frames this as a separate use of privileged information for estimating continuation return before token credit assignment.Source: §1 · p. 2
  • Critic prediction benefits grow toward the response end, while self-teacher divergence is high initially, falls mid-generation, and rebounds at the end.These positional profiles motivate using privileged information for both continuation-value estimation and token guidance.Source: §1 · p. 2
  • VASPO shares privileged information between a self-teacher and critic: the critic supplies PPO advantages, teacher preferences modulate selected positive advantages, and the actor remains prompt-only.Both readers evaluate privileged information at student-generated prefixes.Source: §1 · p. 2
  • 2.59 percentage points on Qwen3 and 1.81 points on OLMo3: VASPO improves mean score over PPO across six mathematical benchmarks and has the highest average accuracy on Knights and Knaves.The mathematical and logical reasoning evaluations use Qwen3; mathematical reasoning also uses OLMo3.Source: §1 · p. 2

2 Preliminaries

The preliminaries introduce outcome-based policy optimization and privileged token-level self-distillation, then compare how privileged information benefits the critic and self-teacher across response positions.

2.1 Reinforcement Learning with Verifiable Rewards

  • RLVR uses automatically checkable task outcomes as rewards; PPO uses a value function and generalized advantage estimation to assign token-level credit from the terminal reward.The value function provides a prefix-dependent baseline, while the verifier determines the realized outcome.Source: §2.1 · pp. 2–3

2.2 On-Policy Self-Distillation

  • OPSD evaluates the student and a detached privileged self-teacher on the same student-generated prefixes, then minimizes token-wise divergence along the student’s own trajectories.Gradients propagate only through the student distribution; the two readers provide a privileged next-token signal and a verifier-grounded return, respectively.Source: §2.2 · p. 3

2.3 The Positional Divide of Supervision

  • Across response positions, critic prediction benefits grow toward the end, while teacher divergence is high early, falls through the middle, and rebounds at the end.Figure 2 plots solution-induced reduction in critic squared error and KL divergence between the clean actor and privileged self-teacher, with 95% confidence intervals.Source: §2.3 · p. 3, Figure 2 · p. 3
  • These differing profiles motivate using the critic for outcome-grounded advantages and the self-teacher to modulate token updates.The section compares ordinary and solution-conditioned critics on fixed responses alongside the actor as its own solution-conditioned teacher.Source: §2.3 · p. 3

3 Methodology

VASPO retains PPO’s prompt-only actor while sharing privileged information between a critic and a self-teacher at student-generated prefixes. The critic supplies token-level advantages, and bounded teacher weights modulate selected positive advantages.

3.1 Overview of VASPO

  • VASPO keeps the actor prompt-only while privileged readers provide critic values for GAE and teacher weights for advantage modulation.Both readers access the same privileged information at student-generated prefixes.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 Shared Privileged Information

  • Privileged information is fixed before rollout and shared by the teacher and critic, while the student generates responses using only the standard prompt.At each student-generated prefix, both readers receive the same privileged context.Source: §3.2 · p. 4

3.3 Privileged Value Estimation

  • The privileged critic estimates the return of a student continuation without privileged input, using verifier returns to train its prefix-dependent value predictions.After a critic pre-update, refreshed values provide the baseline for GAE advantages; those advantages are detached and held fixed during the actor update.Source: §3.3 · pp. 4–5

3.4 Teacher-Guided PPO Update

  • Clipped teacher–student probability ratios scale selected positive advantages, while negative advantages and unselected tokens retain their original credit.Teacher modulation is gated by positive GAE advantages and a sufficiently large teacher–student gap; the bounded reweighting limits its effect on critic-derived credit.Source: §3.4 · p. 5

4 Experiments

VASPO surpasses the evaluated baselines across mathematical benchmarks on both backbones and achieves the highest average accuracy on Knights and Knaves. Experiments and ablations assess the roles of its critic and self-teacher, while analysis finds their benefits vary across response positions.

4.1 Experimental Setup

  • Experiments use Qwen3-4B and OLMo3-7B for mathematics, and Qwen3-4B for Knights and Knaves, including larger-role out-of-distribution tests.Mathematics uses DeepMath-103K; KK training uses 4–8-role puzzles, with evaluation extending to 9, 10, and 11 roles.Source: §4.1 · pp. 5–6

4.2 Main Results

  • VASPO surpasses all baselines on mathematical benchmarks for both backbones and leads Knights and Knaves overall, though RLSD tops 9-role puzzles.The mathematical comparison covers both Qwen3 and OLMo3; the KK result includes an exception at 9 roles.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

4.3 Ablation Study

  • Default teacher-modulation clip width ϵw = 0.2 yields higher final AIME accuracy than disabling modulation or widening the width to 0.5.Figure 5 tracks AIME 2024 and AIME 2025 avg@16 accuracy over DeepMath training steps; ϵw = 0 disables teacher modulation.Source: Figure 5 · p. 7, §4.3 · p. 8
  • The privileged critic alone improves over PPO, teacher modulation adds gains, and the full method matches or outperforms the critic-only variant across benchmarks.Removing the critic leaves OPSD as the objective, and that variant has lower accuracy.Source: §4.3 · p. 7, Table 3 · p. 8
  • Removing the token-level mask or replacing solution-plus-answer input with answer-only input lowers accuracy across evaluated benchmarks and role groups.The mask result holds across both domains; the input-content change is applied to both readers while the actor keeps the standard prompt.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • Removing critic pre-update or applying global whitening after teacher modulation degrades accuracy across mathematical benchmarks and KK role groups.The authors link pre-update to fitting the critic on current actor trajectories, and warn whitening can invert some fused advantage signs.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Analysis

  • On frozen-actor continuations, the solution-conditioned critic has the highest explained variance and lowest Brier error, while the baseline critic has the lowest ECE.These point-estimate rankings do not establish a strict overall advantage; solution conditioning improves value point estimates but not calibration.Source: §4.4 · p. 8
  • Critic prediction benefits from solution conditioning grow toward response ends, while teacher divergence is high initially, falls mid-response, and rebounds at the final position.The critic’s early-prefix accuracy degrades and middle results are mixed; its late gain is largest for incorrect source responses, but uncertain for long responses.Source: §4.4 · p. 8

5 Conclusion

VASPO combines privileged critic credit with bounded self-teacher modulation while training actors on standard prompts. It achieves top average accuracy across the reported mathematical and logical reasoning benchmarks.

5 Conclusion

  • VASPO combines shared reference-solution information for continuation-value estimation and token guidance with bounded self-teacher modulation.The actor is trained using only standard prompts.Source: §5 · pp. 8–9
  • VASPO achieves top average accuracy across Qwen3 and OLMo3 mathematical and logical reasoning benchmarks.Analyses report a trade-off in which privileged conditioning worsens early-prefix error and calibration but improves late-prefix value prediction; ablations identify full solution derivations and selective, moderate teacher modulation as crucial.Source: §5 · p. 9
  • VASPO's analyses find that privileged conditioning trades higher early-prefix error and worse calibration for improved late-prefix value prediction.Ablations also confirm the importance of full solution derivations and selective, moderate teacher modulation.Source: §5 · p. 9

Limitations

The supplied material covers limitations, related training paradigms, critic pretraining, and datasets and models. Its scope includes evidence timing, supervision quality, computational cost, and single-turn generation.

Limitations

  • The current protocol cannot use feedback received at response completion as privileged context for earlier prefixes of that same trajectory.VASPO is currently designed for single-turn generation; applying it to multi-turn dialogue with evolving history and evidence remains future work.Source: Limitations · p. 9

A LLM Usage

  • LLMs were used only to polish grammar and improve manuscript clarity; the authors conducted the research ideas, experiments, and analyses.Source: A LLM Usage · p. 11

B.1 RLVR and Value-Based Credit Assignment

  • GRPO shares response-level advantages across tokens, whereas PPO uses a learned value function for prefix-dependent advantages; VASPO conditions its continuation-return critic on training evidence (Schulman et al., 2017).The paper distinguishes this learned critic from a distillation-objective variance-reduction baseline and from other value-estimation approaches.Source: §B.1 · p. 11

B.2 On-Policy Distillation and Self-Teaching

  • On-policy distillation gives teacher feedback on student-generated prefixes, while privileged context can produce unreliable targets and must respect evidence timing.The shared evidence bank is fixed before rollout; feedback from the completed response is not an admissible critic input for that same trajectory.Source: §B.2 · pp. 11–12

B.3 Combining Distillation with Verifiable Rewards

  • Distillation and outcome optimization can be combined through additive objectives, advantage weighting, or response routing (Xiao et al., 2026; Liu et al., 2026b; Ding et al., 2026; Li et al., 2026).These designs vary in signal granularity, scale, and timing; reward-gated distillation uses verifier agreement as a gate rather than an additive reward term (Akhondzadeh et al., 2026).Source: §B.3 · p. 12
  • Unlike group-relative methods that modulate shared outcome advantages, VASPO uses privileged information in a learned PPO critic to produce prefix-varying advantages before teacher modulation (Yang et al., 2026a; Pan et al., 2026; Wu et al., 2026b).Its bounded fusion preserves the sign of the raw privileged-critic advantage.Source: §B.3 · p. 12

C Critic Pretraining

  • Critics are pretrained for two epochs on token-level predictions of terminal verifier returns; privileged critics additionally read the fixed dataset solution and answer.At the final recorded checkpoint, answer- and solution-conditioned critics have lower loss and higher explained variance than the clean critic; Figure 6 shows the recorded dynamics.Source: C Critic Pretraining · pp. 12–13, Figure 6 · p. 13

D.1 Datasets and Models

  • The study uses DeepMath-103K for mathematical reasoning and the complete 5,000-problem 4–8-role training split for Knights and Knaves, with dataset solutions and answers as privileged information.Qwen3 and OLMo3 are used for mathematics, while logical reasoning uses Qwen3; all compared methods share task-specific training data within each backbone.Source: §D.1 · p. 13

D.2 Evaluation Details

Evaluation uses task-specific verifiers, repeated sampling, and equally weighted benchmark averages. The comparisons include the initial checkpoint, RL baselines, and reader-component variants.

D.2 Evaluation Details

  • Mathematics training rewards are binary answer-verification outcomes, while Knights and Knaves responses are correct only when every role assignment matches; math test answers receive a second verification pass when initially judged incorrect.The table lists dataset and model assets, licenses, and question counts.Source: §D.2 · p. 13, Table 4 · p. 14
  • Math Average weights six benchmarks equally, and KK Average weights the in-distribution and three out-of-distribution scores equally; reported accuracy estimates single-sample correctness from repeated draws.All methods use the same decoding and verification procedure within each backbone, and test scores are excluded from checkpoint and hyperparameter selection.Source: §D.2 · pp. 13–14

D.3 Baselines

  • The baselines include the initial checkpoint, DAPO, PPO, OPSD, and RLSD; reader ablations remove privileged critic updates, teacher modulation, or both.PPO, Critic-only, and VASPO share critic architecture, initialization, targets, and optimization settings, while each generates its own on-policy responses.Source: §D.3 · p. 14

D.4 Training Details

Training uses PPO with clipped teacher-ratio reweighting and selected-token modulation. The implementation specifies optimization, reader-context, and rollout budgets.

D.4 Training Details

  • VASPO uses AdamW with batch size 128, PPO mini-batches of 32, actor and critic learning rates of 1×10−6 and 5×10−6, and PPO clipping ϵclip = 0.2.It uses γ = λGAE = 1 and no advantage normalization.Source: §D.4 · pp. 14–15
  • Teacher modulation uses full clipped probability-ratio reweighting with ϵw = 0.2 and token-selection parameters τ = δ = 0.02.The self-teacher uses EMA decay 0.95 after each actor optimizer step.Source: §D.4 · p. 15
  • Rollouts sample eight responses per prompt with a maximum response length of 8,192 tokens; privileged reader inputs must fit their context window.Training uses 50-step linear warmup, one PPO epoch per batch, and 200 rollout-batch updates; oversized or invalid privileged inputs are handled before the training manifest is frozen.Source: §D.4 · p. 15

D.5 Training-time Analysis

VASPO’s training-time analysis reports component timers alongside the end-to-end step time, while its prompts separate the prompt-only actor from privileged readers that score the actor’s sampled tokens.

D.5 Training-time Analysis

  • 412.28 s is the recorded end-to-end step time; student rollout is the largest component timer at 191.97 s.Reward verification takes 2.68 s, student log-probability computation 20.72 s, and self-teacher scoring 22.29 s per step.Source: §D.5 · p. 15, Figure 7 · p. 16
  • 492.42 s summed component means exceed the separately recorded 412.28 s step time, so the timers are not an additive partition.Advantage computation and teacher fusion are timed jointly at 91.37 s, not as isolated fusion overhead.Source: §D.5 · p. 15

D.6 Prompts

  • The actor receives the problem without privileged information, while the self-teacher and critic use privileged inputs to score the actor’s sampled tokens.Braced fields denote per-example substitutions; the critic templates append the ground-truth answer, and the solution-conditioned version also appends the reference solution.Source: §D.6 · pp. 15–17

E Additional Method Details

The critic is trained on student-rollout returns, with a pre-update value snapshot serving as the fixed clipping center. After the pre-update, values and GAE advantages are recomputed for token selection and teacher fusion.

E Additional Method Details

  • The training driver first evaluates values and constructs return targets, then uses those initial predictions as fixed clipping centers during the critic pre-update.It re-evaluates values and recomputes actor advantages on the same rollout batch; only the recomputed advantages enter token selection and teacher fusion.Source: E Additional Method Details · p. 17
  • With terminal-only rewards and γ = λ_GAE = 1, the return target is G_i at every valid token, and the refreshed advantage is G_i − v_z,+.The critic regresses to student-rollout returns; privileged information remains fixed throughout the response and the critic excludes the current token and subsequent continuation.Source: E Additional Method Details · p. 17
  • When z = f(x) is deterministic, privileged conditioning changes the critic’s representation without adding information beyond the prompt.The stated question is whether this explicit representation helps a finite critic predict student returns.Source: E Additional Method Details · p. 17

F Complete Implementation Details

Implementation details specify the per-batch critic and teacher-fusion schedule and optional loss variants. Additional experiments examine response-group signal profiles and critic predictions on independently sampled continuations.

F Complete Implementation Details

  • The per-batch critic schedule stores initial values as clipping centers, performs the clipped value update, then recomputes values and GAE before teacher fusion.The ordinary later critic-update call is skipped, so this changes update timing without adding a second update phase.Source: F Complete Implementation Details · p. 17
  • Teacher probabilities are cached before each actor update, and selected positive advantages receive the clipped ratio while negative, zero, and below-threshold advantages retain their GAE values.The default uses raw GAE advantages without whitening; an optional post-fusion whitening uses statistics over all valid tokens and can change advantage signs.Source: F Complete Implementation Details · pp. 17–18
  • A supported variant recomputes the teacher-gap selection and fusion at each actor forward pass using current student probabilities, while the default keeps batch-update fusion fixed.The recomputed fusion remains detached, teacher probabilities stay cached, and PPO’s update ratio still compares the current actor with the behavior policy.Source: F Complete Implementation Details · pp. 17–18

G.1 Signal Profiles on Fixed Responses

  • On fixed responses, critic errors decrease throughout incorrect responses but improve for correct responses only near the end; teacher KL falls toward mid-generation and rebounds at the end in both groups.The pooled trend hides outcome-dependent critic behavior, and teacher KL measures disagreement rather than useful guidance.Source: §G.1 · p. 18, Figure 8 · p. 19

G.2 Prediction Quality under Independent Continuations

  • Solution conditioning has the highest EV and lowest Brier point estimates, whereas the clean critic has the lowest ECE; these rankings alone do not establish a paired overall advantage.Full-response rescoring confirms every evaluation label (Appendix G.4).Source: §G.2 · p. 19

G.3 Where the Critic Improvement Occurs

The diagnostic compares clean and solution-conditioned critic Brier improvement at sampled response positions and across source-response groups. The reported gains vary by correctness and length, with important uncertainty around late gains.

G.3 Where the Critic Improvement Occurs

  • The diagnostic compares clean-minus-solution Brier improvement at three sampled positions, not continuous trajectory regions.Final-answer and terminal-region exclusions were unavailable because those regions were not marked.Source: §G.3 · p. 19
  • The positional contrast is supported in medium and long responses but uncertain in short ones; the long-response late-gain interval includes zero.A positive positional contrast can reflect the pronounced early cost rather than a significant late gain in every length stratum.Source: §G.3 · p. 19, Figure 10 · p. 21

G.4 Diagnostic Protocol

  • The continuation study evaluates 128 problems, 384 prefixes, and 3,072 independent continuations; full-response rescoring changed none of the evaluation labels.Uncertainty intervals are pointwise whole-problem bootstrap intervals without multiplicity correction, and continuation comparisons share sampled indices.Source: Figure 9 · p. 20, §G.4 · p. 20

G.3 Where the Critic Improvement Occurs

  • The pooled comparison shows an early cost and late benefit, with an uncertain middle-position effect; the late gain is much larger for incorrect source responses.The correct-source late interval includes zero, although both correctness groups have positive positional contrasts.Source: §G.3 · p. 19, Figure 10 · p. 21

Related papers

Loading 2610.20516v1…