PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Yanjiang Liu, Jie Lou, Xinyan Guan, Yuqiu Ji, Hongyu Lin, Ben He, Xianpei Han, Le Sun, Xing Yu, Yaojie Lu

arXiv:2605.30833v1cs.CLcs.AI

TL;DR

On-policy distillation uses teacher feedback on student-generated reasoning, but that feedback degrades as prefixes lengthen. Lookahead Group Reward evaluates candidate tokens by the teacher confidence they induce at the next step, using group normalization and entropy-triggered tree attention. Across six benchmarks, it improves mean@8 over OPD for a 7B student, with larger gains on longer reasoning tasks.

  • Problem

    On-policy distillation treats teacher supervision as unaffected by student-generated prefixes, although teacher confidence and its corrective signal weaken as those prefixes lengthen.

    Source: §1 · pp. 1–2

  • Method

    Lookahead Group Reward assigns group-normalized rewards to top-K student tokens based on the teacher confidence they induce at the next position, with entropy-triggered tree attention for efficiency.

    Source: §1 · p. 2

  • Results

    For the 7B student, LGR improves mean@8 by 2.57% over OPD on average across benchmarks, with gains growing on longer reasoning tasks.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    LGR actively steers students toward trajectories where the teacher maintains higher supervision fidelity rather than passively avoiding weak-supervision regions.

    Source: §6 · p. 9

    LGR requires white-box access to teacher logits, and its confidence ranking may weaken on extremely out-of-distribution student trajectories or with poorly calibrated teachers.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens.

1 Introduction

On-policy distillation trains students on their own reasoning trajectories with teacher feedback, but that feedback degrades as student prefixes lengthen. The paper introduces Lookahead Group Reward to preserve useful teacher supervision and reports larger gains on longer reasoning tasks.

1 Introduction

  • As student prefixes lengthen, teacher supervision quality degrades, a failure mode the paper names Supervision Fidelity Decay (SFD).The introduction frames this as a challenge to treating the teacher as a static oracle.Source: §1 · p. 1
  • Teacher completion accuracy and peak next-token probability both decline with longer student prefixes, while teacher-generated prefixes retain higher fidelity.The handoff confidence jump indicates token choices affect next-step teacher confidence.Source: §1 · p. 2, Figure 2 · p. 2
  • As teacher confidence falls, reverse-KL’s corrective signal weakens, allowing student drift to compound across long reasoning chains.The paper links lower teacher confidence to less discriminative log-probabilities and a student-only signal.Source: §1 · p. 2
  • Lookahead Group Reward scores top-K student token candidates by subsequent teacher confidence, using group-normalized rewards and entropy-triggered tree attention.The approach uses higher next-position confidence as a source of more discriminative future supervision.Source: §1 · p. 2
  • LGR’s gains grow with reasoning length and are pronounced on long-reasoning tasks compared with OPD and alternative distillation methods.The supplied introduction reports this trend without giving a numerical comparison.Source: §1 · p. 2

2 Supervision Fidelity Decays Along Student Trajectories

This section traces how student-generated trajectories weaken teacher supervision and how that weakening affects reverse-KL learning. It formalizes supervision fidelity, analyzes signal collapse and compounding drift, and describes a shrinking useful-learning horizon.

2.1 On-Policy Reverse-KL as Policy Gradient

  • On-policy reverse-KL distillation has a policy-gradient form: each token’s reward compares its teacher and student log-probabilities.The derivation assumes generated sequences are detached from the loss computation graph.Source: §2.1 · p. 3

2.2 The Supervision Capability Functional and Supervision Fidelity Decay

  • The supervision capability functional averages teacher peak next-token probability over student prefix contexts; empirically, teacher completion accuracy and confidence fall with prefix length.Teacher confidence on its own prefixes remains higher, supporting student drift as the source of the observed decay.Source: §2.2 · p. 3

2.3 Theoretical Analysis: Signal Collapse and Compounding Drift

  • When teacher distributions become diffuse, the teacher’s discriminative gradient contribution vanishes, leaving a student-only signal that reinforces existing modes without correction.The proposition states that the teacher signal scales with the variance of teacher log-probabilities and tends to zero as that variance collapses.Source: §2.3 · p. 4

2.4 Training Dynamics: The Vicious Cycle and Supervision Boundary Contraction

  • The paper describes a vicious cycle in which positions beyond the effective learning horizon receive little useful supervision and may further shrink that horizon.The fitted supervision boundary contracts from 7.43k to 7.04k during OPD training, while the transition slope steepens from 0.21 to 0.26.Source: §2.4 · p. 4

3 Lookahead Confidence as a Supervision Signal

When local teacher feedback loses discrimination, the paper proposes using next-step teacher confidence to compare student token choices. It develops a group-normalized confidence reward and an entropy-triggered tree-attention procedure for computing it efficiently.

3.1 From Gradient Failure to One-Step Lookahead

  • Even when the teacher’s current distribution is nearly uniform, candidate tokens can lead to different next-step teacher distributions, preserving a lookahead comparison.The paper’s proposition states that this next-step discriminability can remain large even when local teacher entropy is maximal.Source: §3.1 · pp. 4–5

3.2 Confidence Reward Design

  • LGR rewards student candidates according to the teacher’s peak next-token probability after each candidate, then normalizes their relative ranking within the group.The reward targets the next step’s supervision quality rather than attempting to restore an in-distribution state.Source: §3.2 · p. 5

3.3 Efficient Computation

  • The entropy trigger evaluates confidence rewards only at high-entropy positions, while tree attention batches candidate branches without letting them attend to one another.The trigger typically selects 0.15L–0.25L positions early in training and stabilizes around 0.10L, reducing computational overhead by 4–7 times.Source: §3.3 · p. 6

4 Experiments

Across two student–teacher configurations, LGR improves aggregate benchmark performance over OPD, with a larger advantage for the 7B student. Training dynamics and ablations report higher teacher log-probability, more stable entropy, and sensitivity to the confidence metric and reward weight.

4.1 Experimental Setup

  • The evaluation uses 1.5B and 7B student configurations across math and code benchmarks, comparing LGR with OPD and other distillation baselines.Benchmarks include AIME, HMMT, and LiveCodeBench v6; baselines include GRPO, OPD topk, JSD, and REOPOLD.Source: §4.1 · pp. 6–7

4.2 Main Results

  • For the 7B student, LGR achieves 48.35% average mean@8, improving by 2.57% over OPD 9k; for 1.5B, it reaches 33.18%, up 1.61% absolute.For the 1.5B student, gains are consistent across all six evaluation sets, with the largest improvements on AIME-26 and LCB_v6.Source: §4.2 · p. 7
  • LGR (topk) improves pass@8 on several 1.5B benchmarks, while LGR without top-K generally performs better in the 7B setting.The reported 1.5B AIME-25 comparison is 56.67% versus 50.00%; the text describes top-K gains as complementary but not strictly additive.Source: §4.2 · pp. 7–8

4.3 LGR Addresses Supervision Fidelity Decay

  • During training, LGR maintains higher teacher log-probability and more stable entropy than OPD variants.The paper interprets the teacher log-probability pattern as student contexts remaining closer to the teacher's in-distribution manifold.Source: Figure 3 · p. 8, §4.3 · p. 8

4.2 Main Results

  • JSD and REOPOLD achieve comparable or lower performance than OPD, indicating that stabilization or divergence choice alone does not address SFD.The reported interpretation is that degraded teacher supervision on student-generated contexts requires a structurally different solution.Source: §4.2 · p. 8

4.3 LGR Addresses Supervision Fidelity Decay

  • The average confidence reward rises during training, while the entropy-triggered applied ratio stabilizes around 10%.The authors report that the trigger identifies a consistent fraction of high-entropy decision points without manual tuning over training.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4 Ablation Studies

  • 46.67 on AIME-24 for Max-p exceeds 43.75 for Sampled-p, 42.08 for entropy, and 37.92 for PPL.The paper says Max-p directly measures proximity to the teacher's in-distribution regime, while PPL and entropy can be inflated by irrelevant low-probability tokens.Source: §4.4 · pp. 8–9, Table 3 · p. 9
  • Confidence weight γ performs best in the 1.0–1.5 range; γ=10.0 causes an accuracy decline associated with repetitive confidence-seeking sequences.At γ=0.1, the lookahead signal has negligible influence and performance remains close to standard OPD.Source: Table 3 · p. 9, §4.4 · p. 9

5 Related Work

Prior work establishes on-policy distillation as a paradigm for transferring capabilities using student-generated data, while extending it through stability and objective-design methods. Concurrent studies also identify reliability or reward-quality degradation with student trajectory depth; this work distinguishes itself with formal SFD analysis and a one-step lookahead remedy.

On-Policy Distillation

  • MiniLLM and GKD support on-policy training on student-generated rollouts to address distribution mismatch and forward-KL mode averaging.The passage describes these works as establishing OPD as the standard paradigm for reasoning-model compression.Source: §5 · p. 9
  • Later work explores stability through REOPOLD and objective design through G-OPD and TSD-KD.REOPOLD uses reward clipping and dynamic sampling; G-OPD proposes reward extrapolation, while TSD-KD selectively applies KL loss to high-entropy tokens.Source: §5 · p. 9

Related Concurrent Work

  • Revisiting OPD [36] studies student-generation unreliability, while Rethinking OPD reports reward-quality degradation with trajectory depth; this work adds formal SFD analysis and one-step lookahead.The passage characterizes the lookahead remedy as orthogonal to divergence modification and applicable on top of the OPD objective.Source: §5 · p. 9

6 Conclusion

Lookahead Group Reward augments reverse-KL distillation with a group-normalized confidence reward, while entropy-triggered tree attention makes the approach computationally practical. The method requires teacher-logit access and relies on candidate rankings that may weaken in some settings.

6 Conclusion

  • LGR steers students toward trajectories where the teacher retains high supervision fidelity, rather than merely avoiding weak-supervision regions.It augments standard reverse-KL distillation with a group-normalized confidence reward and uses entropy-triggered tree attention for efficiency.Source: §6 · p. 9
  • The confidence reward requires white-box access to teacher logits, which may not always be available.Source: §6 · p. 9
  • The approach uses fixed candidate count K and entropy threshold τ; rankings may weaken on extremely out-of-distribution trajectories or with poorly calibrated teachers.The authors also identify multimodal reasoning as a direction for future work.Source: §6 · p. 10

A Training and Evaluation Details

The experiments use specified training and evaluation settings, with DeepSeek-R1-Distill students as the main testbed. The paper also contrasts homogeneous and heterogeneous teacher–student pairs, finding a wider distribution gap in the heterogeneous setting used throughout its experiments.

A Training and Evaluation Details

  • Training uses Polaris for math and DeepScaler code data, with 32 H20 GPUs; evaluation reports mean@8 and pass@8 over eight responses per problem.The evaluation generation limits are 39k tokens for math and 36k for code, at temperature 1.Source: A Training and Evaluation Details · pp. 13–14
  • Both student and teacher rollout temperatures are fixed at 1.0 because the two tested non-unit configurations degraded performance.Source: A Training and Evaluation Details · p. 13
  • The top-K OPD and LGR candidate set unions student and teacher top-10 tokens and renormalizes them; the paper says renormalization is essential to avoid divergence.GRPO comparisons use publicly available RL-trained checkpoints initialized from the corresponding student base models.Source: A Training and Evaluation Details · p. 13
  • The main experiments use DeepSeek-R1-Distill students at 1.5B and 7B; Qwen3-1.7B OPD training improved initially but then degraded progressively.The authors attribute Qwen3 instability to its prior integrated multi-domain RL fusion and use DeepSeek-R1-Distill models as a cleaner testbed for SFD.Source: A Training and Evaluation Details · p. 14

B.1 Homogeneous vs. Heterogeneous Teacher–Student Pairs

  • The heterogeneous teacher–student setting has a substantially wider student/teacher log-probability scatter, with students often visiting teacher low-probability regions.The paper uses this setting throughout as its primary testbed; in the homogeneous case, the distributions remain tightly correlated.Source: §B.1 · p. 15, Figure 6 · p. 15, Figure 5 · p. 15

B.2 Effect of Renormalization in Top-K Training

The appendix examines top-K renormalization, rollout temperature, token-level KL behavior, future-KL weighting, and alternative divergence objectives. Its findings support renormalized top-K training and reverse-KL as the stronger choices among those tested, while future-KL does not improve over standard OPD.

B.2 Effect of Renormalization in Top-K Training

  • Unnormalized top-K training underperforms renormalized training, and can inflate candidate probabilities while allowing probability mass to escape the KL penalty.Larger unnormalized candidate sets improve performance over smaller ones, but do not match renormalized training.Source: Figure 7 · p. 16, §B.2 · p. 16

B.3 Effect of Rollout Temperature

  • Both tested non-unit rollout temperatures degrade performance relative to the temperature-1.0 baseline.A sharpened teacher increases mismatch with the student’s on-policy distribution; symmetric low temperatures reduce diversity and weaken the confidence reward.Source: Figure 8 · p. 17, §B.3 · p. 17

B.4 Per-Token Top-K Logit Visualization

  • High-RKL tokens have dispersed student probability mass while the teacher concentrates on one or two tokens, and one OPD update reduces KL only slightly.Low-RKL tokens are already peaked on the same top-1 token as the teacher and contribute near-zero loss and gradient; their growing share dilutes high-RKL tokens’ relative influence in the batch average.Source: §B.4 · pp. 17–18, Figure 9 · p. 18

B.5 Future-KL with GAE Weighting

  • Future-KL with GAE weighting fails to improve over standard per-token RKL across the tested discount settings.The authors hypothesize that accumulated noise over many future tokens makes early-position gradients uninformative, favoring one-step lookahead instead.Source: §B.5 · pp. 18–19, Figure 10 · p. 19

B.6 Comparison of KL Divergence Objectives

  • Reverse-KL is the strongest of the three tested objectives; JSD is moderately stable but underperforms OPD, while forward-KL collapses.Forward-KL’s AIME-24 performance drops toward zero and does not recover; reverse-KL maintains stable entropy and the highest teacher log-probability.Source: §B.6 · p. 19, Figure 11 · p. 20

C Sigmoid Fit of the SFD Curve

The sigmoid fit analysis reports shifts in the SFD curve after OPD training, including a contracted supervision boundary. The appendix also argues that disabling the confidence reward at low-entropy positions loses little information while saving teacher evaluations.

C Sigmoid Fit of the SFD Curve

  • After OPD training, the fitted supervision boundary shifts from 7.43k to 7.04k, the transition steepens, the floor drops from 0.41 to 0.23, and the amplitude rises from 0.34 to 0.55.The authors say these shifts align with the predicted vicious-cycle direction, but two snapshots do not support precise estimates of logistic growth parameters.Source: C Sigmoid Fit of the SFD Curve · pp. 20–21

D Entropy-Triggered Activation is Near-Lossless

  • At low-entropy positions, the top-1 token dominates and the confidence-reward gradient is negligible, so disabling the reward saves K teacher forward-pass evaluations per position with near-zero information loss.Source: D Entropy-Triggered Activation is… · p. 21

E Tree Attention: Cost Analysis and Practical Segmentation

Tree attention batches candidate-confidence evaluations into one teacher pass, reducing the cost of lookahead; the proof section explains how teacher discrimination vanishes as its distribution becomes diffuse.

E Tree Attention: Cost Analysis and Practical Segmentation

  • Tree attention appends each high-entropy position’s top-K candidates as branches, letting one teacher forward pass evaluate them with separate causal-prefix attention.The main sequence is prefilled once, and each candidate attends to its own prefix.Source: Figure 12 · p. 21, E Tree Attention: Cost Analysis and… · p. 22
  • At K = 8 and L = 16k, tree attention yields an estimated 4741× speedup; speedup grows linearly with L because naïve cost scales as O(L^3), while tree attention scales as O(L^2).Even with segmentation, the speedup remains large because the O(L^3) vs. O(L^2) gap dominates.Source: E Tree Attention: Cost Analysis and… · p. 22
  • With L = 16k, K = 8, and about 3.2k selected positions, the experiments use 8 segments to fit within GPU memory.Even after segmentation, the O(L^3)-versus-O(L^2) cost gap leaves a large speedup.Source: E Tree Attention: Cost Analysis and… · p. 22

F.1 Proof of Proposition 1

  • When the teacher distribution is uniform, its log-probability is constant across tokens, so the teacher provides no corrective signal.In this case, the per-token advantage depends only on the student and a constant.Source: §F.1 · p. 22
  • As the teacher distribution becomes more diffuse under SFD, its discriminative signal and signal-to-noise ratio decrease toward zero.The teacher’s contribution depends on variation in its log-probability across student token choices.Source: §F.1 · p. 23

F.2 Proof Sketch of Proposition 2

The proof sketch shows how reverse-KL can reinforce student modes when teacher discrimination vanishes and sustain drift below a threshold; forward-KL avoids SFD but incurs exposure bias.

F.2 Proof Sketch of Proposition 2

  • When teacher discriminability is zero, reverse-KL advantage reinforces high-probability student tokens and sharpens the student mode without teacher correction.The sketch attributes this behavior to the advantage increasing with the log probability of the student token.Source: §F.2 · p. 23
  • When teacher discriminability falls below δcrit, expected drift does not decrease, sustaining a positive feedback loop under the stated drift assumption.The sketch links increased drift to lower teacher discriminability, which weakens corrective signal and leaves student-selected tokens less guided by the teacher.Source: §F.2 · pp. 23–24
  • Forward-KL avoids SFD by using teacher-generated training sequences, but creates exposure bias because inference uses student-generated prefixes.Source: §F.2 · pp. 23–24

F.3 Proof of Proposition 3

This section analyzes one-step-ahead discriminability and group normalization, and records checklist disclosures on released assets, human-subject research, and LLM use.

F.3 Proof of Proposition 3

  • Even with a uniform teacher distribution at position t, one-step-ahead discriminability can be arbitrarily large, so it is independent of current-position entropy.The construction gives D_ahead(t) → 2/3 as ε → 0 and extends to any vocabulary size |V| ≥ 2 at maximum local entropy.Source: §F.3 · p. 24

F.4 Proof of Proposition 4

  • Higher max-p at t+1 for a candidate indicates less additional drift from the teacher’s in-distribution manifold, not a return to it.The bound uses β-smoothness to relate teacher distribution shifts to distance from the nearest in-distribution prefix.Source: §F.4 · p. 25

F.5 Group Normalization: Design Rationale and Formal Properties

  • Group normalization cancels position- and task-level confidence confounds, retaining relative candidate rankings; when candidates have similar confidence, the reward approaches zero.This suppresses the signal where the teacher cannot discriminate among candidates, without external gating.Source: §F.5 · pp. 25–26

NeurIPS Paper Checklist

  • The checklist reports no error bars, while marking proofs, experimental reproducibility and settings, compute resources, and societal-impact discussion as provided.It says the proofs and experimental details are in the appendix, source code is provided through an anonymized link, societal impacts are discussed in Limitations, and safeguards are not applicable because the paper poses no such risks.Source: NeurIPS Paper Checklist · pp. 27–30
  • The authors report that they release no new assets.The checklist marks documentation of new assets as not applicable because none are released.Source: NeurIPS Paper Checklist · pp. 31–32
  • The paper does not involve crowdsourcing or research with human subjects.The checklist gives this justification for both its crowdsourcing and IRB questions.Source: NeurIPS Paper Checklist · p. 32
  • LLMs are used only for writing, editing, and formatting, not as an important, original, or non-standard component of the core method.The checklist says declaration is unnecessary when LLM use does not affect the core methodology, rigor, or originality.Source: NeurIPS Paper Checklist · p. 33
Loading 2605.30833v1…