PaperlayerView exact revision on arXiv

Source-linked AI summary

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, Ge Liu

arXiv:2605.11182v2cs.AI

TL;DR

OPD and OPSD have shown mixed results, leaving unclear when they work and why they fail. This paper studies both methods across task settings, identifies failure mechanisms, and finds that effectiveness depends on task structure and privileged information, with several stabilizers improving training stability.

  • Problem

    The empirical conditions and mechanisms governing when OPD and OPSD succeed or fail remain poorly understood despite mixed prior results.

  • Method

    The paper empirically studies OPD and OPSD across tasks, analyzes their failure mechanisms, and evaluates stop-gradient TopK losses, RLVR-adapted teachers, and SFT-stabilized students.

  • Results

    Effectiveness depends on task structure and privileged information: OPSD fails on tested mathematical reasoning but works for system-prompt internalization and alignment, while OPD is conditionally beneficial.

  • Takeaways & Limitations

    OPD and OPSD require task-appropriate privileged information and objective design, while stop-gradient losses, improved teachers, and SFT can improve practical training stability.

  • Takeaways & Limitations

    The stop-gradient TopK surrogate mitigates local gradient bias but no longer faithfully approximates the full-vocabulary reverse KL.

Abstract

from arXiv · show

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

1 Introduction

This paper studies when OPD and OPSD succeed or fail across reasoning, system-prompt internalization, and alignment, addressing mixed prior findings. It identifies three failure mechanisms and proposes stabilizers based on stop-gradient Top-K KL, RLVR-adapted teachers, and student SFT.

  • Method background: OPD trains students on their own-policy trajectories with dense token-level supervision from stronger teachers, enabling capability integration and potentially mitigating catastrophic forgetting.This setup differs from training on off-policy responses and uses one or more teachers along student-generated rollouts.
  • Motivation: OPD and OPSD have mixed empirical behavior: prior studies report successful context internalization and reasoning, whereas recent work finds instability and performance degradation.These conflicting results motivate examining when on-policy (self) distillation works, fails, and why.
  • Empirical scope: OPD offers only conditional benefits on reasoning tasks, while OPSD fails in the tested mathematical-reasoning settings but succeeds for system-prompt internalization and alignment.The contrast arises because system prompts and alignment preferences induce shared latent behavior that can be compressed into a PI-free policy.
  • Failure mechanisms: The study identifies three failure mechanisms: prefix-induced teacher–student mismatch, biased Top-K reverse-KL gradients, and PI-free OPSD aggregation across instance-specific privileged information.Student-generated prefixes can place teachers in locally incompatible states, while OPSD’s single consensus policy is insufficient when privileged information varies by instance.
  • Practical fixes: The paper proposes a stop-gradient Top-K KL surrogate, RLVR adaptation of teachers, and student SFT to reduce biased gradients, improve task performance, and stabilize formatting and response length.The stabilizers target optimization instability, teacher quality, output well-formedness, and response-length dynamics.

2 Related Work

Prior work establishes on-policy distillation as dense supervision on student-sampled trajectories, with demonstrated benefits for distillation and reasoning. Related approaches include context distillation and reinforcement learning augmented by textual or privileged feedback.

  • On-Policy Distillation and Context Distillation: On-policy distillation trains students on their own sampled trajectories with dense teacher supervision, outperforming standard off-policy distillation in some settings and showing effectiveness for math reasoning.Context distillation similarly turns in-context behaviors into model parameters.
  • On-Policy Distillation and Context Distillation: OPSD constructs the teacher from the student and privileged information, whereas OPD uses a stronger teacher and does not require privileged information; reverse KL is mode-seeking and forward KL is mode-covering.These distinctions are summarized in Figure 2.
  • Reinforcement Learning from Textual Feedback: Textual-feedback reinforcement learning provides richer supervision over intermediate behaviors, while POPE uses privileged guidance to improve exploration on difficult reasoning problems.These methods motivate studying the role of additional textual information.

3 Preliminary: On-Policy Distillation

OP(S)D trains on student-generated trajectories using token-level teacher supervision, with OPSD augmenting a student-derived teacher with privileged information. The section distinguishes full-vocabulary and sampled-token objectives while illustrating OPSD failure and unnormalized Top-20 reverse-KL collapse.

  • OP(S)D samples trajectories from the student policy and applies teacher supervision at every generated prefix.In OPD, the teacher is an external stronger model; in OPSD, it is student-derived and augmented with privileged information.
  • The section presents full-vocabulary KL and sampled-token KL as alternative token-level distillation objectives.Sampled-token KL treats the teacher–student log-probability gap as an advantage.
  • Full-vocabulary KL: Reverse KL is mode-seeking, preserves the student’s high-probability modes, and is less prone to catastrophic forgetting than forward KL.Forward KL is undesirable for OPD because it pushes the student toward teacher-preferred but student-unlikely tokens.
  • OPSD fails to improve the student in the Qwen3-1.7B OpenThoughts experiment.
  • Unnormalized Top-20 reverse KL causes verbosity, repetitive “maybe” outputs, and declining evaluation accuracy as response length reaches the limit.Repetitive tokens dominate as the repeat ratio approaches one.

4 Experiments

Experiments reveal that OPD and OPSD can fail catastrophically on mathematical reasoning but succeed when privileged information is shared across instances, including alignment and system-prompt internalization. OPSD also outperforms GRPO and PPO on style alignment, while reducing reasoning response length without harming accuracy.

  • Experimental Scope: The experiments cover reasoning, system-prompt internalization, and alignment, spanning both failure and success regimes for OPD and OPSD.The tested settings include Qwen3 students and OpenThoughts-based training, with verifiable mathematical answers and question-specific alignment prompts where applicable.
  • Reasoning: On mathematical reasoning, OPSD with either answer-only or full-response privileged information produces no stable gains on Math500, AIME24, or AIME25.Full-response privileged information performs worse than answer-only privileged information, suggesting richer conditioning can increase mismatch.
  • Reasoning: OPD initially improves mathematical reasoning but later collapses into repetitive “maybe” outputs, driving Math500, AIME24, and AIME25 accuracy nearly to zero.Around step 700, rollout length and revision tokens such as “wait” and “maybe” rise sharply; by step 1000, degeneration is evident.
  • Alignment: Under the same sampling budget, OPSD improves and converges faster than GRPO and PPO on CharacterBench and EmotionBench.The comparison uses Qwen3-4B-Instruct and Qwen3-8B students; Figure 5 reports training-reward and evaluation-score curves.
  • System Prompt Internalization: With a fixed system prompt shared across questions, OPSD preserves reasoning accuracy while substantially reducing response length.This contrasts with question-specific privileged information in the alignment experiments; the result is shown for reasoning compression in Figure 6.

5 Mechanism of On-Policy Distillation

The paper identifies three mechanisms behind OPD and OPSD failures: student prefixes can distort teacher behavior, unnormalized TopK reverse-KL can destabilize optimization, and instance-specific privileged information can make OPSD learn an insufficient aggregate policy.

  • Student Prefixes Distort the Teacher State: OPD teacher conditioning on student-generated prefixes can force inconsistent intermediate reasoning states, causing branch switching instead of refinement.This conflict often appears as elevated probability on revision tokens such as “wait” and “but”.
  • The Pitfall of Unnormalized TopK Reverse KL: TopK reverse-KL truncation retains a +1 gradient term, promoting tokens only when πT(v | x, y<t, I) > e πS(v | x, y<t).This biases updates away from the teacher distribution and can drive unstable low-probability continuations or collapse.
  • Privileged Information in OPSD: OPSD’s optimal student is the normalized geometric mean of PI-conditioned teachers, suppressing outputs that are not consistently supported across privileged-information settings.Thus, problem-specific PI can induce incompatible teacher behaviors whose common aggregate is insufficient for individual math instances.
  • Privileged Information in OPSD: Both answer-PI and response-PI OPD underperform vanilla OPD with a stronger teacher on OpenThoughts, with response-PI performing worst.Figure 10 also shows higher initial distillation losses for response-PI and answer-PI than vanilla OPD.

6 Solutions

The proposed fixes target TopK reverse-KL instability, teacher–student distribution mismatch, and practical TopK truncation constraints. Stop-gradient and renormalized objectives stabilize training, while RLVR-adapted teachers and SFT-based stabilization address distributional and optimization failures.

  • TopK reverse-KL objectives: Unnormalized TopK reverse KL collapses, whereas stop-gradient and renormalized variants achieve comparable stable performance; policy-gradient distillation performs similarly to Top1 stop-gradient reverse KL.The stop-gradient version treats the weighting term as an advantage rather than differentiating through it; sampled-token policy-gradient supervision is stable but captures only sparse teacher signal.
  • TopK reverse-KL objectives: Renormalized TopK reverse KL removes the unnormalized gradient bias but matches only relative probabilities within TopK, ignoring selected-set mass and full-vocabulary behavior.Thus, it mitigates local gradient bias without faithfully approximating full-vocabulary reverse KL.
  • TopK truncation: Because SGLang cannot query each position’s ideal TopK set, the method queries their global union, increasing memory by min(|V|, TK)/K in the worst case.Backpropagating only through tokens shared by teacher and student TopK sets preserves effectiveness according to prior work.
  • RLVR-adapted teachers: RLVR-adapted Qwen3-1.7B-GRPO significantly outperforms Qwen3-8B as an OPD teacher despite comparable reasoning performance, indicating student-distribution proximity matters beyond accuracy.RLVR improves training-set task performance and can make teacher outputs locally more compatible with student-generated prefixes.
  • SFT-stabilized students: SFT-based stabilization has been used to reduce length inflation and teacher–student mismatch, while the Qwen3-1.7B student can otherwise produce garbled non-English Unicode outputs.Prior work combines OPD with SFT on gold answers or teacher-generated traces; the reported instability is distinct from ordinary incorrect reasoning.

7 Conclusion

The study shows that OPD and OPSD effectiveness depends on task structure and the nature of privileged information. OPSD fails in tested mathematical reasoning settings but works for system-prompt internalization and alignment tasks involving shared latent rules.

  • 7 Conclusion: OPD and OPSD effectiveness depends on task structure and the nature of privileged information.The conclusion presents an empirical study of both methods for large language models.
  • 7 Conclusion: OPSD is ineffective in the tested mathematical reasoning settings, where privileged information is largely instance-specific.
  • 7 Conclusion: OPSD works well for system-prompt internalization and alignment tasks, where privileged information reflects shared latent rules.
  • 7 Conclusion: The study identifies teacher-student distribution mismatch as one failure mechanism for OPD and OPSD.

A Appendix · A.1 Experiment Setup

The appendix evaluates models with fixed decoding settings and reports default training configurations for OPD, OPSD, GRPO, and PPO. Experiments generally use stop-gradient, renormalized Top-K reverse-KL training, with OPSD using a fixed step-0 student checkpoint as teacher.

  • A.1 Experiment Setup: Evaluation uses a maximum response length of 16384, temperature 1.0, and top-p 0.95.
  • A.1 Experiment Setup: Unless specified otherwise, OPD and OPSD use one rollout per prompt and optimize a stop-gradient, renormalized Top-K reverse-KL objective.
  • A.1 Experiment Setup: By default, OPSD disables thinking mode and uses the step-0 student checkpoint as its fixed teacher, while experiments run on 10 NVIDIA RTX PRO 6000 Blackwell GPUs.Table 1 summarizes the default training hyperparameters for OPD, OPSD, GRPO, and PPO.

A.2 Evaluation Metrics

This section defines metrics for analyzing token-level behavior, repetition, and teacher–student agreement. It measures sampled-token log-probabilities, repetition patterns, TopK overlap, and teacher-side sampled-token rank, including repetition-conditional averages.

  • Log-probability metrics: The metrics track teacher and student log-probabilities for each sampled token and their log-probability gap under the generated prefix.The teacher conditions additionally on privileged information I, whereas the student does not.
  • Repetition: Repetition is identified when a response-ending n-gram, using default n = 3, appeared earlier in the same response.The section defines a repetition ratio over valid response tokens and partitions positions into repetitive and non-repetitive sets.
  • Teacher-student agreement: Teacher–student agreement is measured by overlap between their TopK candidate sets at each position, using default K = 50.The overlap compares the teacher’s and student’s local candidate distributions through their TopK sets.
  • Teacher-student agreement: The evaluation also records the teacher-side rank of each sampled token and reports overlap and rank averages separately for repetitive and non-repetitive positions.Sampled tokens inside the teacher TopK set receive their rank; others receive K + 1.

A.3 Design Space of OP(S)D

OP(S)D methods vary along three largely orthogonal axes: teacher construction, privileged information design, and distillation loss. The first two mainly distinguish OPSD, while distillation loss is central to both OPSD and OPD.

  • Design axes: OP(S)D design comprises teacher construction, privileged information design, and distillation loss; the first two primarily apply to OPSD, whereas loss choice affects both methods.In standard OPD, the teacher is typically fixed and no privileged information is introduced.

A.3.1 Teacher Construction

Teacher construction trades off stability, adaptability, and privileged-information use across self, frozen, and EMA teachers. A frozen teacher stabilizes targets, whereas jointly updated or EMA teachers can track learning but do not guarantee stronger task performance.

  • Self-Teacher: A self-teacher shares the student’s parameters and updates jointly throughout training, a setup used in several papers.The teacher and student therefore remain fully coupled during optimization.
  • Frozen teacher: A frozen teacher keeps targets stable by fixing initialization, a stronger pretrained model, or separately trained expert parameters, but may become stale and mismatch student-induced states.Its capability is inherently bounded, despite improved optimization stability.
  • EMA teacher: An EMA teacher tracks the student while smoothing high-variance updates, providing a more stable target than using the current student directly.This preserves self-distillation while following the student’s learning progress.
  • Takeaway: EMA and self-teachers are often expected to exploit privileged information better, but plain OP(S)D is not guaranteed to surpass the original base teacher without task-directed optimization such as RL.The advantage of adaptive teachers is therefore conditional rather than automatic.

A.3.2 Privileged Information Design … A.23.1 The PPL and NLL statistics.

The appendix specifies OP(S)D design choices, gradient approximations, engineering constraints, and empirical failure modes, while documenting cases where privileged information transfers successfully. It also reports evidence that student prefixes distort teacher supervision and that SFT improves trace modeling.

  • A.3.2 Privileged Information Design; A.3.3 Distillation Objectives: Teacher privileged information can include math reasoning and answers, environment feedback, or other task-dependent signals, while students match teachers using full-vocabulary or sampled-token objectives.Full-vocabulary KL reshapes the entire next-token distribution, whereas sampled-token objectives act more like reinforcement learning with dense token-level rewards.
  • A.4 General OP(S)D Gradient Decomposition.; A.5 Detailed Gradient computation.: Ignoring the score-function term yields a biased but lower-variance gradient, while retaining it gives an unbiased estimator with substantially higher variance; full-vocabulary and sampled-token derivations expose this tradeoff.In practice, full-vocabulary KL often omits the rollout-distribution term, as in SDPO and SDFT.
  • A.6 TopK Engineering Challenge: Student TopK teacher queries require flattening per-position token sets into a sorted unique union because SGLang expects one token list shared across positions.The original position-dependent representation creates an [L, K] index tensor instead of the expected one-dimensional shape and breaks downstream gathering.
  • A.3.3 Distillation Objectives; A.7 Additional OPD Experiment using Top20 reverse KL: TopK approximations reduce memory requirements but can be unstable: increasing K from 5 to 20 does not prevent reverse-KL collapse, whereas stop-gradient remains stable.Tail-augmented objectives preserve probability mass outside the selected TopK set and can better approximate the full distribution.
  • A.8 Evaluation Biases in OPD; A.9 Teacher Signal Analysis (On Policiness); A.11 Visualizing token-level supervision on an example response; A.12 Experimental Results on General Reasoning Tasks: OPD evaluation can be misleading because short validation limits truncate longer reasoning, and early OPD gains may obscure GRPO’s later improvement and higher performance ceiling.Teacher signals also vary with capability gap, response correctness, token position, and teacher scale; incorrect trajectories can receive stronger supervision than correct ones.
  • A.13 OPSD Fails on Persuasion Tasks; A.14 Thinking Mode Hacking in OPSD; A.15 OPSD Response Length Collapse; A.16 OPD Response Length Explosion - step 700; A.17 OPD Response Length Explosion - step 1000: OPSD fails in several reasoning and persuasion settings, producing response collapse, direct-answer behavior, control-token leakage, or verbose and repetitive outputs.On Persuasion for Good, GRPO is described as stable and gradually improving, while OPSD does not show the same behavior; examples include a response containing only “maybe” and another repeating “but” 10 times.
  • A.18 OPSD Safety Alignment System Prompt; A.19 OPSD Reasoning Compression System Prompt.; A.20 OPSD Language Style Alignment: Privileged Information Design and Model Outputs; A.21 The correlation between entropy and teacher supervision: OPSD can transfer shared style or safety behavior without exposing the privileged profile at inference, but entropy offers only mild evidence of correlation with teacher supervision.The style-transfer setup prepends style-specific information only to teacher prompts, while the student receives the plain prompt; an example receives an LLM-judge reward of 1.0.
  • A.22 Experimental Evidence for Student Prefixes Distorting the Teacher’s Reasoning State; A.23 SFT Experiment Setup in Section 6.3.; Data Preparation; SFT Stage; A.23.1 The PPL and NLL statistics.: 62.12% versus 45.96%: conditioning the teacher on student prefixes reduces GPQA-Diamond accuracy by 16.16 points and 32 correct cases, while SFT lowers NLL from 0.640 to 0.335 and PPL from 1.896 to 1.397.Prefix continuation also changes 40 originally correct predictions to wrong versus only 8 wrong-to-correct changes, and lowers format correctness from 98.48% to 78.79%.
Loading 2605.11182v2…