The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

1 Introduction

• OPD provides teacher supervision on student-sampled trajectories; OPSD instead uses the student itself conditioned on additional privileged information.Privileged information can include ground-truth answers, system prompts, or user preferences.Source: §1 · p. 1
• Prior self-distillation successes in context internalization and reasoning contrast with reports of instability and performance degradation, motivating the study.The study asks when OPD and OPSD work, when they fail, and which mechanisms explain the difference.Source: §1 · p. 1
• OPD offers conditional reasoning benefits, while OPSD fails in tested math settings but succeeds in system-prompt internalization and alignment.Reasoning OPD can exhibit length explosion, repetition, and teacher–student mismatch. Successful OPSD settings use privileged information that induces shared latent behavior compressible into a PI-free test-time policy.Source: §1 · p. 2
• OPSD learns a single PI-free consensus policy across PI-conditioned teachers, which is insufficient when privileged information is instance-specific.This aggregation limitation distinguishes instance-specific privileged information from the shared latent behavior underlying successful internalization and alignment.Source: §1 · p. 2
• Student-generated prefixes can make teacher supervision locally incompatible with student trajectories, while Top-K reverse-KL approximations can introduce biased gradients and destabilize optimization.These constitute two additional failure mechanisms beyond OPSD’s aggregation of privileged-information-conditioned teachers.Source: §1 · p. 2
• Proposed stabilizers combine stop-gradient Top-K KL, RLVR-adapted teachers, and student SFT.The stop-gradient surrogate removes the biased gradient term; RLVR adaptation targets teacher task performance. Student SFT targets output well-formedness and response-length stability.Source: §1 · p. 2

On-Policy Distillation and Context Distillation

• On-policy distillation provides dense teacher supervision on student-sampled trajectories, outperforming standard off-policy distillation in Agarwal et al. and demonstrating effectiveness for mathematical reasoning.Context distillation instead transfers in-context behaviors into model parameters.Source: §2 · p. 2
• Context distillation internalizes instructions and knowledge as persistent capabilities, with recent work extending on-policy distillation to continual learning.Source: §2 · p. 2
• OPSD constructs its teacher from the student and requires privileged information, whereas OPD uses a stronger teacher with optional privileged information.Figure 2 also contrasts mode-seeking reverse KL with mode-covering forward KL.Source: Figure 2 · p. 3

Reinforcement Learning from Textual Feedback

• Textual feedback improves intermediate behaviors and agent capabilities, while POPE uses privileged guidance to improve exploration on hard reasoning problems.These approaches augment reinforcement learning beyond scalar rewards; this paper instead studies additional textual information through on-policy distillation objectives.Source: §2 · p. 3

• OPD uses a stronger external teacher; OPSD uses a student-derived teacher with privileged information, and both provide token-level supervision on student-generated trajectories.Teachers supervise each student-generated prefix, conditioning on privileged information when available.Source: §3 · p. 3
• Full-vocabulary reverse KL is mode-seeking and tends to preserve student high-probability modes; forward KL covers teacher modes and pushes toward student-unlikely tokens.Reverse KL is less prone to catastrophic forgetting. Under unnormalized Top-20 reverse KL, outputs become verbose, then repeat “maybe” as response length reaches its limit, evaluation accuracy drops, and the repeat ratio approaches one.Source: §3 · p. 3, Figure 4 · p. 4
• Sampled-token KL uses a token sampled from the student and treats the teacher–student log-probability gap as an advantage.Source: §3 · p. 3
• OPSD fails to improve Qwen3-1.7B trained on OpenThoughts.Source: Figure 3 · p. 4

4.1 Math Reasoning

• Math OPSD produces no stable improvements on Math500, AIME24, or AIME25 with either answer-only or full-response privileged information.Using Qwen3-1.7B, a step-0 teacher, and stopgrad Top20 reverse KL on OpenThoughts, full-response PI performs worse than answer-only PI, suggesting richer PI can increase mismatch. A GRPO-trained full-response-PI teacher performs even worse.Source: §4.1 · p. 4
• Nearly zero accuracy on Math500, AIME24, and AIME25 follows OPD’s initial gains and collapse into repetitive “maybe” outputs by step 1000.This occurs with a Qwen3-1.7B student, Qwen3-8B teacher, thinking disabled, and unnormalized Top20 reverse KL. Around step 700, rollout length and revision tokens surge, accompanied by rising repetition and collapsing teacher–student overlap.Source: §4.1 · p. 4

4.2 Alignment & System Prompt Internalization.

• OPSD improves and converges faster than GRPO and PPO on CharacterBench and EmotionBench under the same sampling budget.Teachers receive question-specific alignment prompts as privileged information.Source: §4.2 · pp. 4–5, Figure 5 · p. 5
• OPSD preserves reasoning accuracy while substantially shortening responses and using samples more efficiently than length-penalized GRPO, but requires a sufficiently capable model.An 8B-scale model provides sufficient capability, consistent with. Token-wise supervision permits shorter rollouts than RL, which needs long generations to obtain final answers.Source: Figure 6 · p. 5, §4.2 · p. 5
• In safety alignment experiments with Qwen3-1.7B and Wildguardmix, OPSD improves rapidly early but is ultimately teacher-limited, while GRPO improves gradually and continues gaining after OPSD saturates.Reasoning compression and safety alignment use a fixed system prompt as privileged information across all questions, unlike the question-specific privileged information used in the preceding alignment tasks.Source: §4.2 · p. 5

5.1 Why Does OPD Sometimes Fail?

• On GPQA-Diamond, Qwen3-14B achieves 46.0% accuracy when conditioned on Qwen3-1.7B student-generated prefixes, compared with 62.1% accuracy standalone, demonstrating that student-prefix conditioning can weaken teacher supervision relative to its standalone capability.Student prefixes can force the teacher into reasoning states it would not reach independently, creating conflict between the student’s current path and the teacher’s preferred path. In mathematical reasoning, such unreliable supervision encourages verbose and inconsistent reasoning.Source: §5.1 · p. 5
• Unnormalized TopK reverse KL promotes tokens only when teacher probability exceeds e times student probability, suppressing smaller teacher-preferred margins and risking instability.The +1 gradient term remains after truncation, changing the update rule and potentially biasing the student toward unstable, low-probability continuations.Source: §5.1 · p. 6

5.2 OPSD Learns a PI-Free Policy.

• OPSD’s optimal PI-free student is the normalized geometric mean of PI-conditioned teacher distributions, suppressing outputs unsupported across privileged information.Outputs assigned high probability under some PI but low probability under others are suppressed.Source: §5.2 · pp. 6–7
• Both answer-PI and response-PI OPD underperform vanilla OPD with a Qwen3-8B teacher and Qwen3-1.7B student, with response-PI worst.On OpenThoughts, step-0 distillation losses follow response-PI > answer-PI > vanilla OPD, suggesting that longer PI increases teacher–student mismatch rather than improving supervision.Source: Figure 10 · p. 7, §5.2 · p. 7
• Instance-specific PI can induce incompatible teachers and weaken OPSD, whereas shared latent rules enable reusable behavior in prompt internalization and alignment.With problem-specific mathematical PI, the student learns common patterns across incompatible teacher policies and becomes substantially weaker than the PI-conditioned teacher.Source: §5.2 · p. 7

6.1 Designing an appropriate OPD loss.

• Stop-gradient and renormalized TopK reverse KL stabilize training where unnormalized reverse KL collapses; policy-gradient distillation is similarly stable.Stop-gradient blocks differentiation through the log-probability term, aligning the weighting with an advantage interpretation. At K = 5, both modifications achieve comparable stable performance; policy-gradient distillation performs similarly to stop-gradient Top1.Source: §6.1 · pp. 7, 9, Figure 11 · p. 8, Figure 12 · p. 8
• Renormalization removes local gradient bias but ignores TopK probability mass; sampled-token policy gradients avoid the bias but provide sparse rather than vocabulary-distribution supervision.Renormalized TopK reverse KL matches only relative probabilities within the selected set, so it no longer faithfully approximates full-vocabulary reverse KL.Source: §6.1 · p. 8

6.2 Enhance teacher with RLVR.

• Qwen3-1.7B-GRPO produces stronger OPD students than Qwen3-8B despite comparable teacher reasoning performance, showing teacher accuracy alone does not predict distillation effectiveness.The teachers perform comparably on Math500, AIME24, and AIME25. RLVR adaptation reduces teacher–student distribution mismatch and provides an effective alternative to privileged-information-based OP(S)D.Source: Figure 13 · p. 9, §6.2 · p. 9

6.3 Stabilize Student with SFT.

• Teacher-trace SFT stabilizes Qwen3-1.7B-Base outputs and response length, allowing subsequent OPD from Qwen3-4B to improve accuracy where direct OPD is unstable.The initial student sometimes generates nonsensical, non-English Unicode sequences that prevent meaningful teacher feedback. SFT keeps on-policy samples in well-formed regions where that feedback remains effective.Source: Figure 14 · p. 9, §6.3 · p. 10

6.1 Designing an appropriate OPD loss.

• To address SGLang’s costly global TopK unions, training backpropagates only through tokens appearing in both teacher and student TopK sets.SGLang requires querying a global union U at every position, increasing the queried tensor from T × K to T × |U|. Prior work finds that the intersection performs comparably to student TopK.Source: §6.1 · pp. 8–9

7 Conclusion

• OPSD is ineffective in tested mathematical reasoning settings with largely instance-specific privileged information, but works well for system prompt internalization and alignment tasks involving shared latent rules.The nature of privileged information distinguishes these task settings.Source: §7 · p. 10
• Three failure mechanisms are teacher-student distribution mismatch, biased TopK reverse-KL gradients, and OPSD learning only the marginal distribution over instance-specific privileged information.Source: §7 · p. 10
• Stop-gradient TopK surrogate loss, RLVR teacher improvement, and SFT stabilization can improve practical training stability.Source: §7 · p. 10
• Findings cover a limited set of model families and scales, so larger-scale settings may exhibit different behavior.Source: §7 · p. 10
• Future work could explore iterative self-improvement pipelines combining SFT initialization, RL task-specific improvement, and OPD distillation of improved on-policy behavior back into the student.Source: §7 · p. 10

A.1 Experiment Setup

• 16384 is the maximum response length for evaluation, with temperature 1.0 and top-p 0.95.Source: §A.1 · p. 13
• OPD and OPSD optimize a stop-gradient, renormalized Top-K reverse-KL objective unless otherwise specified.Source: §A.1 · p. 13
• One rollout per prompt is the default for both OPD and OPSD unless otherwise specified.Source: §A.1 · p. 13
• OPSD uses the step-0 student checkpoint as its fixed teacher model by default.Source: §A.1 · p. 13
• OPSD disables thinking mode by default.Source: §A.1 · p. 13
• 10 NVIDIA RTX PRO 6000 Blackwell GPUs equip the server used for all experiments.Source: §A.1 · p. 13

A.2 Evaluation Metrics

• Sampled-token log-probabilities condition both models on the prompt and generated prefix, while only the teacher additionally conditions on privileged information I.The metrics also include the teacher–student log-probability gap.Source: §A.2 · p. 13
• Repetition is identified by previously occurring n-grams within the same response, using n = 3 by default.Token-level quantities are averaged separately over repetitive positions R and non-repetitive positions O.Source: §A.2 · p. 13
• Teacher–student agreement compares their TopK candidate sets at each token position, using K = 50 by default.The sampled token’s teacher-side rank is recorded within the teacher’s TopK set and assigned K + 1 otherwise. Overlap and rank are averaged separately over repetitive and non-repetitive positions.Source: §A.2 · p. 14

• Distillation loss is a central design choice for both OPD and OPSD.Source: §A.3 · p. 14
• Teacher construction is primarily an OPSD design axis, since standard OPD typically uses a fixed teacher model.Source: §A.3 · p. 14
• Privileged information design primarily concerns OPSD, since standard OPD introduces no privileged information.Source: §A.3 · p. 14

Self-Teacher

• Self-teachers use exactly the student's model and share its parameters, so teacher and student update jointly throughout training, as in.The setup denotes privileged information by c.Source: §A.3.1 · p. 14

Frozen teacher

• Frozen teachers keep parameters fixed, stabilizing targets but risking staleness and mismatch with the student-induced state distribution.The teacher can copy the student's initialization or use a stronger pretrained model or separately trained expert policy.Source: §A.3.1 · p. 14

exponential moving average (EMA) teacher

• EMA teachers track student parameters through moving averages, smoothing high-variance updates and providing more stable targets than fully coupled self-teachers.This preserves self-distillation while tracking the student's learning progress.Source: §A.3.1 · p. 14

Takeaway

• EMA and self-teachers do not guarantee better use of privileged information, and plain OP(S)D alone need not improve on the original base teacher.Without direct task-objective optimization, such as RL, there is little reason to expect stronger privileged-information use; frozen teachers also have inherently bounded capability.Source: §A.3.1 · p. 14

A.3.3 Distillation Objectives; A.3.2 Privileged Information Design

• Full-vocabulary objectives reshape the entire next-token distribution, whereas sampled-token objectives update only sampled tokens and resemble reinforcement learning with dense token-level rewards.PI can provide mathematical reasoning or answer-only hints, environmental feedback, or summarized experience. TopK reduces vocabulary-computation costs, tail aggregation preserves omitted probability mass, and OP(S)D can supplement RL through regularization or token-level advantages.Source: §A.3.3 · pp. 15–16, §A.3.2 · p. 15

A.4 General OP(S)D Gradient Decomposition.; A.5 Detailed Gradient computation.

• Ignoring the rollout-distribution score-function term introduces gradient bias but often reduces variance and improves optimization stability.The retained direct term backpropagates through token-level losses on fixed sampled prefixes; sequence-level RL estimators retain the score-function term for unbiased, higher-variance gradients. Full-vocabulary KL implementations in SDPO and SDFT use the omission.Source: §A.4 · p. 17, §A.5 · p. 18

A.7 Additional OPD Experiment using Top20 reverse KL

• Increasing K from 5 to 20 does not prevent reverse-KL collapse, whereas the stop-gradient version remains stable.The experiment uses Qwen3-1.7B-GRPO as teacher and Qwen3-1.7B as student, both without thinking, on DAPO.Source: Figure 15 · p. 19, §A.7 · p. 19

A.6 TopK Engineering Challenge

• Flattening per-position student TopK token IDs into a sorted unique union resolves SGLang’s teacher-probability query incompatibility without changing the intended candidate queries.SGLang expects one flat token list shared across positions, so nested per-position lists cause a gather-index shape mismatch. The union produces a sequence-length-by-union-size probability tensor from which each position’s requested teacher probabilities can be recovered.Source: §A.6 · p. 19

A.8 Evaluation Biases in OPD

• Validation length limits and early-training comparisons can misrepresent OPD gains by confounding response completion with reasoning quality and ignoring GRPO’s eventual performance ceiling.OPD can lengthen or repeat reasoning, causing premature validation truncation. Although OPD may improve faster initially, observed GRPO runs continue improving longer and can ultimately perform better.Source: §A.8 · pp. 19–20

A.12 Experimental Results on General Reasoning Tasks; A.9 Teacher Signal Analysis (On Policiness); A.11 Visualizing token-level supervision on an example response

• OPD with Qwen3-8B teaching Qwen3-1.7B yields inconsistent GPQA-Diamond performance and only marginal MMLU-Pro improvement with thinking disabled.Training uses the Science subset of Mixture of Thoughts; supervision concentrates on early tokens and incorrect responses, especially final-answer tokens, paralleling mathematical-reasoning observations. Qwen3-8B supervises Qwen3-1.7B more strongly than itself, while teacher scale and reasoning depth fundamentally shape supervision beyond PI refinements.Source: §A.9 · p. 20, Figure 19 · p. 21, §A.12 · p. 21

A.13 OPSD Fails on Persuasion Tasks

• Approximately 20–30 steps precede Qwen3-1.7B OPSD collapse on Persuasion for Good, whereas GRPO remains stable and gradually improves.PI includes persuadee background, personality, and persuasion strategies; Qwen3-4B suffers more severe degradation. OPSD drives the truncation ratio nearly to 1.0 and decreases student token log-probabilities, indicating excessively long generations and increasing uncertainty.Source: Figure 21 · p. 22, §A.13 · p. 22

A.14 Thinking Mode Hacking in OPSD; A.15 OPSD Response Length Collapse

• OPSD can produce unintended thinking-mode activation under mismatched configurations or collapse to direct answers when privileged information supplies only final answers.With student thinking disabled and teacher reasoning enabled, Qwen3-1.7B can imitate control tokens, producing malformed patterns such as <think> ...</think> ...<think>. On DAPO-Math-17k, answer-only hints can make Qwen3-1.7B bypass intermediate reasoning.Source: §A.14 · pp. 22–23, §A.15 · p. 23

A.15 OPSD Response Length Collapse

• The displayed assistant output shows an empty thinking block followed by the boxed answer 7, with no intervening reasoning or explanation accompanying the answer in the provided passage.Source: §A.15 · p. 23

A.16 OPD Response Length Explosion - step 700

• An OPD example with a Qwen3-1.7B student distilled from a Qwen3-8B teacher illustrates response length explosion: the unusually verbose response contains 10 occurrences of the word “but”.The response repeatedly reformulates directional step counts and net movement while restating the 10-step constraint.Source: §A.16 · pp. 23–24

A.17 OPD Response Length Explosion - step 1000

• In an OPD example, a Qwen3-1.7B student distilled from a Qwen3-8B teacher exhibits response-length explosion, with the illustrated response containing only the word “maybe”.Source: §A.17 · p. 25

A.18 OPSD Safety Alignment System Prompt

• The OPSD safety system prompt prohibits assistance with harmful or illegal actions, requests polite refusals and safe alternatives, and preserves helpful neutrality for legitimate requests.Safe alternatives are requested when possible; this passage supplies instructions rather than alignment results.Source: §A.18 · p. 25

A.19 OPSD Reasoning Compression System Prompt.

• Reasoning-compression OPSD gives teachers concise, correct, step-by-step instructions, while students receive no conciseness instruction.Teacher instructions prohibit unnecessary elaboration, redundant steps, and restating the problem, and require the final answer on its own “Answer:” line.Source: §A.19 · pp. 25–26

A.20 OPSD Language Style Alignment: Privileged Information Design and Model Outputs

• 1.0 LLM-judge reward accompanies the illustrated Qwen3-4B-Instruct OPSD EmotionBench response, which expresses anger without the student prompt specifying it.Teacher PI specifies anger caused by dismissal of personal opinions. The student conveys it through recalled incidents, frustrated contrasts, and an escalating tone of personal hurt.Source: §A.20 · pp. 26–27
• In the illustrated CharacterBench example, the OPSD-trained Qwen3-4B-Instruct student receives no character profile in its input, and its response has an LLM-judge reward of 1.0.The student recounts reciting poetry at a village market and wandering between villages, castles, and remote mountain wilds.Source: §A.20 · pp. 27, 29
• The OPSD-trained student reproduces the character’s traditionalist poetic stance, medieval framing, and reflective narrative style without accessing the character profile at inference.This example illustrates transfer from the teacher’s profile-conditioned distribution into the student’s profile-free behavior.Source: §A.20 · p. 30

A.21 The correlation between entropy and teacher supervision

• Token entropy and ∆logprob show only a mild negative trend, providing no strong evidence of robust correlation in OP(S)D.The entropy-guidance investigation was inspired by.Source: §A.21 · p. 30

A.22 Experimental Evidence for Student Prefixes Distorting the Teacher’s Reasoning State

• 16.16 points of GPQA-Diamond accuracy are lost when the teacher continues student-generated prefixes: 62.12% standalone versus 45.96% prefix-conditioned.The experiment uses Qwen3-14B as teacher and Qwen3-1.7B as student; prefixes turn 40 correct predictions wrong and only 8 wrong predictions correct. Format correctness falls from 98.48% to 78.79%, showing that student conditioning can weaken the teacher signal used in OPD.Source: §A.22 · pp. 30–31

Data Preparation

• Data preparation separates boxed-answer OpenThoughts[22] prompts into disjoint SFT and OPD splits, with quality-filtered Qwen3-4B demonstrations for warm-up.The first 30,000 prompts supply at most 20,000 demonstrations; remaining prompts support OPD. Filtering removes incorrect answers, missing boxed answers, and repetitive responses.Source: Data Preparation · p. 31

SFT Stage

• Qwen3-1.7B-Base undergoes SFT using ShareGPT formatting without a system prompt, with thinking disabled to match the nonthinking teacher’s output format.Training uses four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs with DeepSpeed ZeRO Stage 2.Source: SFT Stage · p. 31

A.23.1 The PPL and NLL statistics.

• 0.640 to 0.335 is the NLL reduction after SFT for Qwen3-1.7B-Base on Qwen3-4B traces; PPL falls from 1.896 to 1.397.Lower NLL and PPL indicate that teacher-generated traces are closer to the student’s distribution.Source: Table 3 · p. 32, §A.23.1 · p. 32