Source-linked AI summary
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, Ning Ding
TL;DR
OPD is widely used for LLM post-training, but its training dynamics and failure conditions remain poorly understood. The paper systematically studies these dynamics and mechanisms, finding that success requires compatible thinking patterns and genuinely new teacher knowledge, while effective alignment concentrates on shared high-probability tokens. It also proposes recovery strategies and identifies reward degradation over long trajectories as a practical limitation.
Problem
OPD is widely used for LLM post-training, yet its token-level mechanisms and conditions for success or failure remain poorly understood.
Method
The paper systematically analyzes OPD phenomenology, token-level training dynamics, practical recovery strategies, and dense-reward behavior across trajectory depth.
Results
Successful OPD requires compatible thinking patterns and genuinely new teacher knowledge, with progressive alignment on shared high-probability tokens concentrating 97–99% of combined probability mass.
Takeaways & Limitations
Off-policy cold start and teacher-aligned prompt selection can recover failing OPD by improving alignment dynamics.
Takeaways & Limitations
Reward quality degrades with trajectory depth, limiting current OPD for long-horizon reasoning and agentic settings.
Abstract
from arXiv · showhide
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.
1. Introduction
OPD is widely used for LLM post-training but remains fragile and poorly understood. This study identifies conditions for success, characterizes token-level dynamics, proposes recovery strategies, and exposes limitations for long-horizon distillation.
- Motivation: OPD has become a core LLM post-training technique and a practical alternative to conventional SFT and outcome-reward RL.Recent industry efforts report substantial gains, while an independent replication finds the recipe efficient in RL compute.
- Phenomenology: Two conditions govern OPD effectiveness: compatible student-teacher thinking patterns and genuinely new teacher knowledge beyond the student’s training exposure.Higher teacher benchmark scores alone do not guarantee useful transferable signal.
- Mechanism: Successful OPD progressively aligns student and teacher distributions on student-visited states, with overlap rising from 72% to 91%.The entropy gap narrows as high-probability tokens increasingly coincide.
- Mechanism: The shared top-k tokens concentrate 97–99% of the combined probability mass, and overlap-token supervision alone matches full top-k performance.This identifies the overlap set as the principal locus of OPD’s gradient signal.
- Practical recipe: Two strategies recover otherwise failing OPD: off-policy cold start and teacher-aligned prompt selection.Cold start bridges the thinking-pattern gap, while teacher-aligned prompts sharpen alignment but may require out-of-distribution mixing to avoid entropy reduction.
- Limitations: Reward quality degrades with trajectory depth, revealing a tension between dense supervision and reliability for long-horizon reasoning and agentic settings.Instability begins at later tokens and propagates backward through the trajectory.
2. Preliminaries
OPD distills a teacher into a student using student-generated trajectories and token-level distribution matching. The framework includes sampled-token, full-vocabulary, and top-k variants, alongside metrics that track candidate-space and distributional alignment.
- On-Policy Distillation: OPD evaluates teacher and student next-token distributions on prefixes from responses sampled by the student.At each student-generated prefix, it obtains the student distribution p_t and teacher distribution q_t over the vocabulary.
- On-Policy Distillation: The sequence-level reverse-KL objective decomposes exactly into token-level terms along autoregressive student trajectories.This decomposition supports implementations that compute supervision at different granularities.
- Supervision Granularities: Sampled-token OPD uses the student-sampled token as an unbiased single-sample estimator of each token-level reverse KL.It is lightweight because the per-token loss evaluates only the sampled token.
- Supervision Granularities: Full-vocabulary OPD computes divergence over the entire vocabulary, providing denser gradients at O(BTM) memory cost.Here B is batch size, T sequence length, and M=|V| vocabulary size.
- Supervision Granularities: Top-k OPD restricts divergence to the student’s highest-probability tokens, reducing teacher-query cost while approximating full-vocabulary reverse KL.The subset formulation discards probability mass outside the selected token set but retains multi-token supervision in the student’s high-probability region.
- Dynamic Metrics: Dynamic metrics measure top-k overlap, agreement within shared tokens, and entropy-gap convergence on student rollouts.Overlap nearing 1.0 indicates shared support, overlap-token advantage near zero indicates aligned confidence, and a shrinking entropy gap indicates matched uncertainty profiles.
3. Phenomenology of On-Policy Distillation
OPD effectiveness depends on compatible student–teacher thinking patterns and genuinely new teacher capabilities, not benchmark strength alone. Controlled comparisons and reverse distillation show that aligned, novel capabilities transfer more effectively.
- Governing Conditions: Two conditions govern OPD: compatible thinking patterns and teacher knowledge beyond what the student has already seen.A stronger teacher can fail when reasoning patterns mismatch or when higher scores reflect no genuinely new capability.
- Thinking-Pattern Consistency: Higher initial overlap with the student corresponds to stronger OPD, even when teachers have broadly comparable benchmark performance.The GRPO teacher outperforms the non-thinking teacher in distillation while showing higher initial overlap.
- New Knowledge, Not Just Scale: Post-trained teachers produce substantially stronger gains and higher gap recovery rates than same-pipeline teachers across DeepSeek and Qwen families.Because the teachers derive from the same base checkpoints, the result is attributed to capabilities acquired through additional RL rather than reduced pattern compatibility.
- Validation via Reverse Distillation: Reverse distillation makes JustRL-1.5B regress to its pre-RL level with either same-family teacher, despite R1-Distill-7B scoring higher.The nearly identical trajectories imply similar local target distributions on student-visited states.
- Conclusions: The findings indicate that OPD learns teacher thinking patterns and requires genuinely new knowledge for improvement.Thinking-pattern consistency is associated with higher initial overlap, while further post-training enables larger transferable gains when overlap is already high.
4. Mechanism of On-Policy Distillation
Successful OPD progressively aligns student and teacher distributions on high-probability tokens at student-visited states. Ablations show that this shared region is both the signature and operative locus of optimization.
- Training Dynamics: Successful distillation improves overlap-token advantage toward zero, whereas failing runs retain weak gradients and persistent probability discrepancies.These dynamics distinguish improvement from failure across the compared teacher–student runs.
- Optimization Support: Optimizing only the overlap Top-k region nearly recovers the full benefit of standard Student Top-k OPD on all three benchmarks.Non-Overlap Top-k remains consistently weaker, indicating that non-overlap tokens contribute little to the main gains.
- Optimization Support: Overlap optimization is self-reinforcing: overlap rises from about 72% to above 91% as reverse-KL updates displace competing non-overlap tokens.The shared high-probability region expands because optimization concentrates probability on teacher-supported tokens.
- Unified Mechanism: The unified mechanism is progressive refinement of the student distribution over teacher-supported high-probability tokens at student-visited states.This alignment is both the observed signature of success and the region where restricting optimization remains effective.
5. Practical Recipe
Two training-design strategies recover OPD when student–teacher alignment is poor: teacher-rollout cold starts and teacher-aligned prompts. Prompt alignment improves supervision but can suppress student entropy, requiring mixing with out-of-distribution prompts.
- Off-Policy Cold Start: Off-policy cold start uses SFT on teacher-generated rollouts before OPD to close the initial thinking-pattern gap.The SFT-initialized student begins with higher overlap and a closer confidence profile to the teacher.
- Off-Policy Cold Start: SFT+OPD consistently outperforms pure OPD, and its performance advantage persists throughout training.The cold start improves both early optimization and the final performance ceiling of subsequent OPD.
- Teacher-Aligned Prompts: Teacher-aligned prompt templates improve validation performance and produce higher overlap throughout training.The aligned template changes task presentation while keeping the underlying math problems unchanged.
- Teacher-Aligned Prompts: Teacher-aligned prompt contents can concentrate more student probability mass on fewer shared tokens despite a lower overlap ratio.This yields stronger effective alignment on high-probability tokens.
- Teacher-Aligned Prompts: Teacher-aligned prompts substantially lower student entropy, so robust OPD should mix them with out-of-distribution prompts.The mixture is proposed to preserve entropy and maintain exploration capacity.
6. Discussion
The discussion finds that OPD’s dense token-level supervision is effective on moderate-length traces but degrades as trajectories lengthen, while globally informative rewards may remain locally difficult to optimize. It also shows that sampled-token OPD performs well without large support sets, provided the biased Top-1 setting is avoided.
- Response-length tradeoff: Moderate response lengths of 3K and 7K yield the strongest OPD results, while very short responses underprovide supervision and 10K–15K responses plateau or decline.Training dynamics show smoother overlap growth at moderate lengths.
- Response-length tradeoff: Later-token instability propagates backward through long responses, with entropy first rising at the suffix before spreading toward earlier positions.This pattern appears in the 15K setting during training.
- Response-length tradeoff: Dense reward reliability declines with trajectory depth as student prefixes drift farther from states familiar to the teacher, limiting longer-horizon OPD.The paper identifies extended chain-of-thought and agentic multi-turn interaction as potential boundary cases.
- Global versus local reward: Both successful and failing teachers assign higher sequence mean reward to correct rollouts, with comparable AUROC values of 0.73 and 0.75.Thus, the failing teacher retains a globally informative correctness signal.
- Global versus local reward: The authors hypothesize that large per-token advantages can coexist with small gradients when reward structure is anisotropic across sequence positions, but they leave this unverified.They frame locally exploitable reward geometry as an open question.
- Top-k OPD: Sampled-token OPD matches larger-support Top-k settings on average, whereas Top-1 underperforms because argmax selection concentrates reward on one unstable mode.Increasing k beyond 4 adds negligible gain while increasing computational overhead.
7. Related Work
Related work situates OPD as a response to exposure bias in conventional distillation and reviews evidence that capacity gaps can harm distillation. It emphasizes that capacity-gap and distillability issues remain underexplored specifically for OPD.
- Knowledge distillation: Conventional off-policy distillation trains on teacher or reference sequences, creating train-inference distribution mismatch and exposure bias during student generation.This motivates shifting distillation toward the student’s own on-policy distribution.
- On-policy distillation: MiniLLM formalized OPD for LLMs with reverse-KL policy optimization, while later work unified on-policy and off-policy data across divergences.The cited literature reports gains over other distillation baselines.
- Capacity gap and distillability: Prior distillation studies show that large teacher-student capacity gaps can degrade or reverse benefits, motivating intermediate teacher assistants and scaling-law analyses.These findings provide context for OPD’s teacher-size sensitivity.
- Capacity gap and distillability: Capacity-gap and distillability issues remain underexplored in on-policy distillation.
8. Conclusion and Future Work
The paper concludes that OPD success depends on thinking-pattern consistency and genuinely new teacher knowledge, with two recovery strategies for mismatched settings. It also identifies reward degradation over long trajectories and several directions for extending the analysis.
- Conclusion: OPD success is governed by thinking-pattern consistency and genuinely new knowledge beyond what the student has already seen.When these conditions fail, the paper proposes off-policy cold starts and teacher-aligned prompt selection.
- Conclusion: The study identifies reward degradation over long trajectories as a practical ceiling for OPD.
- Future work: All experiments use mathematical benchmarks, leaving the applicability of these OPD conditions and mechanisms to code and open-ended domains unresolved.
- Future work: Future work proposes isolating pre-training-data effects, extending the analysis to self-distillation, and studying long-horizon and agentic settings.
A.3. Benchmark-wise breakdown of thinking-pattern compatibility
The benchmark-wise analysis shows that the advantage of Qwen3-4B-Base-GRPO distillation is broadly distributed across AIME 2024, AIME 2025, and AMC 2023. Overlap tokens also carry nearly all probability mass for both student and teacher distributions.
- Benchmark-wise breakdown: The cross-benchmark pattern supports the interpretation that better early thinking-pattern compatibility improves downstream distillation performance.The paper states that early mismatch losses are not fully recovered later in training.
- Benchmark-wise breakdown: Qwen3-4B-Base-GRPO distillation consistently matches or outperforms Qwen3-4B Non-thinking distillation across AIME 2024, AIME 2025, and AMC 2023.The advantage is more pronounced on AMC 2023 and AIME 2024, and smaller but generally present on AIME 2025.
- Overlap probability mass: 97%–99% of both student and teacher probability mass lies on their shared top-k overlap tokens throughout training.This indicates that the overlap dominates the probability distributions, not merely the token sets.
B.2. Auxiliary Optimization Dynamics
Auxiliary diagnostics distinguish successful from failing OPD through the strength and usefulness of the optimization signal. Successful runs reduce mismatch, maintain stronger gradients, and progressively correct high-advantage token discrepancies, while failing runs do not.
- Optimization diagnostics: Successful OPD steadily reduces training loss, whereas the failing R1-Distill-7B run changes only modestly despite starting with a smaller mismatch.The successful JustRL-1.5B run begins with a larger mismatch but decreases steadily before flattening at a low value.
- Optimization diagnostics: Successful OPD maintains a substantial gradient norm, while the failing run receives a consistently weaker update signal.The stronger sustained gradient indicates that the student continues receiving a meaningful corrective signal.
- Optimization diagnostics: Successful OPD reduces the probability discrepancy on the token with the largest absolute advantage, whereas failing runs retain a larger gap.This tracks whether the student corrects the local disagreement carrying the strongest teacher-induced optimization signal.
- Interpretation: The auxiliary results reinforce that useful OPD learning is concentrated on high-probability tokens at student-visited states.Successful training combines increasing overlap with gradients strong enough to reduce important local distributional mismatches.
- Cross-model validation: Across model pairs, successful distillation coincides with rising high-probability-token alignment, while stagnating runs show poor or unstable alignment.With Skywork-OR1-Math-7B, performance improves alongside increasing overlap ratio, near-zero overlap-token advantage, and a small entropy gap; R1-Distill-14B shows little improvement.
C.2. Additional Analysis of Overlap Mass
Overlap mass analysis shows why overlap-token advantage alone can misrepresent alignment quality. Teacher-aligned prompt templates improve benchmark performance and teacher-performance recovery by producing more compatible student-generated states.
- Overlap mass: SFT-initialized students maintain consistently high student and teacher overlap mass throughout training.The overlap tokens therefore cover most of the important probability mass on both sides.
- Overlap mass: Overlap-token advantage can appear favorable even when the overlap set misses substantial high-probability teacher tokens.Overlap mass complements this metric by measuring whether shared support covers the important parts of both distributions.
- Prompt-template alignment: Teacher-aligned templates consistently improve performance across datasets, with larger gains on both AIME sets and a smaller positive effect on AMC 2023.The student recovers roughly 85% rather than roughly 80% of the teacher’s performance.
- Prompt-template alignment: Teacher-aligned prompt selection improves OPD by making the student’s generated states more compatible with the teacher.The comparison uses in-domain prompts deduplicated against the teacher’s post-training data to isolate template alignment.
D.1. Teacher entropy by output position
Teacher entropy increases first at later decoding positions and then propagates toward earlier output tokens during OPD training.
- Teacher entropy: Teacher entropy first rises in the suffix before progressively spreading toward earlier output positions.This pattern is observed across training steps under the 15K maximum response-length setting.