A Survey of On-Policy Distillation for Large Language Models

• Large language models’ rising capability and cost make transferring frontier reasoning into smaller deployable students a central pipeline challenge, exemplified by DeepSeek-R1 distilling a 671B teacher into 1.5B–70B students.Knowledge distillation has expanded from same-architecture compression to capability transfer across model scales and architectural families.Source: §1 · p. 1
• Off-policy distillation trains on flawless teacher prefixes, but student-generated inference states create compounding error of O(ϵT^2), especially when early mistakes derail long reasoning trajectories.This mismatch can produce students that reproduce surface style without acquiring underlying reasoning competence.Source: §1 · pp. 1–2
• OPD samples the student’s evolving trajectories and applies feedback ranging from white-box token distributions to black-box rewards, preferences, or teacher-free contrastive signals, targeting O(ϵT) compounding.The training distribution therefore co-evolves with the model, correcting deployment-relevant distribution gaps during training rather than at inference.Source: §1 · p. 2
• The literature now spans divergence design, KL-constrained reinforcement learning, self-play, multi-teacher debate, agentic trajectories, and cross-modal transfer, with OPD adopted in systems including Qwen3, DeepSeek-V4, Gemma 2, and MiMo-V2-Flash.DeepSeek-V4 notably replaced its mixed RL stage with pure multi-teacher OPD for model consolidation.Source: §1 · p. 2
• The survey addresses fragmented treatment across knowledge distillation, RLHF, and interactive imitation learning by presenting OPD as a distinct paradigm with a unified f-divergence framework over student-sampled trajectories.Its contributions also include a three-axis taxonomy, a consolidated account of failure conditions, and cross-regime comparisons spanning white-box, black-box, and self-distillation.Source: §1 · pp. 2–3
• The survey covers methods where students generate training data under teacher or verifier supervision, including adjacent imitation-learning, online-RL, and preference-optimization work, while excluding generic off-policy KD and weight-preserving inference methods.The paper proceeds from mathematical foundations through method selection, design axes, failure analysis, industrial deployments, and open problems.Source: §1 · p. 3

2.1 Classical Knowledge Distillation

• Classical KD and Seq-KD rely on static or teacher-generated prefixes, with Seq-KD collapsing the teacher distribution to a beam-search point and discarding uncertainty.Seq-KD remains computationally tractable but is disconnected from the student’s generation behavior, motivating on-policy objectives.Source: §2.1 · pp. 4–5

2.2 Off-Policy Exposure Bias

• Off-policy training can amplify per-step error to O(ϵT2), whereas student-generated training exposes error states and supports recovery strategies.A 10-step proof with 95% per-step accuracy has approximately 60% trajectory-level correctness under independent errors; the DAgger bound predicts strictly worse compounding.Source: §2.2 · pp. 5–6

2.3 The Unified f-Divergence Framework

• The unified framework separates trajectory sampling πmix from f-divergence choice, enabling mode-covering, mode-seeking, or intermediate objectives under student-generated samples.Forward KL favors diversity, Reverse KL concentrates on teacher modes, and JSD interpolates between these behaviors; GKD, MiniLLM, and DistiLLM instantiate different tradeoffs.Source: §2.3 · pp. 6–7
• GKD reports that on-policy sampling λ = 1 outperforms off-policy sampling across tested divergences, while JSD performs best on translation tasks.On summarization and instruction-following, the three divergences achieve comparable results, suggesting sampling policy can matter more than divergence choice when task geometry is not extreme.Source: §2.3 · p. 7

2.4 Distillation Scaling Laws

• Distillation scaling differs from pre-training because compute is divided among teacher inference, student rollouts, and updates, while teacher overcapacity can create a student capacity gap.No framework yet jointly predicts quality from teacher size, student size, data, and rollout budget; rollout cost scales linearly with R while benefits may diminish.Source: §2.4 · pp. 7–8

3.1 Method Landscape

• OPD methods are classified along three axes: objective function, supervisory signal source, and training-dynamics stabilization.Signal sources distinguish external teachers from self-distillation, while external teachers are further separated by white-box versus black-box access; Figure 1 maps representative methods across the taxonomy.Source: §3.1 · p. 8, Figure 1 · p. 9
• The axes constrain feasible combinations: exact token-level forward KL requires white-box teacher distributions, while black-box access favors sequence-level surrogates and sparse-feedback methods.Sampling policies, curricula, interpolation, and token weighting govern training dynamics, and objective choices can reinforce or restrict available signal sources.Source: §3.1 · pp. 8, 10

3.2 Method Comparison Table

• Self-distillation is the largest and fastest-growing OPD category as of early 2026, reflecting a shift from KL-direction debates toward self-improvement and adaptive alternatives.The field’s emphasis moved from objectives in 2023–2024 to self-distillation by mid-2025 and training dynamics in late 2025–2026.Source: §3.2 · p. 10, §3.1 · p. 10

3.3 Method Selection Considerations

• Successful deployments generally make deliberate choices across all three stages rather than optimizing one axis in isolation.The interactions among objectives, signal sources, and dynamics substantially reduce the space of viable configurations.Source: §3.3 · p. 11, Table 2 · p. 11
• Teacher access determines feasible methods, with white-box logits supporting exact divergences, API-only access favoring text- or score-based supervision, and no external teacher motivating privileged or verifier-guided self-distillation.Examples include DistiLLM and G-OPD for white-box access, OVD, GAD, and LUFFY for sparse feedback, and OPSD, GATES, and SD-ZERO without an external teacher.Source: §3.3 · p. 11
• Selection should match task, compute, and stability needs: reasoning favors mode-seeking objectives, open-ended generation favors mode-covering objectives, constrained compute favors truncated or cached rollouts, and known failures require targeted stabilization.Stabilizers include reference constraints and rollout mixing for length inflation, reliability filtering for flawed prefixes, and off-policy cold starts for thinking-pattern mismatch; deployments above 5000 GPU-hours typically use multi-stage pipelines.Source: §3.3 · pp. 11–12

4.1 Fixed Divergence Objectives

• Moderate on-policy mixing (λ ≥0.5) outperforms pure off-policy training, while source-aware asymmetric divergences improve over symmetric baselines by matching divergence direction to data origin.GKD interpolates between off-policy and on-policy trajectories with Forward KL, Reverse KL, or JSD; DistiLLM-2 uses Forward SKL on teacher data and Reverse SRKL on student data.Source: §4.1 · pp. 12–13
• AntiSD reaches GRPO-level accuracy in 2–10× fewer steps and exceeds GRPO by up to 11.5 points on math reasoning, while vOPD averages +3% over vanilla single-sample OPD and cuts wall-clock time by up to 57.7%.These methods redesign fixed-divergence optimization through divergence ascent, entropy gating, control variates, and top-k value baselines; sequence-level methods trade lower bias for higher variance, while full-vocabulary KL improves stability when infrastructure permits.Source: §4.1 · pp. 14–15

4.2 Adaptive Divergence Objectives

• Adaptive objectives select divergence or gradient behavior per position, with EOPD outperforming fixed baselines and AOPD gaining +4.09 with strong initialization and +8.34 with weak initialization over standard on-policy baselines.EOPD routes by teacher entropy, whereas AOPD preserves exploitation for positive advantages and applies top-K forward-KL guidance to non-positive advantages, addressing variance, vanishing gradients, and exploration black holes.Source: §4.2 · pp. 16–17

4.3 RL-Augmented Objectives

• RL-augmented objectives can exceed teacher behavior: ExOPD produces a unified student surpassing all same-size domain teachers, while RLKD surpasses standard SFT-RL using only 0.1% of the data under pure RL.G-OPD frames OPD as KL-constrained RL with reward extrapolation; KD+RL combines dense imitation with reward maximization, REOPOLD reports 6.7–12× greater sample efficiency and lets a 7B student match a 32B teacher, and CoDistill-GRPO reaches 32% on Minerva, 11.6 points above its base model.Source: §4.3 · pp. 18–19
• 79.3% vs. 75.9% on MATH and 25.2 vs. 19.8 on AIME 2024 show that distilling an RL-improved teacher through a dense bridge outperforms direct student-side GRPO at fixed Qwen3-1.7B size.The bridge also raises subsequent student-side GRPO performance on MATH from 75.4% to 78.5%.Source: §4.3 · p. 20
• Joint KD+RL methods mitigate interference by coupling teacher guidance with policy optimization, progressing from KDRL’s KL regularization to CMDP-KD’s hard budget and RLAD’s selective TRRD trust region.TGPO embeds teacher likelihood-ratio guidance directly into the policy gradient, avoiding the balancing hyperparameters required by additive KD+RL objectives; dGRPO instead adds fixed-teacher token guidance to GRPO for long-context reasoning up to 128K tokens.Source: §4.3 · pp. 19–20
• Token-level KL, step-level structural matching, and sequence-level preference ranking form a common KL-constrained optimization spectrum, selected according to signal availability and teacher-expertise granularity.DPO-style methods are useful when preference pairs are cheaper than full teacher logits, while PBSD uses a reward-tilted context-augmented teacher target that can improve over the teacher under a non-degenerate reward.Source: §4.3 · pp. 20–21
• Fine-grained credit assignment concentrates supervision on causal failures: SuperCorrect localizes divergent reasoning steps, SCoRe corrects only the earliest error, and StepOPSD aligns step-level scoring with token-level distillation in multi-turn agents.AMR-SD stabilizes self-distillation with meta-reflective per-token supervision, while OPPO recursively estimates per-token advantages without a separate critic network.Source: §4.3 · pp. 21–22
• Other RL-integrated designs weight distillation by verifier rewards, mask judge distillation to self-evaluation tokens, and use correct-versus-wrong teacher hints to reinforce or suppress reasoning strategies.RWOPD preserves partial-success gradients in NL-to-SystemVerilog generation; RLCSD adds contrastive reverse-KL guidance without a discriminator or preference-pair construction.Source: §4.3 · p. 22
• The design space progresses from fixed divergences, to adaptive divergences, to higher-variance RL augmentation and environment reconstruction, with the choice depending on teacher quality, task demands, and access to the teacher’s training process.X-KD reconstructs the teacher’s latent reward landscape, while TSD-KD combines preference reranking with selective KL matching at high-entropy tokens.Source: §4.3 · p. 22

5 Signal Source and Teacher Architecture

• OPD classifies signal sources by teacher identity and access level, separating external from self-distillation and white-box from black-box supervision.These dimensions impose different objective constraints and performance trade-offs, while increasing autonomy generally reduces signal density.Source: §5 · pp. 22–23

5.1.1 White-Box Logit Supervision

• White-box innovations extend OPD beyond shared-family logits through tokenizer alignment, multi-token matching, representation transfer, and teacher-signal selection, enabling heterogeneous teacher-student pairs.SimCT outperforms shared-vocabulary and coarser cross-tokenizer baselines across heterogeneous pairs, while the broader methods remove the historical shared-vocabulary restriction.Source: §5.1.1 · pp. 24–25

5.1.2 Black-Box and API-Constrained Distillation

• Black-box distillation recovers substantial teacher knowledge from text, scores, preferences, adversarial feedback, or rubrics despite lacking full token distributions.Distilling Step-by-Step (Hsieh et al., 2023) showed a 770M-parameter student outperforming a 540B-parameter teacher with over 700× fewer parameters, while ROPD matches or surpasses logit-based OPD with approximately 10× sample efficiency.Source: §5.1.2 · pp. 27–28

5.1 External Teacher Distillation

• External teacher distillation offers the densest supervision through full-vocabulary KL but requires co-hosting, whereas API-based methods trade signal density for deployment flexibility.This creates an unresolved tension because proprietary frontier teachers are often API-only, where dense per-token feedback is hardest to obtain.Source: §5.1 · p. 23, §5.1.2 · p. 29

5.2 Self-Distillation

• Self-distillation removes the external teacher by aligning different views of the same model, including privileged-information, rollout-diversity, and verifier- or environment-anchored signals.This scope excludes game-theoretic self-play methods such as SPIN (Chen et al., 2024) and IRIS (Liao et al., 2026), whose adversarial checkpoint competition targets Nash-equilibrium convergence rather than distributional alignment.Source: §5.2 · p. 29

5.2.1 Privileged Information

• Privileged-information distillation uses training-only context to create a stronger self-teacher, but succeeds only when the signal encodes reusable instance-independent structure rather than an unaggregatable answer.Signals range from ground-truth answers and documents to prompts, masks, critiques, future observations, and structured feedback; the common mechanism remains token-level matching on student rollouts.Source: §5.2.1 · pp. 29–30
• OPSD matches or exceeds GRPO at 4B and 8B while using one rollout instead of eight, but trails GRPO by 0.5 points at 1.7B because meaningful self-rationalization requires sufficient capacity.Its gains are +0.9 average points at 8B and +0.8 at 4B; ground-truth-conditioned supervision also has a difficulty ceiling and can fail on cliff prompts when all rollouts fail.Source: §5.2.1 · p. 30
• PI methods broaden supervision beyond ground truth: consensus gating suppresses uncertain tutors, negative privileged information turns failures into diagnostic playbooks, and behavioral or cross-lingual context transfers reusable capabilities.GATES filters unreliable document-conditioned self-supervision, RESD reports approximately 8× greater sample efficiency than GRPO on low-pass-rate tasks, and COPSD transfers English reasoning across 17 low-resource African languages.Source: §5.2.1 · pp. 31–32
• Privileged signals also support reasoning compression, multimodal grounding, adaptive exposure, sparse token routing, and cross-lingual safety, with CRISP reducing MATH-500 chain-of-thought tokens by 57–59% while improving accuracy by 9–16 points.GUI-SD, TRACE, MSD, and ATESD show that spatial masks, critical-span routing, English reasoning, and controlled teacher exposure can respectively improve dense supervision, OOD behavior, safety transfer, or reasoning performance.Source: §5.2.1 · pp. 31–32
• Corrective mechanisms prevent privileged teachers from reinforcing confident mistakes: PBSD reward-reweights teacher targets, while gated, calibrated, or advantage-conditioned objectives combine dense distillation with outcome direction.PBSD matches OPSD’s token efficiency, surpasses its peak accuracy, and avoids the documented post-peak decline; AR-OPD’s residual oracle correction improves by +2.3 points over full oracle OPD and +7.9 over SFT.Source: §5.2.1 · pp. 32, 33, 35

5.2.2 Pure Self-Distillation

• Pure self-distillation derives supervision from rollout diversity or retrospective contrasts, spanning forgetting resistance, architectural conversion, cross-paradigm transfer, reasoning correction, and multi-turn hindsight without auxiliary inputs.SDFT is fully on-policy and trust-region equivalent; MTP-SD achieves > 3× faster decoding with a typical 3–7% accuracy drop, while AR-to-Diffusion reports 15×–7000× greater training-token efficiency than diffusion-LM pretraining.Source: §5.2.2 · pp. 35–36
• UniSD unifies self-distillation around supervision reliability, representation alignment, and training stability, finding on-policy trajectories, multi-teacher agreement, and EMA stabilization central to effectiveness.Across six benchmarks and six models, UniSD* improves the Qwen2.5-7B base model by +5.4 points and the strongest baseline, GKD, by +2.8 without an external teacher.Source: §5.2.2 · p. 36
• Pure self-distillation reconstructs supervision from rollouts, complementary contexts, or specialist peers, extending across coding, reasoning, instruction-following, multimodal, dialogue, privacy, and cross-domain tasks.OPCoD uses competence-gated peer feedback and achieves mutual Pareto improvement on Qwen3-8B without sacrificing domain specialties.Source: §5.2.2 · p. 37
• Pure self-distillation cannot distinguish confident errors from correct outputs without ground truth or environmental verification, so its gains are bounded by information already present in pretrained parameters.This limitation motivates external verification and internally generated privileged information to overcome saturation.Source: §5.2.2 · p. 37, §5.2.3 · pp. 37–38

5.2.3 External Feedback

• 68.3% average accuracy across eight benchmarks (avg@8) lets SD-ZERO outperform GRPO (64.2%) on Qwen3-4B-Instruct by converting binary verification into dense Generator–Reviser self-supervision.SDPO uses structured errors, while RLTF uses free-form judge critiques; both convert sparse outcomes into token-level corrective targets.Source: §5.2.3 · p. 38
• RESD improves sample efficiency by 8× over GRPO on Amazon/UCSD continual agentic learning, while SRPO raises Qwen3-8B’s five-benchmark average by 3.4% over GRPO and 6.3% over SDPO.RESD turns failed rollouts into retrospective reflections and a persistent playbook; SRPO routes correct samples to GRPO and failures to SDPO-style logit correction.Source: §5.2.3 · p. 38
• RLSD reaches 2× sample efficiency by using self-distillation for update magnitude and verifier feedback for direction, while SDAR gates distillation in multi-turn agents and improves over GRPO by +9.4%, +10.2%, and +7.0% on ALFWorld, WebShop, and SearchQA.SDAR attenuates negative teacher rejections while strengthening positive-gap tokens, addressing unstable multi-turn supervision and imperfect skill retrieval.Source: §5.2.3 · pp. 38–39
• External feedback also supports adaptive information disclosure and co-evolution: PAINT gains +2.1 over OPSD and +2.9 over GRPO, while VPD continuously refines teacher targets through variational co-evolution.PAINT reveals more partial solution information when rollouts drift and concentrates loss on high-mismatch positions; CoPD and VPD replace fixed-teacher distillation with jointly improving teacher–student training.Source: §5.2.3 · p. 40
• Prompt perturbation can substitute for an external safety reward: On-Policy Consistency Training aligns perturbed-prompt responses with a frozen-copy teacher’s canonical-prompt behavior and improves safety with minimal capability degradation.The method supplies dense per-token consistency supervision using only a shared backbone and contrasting prompt conditions.Source: §5.2.3 · pp. 40–41
• Verifier-guided gating improves SG-OPD over base GRPO by +1.98 avg@32 and +7.50 pass@32, while CREDIT suppresses input-generic shortcuts through contrastive pointwise-MI correction.SG-OPD phases teacher sampling and removes sign-inconsistent token gradients; CREDIT targets responses that remain likely under unrelated inputs.Source: §5.2.3 · pp. 39, 41

6 Training Efficiency and Stabilization

• On-policy distillation is several times more expensive than off-policy training and suffers from stale rollouts and gradient SNR collapse on hard prompts.These challenges motivate complementary stabilization and acceleration mechanisms rather than a single optimization change.Source: §6 · pp. 41–42

6.1 Token and Sample Weighting

• Token-, step-, and trajectory-level weighting suppress unreliable teacher signals from flawed prefixes, with reported gains including 19.8% over sampled-token OPD and 7.3% relative Pass@32.TIP recovers overconfident-error tokens, SCOPE routes correct and incorrect rollouts differently, and SOD targets tool-call-induced divergence at reasoning-step boundaries.Source: §6.1 · pp. 42–43

6.2 Curriculum and Difficulty Adaptation

• Curricula improve training efficiency by sampling competence-boundary prompts, controlling supervision depth, and using warmup or freshness mechanisms to prevent uninformative or unstable updates.PACED weights prompts by p(1−p), TCOD reports gains of up to +18 points on multi-turn tasks, and off-policy cold starts reduce thinking-pattern mismatch before on-policy refinement.Source: §6.2 · pp. 45–47

6.3 Compute Optimization

• FOPD matches full OPD performance while reducing training FLOP by 2×–47× by distilling on early rollout prefixes and using off-policy KD for later tokens.A scheduled prefix that grows from 1 token by ΔL = 256 per gradient step closely matches full-OPD quality with lower compute.Source: §6.3 · p. 48
• Lightning-OPD delivers 4.0× higher training efficiency than standard OPD by caching teacher log-probabilities, provided the same teacher is used for SFT and OPD.Under teacher consistency, it shares standard online OPD’s optimum, bounds gradient discrepancy, and implicitly limits policy drift without explicit KL penalties.Source: §6.3 · p. 48
• The methods form a fidelity–efficiency frontier and complementary stack: partial rollouts, stale teacher signals, and asynchronous training trade alignment against cost, while token weighting, curricula, and compute optimization can compound 2–4× gains.The combined stack may reduce OPD’s 4–5× cost gap relative to off-policy training toward parity while retaining its quality-ceiling advantage.Source: §6.3 · pp. 48–50
• DistillSpec improves speculative-decoding speed by 10%–45%, while multi-scale deployments can reduce decoding latency by 6–10× with minimal performance drop.It aligns the draft model with the target distribution, and SKD interpolates between supervised and on-policy KD through teacher verification and selective replacement.Source: §6.3 · p. 48
• NPD achieves 8.1× throughput over synchronous on-policy baselines, improves SFT by +8.09% across 11 benchmarks, and reaches 68.73% accuracy versus Qwen3-1.7B’s 63.69%.Its asynchronous pipeline uses ∆-IFD filtering to limit policy lag, and NPD→GRPO provides stronger initializations for subsequent reinforcement learning.Source: §6.3 · p. 49
• Prune-OPD reduces training time by 37.6–68.0% while preserving or improving AMC, AIME, and HMMT performance through drift-aware reward down-weighting and truncation.Unlike fixed truncation, it expands supervision when compatibility remains high and contracts it when student–teacher drift exceeds a budget.Source: §6.3 · p. 50

7.1 Success Conditions

• OPD benefits require compatible teacher–student reasoning patterns and a teacher capability gap, with the productive regime lying between structural mismatch and negligible transfer.High top-k overlap supports progressive alignment, while same-recipe teachers provide little new signal; cold starts and teacher-aligned prompts can reduce pattern gaps.Source: §7.1 · p. 51
• OPSD primarily compresses correct reasoning traces rather than correcting incorrect ones, so SFT, RLVR, then correct-only OPSD is the preferred ordering for cheaper inference.Correct-only OPSD preserves accuracy while shortening traces, whereas incorrect-only OPSD degrades accuracy; post-RLVR correct-only OPSD reached the most favorable accuracy–length position in the reported experiments.Source: §7.1 · p. 52

7.2 Failure Modes

• OPD fails through flawed prefixes, extrapolation cliffs, persistent Rock Tokens, local teachability collapse, self-play saturation, and calibration or diversity loss, each requiring targeted stabilization or truncation.These mechanisms include out-of-distribution teacher conditioning, clip-unsafe reward extrapolation, structurally uninformative high-loss tokens, vanishing local margins, hypothesis-space collapse, and overconfident distributional narrowing.Source: §7.2 · pp. 52–54
• Agentic OPD additionally requires stable teacher dynamics, explicit trajectory-structure regularization, and granularity-matched credit assignment.TT-OPD reports hard-copy teacher resets causing KL collapse, erosion from coordinated tool use toward monologues, and reward-hint runaway; across settings, these pathologies motivate jointly addressing all three requirements.Source: §7.2 · pp. 54–55

7.3 Unified Theoretical Perspectives

• The unified f-divergence view treats Forward KL, Reverse KL, JSD, and total variation as regularization choices whose head–tail weighting controls coverage–precision tradeoffs.Adaptive divergence selection can outperform fixed choices, while lower teacher entropy sharpens outputs but reduces distributional recall.Source: §7.3 · p. 56
• OPD updates occupy a distinct geometric regime: they rapidly lock into narrow low-dimensional channels, touch relatively few coordinates, and remain closer to RLVR than ordinary SFT.Training only discovered subnetworks nearly recovers full-training performance, although adaptive optimization remains important.Source: §7.3 · p. 56
• OPD, RL, and preference optimization form a continuum governed by supervision density, divergence, and signal source, with hybrid objectives combining dense low-variance correction and reward-driven exploration.Standard OPD can be expressed as dense KL-constrained RL; KD reduces policy-gradient variance, while RL avoids suboptimal teacher-mode collapse and textual feedback supplies denser signals than binary outcomes.Source: §7.3 · p. 57

7.4 On-Policy vs. Off-Policy: A Decision Framework

• Off-policy distillation can suffice when an exceptionally strong teacher supplies diverse traces to a relatively small student: R1-Distill-Qwen-7B reaches 55.5% on AIME 2024 pass@1 and R1-Distill-Qwen-32B reaches 72.6%.These results exceed GRPO on Qwen2.5-32B-Base at 47.0%, while on-policy methods become more attractive as student capacity grows and diverges from the static training distribution.Source: §7.4 · p. 57
• Off-policy methods face a distributional-mismatch ceiling that widens with reasoning depth, output diversity, and student capability, because static data cannot cover the combinatorial space of valid paths.On-policy data remains aligned with the student’s evolving generation distribution, preserving a generalization advantage as the student explores beyond fixed training traces.Source: §7.4 · p. 57
• The choice between distillation and direct RL depends on teacher quality, student scale, and the target ceiling: smaller students favor dense teacher supervision, while larger students can exploit RL exploration.G-OPD, KDRL, and RLAD combine teacher logits as dense variance-reducing supervision with sparse rewards, and have outperformed either approach alone in tested settings (Yang et al., 2026e; Xu et al., 2025a; Zhang et al., 2026o).Source: §7.4 · p. 58
• Hybrid pipelines generally provide the best cost-quality tradeoff by using off-policy SFT to establish a foundation, then OPD or RL to close the remaining distribution gap.Weaker students need longer off-policy warmup, whereas stronger students can transition earlier to on-policy rollouts.Source: §7.4 · p. 58
• On-policy training incurs substantial compute overhead because autoregressive student generation dominates forward cost, so it is most justified for reasoning, code generation, and complex mathematics rather than generic knowledge transfer.Industrial pipelines therefore front-load cheaper off-policy learning and reserve expensive on-policy computation for the final quality push.Source: §7.4 · p. 58
• Use off-policy SFT alone for bounded-depth tasks when the teacher is more than 10× larger; switch to OPD for larger students, multi-step reasoning, or stalled off-policy loss despite improving on-policy reward.Otherwise, off-policy warmup followed by on-policy refinement is the recommended default.Source: §7.4 · p. 58
• Operationally, smaller students often benefit from dense logit supervision, whereas larger students with reliable verifiers benefit from reward-guided OPD that can discover behavior beyond the teacher ceiling.OPD and RLHF share on-policy rollouts, trajectory-level signals, and policy-gradient updates; reward-guided OPD can therefore reuse RLHF infrastructure such as OpenRLHF (Hu et al., 2024), while white-box OPD additionally requires teacher co-hosting and full-vocabulary logit transfer.Source: §7.4 · pp. 58–59

8.1 Industrial Deployment

• TCOD gains up to +18 points over vanilla multi-turn OPD, OPAD lifts the agentic average by +2.4% over single-teacher OPD, and Skill-SD exceeds GRPO by +14.0% on AppWorld and +10.9% on Sokoban.TCOD controls temporal depth of teacher supervision, while Skill-SD uses importance-weighted Reverse KL with dynamic teacher synchronization for skill-conditioned learning.Source: §8.1 · pp. 60–61
• DAgger-style turn-level interpolation improves Qwen3-4B and Qwen3-8B over pure SFT by +3.9 percentage points on SWE-bench Verified.The method mixes student- and teacher-generated turns before supervised learning on the full trajectory.Source: §8.1 · p. 61
• Industrial OPD now supports five deployment patterns: quality refinement, expert consolidation, multi-budget control, difficult capability domains, and safety evolution.These patterns use on-policy generation rather than static dataset distillation.Source: §8.1 · p. 61

8.2 Emerging Domains

• Multimodal OPD succeeds through cross-modal transfer, self-alignment, bootstrapping, structured temporal supervision, and efficiency-aware agentic feedback, with HyperEyes-30B gaining 9.9% accuracy and reducing tool-call rounds by 5.3×.VOLD requires SFT cold-start alignment, while KEPO addresses near-zero solve rates and VISD reports approximately 2× faster convergence than RLVR baselines.Source: §8.2 · pp. 62–63
• Embodied applications show rising importance for on-policy generation as errors become physically consequential: HY-Embodied-0.5 outperforms similarly sized models on 16 out of 22 benchmarks, while VLA-OPD improves sample efficiency and robustness.Distributional mismatch escalates from benchmark degradation to unsafe physical trajectories, motivating simulator-integrated rollout data.Source: §8.2 · pp. 63–64
• Domain-specific OPD extends to clinical reasoning, protein design, formal verification, speculative decoding, and privacy, including TT-OPD’s best score on 10 of 18 benchmarks and TTS’s 41% average acceptance-length gain.These adaptations redefine privileged information, reward, and trajectory interfaces while retaining the on-policy optimization loop.Source: §8.2 · pp. 64–65

8.3 System-Level Integration

• Production OPD separates rollout generation, teacher scoring, and student updates across specialized parallelism and serving systems, but stale rollouts and full-vocabulary logits create synchronization and communication costs.A 70B teacher with a 7B student on 8×H100 requires approximately 16 GB of logit data per batch under the supplied configuration.Source: §8.3 · p. 65
• OPD-specific systems must support teacher co-hosting, logit-tensor transfer, and staleness tolerance, motivating separate rollout, teacher-scoring, and gradient-update process groups with synchronization barriers.At trillion-parameter scale, emerging frameworks additionally require hidden-state caching, on-the-fly logit reconstruction, and teacher-aware batch scheduling.Source: §8.3 · p. 66
• Industrial pipelines commonly allocate compute across off-policy warm-up, on-policy logit distillation, and reward-guided refinement, front-loading cheap learning and reserving expensive on-policy computation for final quality gains.Warm-up establishes basic capabilities and narrows the student–teacher distribution gap; OPD addresses exposure bias on student trajectories, while reward-guided objectives seek improvements beyond the teacher’s capability ceiling.Source: §8.3 · p. 66
• Qwen3’s staged pipeline uses distillation from larger models as its primary transfer mechanism for smaller variants, achieving comparable performance to RL at roughly 1/10 the GPU cost.The pipeline interleaves cold-start SFT, reasoning RL, thinking-mode fusion SFT, and general RL, using 32B and 235B-A22B teachers.Source: §8.3 · pp. 66–67
• On-policy training is most effective after off-policy warm-up brings the student close to the teacher’s distribution, reducing wasted rollouts caused by a large initial gap.This ordering explains why staged industrial pipelines place standard SFT before on-policy distillation.Source: §8.3 · pp. 66–67

Distillation scaling laws

• On-policy distillation lacks validated scaling laws that jointly characterize student size, teacher size, and rollout budget, including teacher-diversity and capacity-gap effects.The proposed joint power-law is explicitly speculative; controlled grid searches could determine whether additional rollouts compensate for weaker teachers or whether teacher quality imposes a ceiling.Source: §9 · p. 67

Uncertainty-aware feedback

• Uncertainty-aware feedback should discount unreliable teacher signals, suppressing gradients on uncertain positions and eventually predicting unreliable reasoning paths before costly exploration.Process reward models can score reasoning steps, while TIP (Xu et al., 2026c) indicates relative teacher surprise may be more informative than absolute entropy.Source: §9 · pp. 67–68

Agent-level, continual, and lifelong distillation

• Agent-level and lifelong distillation require credit assignment across non-stationary environments, structured tool calls, safety-critical actions, and evolving teacher distributions without catastrophic forgetting or drift.MAD-OPD’s step-level sampling and TCOD’s temporal curriculum suggest temporal decomposition and progressive exposure, but trajectory-depth degradation remains unresolved.Source: §9 · p. 68

Efficiency frontiers

• Efficiency research must identify on-policy data lower bounds and selective teacher inference schemes that prioritize informative student rollouts while reducing expensive feedback.MiniPLM provides off-policy motivation for transferring discrepancy-based Difference Sampling to on-policy training.Source: §9 · p. 68

Latent-space and cross-modal distillation

• Latent-space and cross-modal OPD could bypass vocabulary bottlenecks, but divergence choices, multimodal exposure bias, and continuous-signal foundations remain unresolved.VOLD (Bousselham et al., 2025), X-OPD (Cao et al., 2026), and CORD (Hu et al., 2026) provide empirical starting points without an established theory.Source: §9 · p. 69

Privacy and evaluation methodology

• Privacy-preserving OPD and evaluation need stronger foundations because DP-OPD must scale to reasoning models, while benchmark success can conceal distribution-shift failures and superficial correlations.The precision-recall framework of Cha & Cho (2025) motivates testing causal mechanisms, distributional coverage, and generalization rather than token-level benchmark performance alone.Source: §9 · p. 69

Diagnostic tools and cross-architecture scalability

• OPD needs proactive diagnostics for gradient noise, representation collapse, divergence dynamics, self-distillation saturation, and trajectory-structure erosion during training.Such probes would replace post-hoc benchmark debugging with mid-training intervention while avoiding the full cost of repeated forward passes.Source: §9 · p. 69

Unifying distillation, reinforcement learning, and self-play

• The deepest open question is how distillation and reinforcement learning should dynamically share computation while escaping pure self-distillation’s ceiling through scalable verification and self-play hybrids.G-OPD (Yang et al., 2026e), KDRL (Xu et al., 2025a), RLAD (Zhang et al., 2026o), CoPD (Gu et al., 2026), π-Play (Zhang et al., 2026l), SPIN (Chen et al., 2024), and IRIS (Liao et al., 2026) motivate dynamic schedules, co-evolution, and game-theoretic convergence mechanisms.Source: §9 · pp. 69–70

Survey Organization

• The survey organizes OPD along objective design, signal-source architecture, and training dynamics, spanning fixed-to-adaptive divergences, logits-to-self-distillation, and weighting-to-compute optimization.This framework synthesizes foundational formulations, algorithmic innovations, theoretical analyses, diagnostics, and deployments.Source: §10 · p. 70

Key Findings

• On-policy training commonly improves accuracy across reasoning, code, and instruction following by reducing exposure bias, especially on longer reasoning chains.The student learns to recover from its own errors rather than imitate idealized trajectories.Source: §10 · p. 70
• Divergence selection benefits from task- and position-adaptive choices, while dense teacher supervision combined with sparse rewards narrows the boundary between distillation and reinforcement learning.Reverse KL favors mode-seeking reasoning, Forward KL preserves open-ended diversity, and RL can explore beyond teacher capabilities while KD stabilizes gradients.Source: §10 · p. 70

Practical Takeaways

• White-box settings favor token-level methods for general instruction following and sequence-level or hybrid methods for reasoning, while API-only and self-distillation methods address restricted teacher access.Speculative decoding, prefix truncation, and offline precomputation can reduce on-policy generation overhead.Source: §10 · p. 71

Looking Ahead and Broader Impact

• Future OPD must address missing scaling laws, teacher uncertainty, lifelong adaptation, and safety-aware training, while balancing 4–5× higher training compute against lower lifetime inference costs.MSD and Safactory illustrate multilingual safety alignment and recurring closed-loop safety evolution.Source: §10 · pp. 71–72

Limitations and Reproducibility

• The survey’s scope is limited to student-generated training trajectories, and rapid publication plus inconsistent models, budgets, benchmarks, and rollout counts make cross-method comparisons unreliable.It advocates standardized protocols fixing the base model, compute budget, and evaluation suite.Source: §10 · p. 71