PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of On-Policy Distillation for Large Language Models

Mingyang Song, Mao Zheng

arXiv:2604.00626v4cs.LGcs.CL

TL;DR

Static teacher imitation trains students on idealized prefixes despite their own deployment errors, creating exposure bias that worsens on long reasoning trajectories. This survey formalizes on-policy distillation over student-sampled trajectories, organizes its methods and signal sources, and synthesizes its outcomes, failures, and open scaling and agentic challenges.

  • Problem

    Static teacher-generated-text distillation trains on flawless prefixes while students generate their own histories at inference, making exposure bias particularly problematic for longer reasoning tasks.

  • Method

    The survey formalizes OPD as f-divergence minimization over student-sampled trajectories and organizes methods by objectives, signal sources, and training dynamics.

  • Results

    On-policy training is associated with consistent accuracy gains across mathematical reasoning, code generation, and instruction following, with larger gaps for longer reasoning chains.

  • Takeaways & Limitations

    The synthesis treats OPD as a distinct paradigm linking knowledge distillation, reinforcement learning, and imitation learning while identifying recurring success conditions and failure modes.

  • Takeaways & Limitations

    Multi-turn OPD can collapse tool-use structure or accuracy: reward-hint runaway reduced accuracy from 54.5% to 49.0% in the reported setting.

Abstract

from arXiv · show

As Large Language Models continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become an important engineering problem, and knowledge distillation remains a common technique for this transfer. The prevailing recipe in industrial pipelines, static imitation of teacher-generated text, carries a structural weakness that grows more severe as tasks become longer and more reasoning-intensive. Because the student is trained on flawless teacher prefixes but generates its own at inference, small errors tend to accumulate into trajectories it has rarely been trained to recover from, and the resulting exposure bias has been shown to scale roughly with the square of sequence length. On-Policy Distillation reorganizes the training loop around this observation by having the teacher provide feedback on what the student actually produces, with the goal of reducing the compounding term toward linear and reframing distillation as an iterative correction process rather than single-pass imitation. The resulting literature has expanded along divergence design, reward-guided optimization, and self-play, yet contributions remain scattered across the knowledge distillation, RLHF, and imitation learning communities without a unified treatment. This survey provides such a treatment. We formalize OPD as f-divergence minimization over student-sampled trajectories, organize the field along three design axes (what to optimize, where the signal comes from, and how to stabilize training in practice), and consolidate success conditions, recurring failure modes, and the connection between OPD and KL-constrained reinforcement learning. We close with open problems that emerge from this synthesis, including distillation scaling laws, uncertainty-aware feedback, agent-level distillation, and the growing overlap between knowledge distillation and RL.

1 Introduction

On-policy distillation addresses the exposure bias of static teacher imitation by training on student-generated trajectories. This survey unifies the scattered literature through a theoretical framework, design taxonomy, failure analysis, and explicit scope.

  • Motivation: Static distillation trains on flawless teacher prefixes, whereas deployment follows student-generated histories that may diverge from those training states.This mismatch is especially consequential for autoregressive reasoning and other long-horizon tasks.
  • Motivation: O(ϵT^2) expected trajectory discrepancy captures how per-step imitation error compounds over a horizon T under the learner’s own state visitation.A single early error can derail an entire proof or program.
  • On-Policy Distillation: On-policy distillation shifts training samples to the student’s evolving policy, so teacher feedback targets states the student will actually visit at deployment.Available supervision ranges from full token-level distributions to weaker teacher-access settings.
  • Survey Framework: The survey recasts OPD as f-divergence minimization over student-sampled trajectories and organizes methods by objective, signal source, and training dynamics.This supplies a common vocabulary for methods previously studied across separate communities.
  • Survey Framework: The review consolidates recurring breakdowns including flawed prefixes, self-play saturation, diversity collapse, calibration-capability gaps, and agentic failures.Its stated scope includes student-generated-data methods from distillation, imitation learning, online RL, and preference optimization, while excluding generic off-policy KD and weight-preserving inference methods.

2 Background and Unified Math

The background contrasts dataset-state distillation with student-policy training and develops a unified f-divergence view of OPD. It then connects divergence and sampling choices to task geometry, stability tradeoffs, and unresolved compute scaling.

  • On-Policy Definition: OPD defines training data as samples from the student’s current policy, creating a non-stationary optimization landscape that requires fresh rollouts as parameters change.The rollout cost is a central systems challenge.
  • Off-Policy Exposure Bias: Classical autoregressive KD optimizes next-token matching on static, precomputed prefixes, assuming an error-free history rather than the student’s deployment state distribution.Sequence-level KD reduces intractable sequence KL to teacher-generated outputs but discards uncertainty and alternative modes.
  • Off-Policy Exposure Bias: O(ϵT^2) expected discrepancy follows when a policy with per-step error ϵ is evaluated over trajectories of length T under its own state visitation.This formalizes the compounding exposure-bias problem motivating on-policy training.
  • Unified f-Divergence Framework: The f-divergence framework separates divergence geometry from sampling policy, with Reverse KL favoring mode-seeking solutions and Forward KL preserving diversity across acceptable outputs.These divergences can be estimated under student sampling without importance-weight or off-policy corrections.
  • Method Comparisons: GKD interpolates between dataset and student sampling through λ, while DistiLLM adds mixture targets, replay, and adaptive scheduling for numerical stability and sample efficiency.The comparison traces a move from GKD’s simplicity through MiniLLM’s variance-management needs to DistiLLM’s engineering sophistication.
  • Scaling and Capacity: Distillation scaling remains unresolved because teacher inference, student rollouts, data, and updates interact, while rollout benefits likely diminish with additional compute.Stronger teachers initially help through denser signals, but student absorption saturates beyond a critical teacher capacity.

3 Landscape and Method Selection

The survey organizes on-policy distillation around objective, supervisory signal, and training dynamics, treating these as sequential and interacting design decisions. Method selection depends on teacher access, task characteristics, compute budget, and stability requirements.

  • Method Landscape: OPD methods are organized by objective function, signal source, and training-dynamics stabilization mechanism.The signal-source axis distinguishes external versus self-distillation and white-box versus black-box teacher access.
  • Method Landscape: These axes correspond to the f-divergence, teacher-signal interface, and on-policy sampling or stabilization choices in the unified objective.The framework separates the divergence generator from the identity and access level of the entity supplying teacher probabilities.
  • Method Landscape: Methods are assigned to one primary category according to their core contribution, reducing ambiguity when papers address multiple dimensions.Loss-function innovations map to Objective, signal architectures to Signal, and stability or efficiency contributions to Dynamics.
  • Method Selection Considerations: Teacher access constrains feasible objectives: white-box logits support exact token-level divergences, whereas API-only access favors text- or score-based supervision.Examples include DistiLLM, G-OPD, and DSKD for white-box access, and OVD and GAD for sparse feedback.
  • Method Selection Considerations: Reasoning and mathematical tasks favor mode-seeking objectives, open-ended generation favors mode-covering objectives, and instruction following occupies an intermediate regime.Instruction-following settings are associated with moderate on-policy mixing (λ ≥0.5) with JSD in the cited selection guidance.
  • Method Selection Considerations: Compute and stability requirements shape deployment choices, from truncated rollouts and caching under <500 GPU-hours to multi-stage pipelines above 5000 GPU-hours.Reference constraints, rollout mixing, and token-level reliability filtering address length inflation and misleading gradients from flawed student prefixes.

4 Objective Functions and Optimization

Objective design in OPD has progressed from fixed divergences to adaptive, reward-augmented, and structurally higher-level supervision. The survey links these choices to token geometry, teacher access, reward density, optimization stability, and whether students should match or exceed teacher behavior.

  • Fixed Divergence Objectives: GKD samples student-influenced trajectories with a mixture policy and applies Forward KL, Reverse KL, or JSD per token, using λ to control on-policy exposure.λ = 0 is fully off-policy, while λ = 1 is fully on-policy.
  • Fixed Divergence Objectives: Skewed KL bounds the target density with a student-dependent mixture, avoiding gradient explosion when student-generated tokens receive near-zero teacher probability.DistiLLM-2 extends this source-aware strategy by using Forward SKL on teacher data and Reverse SRKL on student data, improving over symmetric baselines.
  • Fixed Divergence Objectives: A single divergence can mismatch token-level geometry: Reverse KL suits concentrated operator distributions, whereas Forward KL suits high-entropy positions with interchangeable words.This per-position mismatch motivates adaptive divergence routing.
  • Adaptive Divergence Objectives: EOPD outperforms fixed-divergence baselines across six competition-math benchmarks, using teacher uncertainty to route divergences independently of student quality.AOPD similarly targets failures from unbounded negative advantages, vanishing gradients, and exploration black holes.
  • RL-Augmented Objectives: RL-augmented objectives can extrapolate beyond teacher probability mass or use structural rewards, shifting distillation from distributional matching toward discovering reasoning paths and transferring behavioral organization.ExOPD permits reward extrapolation, while RLKD supervises step structure rather than token distributions.
  • RL-Augmented Objectives: Dense token-level teacher rewards are effective for compressing teacher improvements, while sparse sequence rewards support capability discovery; a GRPO-to-OPD bridge outperforms direct student-side GRPO.For Qwen3-1.7B, the bridge yields 79.3% versus 75.9% on MATH and 25.2 versus 19.8 on AIME 2024.

5 Signal Source and Teacher Architecture

OPD signal sources range from white-box teacher logits to black-box feedback and self-distillation, with each access regime imposing distinct supervision and deployment trade-offs. Privileged-information methods broaden self-teaching across tasks and modalities, while recent approaches target selective supervision, reward regularization, agentic granularity, and cross-domain transfer.

  • External Teacher Distillation: White-box logit access supports exact token-level divergence objectives, while API-only access restricts supervision and motivates sequence-level or sparse-feedback methods.White-box access is assumed by most cataloged objectives; API access exposes mainly teacher text or limited top-k probabilities.
  • External Teacher Distillation: Text-only observations can recover a substantial portion of teacher knowledge, despite black-box methods receiving less dense supervision.Black-box methods trade signal density for deployment flexibility, while white-box methods require co-hosting the teacher.
  • External Teacher Distillation: Distilling Step-by-Step showed that a 770M-parameter student could outperform a 540B-parameter teacher while using substantially less training data by supervising intermediate rationales.The approach used answer prediction and rationale generation, with 50% fewer examples on average and up to 85% fewer on some tasks.
  • Privileged Information: Privileged information gives a self-teacher training-time context unavailable at inference, while the student learns from its own rollouts under token-level matching.Examples include answers, retrieved documents, spatial masks, judge feedback, future observations, and critiques.
  • Privileged Information: PI distillation is bounded by whether the privileged signal is instance-specific or a rule that can be aggregated across examples and internalized by the policy.System prompts, constitutions, and grading rubrics can be internalized more readily than individual answers.
  • Privileged Information: OPSD matched or exceeded GRPO at 4B and 8B scales using one rollout per problem instead of eight, but underperformed at 1.7B.The reported differences were +0.9 average points at 8B, +0.8 at 4B, and -0.5 at 1.7B.
  • Recent Extensions: Recent methods refine the supervision granularity and objective through critical-span routing, cross-lingual PI, reward regularization, multi-view signals, and agentic trajectory-to-skill alignment.These approaches include TRACE, COPSD, MSD, PBSD, AVSD, and TT-OPD, respectively.
  • Recent Extensions: Agentic distillation can match its temporal unit to the level at which task errors compound, spanning trajectories, turns, steps, and skills.TT-OPD is presented alongside TCOD, Skill-SD, and MAD-OPD as completing this granularity spectrum.

6 Training Efficiency and Stabilization

Training efficiency and stabilization address noisy supervision, unsuitable prompts, stale or costly rollouts, and policy drift through complementary weighting, curriculum, compute, and approximation strategies.

  • Engineering challenges: On-policy generation creates non-stationary data, gradient SNR collapse on hard prompts, and substantial autoregressive compute overhead.These challenges motivate online scoring, curriculum strategies, and systems-level architectural solutions.
  • Teacher reliability: Flawed prefixes make teacher guidance noisy because later predictions condition on student errors that the teacher has not encountered during training.Naively matching this corrupted signal can propagate errors instead of correcting them.
  • Token and sample weighting: TIP recovers structurally informative Q3 tokens that entropy-only weighting misses; training exclusively on fewer than 20% of tokens surpasses full-token OPD on DeepPlanning.Its parameter-free selection uses student entropy and teacher-student divergence to distinguish token roles.
  • Token and sample weighting: AdaSwitch uses recent divergence to trigger a single switch from on-policy exploration to off-policy guidance when generation drifts into high-divergence regions.The adaptive threshold avoids both over-intervention on easy prefixes and under-intervention on genuinely problematic states.
  • Teacher reliability: Teacher top-K local support matching addresses imbalanced supervision, unreliable prefixes, and tokenizer mismatch, while SCOPE routes incorrect and correct rollouts to different objectives.EGRSD similarly down-weights uncertain teacher positions while retaining a nonzero weight for every token.
  • Curriculum and difficulty adaptation: PACED weights prompts with w(p) = p^α(1 − p)^β, with the symmetric case peaking at p*=0.5, because binary-reward gradients vanish at both extremes.Off-policy cold starts reduce the thinking-pattern mismatch before on-policy refinement when the initial student policy is too far from the teacher.
  • Compute optimization: Partial rollouts and stale teacher signals can preserve much of OPD’s alignment benefit, defining a fidelity-efficiency frontier rather than a binary on-policy requirement.This approximation trades on-policy fidelity against computational efficiency when teacher consistency and policy-drift control hold.
  • Compute optimization: DistillSpec improves speculative-decoding speed by 10%–45% and can reduce multi-scale decoding latency by 6–10× with minimal performance drop.It aligns the draft model with the target model and supports control over the latency-performance tradeoff.

7 Understanding OPD: Theory, Failure Modes, and Cost

OPD succeeds when teacher and student share compatible reasoning patterns while the teacher provides genuinely new capability, but it faces distinct optimization, calibration, self-play, and agentic failure modes. Evidence also suggests that OPD can compress known solutions more reliably than correct flawed reasoning, with trajectory-specific supervision and structural regularization needed for stability.

  • Success Conditions: Effective OPD requires compatible teacher–student reasoning patterns and teacher capabilities beyond those already acquired by the student.Low top-k overlap cannot be recovered through training, while same-recipe teachers provide little transferable signal.
  • Success Conditions: OPSD preserves accuracy while shortening correct reasoning traces, but incorrect-only OPSD degrades accuracy, indicating compression rather than reliable correction.The reported experiments rule out richer teacher context, feedback reinjection, and extended training as alternative explanations.
  • Failure Modes: The flawed prefix trap combines unreliable teacher guidance on out-of-distribution prefixes, imbalanced one-token signals, and tokenizer mismatch distortions.Token-level OPD is biased relative to sequence-level Reverse KL but has tighter worst-case variance bounds.
  • Failure Modes: Reward extrapolation above the clip-safety threshold λ⋆ switches structured-output training from format-preserving to format-collapsing.On calibrated Amazon Fashion listwise JSON tasks, operating just below λ⋆ brought a 1.7B Qwen3 student to in-domain parity with an 8B-SFT baseline at one-fifth the parameters.
  • Failure Modes: Up to 18% of tokens can remain persistently high-loss after apparent OPD saturation, while contributing disproportionately to gradient norms but negligibly to reasoning performance.These Rock Tokens are characterized as structural or discourse residuals rather than reasoning-informative tokens.
  • Failure Modes: OPD and OPSD failures arise from prefix distribution mismatch, biased top-K reverse-KL gradients, and privileged-information types that cannot be aggregated without instance-specific answers.Stop-gradient top-K objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate some failures.
  • Failure Modes: Gradient SNR vanishes when prompt pass rates approach zero, while self-distillation can suppress hedging and uncertainty markers, producing shorter but less calibrated reasoning.These mechanisms motivate competence-boundary curricula and connect calibration degradation to unreliable self-distillation targets.
  • Failure Modes: Local teachability collapse occurs when teacher margins over the student’s top-K candidates narrow, even though the teacher remains well-calibrated.A BIC-style change-point rule truncates supervision where dense token-level feedback loses discriminative information.

8 Applications, Systems, and Emerging Domains

OPD is increasingly deployed across industrial, agentic, multimodal, embodied, and specialized domains. These systems adapt supervision granularity, teacher access, and optimization to the way errors compound in each setting.

  • Industrial Deployment: Qwen3 combines off-policy cold-start with on-policy refinement and reports better performance than direct reinforcement learning at roughly one-tenth the GPU hours.It also improves pass@64 on AIME benchmarks where reinforcement learning from the same off-policy checkpoint does not.
  • Industrial Deployment: CoPD uses co-evolutionary RLVR and bidirectional OPD to keep multiple experts behaviorally compatible before consolidating their complementary capabilities.The method produces a single model integrating text, image, and video reasoning capabilities that surpasses mixed RLVR and sequential multi-teacher OPD.
  • Agentic Systems: Turn-level interpolation combines student-generated and teacher-filled turns, pairing on-policy distributional alignment with dense supervised-learning feedback.This revisits DAgger for multi-turn LLM agents.
  • Agentic Systems: Agentic methods match distillation granularity to error-compounding granularity, progressing from trajectory-level control through step-level isolation to skill-level decomposition.Coarse trajectory supervision can underperform when errors compound locally, while token-level supervision can waste compute on already reliable positions.
  • Emerging Domains: OPD supports cross-modal transfer, self-alignment, multimodal search, GUI completion, and embodied reasoning, but each domain introduces distinct supervision or credit-assignment challenges.Examples include text-only teachers guiding visual reasoning, text modes supervising audio modes, and invalid GUI outputs limiting teacher evaluation.
  • Emerging Domains: Dense teacher supervision improves learning efficiency and robustness in settings where sparse rewards or offline supervised learning are inadequate.This pattern is reported for physical-control trajectories and multimodal agentic search.

9 Open Problems and Future Directions

Open problems concern how OPD should scale, adapt feedback, handle long-lived agents and evolving teachers, and unify distillation with reinforcement learning. The survey emphasizes unresolved compute, alignment, evaluation, and cross-architecture constraints.

  • Distillation Scaling Laws: No validated scaling law currently describes on-policy distillation loss as a function of student size, teacher size, and rollout budget.A conjectured joint power law remains speculative, with capacity-gap interference and teacher diversity requiring further modeling.
  • Distillation Scaling Laws: Future experiments must disentangle whether additional on-policy data compensates for a weaker teacher or whether teacher quality imposes a hard ceiling.Controlled variation of student size, teacher size, and rollout budget is proposed under fixed compute budgets.
  • Uncertainty-Aware Feedback: Uncertainty-aware feedback could let students discount teacher signals on uncertain tokens and avoid reasoning paths likely to enter regions of teacher incompetence.The survey distinguishes current-prefix entropy from predictive uncertainty several tokens ahead.
  • Agent-Level and Lifelong Distillation: Agent-level and lifelong distillation require credit assignment for long-horizon tool use, state tracking, error recovery, and teacher evolution without catastrophic forgetting or distributional drift.Existing methods represent early steps rather than a complete solution.
  • Efficiency Frontiers: On-policy compute overhead remains a barrier, while the information-theoretic minimum amount of on-policy data needed for a target quality level is unknown.Selective teacher inference is identified as an underexplored efficiency direction.
  • Latent and Cross-Architecture Distillation: Cross-modal and cross-architecture distillation still relies on vocabulary-space alignment, leaving shared latent-space transfer and extreme capacity gaps unresolved.Existing projection and transport methods have mainly been validated across relatively small scale differences.
  • Unifying Distillation and Reinforcement Learning: The distillation–reinforcement-learning boundary is narrowing through interleaved dense KD and reward signals, co-evolutionary training, and self-play formulations.These approaches address complementary weaknesses: teacher saturation and instability under sparse rewards.

10 Conclusion

The survey unifies OPD around adaptive objectives, student-generated signals, and trajectory-aware training, synthesizing findings across algorithms, theory, diagnostics, and deployments. It concludes that OPD improves alignment but remains costly, difficult to compare fairly, and bounded by unresolved scaling and adaptation problems.

  • Conclusion: The survey organizes OPD along three design axes: objective-function design, signal source, and training dynamics.Training dynamics includes token weighting, curriculum adaptation, and compute optimization.
  • Key Findings: On-policy distributions have been associated with consistent accuracy gains across mathematical reasoning, code generation, and instruction following, especially for longer reasoning chains.The survey attributes this pattern to reduced exposure bias and learning to recover from student-generated errors.
  • Key Findings: Divergence selection benefits from task- and token-position adaptation: Reverse KL tends to suit reasoning, while Forward KL preserves diversity for open-ended generation.Adaptive divergence methods are presented as the current frontier.
  • Practical Takeaways: Practical recommendations favor token-level methods for general instruction following and sequence-level or hybrid methods for reasoning-intensive tasks.The latter are preferred despite higher gradient variance from REINFORCE-style estimators.
  • Limitations: The survey’s scope is limited to training methods involving student on-policy generation and excludes orthogonal compression techniques and inference-only student-generated data.Coverage is stated as current through June 2026.
  • Limitations: Fair comparison remains difficult because studies vary in base models, compute budgets, benchmark versions, and rollout counts.The survey reports original-paper results rather than treating published numbers as directly comparable.
  • Broader Impact: OPD can lower access barriers through smaller deployable models, while its 4–5× compute overhead over off-policy SFT raises environmental concerns.The survey notes that downstream inference savings may lower lifetime carbon footprint when amortized across billions of calls.
  • Broader Impact: Teacher supervision can provide a point for injecting safety constraints, alignment objectives, and capability boundaries during distillation.MSD is cited as a safety-aware OPD example for transferring alignment to low-resource languages.
Loading 2604.00626v4…