PaperlayerView exact revision on arXiv

Source-linked AI summary

DOPD: Dual On-policy Distillation

Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang, Shuai Dong, Kaiwen Tuo, Xiangyu Zeng, Kaituo Feng, Qunzhong Wang, Yang Shi, Xiaobin Hu, Xiangyu Yue, Jiaqi Wang, Shuicheng Yan

arXiv:2606.30626v1cs.AI

TL;DR

Privileged information can make distillation gains appear larger than the transferable capability they provide, while pivotal supervision is unevenly distributed across tokens. DOPD routes token-level supervision between privileged teacher and student policies using advantage gaps and relative probabilities, outperforming Vanilla OPD by 7.5 points on LLM setups and 6.0 points on VLM setups across eight benchmarks.

  • Problem

    OPD depends on supervision signals reflecting capabilities beyond the student, but privileged contexts can conflate transferable capability gaps with information-asymmetry gaps.

  • Method

    DOPD dynamically routes token-level supervision between privileged teacher and student policies using their advantage gap and relative predicted probabilities.

  • Results

    Across eight benchmarks, DOPD outperforms Vanilla OPD by 7.5 points on LLM setups and 6.0 points on VLM setups.

  • Takeaways & Limitations

    DOPD provides a selective distillation framework for transferring capability-bearing signals while using lighter supervision for tokens dominated by privileged information.

  • Takeaways & Limitations

    DOPD requires costly privileged information, adds one student forward pass over Vanilla OPD, and relies on heuristic routing mechanisms.

Abstract

from arXiv · show

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.

1 Introduction

The introduction identifies privilege illusion and non-uniform token supervision as key limitations of privileged on-policy distillation. It proposes DOPD, which adaptively combines privileged teacher and student supervision and improves performance across LLM and VLM settings.

  • Motivation: OPD rolls out from the evolving student policy and uses teacher-provided token-level signals, avoiding the state-distribution mismatch associated with off-policy distillation.This contrasts OPD with conventional distillation based on off-policy trajectories.
  • Problem: Privileged information can raise the apparent distillation ceiling without producing transferable signals, creating the failure mode termed privilege illusion.The passage cites verified reasoning hints for LLMs and structured visual annotations for VLMs as examples of privileged information.
  • Problem: Because decisive capability-bearing information is concentrated in a small subset of tokens, treating all tokens with identical supervision can amplify privilege illusion.Other tokens may provide low-value, privilege-dependent supervision.
  • Method: DOPD dynamically routes token-level supervision between privileged teacher and student policies using their privilege advantage gap and relative predicted probabilities.Credible teacher advantages receive stronger teacher distillation, while complementary self-based supervision addresses the remaining tokens.
  • Results: 7.5 and 6.0 points: DOPD outperforms Vanilla OPD across eight benchmarks on LLM-based and VLM-based setups, respectively.Improvements remain 6.2-10.6 points across five model pairs of varying sizes, with further gains in continual learning, out-of-distribution evaluation, and training stability.

2 Related Works

Teacher–student distillation has expanded from label transfer to rationales, trajectories, preferences, and broader behaviors, while OPD has developed into a structured post-training paradigm. However, existing methods face teacher-imposed performance ceilings and may mistake privileged-information effects for genuinely transferable capacity.

  • Teacher–Student Distillation: Teacher–student distillation transfers capabilities from stronger teachers to weaker students through labels, rationales, trajectories, preferences, and broader behavioral patterns.The passage frames distillation as more nuanced than simple teacher imitation.
  • On-Policy Distillation: OPD unifies on-policy learning’s distributional consistency with dense supervision and has evolved across standard, structured, and adaptive distillation directions.Adaptive distillation modulates supervision based on student state or other factors.
  • Limitations: Vanilla OPD imposes a theoretical ceiling on student performance through the teacher policy, especially when the teacher performs poorly on challenging tasks.The passage identifies this limitation as inherent to Vanilla OPD.
  • Limitations: Privileged-information methods assume direct integration can enhance transferable capabilities, but overlook privilege illusion and may fail to identify and distill genuine inherent capacity.They also rely on uniform distillation mechanisms receiving supervision consistently from a monolithic source.

3 Methodology · 3.1 Background

The background identifies privilege illusion as the central limitation of naively adding privileged information to OPD: apparent gains may reflect information asymmetry rather than transferable capability. It motivates the privilege advantage gap, which measures teacher–student prediction discrepancy under identical privileged conditions to identify high-value supervision tokens.

  • 3.1.1 Privilege Illusion: Existing OPD assumes that a stronger teacher provides richer, more informative supervision, motivating privileged inputs for teachers or students.Examples include verified reasoning hints and object bounding boxes.
  • 3.1.1 Privilege Illusion: Privileged information granted separately to either policy yields only modest early-training improvements over Vanilla OPD, while the resulting information asymmetry creates a failure mode.The comparison considers teacher-only, student-only, and teacher-and-student privileged-information variants.
  • 3.1.1 Privilege Illusion: Privilege illusion occurs when privileged inputs produce ostensible gains arising from information asymmetry rather than genuine capability enhancement.This distinction separates apparent performance benefits from transferable capability.
  • 3.1.2 Privilege Advantage Gap: When both policies receive privileged inputs, their relative advantage serves as a proxy for the privilege-conditioned prediction gap: large gaps indicate capability discrepancies, while small gaps suggest information-driven teacher advantages.The proposed interpretation is intended to disentangle capability gaps from information gaps.
  • 3.1.2 Privilege Advantage Gap: The privilege advantage gap A is defined as the absolute difference between teacher and student token log-probabilities under the same input, privileged context, and preceding tokens.For token y_n, A = ∣log Π_T(y_n ∣x,p,y_<n) − log Π_S(y_n ∣x,p,y_<n)∣.
  • 3.1.2 Privilege Advantage Gap: Under identical privileged conditions, A captures prediction discrepancy caused by teacher–student performance differences and represents idealized learning content.The evaluated token is y_n, while p denotes privileged auxiliary context.
  • 3.1.2 Privilege Advantage Gap: Removing high-advantage tokens harms distillation more than removing random or low-advantage tokens, yet the high-advantage-removal variant still gains 3.4 and 1.5 points.These ablations were conducted across both LLM and VLM settings.
  • 3.1.3 Takeaway: The resulting takeaway is that naive privileged-information injection causes privilege illusion, whereas the privilege advantage gap highlights high-value tokens critical for capacity-centric distillation.This motivates using advantage-aware token supervision to distinguish transferable capability from information asymmetry.

3.2 DOPD: Dual On-policy Distillation

DOPD extends on-policy distillation by adaptively routing token-level supervision between privileged teacher and student policies according to privilege advantage gaps and relative confidence. This design transfers credible capability while limiting privileged shortcuts, uncertain supervision, and unnecessary teacher imitation.

  • OPD samples trajectories from the student policy, enabling knowledge transfer under the target distribution and providing richer supervision than conventional reinforcement learning.The sampled sequence is y ∼ ΠS(⋅∣x), after which the teacher supplies token-level supervision.
  • The four masked objectives exhaustively partition tokens, mitigating privilege illusion by limiting strong teacher supervision to credible capability advantages and preserving confident student exploration.DOPD distinguishes transferable capability from information asymmetry and avoids indiscriminate teacher imitation.
  • DOPD evaluates each token using privileged teacher and student probabilities plus their privilege advantage gap to select an appropriate distillation objective and strength.The privileged student shares parameters with the deployed student, while the privileged teacher remains frozen.
  • Low advantage gaps trigger light teacher distillation for confident tokens, but weak privileged-student regularization for low-confidence tokens where both policies are unreliable.The weak coefficient satisfies βw < βl, and the privileged student acts as a parameter-shared consistency anchor.
  • High advantage gaps route confident teacher-favored tokens to full-vocabulary teacher alignment, while student-favored tokens receive light privileged-student Top-K reverse-KL distillation.Full-vocabulary alignment provides denser distributional signals; student-side distillation avoids overregularization.

4 Experiments · 4.1 Settings

Experiments use Qwen3/Qwen3-VL teacher–student pairs, task-specific datasets and benchmarks, and broad baseline comparisons. Training fixes the teacher and optimizes the student under specified distillation settings.

  • 4.1.1 Models: Experiments use non-thinking Qwen3 and Qwen3-VL families as teacher and student policies, primarily pairing 8B teachers with 1.7B and 2B students.Additional Qwen3 pairings test generalization across model sizes.
  • 4.1.1 Models: Training covers general, reasoning, and coding tasks for LLMs, and general, visual reasoning, and visual understanding tasks for VLMs.LLM data combines RaR-Science-20K, DAPO-Math-17K, and Skywork-OR1-Coding-14K; VLM training uses ViRL39K.
  • 4.1.1 Models: Privileged VLM inputs use query-related bounding boxes with object labels and quadruple coordinates, producing 32K LLM and 25K VLM high-quality training samples after GPT-5.4 rechecking.Low-quality generated privileged contents are discarded.
  • 4.1.2 Benchmarks: Evaluation spans eight LLM benchmarks across general, reasoning, and coding abilities, plus eight VLM benchmarks across general, visual reasoning, and visual understanding.LLM examples include C-Eval, MATH500, and BFCLv3; VLM examples include RealWorldQA, MathVision, and LogicVista.
  • 4.1.3 Baselines: DOPD is compared with nine LLM counterparts spanning standard, self, and adaptive distillation paradigms.Baselines include Vanilla OPD, OPCD, ExOPD, Uni-OPD, SDFT, OPSD, SDPO, EOPD, and TIP.
  • 4.1.3 Baselines: For VLMs, DOPD is benchmarked against Vanilla OPD, Uni-OPD, Vision-OPD, and VA-OPD, with all baselines rerun on Qwen3/Qwen3-VL models.VA-OPD results use a reproduced version because its codes are not officially released.
  • 4.1.4 Implementations: Experiments run on 8 NVIDIA H200 141GB GPUs with frozen teachers, AdamW student optimization, cosine scheduling, and a 5 × 10−6 learning rate.LLM and VLM batch sizes are 128 and 64, with 4 rollout samples and maximum 200 and 300 steps, respectively.
  • 4.1.4 Implementations: Top-K distillation uses K = 128, while βw = 0.3 and βl = 0.6 regulate distillation strength.These settings are used alongside the reported optimization configuration.

4.2 Main Results

DOPD consistently improves over Vanilla OPD and competing distillation methods across LLM and VLM settings, narrowing teacher–student gaps and outperforming strong baselines. It also generalizes across model scales, supports continual learning, and improves out-of-distribution performance.

  • LLM Results: DOPD gains 12.3 points and recovers 89.8% of the original teacher–student gap in the main LLM-based OPD results.It approaches the teacher policy on average and surpasses it on four challenging benchmarks, especially reasoning and coding tasks.
  • LLM Results: DOPD achieves the best performance across all eight benchmarks, improving over ExOPD, Uni-OPD, and EOPD by 4.4/4.8/5.3 points on average, respectively.Self-distillation baselines provide relatively modest improvements.
  • VLM Results: DOPD delivers a 10.1-point absolute gain and recovers 69.2% of the teacher–student gap in VLM-based OPD.It achieves the best average performance, outperforming Vanilla OPD by 6.0 points and Uni-OPD, Vision-OPD, and VA-OPD by 4.2/2.8/2.1 points, respectively.
  • Scalability: Across five teacher–student model pairs, DOPD consistently outperforms Vanilla OPD, with average improvements of 11.1–14.1 points across all pairs.In the largest scale mismatch, Qwen3-8B →Qwen3-0.6B, DOPD gains 14.1 points and recovers 53.0% of the teacher–student gap, versus Vanilla OPD’s 3.5-point gain.
  • Continual Learning: DOPD further improves OPD’s sustained continual-learning advantage, enabling consistent gains on newly introduced domains with only tiny degradation on previously acquired domains.The three-stage experiment sequentially introduces general, reasoning, and coding data.
  • Out-of-Distribution Generalization: DOPD outperforms the second-best counterparts by 3.1 and 4.3 points on the two cross-domain evaluations, demonstrating superior out-of-distribution generalization.Models are trained separately on coding or reasoning data and evaluated on the other unseen task domain.

4.3 Additional Analyses

Additional analyses show that DOPD improves training stability and efficiency, while its privileged-input formulations, token-aware routing, divergence choices, and distillation intensities each contribute to performance. The results support differentiated supervision that emphasizes critical tokens while retaining auxiliary optimization signals.

  • Training Stability: DOPD maintains stable, superior performance and higher distillation efficiency throughout training against ExOPD, SDPO, and EOPD.The comparison spans standard, self-, and adaptive-distillation paradigms.
  • Privileged Information: Comparative experiments evaluate five LLM and five VLM privileged-information formulations, including final answers, hints, captions, bounding boxes, labels, and no privileged input.The passage frames these experiments as assessing how distinct privileged-information injection strategies affect distillation.
  • Token-Level Analysis: Token distributions reveal distinct functional roles, with low-gap high-probability tokens reflecting self-consistent or local exploration branches that motivate differentiated distillation.Tokens are categorized using privilege advantage gaps and teacher/student predicted probabilities.
  • Ablation Analysis: Token-level ablations isolate contributions from individual and combined token types, as well as the adaptive advantage-aware dual-distillation mechanism, using JS divergence on Top-K tokens.The non-adaptive variant assigns identical distillation weights and strategies to all tokens.
  • Divergence Designs: JS divergence performs relatively better than forward or reverse KL under the reported settings, while divergence objectives and token-selection strategies exhibit trade-offs.The comparison includes sampled-token, Top-K-token, and full-vocabulary strategies under equal distillation across tokens.
  • Intensity Sensitivity: βw = 0.3 and βl = 0.6 provide a favorable intensity trade-off by amplifying critical-token contributions while preserving auxiliary stabilization signals.The study varies distillation intensity across token categories.

5 Conclusion

The paper identifies privilege illusion in on-policy distillation: privileged-teacher superiority may reflect information asymmetry rather than transferable capability, with supervision unevenly distributed across tokens. It proposes DOPD, an advantage-aware dual framework that adaptively routes token-level supervision between teacher-driven capability signals and privileged student policies.

  • Limitations: Privileged-teacher superiority may arise from information asymmetry rather than transferable capability.The conclusion revisits OPD under privileged contexts and identifies this as a fundamental limitation.
  • Limitations: Token-level supervision is not evenly distributed, making some tokens more relevant to capability transfer than others.The conclusion highlights non-uniform supervision signals across tokens.
  • DOPD: DOPD adaptively routes token-level supervision between teacher-driven capability signals and privileged student policies.The proposed framework is described as advantage-aware and dual on-policy distillation.

6 Limitations and Future Directions

DOPD depends on costly, high-quality privileged information and adds computational overhead, while future work should develop more reliable acquisition and more principled routing mechanisms.

  • DOPD requires privileged information whose annotation, generation, and filtering incur additional costs.
  • DOPD introduces computational overhead over Vanilla OPD by requiring one additional student-model forward pass.
  • Future work should pursue more reliable, cost-effective privileged-information acquisition and more principled or learnable distillation routing.
  • Dynamic distillation from teacher and student may support selective capacity transfer beyond LLMs and VLMs, with more interpretable, efficient, and trustworthy methods.

Appendix · A Details of Privileged Input

The appendix details how privileged inputs are presented for language and vision-language tasks. Examples include geometric reasoning, image-topic recognition, and prompts conditioned on questions with ground-truth labels.

  • A Details of Privileged Input: LLM-based privileged input is demonstrated with a solid square pyramid cut through the plane defined by ACE.The original question asks which shapes result from the cut.
  • A Details of Privileged Input: The LLM example asks whether the two resulting pieces are tetrahedra, squares, square pyramids, circles, triangular prisms, rectangular prisms, or different shapes.It specifies a square pyramid with base vertices A, B, C, D and apex E.
  • A Details of Privileged Input: The appendix includes demonstrations of LLM-based privileged input.These demonstrations are identified in Figure 11.
  • A Details of Privileged Input: VLM-based privileged input is demonstrated through an image question asking for the main topic.The figure is accompanied by a concise image-topic prompt.
  • A Details of Privileged Input: The appendix includes demonstrations of VLM-based privileged input.These demonstrations are identified in Figure 12.
  • A Details of Privileged Input: Privileged input generation is framed around a question together with its corresponding ground-truth label.This pairing defines the information supplied for generation.
  • A Details of Privileged Input: The appendix presents prompts for privileged input generation.These prompts are identified in Figure 13.
Loading 2606.30626v1…