DOPD: Dual On-policy Distillation

Introduction

• OPD rolls out current student policies and uses teacher token-level signals, mitigating distribution shift while providing dense supervision [35] [42] [44].Its achievable upper bound nevertheless depends on supervision-source quality [7] [22] [25].Source: §1 · p. 2
• Privilege illusion conflates transferable teacher–student capability gaps with information-asymmetry gaps, so imitating privileged outcomes can cause entropy collapse, reduced exploration, and poor distillation.Privileged inputs may raise the apparent distillation ceiling without providing transferable supervisory signals.Source: §1 · p. 2
• Because decisive capability signals occur in only a small subset of tokens, uniformly distilling every token from one policy can amplify privilege-dependent shortcuts over transferable capability.Vanilla and many OPD variants apply the same supervision source and objective across tokens.Source: §1 · p. 2
• DOPD dynamically routes token-level supervision by privilege advantage gaps and relative probabilities, using stronger teacher distillation for credible capability advantages and lighter supervision otherwise.It adapts supervision source, strength, and granularity to support stability and favorable exploration.Source: §1 · p. 2
• Across eight benchmarks, DOPD outperforms Vanilla OPD by 7.5 points on LLM-based setups and 6.0 points on VLM-based setups.It also improves by 6.2–10.6 points across five model pairs and shows gains in continual learning, out-of-distribution evaluation, and training stability.Source: §1 · pp. 2–3

2.1 Teacher-student Distillability

• Distillability extends beyond labels to rationales, trajectories, preferences, and behaviors [1] [13], but an overly powerful teacher may provide signals a limited student cannot absorb [4] [26] [45].OPD studies also report that a more compatible initial distribution may improve distillation [1] [7] [22] [25].Source: §2.1 · p. 3

2.2 On-Policy Distillation

• OPD research includes standard teacher-to-student distillation, self-distillation under different contexts, and adaptive methods that modulate supervision based on student state or other signals.Examples include Fast OPD [60], SDFT [34], and EOPD [18], respectively.Source: §2.2 · p. 3
• Vanilla OPD’s student performance has a teacher-policy ceiling [25] [35], while privileged-information methods [15] [28] [30] [34] [36] [49] [57] [61] risk privilege illusion by assuming direct capability gains and uniform supervision.The paper argues these methods may fail to distinguish genuine inherent capacity from information that cannot be replicated.Source: §2.2 · pp. 3–4

3.1.1 Privilege Illusion

• Giving privileged information to only the teacher or student yields early modest gains but later performance degradation and entropy collapse; giving it to both still produces only marginal gains below Vanilla OPD.With privilege available to both policies, uniform token distillation leads students to passively adapt to privileged information rather than internalize core competencies.Source: §3.1.1 · p. 4

3.1.2 Privilege Advantage Gap

• At step 100, removing high-advantage tokens yields approximately 50% of Vanilla OPD’s performance gain; in multimodal models it yields only about 20%.Pruning random or low-advantage tokens has negligible impact; high-advantage-pruned variants still gain 3.4 and 1.5 points despite underperforming the counterparts.Source: Figure 4 · p. 5, §3.1.2 · pp. 5–6

3.1.1 Privilege Illusion

• Privilege illusion occurs when privileged inputs produce ostensible gains from information asymmetry, prompting students to imitate privileged outcomes rather than acquire transferable abilities.The paper warns that uncurated privileged signals can also create an irreparable teacher–student gap, unstable distillation, and entropy collapse.Source: §3.1.1 · pp. 4–5

3.1.2 Privilege Advantage Gap

• With both policies under identical privileged conditions, a large advantage gap signals capability discrepancy, while a small gap suggests the teacher’s advantage mainly reflects privileged information.This gap is proposed as a way to separate capability differences from information differences and selectively transfer signals reflecting authentic competence.Source: §3.1.2 · p. 5
• For each sampled token, the privilege advantage gap is the absolute difference between teacher and student log probabilities conditioned on the same input, privilege, and preceding tokens.The resulting discrepancy is intended to capture the idealized learning content under shared privileged conditions.Source: §3.1.2 · p. 5

3.1.3 Takeaway

• The takeaway is that naive privileged inputs risk privilege illusion, whereas the privilege advantage gap highlights tokens whose supervision is most critical for capacity-centric distillation.The proposed distinction prioritizes high-value token signals rather than treating all privileged outcomes as transferable knowledge.Source: §3.1.3 · p. 6

3.2.1 Divergence

• Forward KL promotes broad teacher-support coverage, reverse KL concentrates on teacher-favored modes, and JS balances the two directions through an intermediate distribution.The passage describes JS as providing a more balanced optimization signal that improves policy-distillation stability.Source: §3.2.1 · p. 6

3.2.2 OPD

• OPD samples trajectories from the student policy and optimizes token-level teacher signals, mitigating distribution-shift bias while offering richer supervision than conventional reinforcement learning.Supervision can cover the sampled token, the Top-K tokens, or the full vocabulary; denser signals may improve efficiency but increase computational overhead and instability risk.Source: §3.2.2 · pp. 6–7

3.2.3 Advantage-aware Dual Distillation

• DOPD dynamically selects supervision source, strength, and distillation form using token-level privilege advantage gaps to separate capability transfer from privileged-information imitation.It applies strong teacher supervision only when the privileged teacher has a credible capability advantage, and otherwise uses lighter teacher or student regularization or preserves student exploration.Source: §3.2.3 · pp. 7, 9
• For low-gap tokens, DOPD uses light Top-K reverse-KL teacher distillation when both policies are confident, but weak student self-regularization when both assign low probability.The weak student anchor uses stop-gradient and a smaller coefficient than light distillation, avoiding forced imitation of uncertain teacher predictions.Source: §3.2.3 · p. 8
• For high-gap tokens, DOPD applies unit-weight full-vocabulary JS teacher distillation when the teacher is more confident, but light Top-K reverse-KL student distillation when the student is more confident.Full-vocabulary alignment conveys dominant decisions and informative secondary preferences, while student distillation avoids suppressing potentially valid exploration.Source: §3.2.3 · p. 8

4.1.1 Models

• Experiments use Qwen3-8B→1.7B and Qwen3-VL-8B→2B as main pairs, with four additional Qwen3 size pairs to test generalization.The additional pairs are 8B→0.6B, 4B→0.6B, 4B→1.7B, and 1.7B→0.6B.Source: §4.1.1 · p. 9
• LLM training combines RaR-Science-20K [9], DAPO-Math-17K [54], and Skywork-OR1-Coding-14K [10], while VLM training uses ViRL39K [24].GPT-5.4 generated step-wise decomposition hints for LLMs and structured visual annotations for VLMs; after rechecking and discarding low-quality samples, 32K LLM and 25K VLM samples remained. LLM hints use verified rationales without execution traces or final answers; VLM annotations use query-related bounding boxes, object labels, and quadruple coordinates.Source: §4.1.1 · p. 9

4.1.2 Benchmarks

• Eight LLM benchmarks cover general, reasoning, and coding tasks, and eight VLM benchmarks cover general, visual reasoning, and visual understanding, using official metrics and evaluations.LLM benchmarks are C-Eval [14], LiveBench [39], MATH500 [11], AIME25, ZebraLogic [27], AutoLogi [62], BFCLv3, and LCBv5 [16]. VLM benchmarks are RealWorldQA, MMStar, MathVision [37], DynaMath [63], LogicVista [43], MMMU [58], MMMU-Pro [59], and VSI-Bench [50].Source: §4.1.2 · p. 9

4.1.3 Baselines

• LLM comparisons include Vanilla OPD, OPCD [53], ExOPD [51], Uni-OPD [12], SDFT [34], OPSD [61], SDPO [15], EOPD [18], and TIP [46].The paper groups these baselines as standard, self, and adaptive distillation paradigms.Source: §4.1.3 · p. 9
• VLM comparisons include Vanilla OPD, Uni-OPD [12], Vision-OPD [57], and VA-OPD [28]; all baselines were rerun on Qwen3/Qwen3-VL models.Tables 1 and 2 present performance comparisons across general, reasoning, coding, visual reasoning, and visual understanding tasks; Table 2 uses reproduced VA-OPD results because its code was not officially released.Source: §4.1.3 · p. 10, Table 1 · p. 10, Table 2 · p. 10

4.1.4 Implementations

• Experiments used eight NVIDIA H200 GPUs, a frozen teacher, and an AdamW-optimized student with a cosine schedule and learning rate 5 × 10−6.LLM and VLM batch sizes were 128 and 64, each with four rollout samples, for up to 200 and 300 steps, respectively; Top-K was 128, with βw = 0.3 and βl = 0.6.Source: §4.1.4 · p. 10

4.2.1 Distillation Performance

• DOPD achieves the best performance across eight LLM benchmarks, improving over ExOPD [51], Uni-OPD [12], and EOPD [18] by 4.4, 4.8, and 5.3 points on average.It gains 12.3 points and recovers 89.8% of the original teacher-student gap, and surpasses the teacher on four benchmarks, especially reasoning and coding tasks.Source: §4.2.1 · pp. 10–11
• 10.1 points of gain over the VLM student corresponds to 69.2% recovery of the teacher-student gap.DOPD has the best average performance, exceeding Vanilla OPD by 6.0 points and Uni-OPD [12], Vision-OPD [57], and VA-OPD [28] by 4.2, 2.8, and 2.1 points, respectively.Source: §4.2.1 · p. 11

4.2.2 Robustness & Scalability

• Across five teacher-student model pairs, DOPD improves over Vanilla OPD by 11.1–14.1 points on average, a two- to over three-fold improvement.For Qwen3-8B → Qwen3-0.6B, DOPD gains 14.1 points and recovers 53.0% of the teacher-student gap, versus a 3.5-point gain for Vanilla OPD; DOPD maintains increasing gains as teacher size grows.Source: §4.2.2 · pp. 11–12, Table 3 · p. 11, Figure 6 · p. 12

4.2.3 Continual Learning Evaluation

• DOPD supports steady capability accumulation across continual-learning stages, improving on each newly introduced domain with only tiny degradation on previously learned domains.The three stages introduce general, reasoning, and coding data sequentially; DOPD further improves the sustained-learning and reduced-forgetting performance of OPD-based methods.Source: §4.2.3 · p. 13

4.2.4 Out-of-distribution Evaluation

• 3.1 and 4.3 points are DOPD’s gains over the second-best counterparts on cross-domain out-of-distribution evaluations.Models are trained separately on coding or reasoning data and evaluated on the other unseen domain; the selected baselines are ExOPD [51], Uni-OPD [12], and EOPD [18].Source: Figure 7 · p. 12, §4.2.4 · p. 13

4.3.1 Training Stability

• By step 80, DOPD surpasses ExOPD [51], SDPO [15], and EOPD [18] at step 200; its entropy stabilizes after step 110, unlike SDPO’s collapse around step 95.The paper attributes the self-distillation degradation to insufficient, overly homogeneous supervision that limits exploration.Source: §4.3.1 · p. 13

4.3.2 Privileged Information Analysis

• LLM step-level hints without execution yield gains of 8.3 and 10.4 points, while VLM bounding boxes with object labels improve 4.2 and 7.2 points over baseline.Directly providing ground-truth answers causes the most severe information gap and can underperform the baseline without privileged input.Source: Table 4 · p. 14, Table 5 · p. 14, §4.3.2 · p. 14

4.3.3 Token Analysis

• Token categories align with differentiated distillation: high-gap, high-teacher/low-student tokens encode key privileged knowledge, while low-gap tokens reflect consensus or low-information content.High-gap, high-student/low-teacher tokens likely reflect self-consistent or local exploration branches; differentiated treatment targets tokens by functional role.Source: Figure 9 · p. 14, §4.3.3 · pp. 14–15
• Using only high-teacher/low-student tokens beats equal-distillation Vanilla OPD by 4.6 points, while adaptive use of all four token types improves over equal distillation by over 8 points.Naively adding the other token types under equal distillation yields only marginal gains and can degrade performance.Source: Table 6 · p. 15, §4.3.3 · p. 15

4.3.4 Divergence Analysis

• With equal token weights, broader alignment from sampled to Top-K to full vocabulary improves performance but raises memory, and JS outperforms forward and reverse KL in these settings.The authors frame these choices as a trade-off across divergence configurations.Source: Table 7 · p. 15, §4.3.4 · pp. 15–16

4.3.5 Sensitivity & Ablation Studies

• Setting βw = 0.3 and βl = 0.6 balances stronger critical-token contributions with auxiliary signals from other tokens.Ablations find privileged input indispensable, teacher signals the primary performance driver, student signals irreplaceable, and token-wise divergence effective.Source: Figure 10 · p. 15, Table 8 · p. 15, §4.3.5 · p. 16

5 Conclusion

• DOPD consistently outperforms Vanilla OPD and strong competitive baselines across LLM and VLM settings, with gains in robustness, continual learning, out-of-distribution generalization, and training stability.The conclusion also reports superior distillation performance overall.Source: §5 · p. 16
• Privileged-teacher superiority may reflect information asymmetry rather than transferable capability, and supervision signals are unevenly distributed across tokens.Source: §5 · p. 16
• DOPD uses the privilege advantage gap and relative token probabilities to apply strong full-vocabulary teacher distillation to capability-bearing tokens and light or weak distillation otherwise.It also uses auxiliary student self-optimization for tokens without a capacity advantage gap.Source: §5 · p. 16

• Privileged information must be available and high quality, and obtaining it adds annotation, generation, and filtering costs.Source: §6 · p. 16
• DOPD requires one additional student-model forward pass, adding computational overhead relative to Vanilla OPD.Source: §6 · p. 16
• The current routing strategy is intuitive and empirically stable but still relies on heuristics.Source: §6 · p. 16
• Future work may develop more reliable, cost-effective privileged-information acquisition and more principled or learnable routing, including alternative ways to detect advantage gaps.The authors also propose exploring dynamic teacher/student distillation beyond LLMs and VLMs toward more interpretable, efficient, and trustworthy paradigms.Source: §6 · p. 16

A Details of Privileged Input

• Figure 11 demonstrates LLM-based privileged input.Source: Figure 11 · p. 21
• Figure 12 demonstrates VLM-based privileged input.Source: Figure 12 · p. 22
• Figure 13 presents prompts for privileged-input generation.Source: Figure 13 · p. 23