Source-linked AI summary
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan
TL;DR
Reliable OPD remains poorly understood, with insufficient student exploration and unreliable teacher supervision limiting optimization across LLM and MLLM settings. Uni-OPD addresses these bottlenecks through dual-perspective data balancing and outcome-guided margin calibration, with experiments demonstrating effectiveness and versatility across 16 benchmarks.
Problem
OPD reliability is limited by insufficient exploration of informative student-generated states and unreliable teacher supervision for student rollouts.
Method
Uni-OPD combines offline difficulty-aware and online correctness-aware data balancing with outcome-guided margin calibration in a unified LLM and MLLM framework.
Results
Experiments across 16 benchmarks demonstrate Uni-OPD's effectiveness and versatility in multi-teacher, strong-to-weak, and cross-modal settings.
Takeaways & Limitations
Uni-OPD provides a practical foundation for scalable and reliable distillation across models and modalities.
Takeaways & Limitations
Future work is needed to extend Uni-OPD to larger-scale teacher distillation settings and broader capability-merging scenarios.
Abstract
from arXiv · showhide
On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model. Despite its empirical success, the conditions under which OPD yields reliable improvement remain poorly understood. In this work, we identify two fundamental bottlenecks that limit effective OPD: insufficient exploration of informative states and unreliable teacher supervision for student rollouts. Building on this insight, we propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), centered on a dual-perspective optimization strategy. Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training. From the teacher's perspective, we show that reliable supervision hinges on whether aggregated token-level guidance remains order-consistent with the outcome reward. To this end, we develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories. We conduct extensive experiments on 5 domains and 16 benchmarks covering diverse settings, including single-teacher and multi-teacher distillation across LLMs and MLLMs, strong-to-weak distillation, and cross-modal distillation. Our results verify the effectiveness and versatility of Uni-OPD and provide practical insights into reliable OPD.
1 Introduction
The paper frames reliable OPD as requiring both informative student exploration and trustworthy teacher supervision. Uni-OPD addresses these bottlenecks through data balancing and outcome-guided margin calibration, with experiments spanning LLMs, MLLMs, and diverse distillation settings.
- OPD combines on-policy sampling with token-level teacher supervision to train students on their own sampled trajectories.This combines strengths associated with reinforcement learning and supervised fine-tuning.
- Reliable OPD depends on exploring diverse, appropriately difficult student-generated states and preserving order consistency between teacher guidance and outcome reward.Correct trajectories should receive higher aggregated token-level scores than incorrect trajectories.
- Uni-OPD uses offline difficulty-aware and online correctness-aware data balancing to promote exploration of informative student-generated states.These are the student-perspective components of the framework.
- Uni-OPD introduces outcome-guided margin calibration to obtain more reliable teacher supervision.The mechanism is motivated by the need to align token-level guidance with trajectory-level correctness.
- Experiments cover 5 domains and 16 benchmarks, including single-teacher and multi-teacher distillation across LLMs and MLLMs, strong-to-weak distillation, and cross-modal distillation.The reported results verify Uni-OPD's effectiveness and versatility in these settings.
2 Related Work
Related work positions OPD as a way to transfer expert capabilities using feedback on student trajectories, while prior extensions remain limited across modalities and task coverage. Uni-OPD is presented as a broader framework for LLMs and MLLMs.
- Knowledge distillation commonly aligns a student with a larger teacher using off-policy forward KL divergence on a static dataset.
- OPD provides teacher feedback on student on-policy trajectories, combining trajectory sampling with token-level supervision.It is described as an efficient approach for merging capabilities from multiple experts and supporting strong-to-weak distillation.
3 Methodology
Uni-OPD optimizes on-policy distillation from complementary student and teacher perspectives: it promotes informative student exploration and calibrates teacher supervision against outcome correctness.
- Preliminaries: OPD trains on student-sampled trajectories using token-level teacher feedback under a reverse KL objective.The student policy samples trajectories, and the objective minimizes reverse KL divergence to the teacher policy over those samples.
- Teacher supervision: Token-level teacher guidance can become unreliable when trajectory-level aggregation is misaligned with outcome correctness, including in out-of-distribution regions.The analysis identifies noisy teacher likelihoods, overestimation of incorrect trajectories, and underestimation of correct ones as failure modes.
- Overview: Uni-OPD unifies single-teacher and multi-teacher distillation by aggregating supervision from expert teachers while optimizing student exploration and teacher reliability.Its dual-perspective recipe combines data balancing with outcome-guided margin calibration.
- Student exploration: Offline difficulty-aware balancing upsamples mid-difficulty samples instead of removing all-correct or all-wrong samples, preserving diversity and improving math and code performance.The strategy reshapes the difficulty distribution toward a more uniform spectrum and supports broader exploration of informative trajectories.
- Student exploration: Online correctness-aware balancing enforces both correct and incorrect trajectories within rollout groups to avoid outcome-homogeneous samples and uninformative gradients.An appropriate correct/incorrect balance outperforms using only correct samples or an excessively high correct/incorrect ratio.
- Teacher supervision: Outcome-guided margin calibration compares the lowest-scoring correct trajectory with the highest-scoring incorrect trajectory and uses outcome rewards to restore order consistency.A nonnegative prompt-level margin indicates that all positive trajectories rank above all negative trajectories.
4 Experiments and Analysis
Experiments evaluate Uni-OPD across LLM and MLLM distillation settings, including single- and multi-teacher, strong-to-weak, and cross-modal transfer. Across these settings, Uni-OPD reports stronger performance, while ablations support complementary roles for data balancing and margin calibration.
- LLM Distillation: 48.7 on math reasoning and 63.2 on code generation are Uni-OPD’s highest single-teacher LLM scores.Uni-OPD consistently outperforms OPD and ExOPD in the reported single-teacher setting.
- LLM Distillation: 1.5% and 3.4% are Uni-OPD’s multi-teacher gains over OPD on math reasoning and code generation, respectively.These gains indicate improved merging of distinct teacher capabilities into one student model.
- MLLM Distillation: 63.9, 51.7, and 83.7 are Uni-OPD’s single-teacher MLLM scores on math reasoning, logic reasoning, and document understanding.For multi-teacher MLLM distillation, average scores increase from 57.9 to 61.0, 50.0 to 52.5, and 83.4 to 83.9 across those domains.
- Strong-to-Weak Distillation: 42.9 and 60.8 are Uni-OPD’s 4B strong-to-weak scores on mathematical reasoning and code generation, exceeding OPD by 1.7 and 1.8 points.With a 1.7B student, scores reach 25.0 on math reasoning and 52.7 on code generation.
- Cross-Modal Distillation: 65.6 versus 64.1 on code generation and 63.9 versus 62.7 on math reasoning are Uni-OPD’s cross-modal averages versus OPD.The setting transfers capabilities from textual-code and multimodal-math teachers into one multimodal student.
- Ablation Study: Removing either data-balancing strategy reduces exposure to diverse, challenging trajectories, while removing margin calibration misaligns token feedback with outcome rewards.The ablation results attribute exploration to offline and online balancing and supervision reliability to margin calibration.
- Ablation Study: Margin shift slightly outperforms margin mask and is therefore used in the main experiments.Both calibration variants produce consistent gains over the OPD baseline.
5 Conclusion and Future Work
The paper concludes that Uni-OPD unifies reliable distillation across LLMs and MLLMs by addressing student exploration and teacher supervision. Future work targets larger teachers, broader capability merging, and mechanistic understanding of OPD.
- Conclusion: Uni-OPD combines data balancing for student exploration with outcome-guided margin calibration for teacher supervision.The framework is presented as generalizing across LLMs and MLLMs and evaluated across multi-teacher, strong-to-weak, and cross-modal settings.
- Conclusion: The paper identifies insufficient exploration of informative states and unreliable teacher supervision as two key OPD bottlenecks.Its conclusion frames these bottlenecks as the basis for the dual-perspective strategy.
- Future Work: Future work includes larger-scale teacher distillation, broader capability merging, and mechanistic study of training dynamics and parameter geometry.Suggested application areas include agentic planning, tool use, and long-horizon decision making.
Supplementary Material
The supplementary material provides implementation, methodological, evaluation, ablation, and related-background details for Uni-OPD. It includes complementary offline and online data-balancing procedures and margin-calibration analyses.
- Data Balancing: Data-balancing supplements describe offline difficulty-aware and online correctness-aware strategies.Offline balancing shapes prompt-level difficulty before training, while online balancing regulates correct and incorrect rollout composition.
- Margin Calibration: Margin-calibration supplements analyze trajectory-return order consistency, its violations, statistical evidence, and outcome-guided calibration.The listed sections organize the paper’s analysis of reliable teacher supervision.
- Training and Evaluation: Additional supplements document training setup, data, reward acquisition, pseudocode, training dynamics, complexity, benchmarks, and evaluation setup.These materials provide implementation and evaluation details beyond the main text.
- Further Evaluations: Further evaluations include more results, downstream-task evaluation, and additional ablations.The supplementary structure explicitly separates these analyses under further evaluations.
- Related Material: Supplementary sections cover multimodal models, reinforcement learning, and on-policy distillation background.These topics are listed under the supplementary related-material sections.
A.1 Offline Difficulty-Aware Data Balancing
Uni-OPD's offline difficulty-aware strategy preserves the full difficulty spectrum instead of discarding easy or hard samples, then reweights underrepresented regions to improve training coverage.
- Limitations of aggressive difficulty filtering: Aggressive filtering is counterproductive in token-level reward OPD: removing either the easiest or hardest difficulty tier consistently hurts final performance.Easy instances calibrate baseline behavior, while intermediate instances provide contrastive signals between correct and incorrect trajectories.
- Difficulty-aware data balancing: Offline difficulty-aware balancing preserves all difficulty tiers and reweights their regions rather than truncating the training distribution.The strategy is motivated by the finding that each tier contributes a distinct token-level credit pattern.
- Difficulty-aware data balancing: For U-shaped distributions, Uni-OPD upsamples instances with 1–7 correct responses out of N = 8 to fill the underrepresented middle region.This reweighting flattens the overall difficulty distribution.
- Difficulty-aware data balancing: For mirrored-J-shaped distributions, Uni-OPD upsamples all non-trivial instances with 1–8 correct responses to counteract the long tail of easy samples.The resulting reweighting also flattens the overall difficulty distribution.
B.3 Statistical Evidence for Order-Consistency Violations
Order-consistency violations are widespread: incorrect trajectories often outrank correct ones within prompts, despite modest global selectivity. Uni-OPD addresses this reliability problem with masking or additive margin-shift calibration.
- Statistical evidence: At N =4, 24.2% of prompts violated order consistency, compared with 10.4% that remained consistent.The violating fraction reached roughly half of prompts at N =16 as larger rollouts exposed more inverted pairs.
- Statistical evidence: Correct rollouts had only slightly higher mean GOPD than incorrect rollouts, whose distributions overlapped heavily and included a degenerate mass near GOPD = 0.The near-zero mass corresponded to repetitive generations.
- Statistical evidence: The pairwise inversion rate stayed around 45% across rollout sizes, leaving only about 55% of comparable pairs correctly ordered.This measures violations within individual prompts rather than aggregate discrimination.
- Calibration strategies: Margin Mask removes unreliable trajectories, whereas Margin Shift applies a minimal additive correction to enforce m(q)≥δ.Both operate on trajectory-level distillation returns; Mask removes adversarial samples, while Shift preserves them.
- Calibration strategies: MinMax enforces worst-case separation between every positive and negative trajectory, while Mean captures average ordering and is less sensitive to outliers.The two modes define alternative prompt-level margins for calibration.
- Calibration strategies: Greedy Margin Mask sorts correct trajectories by ascending GOPD and incorrect trajectories by descending GOPD before greedily removing the more adversarial side.Removal stops when the target margin is met, beneficial removal ends, or the minimum retention ratio ρ is reached.
C.5 Training Dynamics
Uni-OPD improves response correctness over OPD during training while preserving exploration-related dynamics and adding only trajectory-level computational overhead beyond standard OPD.
- Training dynamics: Uni-OPD peaked at 80.6% response correctness versus OPD’s 75.2%.From comparable starting points, Uni-OPD averaged 75.5% over the final 10 steps versus OPD’s 69.1%, a +6.4-point difference.
- Training dynamics: Uni-OPD maintained mildly rising policy entropy while achieving higher correctness than OPD.The supplied passage frames entropy as an exploration indicator and reports no exploration sacrifice.
- Training dynamics: Online balancing costs O(BN) per iteration, while Uni-OPD’s calibration components operate per prompt rather than per rollout token.The dominant OPD costs scale with total rollout tokens and transformer forward passes.
D.1 Evaluation Benchmarks
The evaluation suite spans textual and multimodal reasoning, code generation, logic reasoning, and document understanding across varied difficulty levels and reporting protocols.
- Coverage: The benchmark suite covers five capability axes across textual and multimodal tasks.It includes textual math reasoning, textual code generation, multimodal math reasoning, multimodal logic reasoning, and document understanding.
- Textual benchmarks: Textual math benchmarks include AIME 2024/2025 and HMMT25, targeting challenging mathematical reasoning.HMMT25 spans algebra, geometry, combinatorics, and related areas.
- Textual benchmarks: Textual code benchmarks include HumanEval+, MBPP+, and LiveCodeBench v6, covering correctness, programming skills, execution, self-repair, and test prediction.LiveCodeBench v6 is described as contamination-free and continuously updated.
- Multimodal benchmarks: Multimodal evaluation includes visual math, logical reasoning, and document-understanding benchmarks such as MathVision, LogicVista, VisuLogic, AI2D, ChartQA, DocVQA, and InfoVQA.These tasks require reasoning over images, diagrams, charts, documents, layouts, and visual elements.
- Evaluation protocol: Textual evaluation samples N =32 solutions for math and N =4 for code, using Math-Verify10 for math verification.Multimodal evaluations follow the official LMMs-Eval protocols and configurations.
E.1 More Evaluation Results
Across LLM and MLLM students, Uni-OPD consistently outperforms standard OPD in single-teacher, multi-teacher, and cross-modal distillation settings.
- LLM and MLLM distillation: Uni-OPD consistently outperformed standard OPD across LLM and MLLM domains in both single-teacher and multi-teacher distillation.Gains appeared in math, code, multimodal math, logic, and document understanding, while multi-teacher results narrowed the gap to the teacher ensemble.
- Cross-modal distillation: In cross-modal distillation, LCB improved from 39.6 to 41.4 and LogicVista Accuracy from 50.0 to 52.0.The setup jointly distilled text-only code and multimodal logic reasoning from two domain-specific teachers.
- Cross-modal distillation: Uni-OPD integrated heterogeneous text-only and multimodal data in a single training run while improving both code and logic-reasoning averages.This result was reported for a Qwen3-VL-4B-Instruct student.
E.2 Downstream Task Evaluation
Uni-OPD retains strong general downstream performance after OPD, extending its gains beyond task-specific math reasoning and code generation.
- Uni-OPD retains strong performance across 8 general benchmarks after OPD.The evaluation is designed to assess whether general-purpose capabilities are maintained.
- The general-capability evaluation complements Uni-OPD’s reported gains on math reasoning and code generation benchmarks.
E.3 Further Ablation
Further ablations show that margin shift improves OPD across hyperparameter choices, while larger rollout groups improve calibrated performance up to N=16.
- Margin shift: All margin-shift variants outperform vanilla OPD across math reasoning and code generation benchmarks.The ablation varies scope, normalization mode, and shift direction.
- Margin shift: 48.2 average performance is achieved by Group + Mean + Spread, the best margin-shift variant across the evaluated tasks.Group-level mean normalization with bidirectional shifting provides the most calibrated return signal among the tested variants.
- Rollout number: 48.2 average performance is reached at N=16 with margin shift, increasing from 45.3 at N=4.Increasing N from 16 to 32 yields comparable performance, so N=16 is used as the default.
- Rollout number: The base OPD method remains stable across rollout numbers at 44.3–44.6 average performance when the global batch size is fixed.In contrast, margin shift benefits from larger rollout groups because more responses per prompt improve relative return estimation.
G Case Studies
Case studies across textual and multimodal tasks show Uni-OPD producing more efficient and often more correct outputs than standard OPD and the Student model.
- The case studies span textual math reasoning, code generation, logical reasoning, multimodal math reasoning, and chart understanding.
- Textual cases: Uni-OPD restores order consistency between correct and incorrect trajectories in the math reasoning case.Standard OPD assigns high returns to incorrect trajectories and low returns to correct ones; margin calibration yields a more reliable supervision signal.
- Multimodal cases: Across multimodal cases, Uni-OPD produces more concise outputs while maintaining correctness on complex reasoning problems.The Student model and standard OPD frequently generate excessively long responses that are truncated before reaching a final answer.
- Multimodal cases: Uni-OPD achieves correct answers on visual reasoning examples where the Student model or OPD fails, including pattern, chart, geometry, and mathematical reasoning tasks.The examples cover LogicVista, VisuLogic, ChartQA, MathVision, and WeMath.
- Textual cases: Uni-OPD balances computational efficiency and code readability by producing a single-pass, well-commented Find Max solution.The Student model has limited readability, while OPD introduces redundant two-pass computation.