Source-linked AI summary
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, Fuli Luo
TL;DR
Integrating capabilities learned through different domain-specific RL pipelines into a single LLM remains an open post-training problem. MOPD independently trains domain teachers and distills them on the student’s own rollouts, providing dense on-policy supervision. On Qwen3-30B-A3B it outperforms the listed integration baselines, while also supporting parallel teacher development and deployment on MiMo-V2-Flash.
Problem
Building one LLM that performs well across multiple domains remains difficult despite specialised RL pipelines reliably improving individual target capabilities.
Method
MOPD independently trains specialised domain RL teachers and distills them into a single student using dense teacher supervision on the student’s own rollouts.
Results
5.5 normalised-score points: MOPD leads the next-best integration method on Qwen3-30B-A3B, reaching 0.937 versus 0.882, and closes 91–95% of student–teacher headroom across domains.
Takeaways & Limitations
MOPD supports parallel, independent domain-teacher development and has been applied to the industrial-scale MiMo-V2-Flash model.
Abstract
from arXiv · showhide
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.
1 Introduction
Integrating domain-specific capabilities into one LLM remains difficult because existing approaches trade off efficiency, peak performance, and stability. MOPD instead distills independently trained domain teachers into a student on its own rollouts, achieving strong multi-domain results and supporting parallel teacher development.
- The capability-integration gap: Existing capability-integration methods trade off learning efficiency, attainable peak performance, and training stability.Mix-RL suffers cross-domain interference, Cascade RL risks capability decay, Off-Policy Finetune induces exposure bias, and Param-Merge is often unstable.
- MOPD: MOPD independently trains specialised RL teachers, then distills their capabilities into one student through on-policy learning.The student generates its own rollouts, while per-prompt routing supplies dense teacher supervision in policy space.
- Practical implications: MOPD enables independent, parallel development of domain teachers rather than requiring a fixed cross-domain training sequence.This separates domain-specific RL development and avoids the coupling typical of joint capability integration.
- Empirical validation: 5.5 points: MOPD leads the strongest baseline on Qwen3-30B-A3B’s normalised score, reaching 0.937 versus 0.882.The comparison covers Math, Instruction Following, and Software Engineering across Mix-RL, Cascade RL, Off-Policy Finetune, Param-Merge, and MOPD.
- Empirical validation: MOPD was also deployed on MiMo-V2-Flash, demonstrating practical effectiveness at industrial frontier-model scale.The paper further reports validation of the method’s top-k distillation variant, same-origin teacher requirement, and iterative refinement benefits.
2 Related Work
Prior work integrates multiple capabilities through mixed or sequential RL, off-policy distillation, or weight-space model merging. These approaches respectively face interference, exposure bias, or parameter-conflict concerns, motivating MOPD’s on-policy multi-teacher setting.
- Mixed and cascade RL: Mixed RL combines samples from multiple domains in shared training batches while retaining domain-specific rewards and advantages.Its central integration strategy is joint optimization across domains.
- Mixed and cascade RL: Cascade RL trains domains sequentially, allowing later stages to alter capabilities learned earlier.The method represents the sequential alternative to mixed multi-domain RL.
- Distillation for LLMs: Classical LLM distillation is off-policy because it trains on fixed teacher completions, allowing student inference trajectories to drift from the training distribution.On-policy distillation instead uses student rollouts with the teacher as a scoring signal, but prior formulations are restricted to one teacher and one domain.
- Model merging: Model merging combines independently trained checkpoints in weight space through weight averaging or task-vector arithmetic, with later methods addressing parameter conflicts.These approaches obtain combined models without additional training.
3 Method
MOPD integrates independently trained domain teachers by distilling their signals on the student’s own multi-domain rollouts. Its policy-space design combines dense token-level supervision with modular teacher training and asynchronous infrastructure.
- Pipeline: MOPD uses three stages: general SFT, parallel domain-specialised RL, and final distillation into one unified model.The Stage-1 checkpoint initializes both the domain experts and the Stage-3 student.
- Pipeline: Each Stage-3 prompt is routed to its matching frozen teacher, while the student generates the trajectory used for distillation.The teacher prefills the student rollout to produce per-token probability information.
- Objective: The student is optimized with per-token reverse KL between its distribution and the dispatched teacher along each student-generated trajectory.This preserves on-policy training while using the teacher as a token-level scoring signal.
- Design benefits: MOPD avoids exposure bias because training uses student rollouts, and dense per-token supervision is intended to lower variance relative to trajectory-level RL rewards.The method also merges capabilities in policy space through prompt routing rather than weight averaging or task-vector arithmetic.
- Efficient distillation: Top-k distillation uses an added correction term so the truncated objective is minimized when student and teacher distributions match on the selected tokens.The top-k form also reduces communication overhead compared with transmitting the full vocabulary distribution for every token.
- Infrastructure: Teacher prefill runs as an asynchronous service outside the RL trainer, overlapping with student sampling so teacher cost is essentially hidden behind rollout generation.This avoids folding teacher computation directly into the RL loop, which would add serial latency and infrastructure complexity.
4 Experiments
Experiments compare MOPD with five capability-integration paradigms across Qwen3-30B-A3B and MiMo-V2-Flash. MOPD achieves the strongest and most uniform integration, converges efficiently, remains stable with same-origin teachers, and improves further through another round.
- Evaluation setup: Experiments evaluate Mix-RL, Cascade RL, Off-Policy Finetune, Param-Merge, and MOPD across math, instruction following, and software engineering.The Qwen3-30B-A3B evaluation uses AIME25/AIME26, IFBench/IFEval, and SWE-bench Verified.
- Main results: 0.937 is MOPD’s normalised score, exceeding Mix-RL’s 0.882 by 0.055 and producing the most uniform per-domain profile.MOPD’s per-domain scores fall in [0.91, 0.95], a range of 0.044; Mix-RL, Cascade RL, and Off-Policy Finetune show wider or distinct domain gaps.
- Main results: 0.328 is Linear averaging’s normalised score, while Task Arithmetic reaches 0.857 but varies widely across domains.Task Arithmetic reaches 1.00 on IF but closes only 73% of Math headroom, showing sensitivity to the merging recipe and benchmark.
- Sample efficiency: 25K IF samples and 30K SWE samples suffice for MOPD to reach teacher-level plateaus, whereas Mix-RL needs 150–180K samples per domain.The paper attributes faster convergence to dense per-token teacher supervision rather than trajectory-level RL reward.
- Scaling: MOPD matches or exceeds the corresponding teacher on most MiMo-V2-Flash benchmarks, with regressions of −2.2 on IFBench and −0.8 on SWE-Bench Verified.The full pipeline uses domain teachers for Math, Code, IF, SWE, and Tool Use.
- Stability analysis: Same-origin teachers yield comparable, stable loss variants, while an external teacher with approximately 5× higher initial KL causes degradation and top-𝑘 collapse around step 18.Same-origin runs have initial reverse KL around 0.04 and stable entropy near 0.30; the external-teacher setting starts near 0.19 and destabilizes.
5 Discussion
MOPD separates domain capability production from capability integration, allowing independent parallel development while isolating training risks across domains.
- Structural decoupling: MOPD decouples domain-specialised RL from multi-domain capability integration into separate stages.Stage-2 teachers are developed independently, while Stage 3 integrates their capabilities through distillation.
- Parallel development: Independent domain teachers let teams iterate concurrently without a fixed ordering.Each team can develop its own rewards, sandboxes, and data pipelines in parallel.
- Recipe-level decoupling: Each domain can choose its own RL algorithm, rollout procedure, reward function, and hyperparameters without cross-domain conflicts.Recipe-level decoupling removes interference between domain-specific optimization choices.
- Risk isolation: Training restarts are confined to the affected domain rather than restarting the entire multi-domain run.This isolates the impact of algorithm or hyperparameter tuning from other teachers’ development.
6 Conclusion
MOPD integrates multiple domain teachers in policy space by distilling dense token-level signals on student-generated trajectories. It achieves strong aggregate capability integration on Qwen3-30B-A3B and transfers to MiMo-V2-Flash.
- Method: MOPD routes each student-generated trajectory to its corresponding frozen domain teacher for dense, token-level log-probability supervision.The student samples its own rollouts, and the teacher scores every position.
- Results: 5.5 normalised-score points separate MOPD from the next-best integration method on Qwen3-30B-A3B.MOPD closes 91–95% of the student–teacher headroom on every domain.
- Frontier-scale deployment: On MiMo-V2-Flash, MOPD matches or exceeds the corresponding teacher on most benchmarks.This applies the same recipe in an industrial-scale setting.
- Analysis: Policy-gradient and top-k loss forms perform comparably, while same-origin teachers are critical for stable optimization.The analysis also supports multi-round student–teacher evolution.
A Training Details
Training uses data from math, instruction-following, and software-engineering domains, with domain-specific sources and sequence budgets. RL hyperparameters differ between Math/IF and SWE.
- Domains: Training spans math, instruction following, and software engineering.The supplied training details identify separate data sources for these three domains.
- Math: Math SFT primarily uses Mixture-of-Thoughts, while RL data include BigMath and ORZ.The maximum sequence length for Math is 32,768.
- Instruction following: Instruction-following SFT prompts follow IFBench construction and are distilled on gpt-oss-120b.Its RL data are described as similarly sourced in the supplied passage.
- RL hyperparameters: RL uses on-policy GRPO with Dynamic Sampling, discarding rollout data after one gradient update.Per-domain RL uses a learning rate of 3 × 10^-6; Math and IF use batch size 144, while SWE uses 80.
B Evaluation Details
Evaluation protocols vary by domain and benchmark, with repeated sampling for difficult math questions and single evaluations for instruction following and software engineering.
- Math: AIME25 and AIME26 use 32 samples per question and report average accuracy, avg@32.Repeated sampling reduces variance in low-sample-count, high-difficulty math evaluation.
- Instruction following: IFBench and IFEval evaluate each instruction-following question once.Both benchmarks use the single-evaluation protocol described for instruction following.
- Software engineering: SWE-bench Verified evaluates each software-engineering task once.The protocol differs from the repeated-sampling procedure used for the math benchmarks.
- Decoding: All benchmarks use sampling temperature 1.0 without top-p or top-k truncation.The decoding configuration is shared across the evaluation suite.