PaperlayerView exact revision on arXiv

Source-linked AI summary

D$^3$-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li, Mu Chuan, Huayu Deng, Guojian Zhan, Wenliang Chen, Yao Hu, Min Zhang

arXiv:2608.24987v1cs.LGcs.AI

TL;DR

MOPD’s fixed domain mixture overlooks asynchronous convergence, wasting compute on fast-converging domains while slower domains remain undertrained. D3-MOPD asynchronously reuses per-domain reverse-KL signals to update sampling without changing the core loop, closing 97% of the average student-to-teacher gap versus 63% for vanilla MOPD.

  • Problem

    MOPD fixes domain proportions before training even though domains converge at different rates, causing compute to remain allocated to domains that have plateaued.

  • Method

    D3-MOPD uses an asynchronous watcher and stratified data source to update domain sampling from per-domain reverse-KL signals while leaving the core training loop unmodified.

  • Results

    97% of the average student-to-teacher performance gap is closed by D3-MOPD, compared with 63% for vanilla MOPD.

  • Takeaways & Limitations

    D3-MOPD improves peak quality and training efficiency by dynamically redirecting sampling as domains progress at different rates.

  • Takeaways & Limitations

    The paper’s analysis formalizes vanilla MOPD, studies isolated per-domain dynamics, and demonstrates static-mixture waste within its evaluated setting.

Abstract

from arXiv · show

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D$^3$-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D$^3$-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D$^3$-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3$\times$ reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.

1 Introduction

MOPD combines domain-expert teachers through reverse-KL supervision on the student’s own trajectories, but fixed domain mixtures misallocate training as domains converge at different rates. D3-MOPD dynamically updates sampling from existing per-domain KL signals without modifying the core training loop.

  • Background: MOPD combines several domain-expert teachers by minimizing per-token reverse-KL divergence on the student’s own trajectories.Each domain teacher produces distributions on corresponding student-generated trajectories, yielding a unified model without serving all teachers at inference.
  • Problem: Fixed mixtures waste compute on domains that plateau early while limiting training for domains that remain far from convergence.The imbalance arises because per-domain learning rates differ substantially during the fixed training budget.
  • Contribution: D3-MOPD repurposes already-computed per-domain reverse-KL signals to update domain mixture weights online through an asynchronous watcher.The watcher combines remaining teacher gap and recent descent velocity to determine updated sampling ratios.
  • Results: 97% of the average student-to-teacher performance gap is closed by D3-MOPD, compared with 63% for vanilla MOPD.On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D3-MOPD also surpasses specialist teachers on three of seven benchmarks.

2 Motivation: Limitations of Fixed Domain Mixtures

Vanilla MOPD fixes domain sampling proportions before training, despite substantial differences in per-domain convergence. As a result, uniform sampling continues allocating data to already-low-KL domains while slower domains still need training.

  • Fixed Domain Mixtures: Vanilla MOPD pools and shuffles domain datasets, fixing each domain’s expected batch share before training.The expected share is pk = Nk/∑i Ni, although realized batch proportions fluctuate around it.
  • Domain-Specific Learning Dynamics: Math plateaus early, Code improves more slowly without converging within budget, and IF continues reducing loss while retaining much higher absolute reverse-KL.IF’s absolute reverse-KL remains one to two orders of magnitude above the other domains throughout training.
  • Computational Inefficiency of Fixed Mixtures: Code, Math, and IF enter low-KL phases at roughly steps 48, 96, and 144, respectively, under uniform vanilla MOPD.Despite these staggered phases, the uniform mixture continues assigning one-third of every batch to each domain.
  • Computational Inefficiency of Fixed Mixtures: Valuable training compute is wasted on domains long past the point where additional samples yield further KL reductions.The stated source of waste is holding pk fixed rather than the data or teachers themselves.
  • Takeaway: The motivation is for a dynamic sampling ratio pk that adapts smoothly to native training signals.This follows from domains reaching low-KL plateau phases at different stages under vanilla MOPD.

3 Method

D3-MOPD adds an off-process watcher and stratified data source that dynamically adapt domain sampling from reverse-KL signals. The design preserves the vanilla training loop while supporting arbitrary numbers of domains.

  • Framework: D3-MOPD uses an off-process watcher to compute per-domain mixture weights from reverse-KL signals and a stratified source to assemble batches accordingly.These are the two components introduced into the vanilla MOPD training loop.
  • Framework: The D3-MOPD pipeline generalizes to any number of domains K.
  • Framework: The watcher recovers per-domain reverse-KL signals from logged per-sample values and runs separately from the trainer.The trainer continues generating student responses and obtaining teacher distributions as in vanilla MOPD.
  • Stratified Data Source: The stratified data source replaces pool-and-shuffle loading and applies the watcher-supplied mixture to batch construction.Its per-batch mixture incorporates batch-level jitter around the target domain ratios.

A. Data pools

D3-MOPD dynamically composes domain batches by converting per-domain reverse-KL histories into sampling mixtures. The watcher combines normalized remaining gap and recent descent velocity, then maps the result to valid, continuously updated domain ratios while preserving warmup, floors, and fallback behavior.

  • Composite signal: The composite signal multiplies remaining gap by descent velocity, favoring domains that retain room to improve and are actively improving.If all domains plateau, normalization defaults to zero signals and the mixture becomes uniform.
  • Remaining gap: The remaining-gap signal normalizes each domain’s smoothed reverse-KL by its initial value, enabling comparisons across domains with different absolute KL scales.The normalized gap lies in (0, 1] and approaches zero as a domain converges.
  • Descent velocity: The descent-velocity signal averages relative KL changes across R non-overlapping windows and is positive only when KL decreases on average.Window averaging reduces sensitivity to short-term noise, while clipping prevents non-decreasing trajectories from receiving positive velocity.
  • Mixture mapping and warmup: A temperature-controlled softmax converts composite signals into a mixture with a per-domain floor, while warmup uses a uniform fallback before sufficient KL history is available.The floor keeps every domain represented, and the watcher begins updating only after 2W observations.
  • Batch construction: Each batch applies optional jitter, renormalizes the target ratios, assigns integer domain counts, and fills the batch from the corresponding domain pools.The allocation procedure distributes any remaining samples according to the largest fractional parts so the batch size is preserved.
  • Dynamic batch composition: D3-MOPD uses a watcher and stratified data source to update domain mixtures without modifying the core training loop.The watcher writes updated ratios to a status file, which the data source reads when assembling batches.

4 Experiments

D3-MOPD is evaluated against vanilla MOPD using four domain-expert teachers, seven benchmarks, and repeated rollout checkpoints. It achieves higher peak accuracy, faster convergence, and stronger student-to-teacher gap closure, while ablations support its scheduler design.

  • Evaluation setup: Qwen3.6-35B-A3B is distilled from four domain-expert teachers covering Math, Code, Instruction Following, and Tool-use.The evaluation uses seven benchmarks spanning these four training domains.
  • Main results: D3-MOPD reaches a higher average peak of 62.3 at step 95 than vanilla MOPD’s 61.4 at step 143.Individual benchmark gains range from +0.2 on AIME 2025 to +2.8 on IFBench.
  • Main results: D3-MOPD surpasses vanilla MOPD’s average peak by step 47, scoring 62.1 versus the baseline’s 61.4 at step 143.It matches or exceeds every vanilla per-benchmark peak within the first 79 steps.
  • Convergence analysis: Code benchmarks peak earliest, instruction-following benchmarks peak latest, and mathematics falls between them, matching the observed r-KL convergence order.LiveCodeBench peaks by step 79, OJBench by step 95, IFBench at step 175, and IFEval at step 191.
  • Ablation studies: The composite signal reaches 62.34, outperforming single-signal variants at 61.41 without velocity and 61.46 without gap.The velocity-only variant peaks at step 255, while the no-velocity variant peaks at step 127.
  • Ablation studies: Removing batch jitter lowers peak average accuracy by 0.71, while smoothing velocity with R=3 raises the average from 62.17 to 62.34.The no-jitter variant keeps the same peak step, whereas smoothing mitigates random r-KL variance.
  • Supplementary analysis: D3-MOPD closes 97% of the average student-to-teacher gap versus 63% for vanilla MOPD and surpasses specialist teachers on three of seven benchmarks.The normalized score exceeds 1.0 on HMMT(N), IFEval, and OJBench C++.

5 Related Work

Related work improves on-policy distillation through local token- or sample-level supervision and extends it to multiple domain teachers. D3-MOPD addresses a complementary problem by dynamically reallocating data across domains.

  • On-Policy Distillation: On-policy distillation trains students on their own generations under teacher supervision using reverse-KL objectives.Recent methods modify token- or sample-level supervision, including entropy-based KL selection and noisy-gradient suppression.
  • Position of this work: D3-MOPD complements local supervision methods by dynamically adjusting per-domain data allocation rather than refining which tokens or samples receive supervision.Its focus is the mixture across domains in multi-teacher training.
  • Multi-teacher OPD: Multi-teacher OPD combines several domain-expert teachers into one student and appears in large-scale capability-integration and staged pipelines.Related systems apply MOPD-style distillation across general and vertical domains.

6 Conclusion

The paper identifies static domain mixing as inefficient when domains converge asynchronously and proposes D3-MOPD to adapt allocation from KL-based signals. Experiments and ablations report improved gap closure, faster convergence, and contributions from each scheduler component.

  • Conclusion: D3-MOPD addresses the mismatch between static domain mixtures and asynchronous per-domain convergence rates in MOPD.The scheduler combines remaining KL gap with descent velocity to update sampling ratios.
  • Conclusion: D3-MOPD closes 97% of the student-teacher performance gap versus 63% for vanilla MOPD and reaches baseline peak accuracy in 3× fewer rollout steps.The experiments use four domains and a Qwen3.6-35B-A3B student.
  • Conclusion: Ablations report that the composite signal, batch jitter, and velocity smoothing each contribute to peak accuracy or convergence speed.Trajectory analysis describes an implicit curriculum that concentrates budget during each domain’s active learning window.

B.2 Student Training

The student-training setup uses a Megatron-backed slime implementation with colocated rollout infrastructure and shared teacher resources. The detached watcher preserves throughput without requiring trainer synchronization.

  • Infrastructure: The student is trained with slime on Megatron, while SGLang handles rollout inference in a colocated engine group.The setup uses seven GPU nodes: four trainer, two rollout, and one shared teacher node.
  • Performance: D3-MOPD and vanilla MOPD have mean throughputs of 520 and 531 tokens/GPU/s, respectively, a 2.1% difference over 256 steps.The throughput difference is attributed to shifted sequence-length distributions rather than synchronization overhead.
  • Performance: The scheduler’s standalone watcher reads logs and writes updated ratios without requiring the trainer to pause.This design adds no synchronization overhead to the training process.

C Best-Average Checkpoint Comparison

At a single best-average checkpoint, D3-MOPD provides a deployable-model comparison that differs from per-benchmark peak reporting. Across model scales, it reaches the best-average checkpoint earlier and improves benchmark performance over vanilla MOPD.

  • Best-average checkpoint: Table 6 evaluates all benchmark values at one best-average checkpoint, unlike Table 2, where each benchmark may use a different peak checkpoint.This distinction reflects the performance of a single deployable model.
  • 35B-A3B comparison: D3-MOPD’s normalized score falls from 0.97 to 0.73 when moving from per-benchmark peaks to its best-average checkpoint.The reduction occurs because no single checkpoint maximizes every benchmark simultaneously.
  • Cross-scale consistency: The 4B improvement pattern carries over from the Qwen3.6-35B-A3B experiments under the replicated four-domain training pipeline.The 4B experiment uses the same configuration with minor scale-related adjustments.
  • 4B benchmark results: D3-MOPD outperforms vanilla MOPD on all seven 4B benchmarks, with gains ranging from +0.9 on LCB-v6 to +4.1 on IFBench.Its normalized score is 1.01 versus 0.86 for vanilla MOPD.
  • 4B convergence: 40 steps earlier, D3-MOPD reaches its best-average checkpoint at step 119 versus step 159 for vanilla MOPD on the 4B student.This corresponds to a 25% reduction in training steps needed to achieve peak average performance.

E Theoretical Analysis of Composite-Signal Allocation

The analysis models per-domain reverse-KL decay and shows that the gap-velocity composite signal ranks domains by normalized marginal KL reduction under stated assumptions. The resulting scheduler allocates budget dynamically while smoothing allocations and retaining a floor for every domain.

  • Proposition 1: The composite signal s_k = f_KLk · v_k is proportional to the marginal reduction in the normalized gap.Under uniform allocation, the proof derives v_k ≈ λ_k B W/K and establishes the corresponding ranking.
  • Exponential decay model: The analysis models each domain’s reverse-KL as exponentially decaying with cumulative training exposure at a domain-specific learning rate.Here, n_k is cumulative allocated samples and λ_k > 0 is the domain-specific learning rate.
  • Proposition 1: Under uniform allocation and small per-window changes, allocating samples to arg max_k s_k maximizes the instantaneous reduction in the average normalized gap.The average normalized gap is defined as 1/K ∑_k f_KLk.
  • Signal interpretation: Initial-KL normalization prevents domains with larger absolute KL from dominating allocation regardless of relative improvement rate.The analysis gives IF’s KL as approximately 65× Math’s as an example.
  • Practical robustness: The softmax-floor mapping distributes samples according to signal strength while guaranteeing every domain a minimum sample rate.Temperature smooths greedy allocation, and the floor ε prevents catastrophic forgetting of temporarily plateaued domains.
  • Assumption boundary: The exponential decay assumption is used for closed-form design rationale, not as a necessary condition for effective scheduling.If a domain unexpectedly plateaus, its descent velocity falls and its allocation is automatically reduced.
  • Empirical trajectories: Across the 256-step run, the scheduler shifts allocation through distinct phases as domains exhibit different gaps and descent rates.IF dominates first, Math follows, Tool-use receives later budget, and IF returns after its accumulated gap reopens.

G Ablation Domain Ratio Trajectories

Ablation trajectories show that the gap and velocity signals jointly shape dynamic domain allocation, while jitter and smoothing primarily affect transition sharpness and noise. Removing gap information produces the clearest instability.

  • Overview: Figure 8 compares per-domain sampling ratios for ablations, vanilla MOPD, and D3-MOPD over 256 rollout steps.The comparison covers variants removing velocity, gap, jitter, or smoothing.
  • w/o velocity: Without velocity, allocation ratios are flatter because gap alone cannot distinguish active descent from a high-KL plateau.Math reaches approximately 0.30 instead of D3-MOPD’s approximately 0.50 peak, and late shifts are weaker.
  • w/o gap: Without gap, early ratios oscillate before step 63 and later swings remain abrupt, including Math dropping to approximately 0.10 around step 100.This variant peaks latest at step 255.
  • w/o jitter: Without jitter, the long-run allocation shape remains similar to D3-MOPD, but watcher updates create sharper step-like transitions.The sharper transitions are most visible in Code near step 200 and Tool-use near step 240.
  • w/o smoothing: Without smoothing, velocity estimates become more reactive and noisier, causing visible oscillations and frequent Math and Code spikes around steps 60–130.The overall allocation pattern remains close to D3-MOPD, and this variant achieves 62.17 (+0.81).
Loading 2608.24987v1…