PaperlayerView exact revision on arXiv

Source-linked AI summary

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim, Juho Lee

arXiv:2609.34745v1cs.LGcs.AI

TL;DR

MOPD’s reliance on student-generated prefixes makes initialization important when integrating teachers with different post-training histories. IM-MOPD starts from a uniform merge and iteratively adds task-vector increments for under-recovered domains; across five domains, it achieves higher aggregate recovery than uniform merging or SFT warm-up.

  • Problem

    It remains unclear how effectively MOPD integrates teachers with different post-training histories, despite their sharing a reference model.

    Source: §1 · p. 2

  • Method

    IM-MOPD begins with a uniform merge and periodically adds fixed task-vector increments for domains whose validation recovery is at or below a threshold.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across five domains, IM-MOPD achieves higher average normalized recovery than uniform initialization or SFT warm-up; on Qwen3-4B, recovery rises from 53.8% with uniform initialization to 86.9%.

    Source: §4.2 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    Teacher contributions can be determined progressively during training rather than through a one-shot continuous coefficient search.

    Source: §1 · p. 2, §3.1 · p. 5

    The studied setting assumes domain teachers share an architecture and originate from a common reference checkpoint before specialization.

    Source: §2 · pp. 2–3

Abstract

from arXiv · show

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.

1. Introduction

MOPD integrates independently specialized teachers, but its reliance on student-generated prefixes makes initialization consequential when teachers have different post-training histories. The paper proposes iterative merging to adjust teacher contributions during distillation, reporting higher average recovery than fixed uniform merging or SFT warm-up in five domains.

1. Introduction

  • MOPD’s ability to integrate teachers with different post-training histories remains uncertain, and student-generated prefixes make initialization consequential for subsequent learning.SFT warm-up can bring the student closer to teachers, but requires choices about mixed-domain data and training schedule.Source: §1 · p. 2
  • Merge initialization combines teacher task vectors with the shared reference model, avoiding an additional mixed-domain SFT stage while potentially outperforming SFT warm-up after MOPD.SFT warm-up can start with higher benchmark performance yet finish below a merge initialization after MOPD.Source: §1 · p. 2
  • Effective merging depends on both domain-wise teacher ratios and global scale; in some settings, λ > 1 improves post-MOPD recovery beyond convex parameter averaging.A good initialization therefore cannot be selected reliably from initial performance alone.Source: §1 · p. 2
  • IM-MOPD starts from a uniform merge and iteratively adds fixed task-vector increments for under-recovered domains, replacing continuous coefficient search with merge-or-not decisions.The relative domain weights and total task-vector contribution can evolve during training.Source: §1 · p. 2
  • Across five domains, IM-MOPD achieves higher average normalized recovery than uniform initialization or SFT warm-up; scores are 86.9 (4B) and 76.0 (1.7B), versus SFT warm-up’s 59.0 and 46.5.The model-size-specific comparisons are 86.9 versus 59.0 on 4B and 76.0 versus 46.5 on 1.7B.Source: §1 · p. 2

2. Preliminaries

MOPD distills teacher predictions at prefixes generated by the student, while model merging represents each teacher’s change from a shared reference as a task vector. The preliminaries describe a shared-origin teacher setting and an iterative method that combines MOPD with selective task-vector corrections.

2. Preliminaries

  • IM-MOPD alternates MOPD with selective merging, adding fixed increments for domains whose recovery falls below a threshold.A merge intervention can lower performance immediately yet produce higher recovery after 50 additional MOPD updates.Source: Figure 2 · p. 3, §2 · p. 3, Figure 3 · p. 4
  • The setting assumes teachers share an architecture and reference checkpoint, then specialize through domain-specific post-training procedures.The goal is to integrate these independently specialized teachers into one student policy.Source: §2 · pp. 2–3
  • MOPD routes each domain’s prompts to its corresponding teacher and minimizes token-averaged reverse KL on student-generated trajectories.Initialization shapes the prefixes encountered during distillation and thus the subsequent MOPD trajectory.Source: §2 · p. 3
  • Model merging represents each teacher’s difference from the shared reference as a task vector and adds weighted vectors to form an initial student.The global merge scale λ is the sum of the coefficients; standard weighted averaging has λ = 1, but larger total contributions are allowed.Source: §2 · p. 3

3. Method

The method examines how student initialization affects MOPD recovery and introduces IM-MOPD, which alternates distillation with selective task-vector corrections. The observations motivate evaluating what an initialization enables during training, not just its initial benchmark performance.

3. Method

  • The evaluation normalizes each domain score so the shared reference is 0 and its corresponding expert is 1, and also reports an average normalized score.Teachers share a reference model and are independently specialized through SFT or RLVR.Source: §3 · p. 4

3.1. Observations

  • Merging teacher task vectors improves subsequent MOPD recovery over base initialization without additional training, while SFT warm-up partially alleviates weak recovery at added training cost.Reference-initialized MOPD struggles to recover SFT-trained Medical and Tool-Use capabilities while approaching teacher-level Finance performance.Source: §3.1 · p. 4
  • Higher initial benchmark performance does not guarantee higher post-MOPD performance: SFT warm-up leads scaled merging on Tool-Use initially, but the ordering reverses after MOPD.A merge intervention can temporarily lower normalized performance yet achieve higher performance after 50 additional MOPD updates.Source: §3.1 · p. 4
  • Effective merge coefficients depend jointly on domain ratios and overall scale, and candidates must be assessed through MOPD rather than initial average normalized score alone.In the five-domain comparison, non-uniform weighting improves recovery at larger scale, while normalizing coefficients to sum to one weakens the result.Source: §3.1 · pp. 4–5

3.2. Iterative Merging for MOPD (IM-MOPD)

  • IM-MOPD alternates MOPD with fixed-scale task-vector corrections for domains whose validation recovery is at or below a threshold, allowing merge weights and total scale to evolve during training.Corrections are assessed at intervals; MOPD resumes from the corrected student, and a domain may receive further corrections if it remains below threshold.Source: §3.2 · p. 5

4. Experiments

The experiments compare initialization strategies for five-domain MOPD and find that iterative merging improves aggregate recovery. Ablations indicate that correction timing matters, while tested settings remain robust across a small hyperparameter range.

4.1. Experimental Setup

  • Five domain teachers are independently specialized from a shared reference model, and IM-MOPD is compared with reference initialization, uniform merging, and SFT warm-up.The experiments use Medical, Law, and Tool Use SFT teachers, plus Finance and Instruction Following RLVR teachers; iterative corrections are triggered at recovery thresholds every 25 MOPD updates, with a fixed correction scale.Source: §4.1 · p. 5

4.2. Main Results

  • 86.9% average normalized recovery for Qwen3-4B with IM-MOPD, versus 53.8% with uniform initialization; Qwen3-1.7B reaches 76.0%, versus 46.5% with SFT warm-up.IM-MOPD scores higher than SFT warm-up on all five benchmarks across both model sizes; Table 1 reports the highest average normalized score among compared initialization methods at both sizes, without additional SFT.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Ablations

  • 86.9% normalized recovery comes from iterative merging, versus 79.0% when the same cumulative task-vector coefficients are applied only at initialization.The largest gain is in Tool Use; the timing ablation compares applying the final coefficients entirely at initialization or after MOPD with the original iterative schedule.Source: §4.3 · p. 6, Table 2 · p. 7
  • 23.3% normalized recovery follows applying IM-MOPD’s final merge coefficients only after MOPD, below Base MOPD and far below IM-MOPD.Together with the initialization-only control, this indicates the final task-vector displacement alone does not explain IM-MOPD’s gains; teacher-guided updates after corrections matter.Source: §4.3 · pp. 6–7
  • On Qwen3-4B, iterative merging yields the largest domain gains in Medical, Tool Use, and Law while already well-recovered domains remain broadly stable.The method starts from Uniform Merge and progressively improves under-recovered domains during MOPD.Source: Figure 4 · p. 7
  • Qwen3-4B maintains over 80% normalized recovery across tested correction sizes, while Qwen3-1.7B performs best with γ=0.4 and δ=0.3.The tested ranges are γ∈{0.4, 0.6} and δ∈{0.1, 0.2, 0.3, 0.4}; Table 3 reports average normalized recovery across five domains.Source: §4.3 · p. 7

5. Analysis

Teacher-continuation analysis tests how well domain teachers continue student-generated prefixes. For Medical and Tool Use, merge-based prefixes preserve teacher continuation better early in generation than SFT-warm-up prefixes, even when standalone student performance is similar or lower.

Teacher continuation

  • The diagnostic has each student generate a prefix, then asks the corresponding domain teacher to complete the response while varying the student-generated fraction.Higher continuation performance indicates greater compatibility between the student prefix and the teacher’s subsequent generation.Source: §5 · p. 7
  • In Medical, teacher continuation is consistently stronger from merge-initialized prefixes than SFT-warm-up prefixes; in Tool Use, IM-MOPD has higher continuation success from short prefixes despite lower student-only performance.Across Medical and Tool Use, SFT warm-up prefixes show a drop in teacher continuation even with short prefixes, while merged-student prefixes better preserve continuation over the early trajectory.Source: §5 · pp. 7–8, Figure 5 · p. 8
  • The 0% and 100% endpoints in Figure 5 denote teacher-only and student-only performance, respectively.The figure compares continuation behavior across prefix lengths for merge-initialized and SFT-warm-up students.Source: Figure 5 · p. 8

6. Related Work

Prior consolidation approaches include mixed and cascade RL, as well as independently trained specialists integrated through merging or MOPD [Blakeman et al., 2025; Wang et al., 2025; Yang et al., 2026; Team et al., 2025; Ma et al., 2026]. This work investigates iterative merging to integrate heterogeneous capabilities while maintaining teacher–student alignment.

On-Policy Distillation for Large Language Models (LLMs)

Consolidating Domain Specialists

  • Mixed RL jointly trains over domains, whereas cascade RL uses sequential domain-specific stages that can accumulate capability drift (Blakeman et al., 2025; Wang et al., 2025; Yang et al., 2026).The approaches differ in how they handle heterogeneous data, rewards, rollout lengths, and optimization dynamics.Source: §6 · pp. 8–9
  • Independently trained domain specialists can be consolidated through parameter merging or multi-teacher on-policy distillation (Team et al., 2025; Ma et al., 2026).The paper situates its work in this specialist-consolidation paradigm.Source: §6 · p. 9
  • The work investigates whether iterative merging can progressively integrate heterogeneous capabilities while maintaining effective teacher–student alignment.Source: §6 · p. 9

7. Conclusion

The paper studies MOPD with teachers sharing a reference model but receiving different post-training procedures. It introduces IM-MOPD and reports higher aggregate recovery than uniform merging or SFT warm-up across five domains and two model sizes.

7. Conclusion

  • Across five domains at both 4B and 1.7B scales, IM-MOPD achieves higher aggregate capability recovery than uniform merging and SFT warm-up without an additional SFT stage.Source: §7 · p. 9
  • Initial benchmark performance is an unreliable criterion for MOPD initialization, and effective merging depends on relative teacher weights and overall merge scale.Source: §7 · p. 9
  • Merge-timing ablations find that applying the same final cumulative coefficients entirely before or after MOPD is less effective than interleaving corrections with distillation.Source: §7 · p. 9

Reproducibility Statement

The reproducibility statement points to appendices for training, data, schedules, and evaluation details. Experiments used NVIDIA A100 80GB GPUs.

Reproducibility Statement

  • Appendices document teacher training and data sources, MOPD and SFT warm-up settings, validation data, merge schedules, evaluation protocols, and additional ablations and diagnostics.Source: Reproducibility Statement · p. 9

A. Experimental Details

A.1. Teacher Training

The appendix describes teacher training at two model sizes, MOPD training settings, and the SFT warm-up baseline. The five domains use distinct SFT or RL data and training configurations.

A.1. Teacher Training

  • Qwen3-4B-Base and Qwen3-1.7B-Base were fine-tuned on OpenThoughts33 to form common references, with domain teachers initialized from the corresponding reference.Tables 4 and 5 summarize teacher configurations at the two model sizes.Source: §A.1 · p. 13, Table 4 · p. 13, Table 5 · p. 13
  • Medical, Law, and Tool Use use SFT data, while Finance and IF use RL prompt sets from FinQA/TAT-QA and the rlvr1 blend, respectively.SFT data include generated MedQA and CaseHOLD/bar-exam samples and AReaL-tau2-da tool-use samples.Source: §A.1 · p. 13
  • SFT uses packing, zero weight decay, gradient clipping, and cosine decay, while Finance and IF use NeMo-RL GRPO with domain-specific rewards.GRPO uses clipped policy updates and overlength penalties without a reference KL penalty.Source: §A.1 · p. 13

A.2. MOPD Training Configuration

  • MOPD uses 100 updates, 128 prompts per batch, uniform domain sampling, one completion per prompt, and a learning rate of 10^-6.The setup uses policy-gradient distillation, caps completions at 16,384 tokens, and omits task-reward and reference-KL terms.Source: §A.2 · p. 14

A.3. SFT Warm-up Baseline

  • The SFT warm-up baseline trains on mixed teacher-generated samples until approximately 30% normalized recovery, then starts MOPD from that checkpoint.Warm-up uses packed sequences and cosine learning-rate decay; Table 6 summarizes its configurations.Source: §A.3 · p. 14, Table 6 · p. 14

A.4. Main Experiment Settings

The appendix specifies IM-MOPD correction and evaluation settings, then defines the merge configurations used in the five-domain initialization comparison.

A.4. Main Experiment Settings

  • For 4B, γ=0.6; for 1.7B, γ=0.4; both use δ=0.3, with corrections after updates 25, 50, and 75.Table 7 lists the selected domains for each configuration.Source: §A.4 · p. 14, Table 7 · p. 14
  • Validation uses 128 Tool Use tasks and 256 prompts from held-out validation data for each other domain.Source: §A.4 · p. 14

A.5. Evaluation Protocol

  • Evaluation covers MedQA, CaseHOLD, FinQA, IFBench, and IFEval; Telecom uses the official v1.0.1 harness on 114 base-split tasks, including 20 IDs flagged for upstream training overlap.Telecom empty responses and execution or context-limit errors count as failures.Source: §A.5 · pp. 14–15

B.1. Initialization Comparison

  • The five-domain comparison reports before/after MOPD scores: λ=1.0 assigns 0.2 to each teacher, while λ=2.1 uses (0.5, 0.3, 0.2, 0.3, 0.8) in Medical, Law, Finance, IF, Tool Use order.All post-MOPD scores use 100 updates; Table 8 gives the results underlying Figure 1.Source: §B.1 · p. 16, Table 8 · p. 16

B.2. Ablation Details

The ablation details describe benchmark sweeps, student-prefix teacher-continuation diagnostics, and two-domain initialization and merge-timing results.

B.2. Ablation Details

  • Ablation tables 9–10 report complete benchmark results for both scales under the validation recovery rule.Source: §B.2 · p. 16
  • Table 9 sweeps correction size for 4B at γ=0.6; Table 10 sweeps the 1.7B recovery threshold and correction size.Source: Table 9 · p. 16, Table 10 · p. 16

B.3. Student-prefix Teacher-continuation Diagnostics

  • Student-generated fractions are α∈{0.1, 0.3, 0.5, 0.7}, with student-only endpoints; for MedQA, each teacher continuation begins after ⌊αL⌋ student tokens in that response.The prefix length is determined separately for each sample.Source: §B.3 · p. 16
  • For Telecom, teacher continuation follows a per-turn budget based on the student's mean standalone turn length because earlier changes alter later multi-turn interactions.Some shorter turns may be generated entirely by the student.Source: §B.3 · p. 16

B.4. Two-domain Initialization and Merge Timing

  • Table 11 compares before/after MOPD on MedQA and IFEval; Norm. averages normalized recovery over both benchmarks, and the last row uses uniform initialization with mid-training merging.IFEval reports prompt-level strict accuracy over 541 prompts.Source: §B.4 · p. 17, Table 11 · p. 17
Loading 2609.34745v1…