PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Self-Distillation for Multi-Teacher Capability Merging

Roy Xie, Dan Friedman, Feng Nan, Yukun Huang, Zhichao Xu, Chengjiu Zhang, Jun Xu, Manaal Faruqui, Vivek Rathod, Bhuwan Dhingra

arXiv:2610.04272v1cs.LGcs.AIcs.CL

TL;DR

It is unclear how much of MOPD’s reported advantage over off-policy distillation persists against controlled, independently tuned baselines, despite MOPD’s higher training and environment-interaction costs. The paper compares four distillation methods and weight merging across two multi-teacher settings, four models, and eleven benchmarks, and revisits four published comparisons. The methods achieve comparable in-domain recovery, while MOPD requires substantially more training GPU-hours and stronger SFT recipes narrow prior reported gaps.

  • Problem

    Systematic comparisons of on-policy and off-policy distillation under controlled conditions remain limited, despite differences in compute and environment-interaction costs.

    Source: §1 · pp. 1–2

  • Method

    The study compares SFT, Soft-KD, Offline-MOPD, MOPD, and weight merging across two multi-teacher settings and four models, and revisits four published SFT–OPD comparisons.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    Across four models and two settings, no consistent in-domain recovery advantage favors MOPD; on Qwen3-8B it uses 14.8× and 23.1× SFT’s training GPU-hours on MIA and Agent×3, respectively.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Carefully tuned off-policy distillation and training-free weight merging are viable, cheaper alternatives for integrating expert capabilities, though merging performance varies with model size and task composition.

    Source: §7 · p. 10

    The study covers four models and two multi-teacher settings; it does not establish whether off-policy distillation remains competitive at larger scales or with less compatible teachers.

    Source: H Limitations · p. 28

Abstract

from arXiv · show

Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inference and environment interaction costs incurred by MOPD, we find that much of its reported accuracy gain is due to certain training design choices and hyperparameter optimization, rather than the algorithm itself. We conduct a controlled self-distillation study across two multi-teacher settings, four models, and eleven benchmarks, comparing off-policy methods, namely supervised fine-tuning (SFT) and soft-label distillation, to hybrid teacher-prefix distillation and MOPD. We found that all four methods achieve \textit{nearly identical} accuracy. However, MOPD uses $14.8$--$23.1\times$ SFT's training GPU-hours. We also revisit four recently published comparisons between on-policy and off-policy distillation and find that the reported on-policy gains shrink substantially when SFT baselines are trained on rejection-sampled teacher trajectories and use independently tuned hyperparameters. As a training-free alternative, we also find that simple weight-merging methods can recover expert capabilities with minutes of CPU merging time, although their accuracy degrades as model size decreases and task interference increases. Overall, our results question recent gains reported due to MOPD and suggest careful tuning of more efficient off-policy baselines as a viable alternative.

1 Introduction

As expert capabilities are increasingly developed in separate models, it remains unclear how much reported MOPD gains come from on-policy supervision rather than training choices. In a controlled comparison, the paper finds tuned off-policy methods reach comparable accuracy at far lower training cost, and that stronger SFT recipes narrow previously reported gaps.

1 Introduction

  • MOPD’s repeated student generation and teacher scoring add compute and infrastructure costs, including live environment interaction for agentic rollouts.Off-policy methods reuse fixed expert trajectories, avoiding training-time environment interaction.Source: §1 · pp. 1–2
  • Four reproduced SFT–OPD comparisons show that rejection-sampled teacher trajectories and independently tuned learning rates substantially reduce reported accuracy gaps.The plotted gaps shrink in every comparison.Source: §1 · p. 2, Figure 1 · p. 2
  • The study compares SFT, Soft-KD, teacher-prefix Offline-MOPD, and MOPD under shared initialization across two multi-teacher settings.MIA combines mathematics, instruction following, and agentic tasks; Agent×3 combines three agentic experts, and the study evaluates four models on eleven benchmarks.Source: §1 · p. 2
  • With careful tuning, SFT, Soft-KD, Offline-MOPD, and MOPD achieve comparable accuracy, while MOPD costs 14.8× and 23.1× SFT’s GPU-hours on MIA and Agent×3.Weight merging offers a training-free alternative with comparable capability recovery, but degrades at smaller model scales and with overlapping expert capabilities.Source: §1 · p. 2

2 Related Work

Prior work contrasts off-policy training on teacher trajectories with on-policy supervision at student-generated contexts, alongside intermediate methods and weight merging. Systematic controlled comparisons of these approaches for experts and students sharing an initialization remain limited.

2 Related Work

  • Off-policy distillation uses teacher-generated sequences, while on-policy distillation queries teachers on student-generated trajectories to address mismatch in visited states.Intermediate approaches interleave teacher and student tokens or combine student sampling with replayed trajectories and prefixes.Source: §2 · p. 3
  • Comparisons across distillation approaches remain limited when multiple teachers and the student share the same initialization.Prior work also identifies teacher–student compatibility and data selection as factors in OPD accuracy.Source: §2 · p. 3
  • Multi-teacher capability integration also uses SFT on expert trajectories or direct weight merging, but prior comparisons did not systematically assess computational costs or independent baseline tuning.Using Open-MOPD’s released data and evaluation protocols, the authors find that tuning and rejection-sampled data can bring SFT to OPD-level accuracy.Source: §2 · p. 3

3 Methodology

The methodology compares four self-distillation objectives and training-free weight merging under a shared expert initialization. It also defines capability recovery and retention to assess in-domain and out-of-domain performance.

3 Methodology

  • Experts and student share a base checkpoint; a trajectory is a single response for single-turn tasks or alternating model actions and environment observations for multi-turn tasks.Off-policy methods reuse a fixed rejection-sampled expert corpus; on-policy methods query teachers at student-generated contexts, with live environment interaction for agentic tasks.Source: §3 · p. 3

3.1 Self-Distillation Objectives

  • SFT trains on recorded teacher tokens, Soft-KD uses full teacher next-token distributions, Offline-MOPD samples student responses from recorded contexts, and MOPD uses complete student trajectories.Offline-MOPD avoids live environment interaction; MOPD uses student-generated interaction histories in multi-turn tasks.Source: §3.1 · p. 4

3.2 Training-Free Weight Merging

  • Training-free weight merging combines expert checkpoints in parameter space using task vectors derived by subtracting the shared base parameters.The merging method and its hyperparameters determine task-vector coefficients and any sparsification.Source: §3.2 · p. 5

3.3 Evaluating Capability Recovery

  • In-domain recovery measures student gains as a percentage of aggregate teacher gains, while out-of-domain retention measures mean score change from the base model.Negative retention values indicate degradation.Source: §3.3 · p. 5

4 Experimental Setup

Experiments use two multi-teacher compositions and four models, with independently tuned distillation methods and evaluation on held-out benchmarks. The setup includes multiple seeds and separates in-domain recovery from out-of-domain retention.

4.1 Multi-Teacher Setup

  • MIA combines mathematics, instruction-following, and agentic experts, whereas Agent×3 combines ALFWorld, WebShop, and InterCode-SQL experts focused on long-horizon agentic capability.Four models are studied under both settings, with teachers trained from each base model.Source: §4.1 · p. 5

4.2 Hyperparameter Tuning

  • Each distillation method is tuned independently, with up to five learning-rate values per method, model, and setup before seed replication or further sweeps.Learning-rate choice alone changes SFT validation score by up to 11.3 percentage points; merge coefficients and sparsity are selected by worst-task validation retention.Source: §4.2 · pp. 5–6

4.3 Evaluation

  • The evaluation covers eight single-turn and three multi-turn benchmarks, with disjoint train, validation, and test sets and three seeds for distillation methods.Models are summarized by in-domain recovery and out-of-domain retention; the Qwen3-8B table reports means and standard deviations across three seeds.Source: §4.3 · p. 6, Table 1 · p. 6

5 Experimental Results

Across models and settings, tuned off-policy and on-policy distillation recover similar expert capability, while MOPD requires substantially more training compute. Weight merging and joint multi-domain RL offer alternatives, with trade-offs in recovery, scale, and iteration.

5.1 Capability Recovery across Models and Settings

  • SFT, Soft-KD, Offline-MOPD, and MOPD recover 96.9–98.2% of teacher gains on MIA and 97.6–99.3% on Agent×3, with no consistent on-policy recovery advantage.Across four models, the maximum within-model recovery spread is 3.5 percentage points on MIA and 4.6 on Agent×3.Source: §5.1 · pp. 6–7, Table 2 · p. 7
  • MOPD improves mean out-of-domain retention over SFT by 1.1 benchmark points, but does not consistently outperform it; Mix-RL and merging average −1.1 and −2.7 points.Out-of-domain retention varies by model and setting.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Distillation Efficiency

  • MOPD uses 14.8× SFT’s training GPU-hours on MIA and 23.1× on Agent×3.For Qwen3-8B, the reported totals are 383.32 versus 25.90 GPU-hours on MIA and 415.80 versus 18.00 on Agent×3; Figure 3 plots accuracy against progress and cumulative GPU-hours.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 Alternatives to Distillation

  • Weight merging recovers 99.0% of teacher gains for Qwen3-8B versus 87.3% for Qwen3-1.7B, and performs better on MIA than Agent×3.Mean recovery across four models is 96.6% on MIA and 87.9% on Agent×3; merged weights take minutes of CPU time to construct, but selecting a merge requires GPU validation.Source: §5.3 · p. 8
  • Mix-RL recovers most teacher gains across eight model–setting pairs, approaching or exceeding teacher-level performance on Qwen3.5-4B and Qwen3-8B.Changing one domain’s recipe may require retuning the joint run, while separate experts support independent iteration before integration.Source: §5.3 · p. 8

6 Analysis and Discussion

The analysis finds no general on-policy recovery advantage from larger teachers and shows that methods with similar recovery can still change model behavior differently. Stronger SFT recipes shrink reported on-policy gains in four reproduced comparisons.

6.1 Do Larger Teachers Favor On-Policy Distillation?

  • With 8B teachers, SFT and Soft-KD recover more in-domain gains than both MOPD variants on MIA and the reported Agent×3 results, but incur larger out-of-domain losses.On Agent×3, SFT reaches 99.3% recovery and Soft-KD 93.0%, versus 78.9% for Offline-MOPD and 73.3% for MOPD; their out-of-domain degradation is 14.8 and 11.7 points, respectively.Source: §6.1 · p. 8

6.2 Parameter Updates and Behavioral Changes

  • On Qwen3-8B, attention-weight changes stay below 0.7%, yet distillation shortens MIA responses by 23–29% and lengthens Agent×3 responses by 22–67%.MOPD shows the largest response-length changes, while merging increases response length by 3% on MIA and 23% on Agent×3.Source: §6.2 · p. 8, Figure 4 · p. 9

6.4 Do Reported On-Policy Gains Persist with Stronger SFT Baselines?

  • OPD’s advantage over rejection-sampled SFT is smaller than published in all four reproduced comparisons.Relative to tuned SFT, rejection-sampled recipes gain 7.0 points on Open-MOPD, 4.4 on TCOD, and 0.6 on ReOPD, and lose 0.8 on SKD.Source: Table 4 · p. 9, §6.4 · p. 9

6.3 What Does It Take to Make SFT a Strong Baseline?

  • SFT validation peaks at 5 × 10−6, with scores spanning 8.5 percentage points on MIA and 11.3 on Agent×3; rejection sampling benefits off-policy methods more.SFT’s recovery gaps are 6.5 points on MIA and 7.0 on Agent×3, versus 3.0 and 3.6 for Soft-KD and 0.5–1.6 for the MOPD variants.Source: §6.3 · p. 9, Figure 4 · p. 9

7 Conclusion

The conclusion finds no consistent in-domain recovery advantage for MOPD over tuned off-policy distillation, while emphasizing MOPD’s higher training cost. The appendices detail four reproduced comparisons and discuss data selection and optimization as influences on supervised baselines.

7 Conclusion

  • Across four models and two task compositions, MOPD has no consistent in-domain recovery advantage over tuned off-policy distillation, while using 14.8× and 23.1× SFT’s GPU-hours on MIA and Agent×3.Weight merging is a training-free alternative, but its performance varies with model size and task composition.Source: §7 · p. 10

A Reproducing Four Published SFT--OPD Comparisons

A.1 Open-MOPD

  • Open-MOPD’s published MixSFT and naive OPD recipes balance different units: response tokens at roughly 1:1:1 versus prompts at about 2:2:1.The comparison reuses released experts and evaluators; SFT runs up to seven epochs, with learning rates and epoch budgets selected on reporting benchmarks.Source: §A.1 · p. 16

A.2 SKD

  • For SKD, rejection sampling retains teacher responses matching gold answers but also changes corpus size and prompt coverage.The reproduction uses response-only SFT loss, independently tuned learning rates, and validation-accuracy checkpoint selection.Source: §A.2 · pp. 16–17

A.3 TCOD

  • For TCOD, SFT supervises each teacher action throughout the episode, replacing the framework default of supervising only the final assistant message.The SFT development split is constructed by turn and can share games with training.Source: §A.3 · p. 17

A.4 ReOPD

  • For ReOPD, teacher histories are reused as student-sampling prefixes, avoiding fresh environment rollouts; rejection sampling yields 5,071 trajectories from 6,335 unfiltered trajectories.The reproduction independently tunes SFT and OPD learning rates.Source: §A.4 · pp. 17–18

B Additional Related Work

  • Prior work describes rejection sampling, reward selection, and filtering as consequential data choices, while baseline rankings can change with hyperparameter-search budgets.This paper examines how data and tuning choices affect reported on-policy gains and capability recovery.Source: B Additional Related Work · p. 18

C Model Merging Methods and Evaluation

The section describes a search over weight-merging methods and the shared-initialization, teacher-data setup used for distillation. It specifies how merge candidates are screened and selected, and how prompts are routed in the main experiments.

C Model Merging Methods and Evaluation

  • Eleven MergeKit methods are evaluated by screening default coefficients, then tuning leading methods’ coefficients and sparsity settings.Candidates are checked for tensor-count mismatches, nonfinite values, distance from the base, and malformed tool calls.Source: C Model Merging Methods and Evaluation · p. 18
  • The selected merge is chosen using the same validation sets as distillation checkpoints, and sweep tables report equally averaged recovery of each expert’s gain over the base.The search spans methods including model soup, task arithmetic, TIES, and Karcher mean.Source: C Model Merging Methods and Evaluation · p. 18

D.1 Teachers and Distillation Corpus

  • In the main distillation experiments, teachers and students share initialization, tokenizer, and chat template; all methods use the same rejection-sampled teacher corpus and prompt sets.Prompts are routed to teachers by known task domain, separating capability integration from router quality.Source: §D.1 · p. 19

D.2 Training

The section reports comparable capability recovery from tuned off-policy methods and MOPD, alongside distinct training procedures, evaluation choices, and split safeguards. It also frames exposure bias as the concern motivating analysis of off-policy distillation.

D.2 Training

  • For Agent×3, MOPD executes student actions to obtain fresh environment observations, whereas Offline-MOPD replays recorded teacher histories; single-turn Offline-MOPD samples a full response from the original prompt.MOPD uses one rollout per prompt and per-token reverse KL.Source: §D.2 · p. 19

D.3 Checkpoint Selection and Evaluation

  • Checkpoints are selected by unweighted mean validation score across setup domains, using generated rollouts under the held-out evaluation harness.Each distillation run starts from the base checkpoint and trains five epochs over shared prompts.Source: §D.3 · p. 22

D.4 Task Splits and Benchmarks

  • Validation splits exclude teacher and student training data; a constructed instruction subset lying entirely within teacher RL training inflated the teacher ceiling by over 40 points.This would inflate the recovery denominator.Source: §D.4 · p. 22, Table 12 · p. 22

E Why Off-Policy Distillation Can Match On-Policy Distillation

  • Off-policy distillation reuses fixed expert trajectories with upfront collection costs and no training-time environment interaction, unlike on-policy rollouts.The paper asks how to consolidate expert capabilities into one general-purpose model.Source: E Why Off-Policy Distillation Can… · p. 23

E.1 Exposure Bias and Prior Work

  • Tuned off-policy methods can recover expert capabilities comparably to MOPD, although off-policy training still raises exposure-bias concerns when student actions shift later states from teacher-training trajectories.Early errors can move subsequent trajectories away from the training distribution.Source: §E.1 · p. 23

E.2 Distillation Losses and Success Probabilities

The appendix relates distillation losses to trajectory-distribution differences and task-success gaps, then qualifies how rejection sampling affects those bounds. Across the reported self-distillation settings, recovery and retention do not consistently favor on-policy methods.

E.2 Distillation Losses and Success Probabilities

  • The analysis assumes a shared environment and stopping rule, bounded trajectory lengths, and finite expected losses; its hybrid comparison replaces student responses from the last turn backward.At each history, it compares student and teacher response distributions.Source: §E.2 · pp. 23–24
  • For any trajectory statistic bounded between 0 and 1, the success gap is bounded using total variation, with bounds from L_off and L_on adding no trajectory-length factor beyond summed losses.The proof uses the KL chain rule and Pinsker’s inequality.Source: §E.2 · p. 24
  • Rejection-sampled SFT corresponds to a teacher distribution conditioned on success, so the unfiltered bounds do not directly apply to filtered Soft-KD and Offline-MOPD losses.A simultaneous fit across domains is not guaranteed, even though each teacher is realizable by the student individually.Source: §E.2 · p. 24

F Per-Model Capability Recovery

  • Across self-distillation settings, the four methods recover within 3.5 points on MIA and 4.6 on Agent×3; MOPD improves mean OOD retention over SFT by 1.1 benchmark points.With 8B teachers, SFT and Soft-KD recover more in domain than both MOPD variants but lose more out of domain, most sharply on Agent×3.Source: F Per-Model Capability Recovery · p. 25

G.1 Training-Step Breakdown

The timing breakdown identifies student rollout as the dominant training-step component, while training GPU-hour accounting excludes teacher-corpus generation. MOPD has the highest full-run training cost in both configurations.

G.1 Training-Step Breakdown

  • 86.5–96.0% of step time is student rollout, while teacher scoring is at most 2.6%; seconds per step are not directly comparable across methods because batch sizes differ.Table 18 reports Qwen3-8B configurations; MIA uses batches of 24/96 prompts for Offline-MOPD/MOPD, respectively.Source: §G.1 · p. 26, Table 18 · p. 27

G.2 Teacher Trajectory Generation

  • Corpus generation uses autoregressive decoding and, for agentic tasks, environment interaction; its shared Cgen/N GPU-hours are excluded from training costs.SFT, Soft-KD, and Offline-MOPD reuse the corpus across hyperparameter settings and seeds.Source: §G.2 · p. 27
  • Training-cost ratios are not end-to-end cost ratios because corpus generation is excluded, although teacher scoring for MOPD and Offline-MOPD is included.Teacher scoring covers forward passes over student-generated trajectories during training.Source: §G.2 · p. 27

G.3 Training GPU-Hours

  • 383.32 GPU-hours on MIA and 415.80 on Agent×3: MOPD’s full-run cost was 14.8× and 23.1× SFT’s, respectively.Training GPU-hours exclude teacher-corpus generation, validation, test evaluation, and other hyperparameter-search runs.Source: §G.3 · p. 27, Table 19 · p. 28

H Limitations

The study is limited to four models from 1.7B to 8B parameters and two multi-teacher settings; it does not establish whether off-policy distillation remains competitive at larger scales or with less compatible teachers.

H Limitations

  • The study covers four models from 1.7B to 8B parameters and MIA and Agent×3; the larger-teacher comparison uses one 1.7B student and same-family 8B teachers.The authors state that results do not establish whether off-policy distillation remains competitive at larger scales or with less compatible teachers.Source: H Limitations · p. 28

Related papers

Loading 2610.04272v1…