PaperlayerView exact revision on arXiv

Source-linked AI summary

Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation

Xiaofei Yin, Tong Chu, Jiyuan Fu, Jun Lan, Shuheng Zhou, Huijia Zhu

arXiv:2610.02324v1cs.LGcs.AI

TL;DR

MOPD can displace a multimodal student from its initialization as it consolidates domain expertise, while a fixed initialization anchor can suppress that acquisition. SF-MOPD uses a slow EMA reference and per-teacher projection; across model scales, it consistently enhances specialized capabilities while better preserving general capabilities than vanilla MOPD.

  • Problem

    MOPD consolidates domain-specific expertise but can drive the student away from its initialization, with general capabilities declining as displacement grows.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    SF-MOPD pairs a fast student with a slow EMA reference and removes only each teacher-induced update component that enlarges slow-fast displacement.

    Source: §1 · p. 1, §2.3 · p. 3, §2.4 · p. 3

  • Results

    Across multiple model scales and specialized multimodal domains, SF-MOPD consistently enhances expert-specific capabilities and better preserves general multimodal competence than vanilla MOPD.

    Source: §1 · p. 1, §4 · p. 4

  • Takeaways & Limitations

    The slow reference gradually absorbs confirmed domain expertise, while projection retains aligned and orthogonal update components that may encode complementary knowledge.

    Source: §1 · p. 1, §2.3 · p. 3

Abstract

from arXiv · show

Foundation multimodal large language models are designed to support a broad spectrum of capabilities across diverse domains. Multi-teacher on-policy distillation (MOPD) provides an effective framework for consolidating domain-specific expertise into a single student model. However, MOPD training gradually drives the student away from its initialization model, and general capabilities decline as the displacement grows, resulting in capability interference. A direct remedy is constraining the student toward its initialization, but this suppresses the acquisition of domain expertise as well. We propose Slow-Fast Multi-Teacher On-Policy Distillation (SF-MOPD), which couples a fast model, the current student updated directly by each teacher, with a slow model, an exponential moving average of the student. The slow model absorbs the learning signal gradually, serving as a moving capability reference that fuses the general foundation with confirmed domain expertise. For each teacher, SF-MOPD computes the teacher-induced update in log-probability space and removes only the component that pushes the fast model further away from the slow model, while retaining aligned and orthogonal components. Experiments across multiple model scales demonstrate that SF-MOPD effectively mitigates capability interference, enhances specialized multimodal capabilities, and reduces the average degradation on general-capability benchmarks, consistently outperforming vanilla MOPD.

1. INTRODUCTION

MOPD consolidates domain-specific expertise but can displace the student from its initialization as it trains. SF-MOPD uses a gradually updated capability reference and selective projection to balance expertise acquisition with general-capability preservation.

1. INTRODUCTION

  • MOPD consolidates complementary domain expertise in one student but can drive it away from its initialization during training.The paper calls the resulting phenomenon capability interference.Source: §1 · p. 1
  • A fixed initialization anchor can suppress both general-capability erosion and domain-expertise acquisition.The paper notes that KL-regularized fine-tuning does not separate the two.Source: §1 · p. 1
  • SF-MOPD pairs a fast student updated by each teacher with a slow EMA student that gradually absorbs learning signals as a moving capability reference.The slow reference is intended to fuse the general foundation with confirmed domain expertise.Source: §1 · p. 1
  • For each teacher-induced update, SF-MOPD removes only the component that enlarges slow-fast displacement, retaining aligned and orthogonal components.The pipeline overview describes the fast-slow student-teacher framework and retained update components.Source: §1 · p. 1, Figure 2 · p. 2
  • Across multiple model scales and specialized multimodal domains, SF-MOPD enhances expert-specific capabilities while better preserving general multimodal competence than vanilla MOPD.The overview reports a consistent balance between expert capability integration and general capability preservation.Source: §1 · p. 1, Figure 2 · p. 2

2. METHOD

MOPD transfers complementary expertise from multiple teachers but can displace the student from its initialization as general capabilities decline. SF-MOPD uses a slow EMA reference and per-teacher update projection to balance capability preservation with domain learning.

2.1. Preliminary

  • MOPD combines multiple teachers’ distillation losses, but training can drive the student from its initialization as general capabilities decline.The paper calls this displacement-associated decline capability interference.Source: §2.1 · p. 2

2.2. Slow-Fast Capability Reference

  • SF-MOPD maintains a fast student updated by each teacher and a slow EMA model that gradually absorbs recurring learning directions while smoothing transient fluctuations.The slow model starts from the same initialization and evolves as a moving reference aligned with expertise it has absorbed.Source: §2.2 · p. 2

2.3. Slow-Fast Projection

  • For each teacher, SF-MOPD projects its update against the slow-fast constraint direction and removes only the component opposing that direction.The projection retains aligned and orthogonal components; the opposing component is identified when the update pushes the fast model further from the slow model.Source: §2.3 · pp. 2–3
  • The projection needs no teacher-specific gradient storage or additional trainable parameters, relying on parameter smoothing and the slow-fast models.The source also describes bounding and re-centering the projected update.Source: §2.3 · p. 3

2.4. Overall Objective

  • SF-MOPD combines the original MOPD loss with a slow-fast correction, allowing domain-specific learning while keeping the fast model close to the capability state confirmed by the slow model.The original loss provides unmodified supervision from all teachers, while the correction adjusts teacher-induced updates along the slow-fast constraint direction.Source: §2.4 · p. 3

3. EXPERIMENT

Experiments evaluate SF-MOPD across three student scales and specialized multimodal domains. It improves domain-specific performance while retaining general capability, and component ablations identify the slow-fast projection as especially important.

3.1. Experiment Setting

  • Students at 2B, 4B, and 8B are distilled from Qwen3-VL-8B experts using domain-specific data and evaluated on specialized and general benchmarks.Training uses AdamW, a 1 × 10^-5 learning rate, batch size 64, one epoch, and EMA coefficient α = 0.999.Source: §3.1 · p. 3

3.2.1. Main Results

  • At 2B, 4B, and 8B, SF-MOPD reaches domain-specific averages of 44.1%, 56.9%, and 60.4%; at 8B, General capability is 76.4%, 3.1% above MOPD.At 4B, the overall average is 64.8%, the best reported; at 8B, the General score narrows the gap to the 77.2% instruct baseline.Source: §3.2.1 · p. 3, Table 1 · p. 4
  • On 8B, SF-MOPD reaches 76.4% General and 60.4% Domain-Specific, versus 73.8% and 59.1% with a fixed initialization reference.The fixed-reference variant improves only slightly over vanilla MOPD, while SF-MOPD uses an EMA reference and per-teacher projection.Source: §3.2.1 · p. 3, Table 2 · p. 4

3.3. Ablation Study

  • Removing Slow-Fast Projection lowers the overall average from 67.7% to 64.0%; removing LMOPD or CenterNorm also lowers the full model's overall score.Across β settings 0.1, 0.3, 0.5, 0.9, and 1.0, overall averages are 67.7%, 66.1%, 67.3%, 67.2%, and 67.0%, respectively.Source: §3.3 · p. 4, Table 2 · p. 4
  • With reduced task heterogeneity, General-purpose capability is 76.5% for MOPD and 77.3% for SF-MOPD, while the data strategy drops ZoomBench to 40.7%.The authors describe increasing fine-grained teacher-data sampling as suboptimal for data construction.Source: §3.3 · p. 4

4. CONCLUSION

SF-MOPD uses a fast student and an EMA slow model as a moving capability reference. Its projection filters teacher-induced updates that push the fast model away from the slow one, and experiments across three scales report improved domain-specific results and a better general-domain trade-off.

4. CONCLUSION

  • The slow model is an EMA of the student initialized from the same starting point, while the fast model learns directly from each teacher.The projection removes only update components that push the fast model away from the slow model, leaving specialized learning uncapped.Source: §4 · p. 4
  • Across 2B, 4B, and 8B scales, SF-MOPD achieves the best domain-specific average and improves the general-domain trade-off.The paper concludes that the framework addresses capability interference during domain-expertise acquisition.Source: §4 · p. 4
  • The authors call for further research on balancing capability acquisition and preservation in multimodal large language models.Source: §4 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The work reports that its experiments were computational numerical simulations and involved no human or animal subjects; therefore, ethical approval was not required.

6. COMPLIANCE WITH ETHICAL STANDARDS

  • All experiments were computational numerical simulations.Source: §6 · p. 5
  • The experiments involved no human or animal subjects.Source: §6 · p. 5
  • The authors state that ethical approval was not required.Source: §6 · p. 5

Related papers

Loading 2610.02324v1…