Source-linked AI summary
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Haoyi Sun, Xiaoxiao Wang, Ning Mao, Qian Wang, Lifu Mu, Wen Zheng, Tao Wei, Wei Chen
TL;DR
VLM scale limits efficient deployment, and existing distillation methods do not explicitly unify modalities despite multimodal knowledge being fused in language space. Switch-KD routes student visual outputs through the teacher’s language pathway and uses DBiLD for adaptive bidirectional alignment, with reported gains across multimodal benchmarks. Its current requirement for feature-space and vocabulary consistency limits heterogeneous-architecture use.
Problem
Existing VLM distillation methods supervise modalities separately, overlooking their intrinsic coupling within the shared language space.
Method
Switch-KD uses Visual-Switch Distillation to route student visual outputs through the teacher’s language pathway and DBiLD to adaptively align informative logits regions bidirectionally.
Results
Switch-KD consistently surpasses state-of-the-art distillation and lightweight VLM approaches across 10 multimodal benchmarks.
Takeaways & Limitations
Switch-KD transfers visual and language knowledge within a shared text-probability space without architectural modification to the distilled model.
Takeaways & Limitations
Stable cross-modal transfer currently requires feature-space and vocabulary consistency between teacher and student, limiting heterogeneous architectures.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.
1. Introduction
Large VLMs are difficult to deploy efficiently, while existing distillation methods supervise modalities separately despite multimodal knowledge being fused in language space. Switch-KD unifies transfer through shared text probabilities using visual-switch distillation and adaptive DBiLD alignment.
- Large VLMs incur latency, memory, and energy costs that limit resource-constrained and real-time deployment.
- Existing VLM distillation methods use non-unified modality supervision that overlooks intrinsic visual-language coupling.
- Switch-KD unifies multimodal knowledge transfer within a shared text-probability space.
- Visual-Switch Distillation routes student visual outputs through the teacher’s language pathway to produce visual-switch logits for implicit visual knowledge transfer.
- DBiLD adaptively aligns teacher–student logits through bidirectional supervision, emphasizing informative regions instead of relying on a fixed top-k boundary.
- Extensive experiments on 10 multimodal benchmarks show consistent gains over existing distillation and lightweight VLM approaches.
2. Related Work
Related work includes unified vision-language architectures and several VLM distillation strategies, but existing approaches can produce fragmented supervision. Switch-KD revisits distillation through shared language-probability alignment.
- Vision-Language Models: Modern VLMs commonly combine a vision encoder, projector, and language model for multimodal understanding.
- Existing VLM distillation methods include architecture enhancement and intermediate-layer supervision strategies.
- Intermediate-layer supervision can lead to fragmented and inconsistent knowledge transfer.
3. Method
Switch-KD unifies multimodal distillation in a shared text-probability space, combining standard alignment with a visual-switch pathway and adaptive bidirectional logits alignment.
- Switch-KD addresses fragmented multimodal supervision by aligning visual and language knowledge within a shared text-probability space.
- Visual-Switch Distillation: The training objective combines autoregressive language modeling, standard alignment, and visual-switch distillation through weighted loss terms.λ1 and λ2 control the relative contributions of standard alignment and visual-switch distillation.
- Visual-Switch Distillation: Visual-Switch Distillation feeds student visual encoder outputs through the teacher’s projector and language model to produce visual-switch logits.This hybrid route uses the teacher’s language pathway to interpret student visual representations.
- Dynamic Bi-directional Logits Difference Loss: DBiLD combines teacher-guided and student-guided branches to provide bidirectional symmetric supervision between teacher and student distributions.The student-guided branch evaluates student-confident predictions against corresponding teacher logits.
- Dynamic Bi-directional Logits Difference Loss: DBiLD dynamically selects informative top-k logits using knee-point detection rather than applying one fixed cutoff across samples.The maximum distance from a reference line identifies the transition from information-rich logits to the long-tail region.
- Dynamic Bi-directional Logits Difference Loss: The selected logits are compared through internal pairwise differences and temperature-scaled softmax distributions.These distributions are aligned with reverse KL divergence in the teacher-guided branch.
4.1. Setup
The experiments largely follow LLaVA-KD’s configuration, using specified visual encoders, projectors, and Qwen2.5 backbones across two training stages.
- The main comparison uses SigLIP-B/14@384px, a two-layer GELU MLP projector, and Qwen2.5 family language models.The setting largely inherits LLaVA-KD’s training configuration.
- A separate comparison with Align-KD uses CLIP ViT-L/14, LDPv2, and a Qwen2.5-1.5B backbone.
- Training is divided into pre-training on LLaVA1.5-558K and distilled fine-tuning on LLaVA-Mix-665K.Teacher and student models share the same overall architecture during both stages.
4.2. Benchmarked Results with the SoTAs
Switch-KD achieves leading or competitive performance across model scales and benchmarks, including strong gains for 1.5B and 0.5B students. It also transfers across architectures while using fewer samples and a lighter backbone.
- Overall comparison: Switch-KD consistently achieves leading or competitive performance across models of different scales.The benchmark comparison covers state-of-the-art and lightweight VLMs.
- Large-scale models: 64.5 Avg7 for Switch-KD-1.5B is comparable to 64.9 for TinyLLaVA-3B, while surpassing MoVE-KD-v1.1-7B by 4.0 on MMBench, 1.5 on VQAv2, and 0.7 on TextVQA.These results are reported for the large-scale group (≥3B parameters).
- Mid-scale models: 1.6 points in Avg7 and 1.3 points in Avg10 are achieved over the TinyLLaVA-1.5B PT-SFT baseline.Switch-KD-1.5B also outperforms LLaVA-MoD and LLaVA-KD by 4.4 and 0.4 points in Avg7, respectively.
- Small-scale models: 4.0 points in Avg7 are gained over SPHINX-Tiny while using only 8% of its training data volume.Switch-KD-0.5B also surpasses comparable-scale LLaVA-MoD and LLaVA-KD by 3.7 and 1.1 points in Avg7, respectively.
4.3. Ablation Study and Analysis
The ablations show that visual switching, dynamic bidirectional logits alignment, and fine-tuning-stage distillation improve performance, while teacher benefits depend on student capacity and current transfer requires architectural consistency.
- Visual-Switch Distillation: Visual-switch distillation raises the 0.5B student’s average score from 58.8 to 60.1 over standard distillation, with gains on VQAv2, GQA, TextVQA, and VizWiz.The reported improvements are +1.8 on VQAv2, +1.9 on GQA, +1.3 on TextVQA, and +3.4 on VizWiz.
- Visual-Switch Distillation: Switch-KD produces attention maps more consistent with teacher semantics than SFT, LLaVA-KD, and Align-KD, whose attention is respectively less selective, diffuse, or partial.The teacher emphasizes semantically critical regions, while Switch-KD aligns the student’s visual focus with that semantic pattern.
- DBiLD Loss: 58.8 average score from DBiLD-RKL exceeds the FKL baseline by 0.5 and BiLD by 0.4, supporting dynamic Top-k selection with reverse-KL bidirectional supervision.FKL and RKL alone both score 58.3; BiLD-FKL reaches 58.4 and BiLD-RKL adds a further 0.2 points.
- Training Scheme: 2.3 points gain from fine-tuning-only distillation exceeds the 0.9-point pre-training gain, while applying distillation at both stages causes a 0.1-point drop.The findings motivate replacing conventional SFT with DFT in a streamlined two-stage pipeline rather than adding distillation to both stages.
- Teacher Model Size: 65.3 on Avg10 is reached by the 1.5B student with a 7B teacher, but the 0.5B student drops to 59.8 versus 60.1 with a 3B teacher.Larger teachers can help larger students, whereas limited student capacity constrains or reverses that benefit.
- Limitations: Stable cross-modal transfer currently requires teacher and student feature-space and vocabulary consistency, limiting applicability to heterogeneous architectures.The paper identifies architecture-agnostic distillation or adapter-based mappings as future directions for relaxing this constraint.
5. Conclusion
Switch-KD unifies vision-language distillation through a visual-switch architecture and DBiLD loss in a shared text-probability space. Across ten multimodal benchmarks, it consistently surpasses state-of-the-art distillation and lightweight VLM approaches.
- Switch-KD switches student visual outputs into the teacher’s language pathway for implicit cross-modal supervision within a shared text-probability space.
- Its DBiLD loss dynamically selects informative top-k logits and aligns their ranks bidirectionally via reverse KL divergence while preserving teacher probability structure.
- Across ten multimodal benchmarks, Switch-KD consistently surpasses state-of-the-art distillation and lightweight VLM approaches.
- Switch-KD is presented as a practical solution for deploying high-performance vision-language models in resource-constrained scenarios.