Source-linked AI summary
Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Zichen Wen, Shaobo Wang, Yufa Zhou, Junyuan Zhang, Qintong Zhang, Yifeng Gao, Zhaorun Chen, Bin Wang, Weijia Li, Conghui He, Linfeng Zhang
TL;DR
Visual tokens impose substantial computational costs in MLLMs, while compressing them can shift the feature-space distribution and make direct training difficult. EPIC uses token-wise and layer-wise progressive consistency distillation with a shared teacher. It achieves comparable average performance to the vanilla model with 128 visual tokens and outperforms it on 4 of 10 visual-understanding benchmarks.
Problem
Visual tokens pose substantial computational challenges for MLLMs, and compression perturbs the feature space in ways that make direct training toward the compressed-token optimum difficult.
Source: §1 · pp. 1–2Method
EPIC uses a shared teacher-student MLLM and token-wise and layer-wise consistency distillation to progressively increase compression difficulty without modifying model architecture.
Source: §1 · pp. 2–3Results
With 128 visual tokens, EPIC achieves comparable average performance to the vanilla model and outperforms it on 4 of 10 visual-understanding benchmarks.
Source: §6 · p. 10Takeaways & Limitations
EPIC is compatible with diverse token-compression strategies and is reported to produce robust, highly generalizable models without architecture changes.
Source: §1 · p. 3The theoretical intuition is developed for a one-dimensional scalar prototype under monotonicity and Lipschitz-slope assumptions.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Visual tokens consume substantial computational resources in multi-modal large models (MLLMs), significantly compromising their efficiency. Recent works have attempted to improve efficiency by compressing visual tokens during training, either through modifications to model components or by introducing additional parameters. However, they often overlook the increased learning difficulty caused by such compression, as the model's parameter space struggles to quickly adapt to the substantial perturbations in the feature space induced by token compression. In this work, we propose to develop Efficient MLLMs via Progressive Consistency Distillation (EPIC), a progressive learning framework. Specifically, by decomposing the feature space perturbations introduced by token compression along the token-wise and layer-wise dimensions, we introduce token consistency distillation and layer consistency distillation, respectively, aiming to reduce the training difficulty by leveraging guidance from a teacher model and following a progressive learning trajectory. Extensive experiments demonstrate the superior effectiveness, robustness, and generalization capabilities of our proposed framework.
1 Introduction
MLLMs pay a substantial computational cost for visual tokens, and compression makes learning difficult because it shifts the feature-space distribution and model optimum. EPIC addresses this with token-wise and layer-wise progressive distillation guided by a shared teacher, without changing model architecture.
1 Introduction
- Visual tokens create substantial computational challenges for MLLMs, especially with high-resolution images and multi-frame videos.Compression can also perturb the feature space and shift the parameter-space optimum, making direct training under heavy compression challenging.Source: §1 · pp. 1–2
- EPIC combines token-wise and layer-wise consistency distillation in a progressive learning framework that uses one weight-sharing MLLM as teacher and student.The framework is presented as compatible with diverse token-compression techniques and does not require model-architecture modifications.Source: §1 · pp. 2–3
- Token Consistency Distillation gradually increases compression over training while keeping the teacher at a slightly lower compression ratio than the student.The compression-ratio gap also increases progressively to ease learning and support the teacher’s guidance.Source: §1 · pp. 2–3
- Layer Consistency Distillation shifts compression from deeper to shallower layers as training progresses, following an easy-to-hard trajectory.The approach is motivated by the observation that visual-token significance diminishes in deeper layers.Source: §1 · pp. 2–3
2 Related Works
Related work describes how MLLMs process visual inputs and why longer visual-token sequences strain inference. It surveys training-free and training-aware compression strategies, including methods that modify architectures or add modules.
2 Related Works
- MLLMs commonly project features from a pretrained vision encoder into an LLM’s input space, while dynamic or arbitrary resolutions increase visual-token counts for fine-grained perception.Longer token sequences challenge inference speed and memory because attention has quadratic complexity.Source: §2 · p. 3
- Visual-token compression methods are broadly training-free or training-aware, with training-aware approaches offering greater performance potential while requiring additional training.Training-free methods commonly use either token importance or token redundancy to guide compression.Source: §2 · p. 3
- Training-aware approaches range from parameter-free strategies to architectural changes and auxiliary modules, including dynamic Q-formers and coarse-to-fine condensation.The related-work discussion notes that recent approaches often overlook learning challenges from feature-space perturbations during compression.Source: §2 · p. 3
3 Methodology
The methodology establishes the MLLM input and generation pipeline, then develops progressive consistency distillation through a scalar theoretical prototype and token-wise and layer-wise curricula. Both curricula use teacher guidance to stage compression during training.
3.1 Preliminary
- An MLLM projects visual-encoder patch features into visual tokens, concatenates them with text tokens, and autoregressively generates language outputs.The projected visual-token dimension matches the language model hidden size.Source: §3.1 · p. 3
3.2 Theoretical Intuition: A 1D Prototype for Progressive Consistency Distillation Learning
- Under its stated assumptions, the scalar prototype shows that a progressive path has strictly smaller total variation than the direct path.The prototype connects progressive distillation to a smoother optimization trajectory and reduced sensitivity to abrupt input-space changes.Source: §3.2 · pp. 4–5
3.3 Token Consistency Distillation
- Token Consistency Distillation uses shared-weight teacher and student passes, progressively increases student compression and teacher guidance, and applies KL distillation alongside supervised fine-tuning.The student’s compression range shifts toward more aggressive compression, while the teacher remains less compressed by a gradually increasing gap.Source: §3.3 · p. 5
3.4 Layer Consistency Distillation
- Layer Consistency Distillation moves compression from deeper to shallower layers over training, using teacher–student KL guidance at the selected layer.Compression begins at the deepest layer and reaches the shallowest designated layer by the end of training; the approach is motivated by deeper layers’ negligible visual-token attention.Source: §3.4 · p. 6
4 Experiments
Across 10 visual benchmarks, EPIC remains comparable to vanilla LLaVA at 128 tokens and surpasses it with 192+ tokens. Its experiments also assess compression efficiency and compare teacher-supervision and progressive-compression settings.
4.1 Experimental Setting
- EPIC is implemented without architecture modifications, using LLaVA’s pretrained projector and Vicuna-v1.5, and is evaluated on 10 visual-understanding benchmarks against token-compression and other MLLM baselines.Training incorporates DART, FastV, and Random token pruning.Source: §4.1 · p. 6
4.2 Experimental Results on Benchmarks
- At 128 tokens, EPIC matches vanilla LLaVAv1.5-7B comparably and surpasses it with 192+ tokens, while retaining flexible 36–256-token control in one trained model.It maintains superior average performance to model-modifying training-aware methods, particularly on MME, MMBench, and VQA V2; performance degrades by 2% at 64 tokens and varies by < 1% at 128–256 tokens.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3 Efficiency
- The Token Consistency Distillation ablation compares teacher supervision with no distillation loss and progressive compression with a fixed 88.9% compression ratio.DART is used for training in this ablation.Source: Table 3 · p. 8
- At 64 tokens, all three inference compression methods reduce KV-cache memory, FLOPs, and latency; Random compression achieves nearly 1.6× speedup without additional computation.EPIC requires supervised fine-tuning, taking approximately 12 hours on 8 A100 GPUs.Source: §4.3 · pp. 7–8, Table 2 · p. 7
5 Analyses
The analyses test the roles of teacher guidance and progressive learning, the framework’s generalization across compression strategies, and whether extreme token compression is worthwhile.
5.1 Ablation Studies
- Without teacher guidance, both token and layer consistency distillation show significant average performance degradation, especially on MME and MMBench.The ablations also fix the compression ratio or layer to remove progressive learning; Table 4 examines layer consistency distillation with DART and fixes the non-progressive setting at the second layer.Source: §5.1 · p. 8, Table 4 · p. 8
5.2 How Well Does Proposed Framework Generalize across Different Methods?
- Token consistency distillation improves performance across tested benchmarks and compression methods, including FastV and Random when training uses DART, and narrows the gap between strategies.The Figure 4 setup follows LLaVA-v1.5 and uses 88.9% token compression at inference for all methods.Source: Figure 4 · p. 9, §5.2 · p. 9
5.3 Is Extreme Token Compression Necessary?
- FLOPs fall from 9.3T to around 2T when visual tokens drop from 576 to 128, while further compression yields smaller FLOPs reductions and marginal latency gains.The study reports that in some cases retaining 64 tokens performs better than retaining 36 or 18.Source: Figure 5 · p. 9, §5.3 · p. 9
- At 64 tokens, performance is largely preserved; below that, latency gains are marginal while performance degrades sharply, defining high- and low-ROI regions.Under heavy compression, GPU compute is underutilized and latency is dominated by memory access.Source: §5.3 · pp. 9–10
6 Conclusion
EPIC improves MLLM efficiency without changing the model architecture, and the conclusion reports comparable average performance using 128 visual tokens. It also reports robustness across compression ratios and strategies, while cautioning against excessive compression.
6 Conclusion
- EPIC integrates with existing token compression strategies without architectural changes and targets efficiency in both training and inference.The conclusion reports comparable average performance to the vanilla model using 128 visual tokens.Source: §6 · p. 10
- Using 128 visual tokens, EPIC achieves comparable average performance to vanilla, and it outperforms vanilla on 4 of 10 visual-understanding benchmarks.The conclusion describes the visual-token reduction as significant.Source: §6 · p. 10
- The experiments support robustness across compression ratios and generalization across strategies, while the analysis finds excessive compression can worsen the latency-performance trade-off.The conclusion argues for balancing compression levels.Source: §6 · p. 10
NeurIPS Paper Checklist
The checklist reports that the paper’s claims match its contributions and scope, and that experimental and reproducibility details are provided in the paper or appendices.
NeurIPS Paper Checklist
- The authors answer that the paper’s claims accurately reflect its contributions and scope, and that it discusses limitations in Appendix E.The checklist also says training and test details are provided in Section 4.1 and Appendix D.Source: NeurIPS Paper Checklist · pp. 15–16
- The authors report that experimental details and computational resources are documented, with resources described in Section 4.3 and Appendix D.They state that code will be released after the review process, with instructions intended to support reproducibility.Source: NeurIPS Paper Checklist · pp. 16–18
- The checklist reports that some experiments, including latency statistics, were repeated and that corresponding error bars or bands appear in Section 5.The authors also answer that the paper provides complete theoretical assumptions and proofs in the main paper and appendix.Source: NeurIPS Paper Checklist · pp. 16, 18
10. Broader impacts
The checklist points readers to Appendix F for broader impacts and says the study involved neither human-subject research nor an important, original, or non-standard LLM component.
10. Broader impacts
- The authors report that broader impacts are discussed in Appendix F.They answer that the paper poses no risks requiring safeguards for high-risk releases.Source: §10 · p. 19
- The authors state that creators or owners of existing assets are credited and that the paper releases no new assets.Source: §10 · p. 20
- The checklist marks human-subject and crowdsourcing questions as not applicable because the paper involves neither.The checklist separately marks declaration of LLM usage as not applicable because core method development does not use LLMs as an important, original, or non-standard component.Source: §10 · pp. 20–21
A More Generalizability Experiments
Experiments using Random and FastV for token consistency distillation show improved inference performance across token-compression strategies. The results support the framework’s generalizability beyond a fixed set of preserved visual tokens.
A More Generalizability Experiments
- Figure 6 applies Random token compression during token consistency distillation training on LLaVA-v1.5-7B; all methods use an 88.9% inference compression ratio.“w/o train” denotes vanilla LLaVA.Source: Figure 6 · p. 22
- Figure 7 applies FastV during token consistency distillation training on LLaVA-v1.5-7B; all methods use an 88.9% inference compression ratio.“w/o train” denotes vanilla LLaVA.Source: Figure 7 · p. 22
- Training with either Random or FastV token compression consistently improves inference performance across compression strategies, indicating adaptation to varying patterns of missing visual tokens.The reported experiments use an 88.9% inference token compression ratio.Source: Figure 6 · p. 22, Figure 7 · p. 22, A More Generalizability Experiments · p. 23
B Detailed Theoretical Analysis
The theoretical analysis bounds the slope ratio along a progressive path, while the integrated method applies token-wise progression across layers. The appendix also describes the compression strategies used during training.
B Detailed Theoretical Analysis
- The detailed theoretical analysis invokes monotonicity and Lipschitz continuity to establish bounds for its terms.The supplied passage introduces these conditions but does not include the expressions being bounded.Source: B Detailed Theoretical Analysis · p. 23
2. Progressive path. From the closed-form minimizer,
- Under convexity, the progressive-path slope ratio satisfies κ ≤ 1; it is strictly below 1 when c is non-affine under the stated condition.The path analysis defines κ using lagged and current slopes and considers appropriately spaced r_t values.Source: §2 · p. 23
C Integrated Progressive Consistency Distillation
- ICD reaches comparable performance on representative visual benchmarks, surpasses vanilla LLaVA on average with 192 visual tokens (66.7% ↓), and outperforms training-aware methods at identical retained-token counts.Table 5 covers 10 visual-understanding benchmarks and uses DART for ICD training and inference.Source: C Integrated Progressive Consistency… · p. 24, Table 5 · p. 24
D.1 Token Compression Techniques During Training
- The training setup uses FastV, DART, and random token pruning, representing importance-based, redundancy-based, and signal-free compression strategies, respectively.Random pruning is used to validate the approach’s effectiveness and generalizability.Source: §D.1 · p. 25
D.2 Training Details
The appendix reports a controlled LLaVA-v1.5 training setup, describes evaluation benchmarks, and reviews general MLLMs and training-aware token-compression methods.
D.2 Training Details
- EPIC follows vanilla LLaVA’s architecture, instruction-tuning data, and hyperparameters; training takes about 12 hours on 8 × A100 GPUs.Ablations use matched data and training settings; variants without progressive ratios or layers fix the second layer as the compression layer.Source: §D.2 · p. 25
D.3 Benchmarks
- The benchmarks cover multimodal perception and cognition, science reasoning, object hallucination, visual reasoning, and text- and document-focused visual question answering.MME evaluates perception and cognition; ScienceQA targets multi-hop science reasoning, and POPE examines object hallucination.Source: §D.3 · p. 26
D.4.1 General MLLMs
- The reviewed general MLLMs include systems for fine-grained vision-language reasoning, unified video and image understanding, and academic-oriented visual question answering.The examples include SPHINX, Video-LLaVA, and LLaVA-v1.5.Source: §D.4.1 · pp. 26–27
D.4.2 Training-aware Token Compression Methods
- The reviewed training-aware methods reduce visual tokens through pooling, query-based projection, token compression, or modality pre-fusion.Examples include QT-LLaVA, MQT-LLaVA, TokenPacker, VoCo-LLaMA, and LLaVA-Mini.Source: §D.4.2 · p. 27
E Limitations and Future Works
The framework is applied only during visual instruction tuning, with extending it to pretraining left for future study. The broader-impact discussion also notes that compressed-input training is not followed by human-preference alignment.
E Limitations and Future Works
- EPIC is applied only during visual instruction tuning; extending it to projector pretraining and other training stages remains future work.The authors hypothesize that projector pretraining with the framework could help adapt its parameters before supervised fine-tuning.Source: E Limitations and Future Works · p. 27
F Broader Impacts
- EPIC is compatible with token-compression strategies without modifying the original model architecture.The authors describe efficiency, robustness, and generalizability as benefits of the approach.Source: F Broader Impacts · pp. 27–28
- The compressed-input-trained models are not further aligned with human preferences, and may carry adversarial vulnerabilities or produce undesirable outputs.The authors note these potential risks despite strong performance across a range of multimodal tasks.Source: F Broader Impacts · p. 28