Source-linked AI summary
Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, Linfeng Zhang
TL;DR
MLLMs incur high computational costs from long vision-token sequences, while importance-based pruning can ignore token interactions, conflict with efficient attention, and sometimes underperform random pruning. DART addresses this by pruning duplicated tokens using a small pivot set; across models and benchmarks, it achieves strong performance under extreme compression, with limitations for black-box models.
Problem
Long vision-token sequences create major computational costs, while importance-based pruning can ignore token interactions and rely on static importance during pruning.
Method
DART selects a small pivot-token subset, measures duplication through cosine similarity, and retains vision tokens with low duplication to the pivots.
Results
Across four MLLMs and over 10 benchmarks, DART outperforms the second-best method by 2.2% (93.7% vs. 91.5%) on LLaVA-1.5-7B with an 88.9% reduction ratio.
Takeaways & Limitations
DART provides training-free token reduction that balances performance and latency while remaining compatible with efficient attention operators.
Takeaways & Limitations
DART cannot be applied to black-box models such as GPT-3.5, more advanced GPT versions, and Claude because their encoded tokens are inaccessible during inference.
Abstract
from arXiv · showhide
Vision tokens in multimodal large language models often dominate huge computational overhead due to their excessive length compared to linguistic modality. Abundant recent methods aim to solve this problem with token pruning, which first defines an importance criterion for tokens and then prunes the unimportant vision tokens during inference. However, in this paper, we show that the importance is not an ideal indicator to decide whether a token should be pruned. Surprisingly, it usually results in inferior performance than random token pruning and leading to incompatibility to efficient attention computation operators.Instead, we propose DART (Duplication-Aware Reduction of Tokens), which prunes tokens based on its duplication with other tokens, leading to significant and training-free acceleration. Concretely, DART selects a small subset of pivot tokens and then retains the tokens with low duplication to the pivots, ensuring minimal information loss during token pruning. Experiments demonstrate that DART can prune 88.9% vision tokens while maintaining comparable performance, leading to a 1.99$\times$ and 2.99$\times$ speed-up in total time and prefilling stage, respectively, with good compatibility to efficient attention operators. Our codes are available at https://github.com/ZichenWen1/DART.
1 Introduction
MLLMs face high computation costs from massive vision-token sequences, while importance-based pruning can ignore token interactions, conflict with efficient attention, and underperform random pruning. DART instead reduces duplication using a small pivot-token set and retains low-duplication vision tokens.
- Massive vision-token sequences, especially from high-resolution images and multi-frame video, create substantial computation costs.
- Importance-based pruning can ignore how token importance changes when similar tokens are removed or retained.
- Importance-based methods conflict with FlashAttention because they require inaccessible attention scores, while disabling it increases latency and memory costs.
- DART selects pivot tokens comprising no more than 2% of tokens, computes cosine similarity, and retains vision tokens with the lowest duplication.
- 93.7% versus 91.5%: DART outperforms the second-best method on LLaVA-1.5-7B at an 88.9% reduction ratio.
2 Related Work
Related work identifies redundancy, low information density, and quadratic attention costs as major challenges in visual-token processing. Existing approaches motivate efficient token representations and longer-context processing, while DART specifically preserves FlashAttention and targets duplication.
- Visual processing is costly because of redundancy, low information density, and the quadratic cost of attention.
- High-resolution image models encode thousands of tokens, while video models process even more tokens across frames.
- DART differs from approaches that neglect token duplication by preserving FlashAttention and targeting duplication for efficient token reduction.
3 Methodology
The methodology critiques static attention-based token importance and introduces DART, which reduces visual tokens by identifying duplication relative to a small pivot subset. It also analyzes efficiency, theoretical error bounds, and compatibility with efficient attention.
- MLLM Architecture: MLLMs encode images into visual tokens that are concatenated with text tokens before sequential language-model decoding.The visual encoder and modality projector produce visual tokens for the LLM input.
- Beyond Token Importance: Attention-based pruning assigns static importance scores and retains Top-k visual tokens, assuming scores remain unchanged after tokens are removed.This independence assumption ignores dynamic interactions among tokens.
- Beyond Token Importance: Attention-based token pruning can bias retention toward later lower-right image tokens, underperform random retention, and conflict with Flash Attention.The paper reports that this bias can increase hallucinations, whereas DART produces more balanced retention and more consistent outputs.
- Token Duplication: DART selects a minimal pivot subset, measures cosine similarity to remaining image tokens, and retains tokens with low duplication relative to the pivots.The pivot subset contains no more than 2% of all tokens, and the threshold is dynamically determined by the desired reduction ratio.
- Token Duplication: Pivot selection is not highly sensitive: K-norm, V-norm2, and random strategies achieve competitive performance across multiple benchmarks.The method can also incorporate attention scores for pivot selection and is described as orthogonal to attention-based importance measurement.
- Theoretical Analysis: Under bounded embeddings and Lipschitz continuity with respect to Hausdorff distance, DART provides a bounded approximation error and output-difference guarantee.The analysis bounds the distance from removed tokens to retained pivots and uses that bound to control the model-output difference.
4 Experiments
DART is evaluated across diverse image and video understanding settings, models, and token configurations. It maintains strong performance under substantial token reduction and outperforms comparison methods in the reported video benchmarks.
- Image understanding task: 98.8% average performance with only 192 tokens, outperforming MustDrop by 1.6%.On image understanding tasks, DART also leads by 2.2% using 64 tokens.
- Image understanding task: 93.9% performance with only 11.1% of tokens on larger and advanced models, outperforming all competitors significantly.The result is reported for models including LLaVA-Next-7B and Qwen2-VL-72B.
- Image understanding task: DART maintains full performance with 192 visual tokens when applied during training, improving the performance-efficiency trade-off.This result is reported for DART† in Table 1.
- Video Understanding Task: 4.0 on MSVD, 46.3% accuracy on TGIF, and 56.7% accuracy on MSRVT, surpassing FastV across all reported video benchmarks.DART achieves an average accuracy of 58.0% and an evaluation score of 3.7.
5 Analysis and Discussion
The analyses examine DART’s efficiency, pivot-token selection, pruning layer, pivot-token count, and pivot-token source. They show that DART combines efficient acceleration with robust selection behavior, while extreme pivot counts and single-modality guidance can hurt performance.
- Efficiency Analysis: 2.99× prefill speedup and 1.99× inference speedup, while POPE performance degrades by less than 3% versus the vanilla model.DART integrates Flash Attention with under 0.08s overhead and achieves a better performance-speed balance.
- Efficiency Analysis: FLOPs alone poorly measure acceleration: SparseVLM increases FLOPs by 2.8% versus DART while its speedup drops 21.6%.The comparison highlights differences in actual latency across methods.
- Pivot token selection: Over 94.9% of vanilla-model performance is achieved across pivot-selection strategies, including random selection.Random pivot selection incurs only a 1.2% drop versus the best strategy and outperforms previous importance-based methods by 2.1%.
- Pivot token selection: Less than 50% overlap between tokens preserved by K-norm-high and K-norm-low strategies still yields highly effective MME results.The finding indicates that multiple distinct groups of tokens can support strong performance.
- Pruned layer: Pruning at layers 10, 15, and 20 can outperform the vanilla model, while deeper-layer pruning increases latency but approaches vanilla performance.At layers 15 and 20, pruning all vision tokens reduces performance by only 0.1%–1.6%.
- Number of pivot tokens: Insufficient or excessive pivot-token counts produce suboptimal outcomes, because too few reduce feature-space diversity and too many reduce selection effectiveness.The reported excessive setting is 20 or more pivot tokens.
- Pivot token sources: Removing pivot tokens from either the visual or text modality causes a noticeable performance decline.The analysis supports contributions from both modalities to DART’s token-reduction process.
6 Conclusion
The paper concludes that token duplication is a more effective reduction target than token importance for balancing performance and latency in multimodal language models.
- Conclusion: DART targets token duplication rather than attention-based importance, achieving a better performance-latency balance across multiple benchmarks and MLLMs.The conclusion also reports distinct retained token sets with under 50% overlap and notes that pruning may reduce hallucinations.
- Conclusion: Importance-based pruning can perform worse than random pruning, exposing limits in using attention scores as the reduction criterion.This is presented as a central insight motivating DART.
7 Limitations
DART cannot be applied to black-box models because their encoded tokens are inaccessible during inference. Some experimental analyses are instead placed in the appendix due to space constraints.
- DART cannot be applied to black-box models such as GPT-3.5, newer GPT versions, and Claude because encoded tokens are inaccessible during inference.
- Space limitations required moving several experimental results, including pivot-selection strategies and larger-model validations, to the appendix.
- Moving these results to the appendix may slightly affect the overall reading experience.
A.1 Supplementary Results on Pivot Token Selection
Supplementary experiments examine pivot-token selection and token-count effects across visual benchmarks. DART remains robust across selection criteria, while both insufficient and excessive pivot counts can reduce performance.
- Pivot Token Selection: DART consistently outperforms SparseVLM and FastV across pivot-token selection criteria with 128 retained vision tokens.The analysis covers GQA, MMB, MME, POPE, SQA, and VQA on LLaVA-1.5-7B.
- Pivot Token Count: Both insufficient and excessive numbers of pivot tokens can produce suboptimal performance on additional benchmarks such as POPE and SQA.
A.3 More Experimental Results on Larger MLLMs
Additional experiments show that DART remains effective on larger multimodal language models, under high pruning ratios, and across audio-recognition settings. It also retains strong performance while reducing visual-token computation.
- Larger MLLMs: DART consistently outperforms FastV and SparseVLM across pruning ratios and downstream tasks on LLaVA-1.5-13B and Qwen2-VL-72B.The method maintains near-Vanilla performance on these larger models.
- Larger MLLMs: 92.2% performance at 88.9% pruning on Qwen2-VL-72B surpasses FastV’s 88.0%.
- Larger MLLMs: 99.5% and 98.9% accuracy are retained at 66.7% pruning on LLaVA-1.5-13B and Qwen2-VL-72B, respectively, with minimal degradation.
- Task Results: 60.9 GQA at 77.8% pruning on LLaVA-1.5-13B and 90.9 ScienceQA at 66.7% pruning on Qwen2-VL-72B both outperform FastV.
- Automatic Speech Recognition: DART maintains lower Word Error Rates than random pruning and FastV on FLEURs-en and LibriSpeech-long under varying reduction ratios.
B.2 Enhancing VLA Efficiency with DART
DART is evaluated for improving VLA efficiency in the SIMPLER tabletop-manipulation benchmark. With only 56 visual tokens, it achieves the highest reported success rates while substantially reducing computational cost.
- Method and Setting: DART is evaluated in SIMPLER, a simulation-based tabletop-manipulation benchmark designed to reflect real-world robot dynamics.The VLA model receives 224×224 RGB observations and natural-language task instructions.
- Results: 75.2% and 64.4% average success rates with 56 retained visual tokens are highest for Visual Matching and Variant Aggregation, respectively.DART outperforms Random Dropping, FastV, VLA-Cache, and vanilla CogACT.
- Results: 44.7% FLOPs and a 1.25× speedup relative to CogACT demonstrate reduced computational cost alongside high task performance.
C.1 Datasets
The experiments use diverse image, video, speech, and robot-manipulation benchmarks, alongside multiple open-source MLLMs and token-reduction baselines.
- Experiments cover ten image-understanding benchmarks, including GQA, MMBench, MME, POPE, VizWiz, SQA, VQAV2, and TextVQA.
- TGIF-QA and MSVD-QA evaluate video question answering with spatio-temporal, frame-based, and open-ended questions.
- FLEURS and LibriSpeech-Long support evaluation of universal speech representations and long-form speech generation.
- SIMPLER evaluates real-world robot manipulation policies in realistic simulated environments with correlation to real-world performance.
- DART is evaluated on open-source MLLMs including LLaVA, Qwen2-VL-7B, and MiniCPM-V-2.6, including high-resolution-image settings.
- The related methods considered reduce MLLM tokens through strategies including token merging, pruning, and adaptive allocation.
C.4 Implementation Details
The implementation evaluates computational cost through FLOPs analysis and reports observations about pruning depth, hallucinations, pivot selection, and sparsification patterns.
- All experiments run on an Nvidia A100-80G GPU using Python 3.10, PyTorch 2.1.2, and CUDA 11.8.
- Computational complexity: Total MLLM FLOPs account for transformer layers, sequence length, hidden dimension, and FFN intermediate size.
- Computational complexity: After pruning, FLOPs use the pruned layer and post-pruning sequence length to estimate theoretical visual-token reduction.
- Future works: Token pruning can reduce hallucinations, and DART performs better than the vanilla model on POPE.
- Sparsification visualization: Across pivot-selection strategies, DART retains spatially scattered tokens without obvious bias, supporting more accurate image understanding and consistent responses.