PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?

Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, Linfeng Zhang

arXiv:2502.11501v2cs.CLcs.CV

TL;DR

MLLM token pruning seeks to reduce severe inference costs, but its design principles and evaluation remain insufficiently understood. Through systematic experiments and analyses, the paper finds that simple spatially uniform selection can outperform complex methods, that language guidance is task-dependent, and that latency is more meaningful than FLOPs for acceleration evaluation.

  • Problem

    Token-pruning research lacks sufficient evidence about token importance, attention scoring, language guidance, importance–redundancy trade-offs, and unbiased efficiency evaluation.

  • Method

    The paper conducts systematic experiments and analyses across token-pruning designs, spatial uniformity, importance versus redundancy, and efficiency evaluation.

  • Results

    Random selection or pooling often outperforms complex pruning designs, while language guidance helps text-driven tasks but can degrade vision-centric tasks; latency is a more practical efficiency metric than FLOPs.

  • Takeaways & Limitations

    Future token pruning should balance simplicity, effectiveness, hardware compatibility, and task-specific adaptability.

  • Takeaways & Limitations

    The experiments focus mainly on LLaVA, LLaVA-Next, and Qwen2-VL, so broader architectures and model scales remain to be evaluated.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown remarkable performance for cross-modal understanding and generation, yet still suffer from severe inference costs. Recently, abundant works have been proposed to solve this problem with token pruning, which identifies the redundant tokens in MLLMs and then prunes them to reduce the computation and KV storage costs, leading to significant acceleration without training. While these methods claim efficiency gains, critical questions about their fundamental design and evaluation remain unanswered: Why do many existing approaches underperform even compared to naive random token selection? Are attention-based scoring sufficient for reliably identifying redundant tokens? Is language information really helpful during token pruning? What makes a good trade-off between token importance and duplication? Are current evaluation protocols comprehensive and unbiased? The ignorance of previous research on these problems hinders the long-term development of token pruning. In this paper, we answer these questions one by one, providing insights into the design of future token pruning methods.

1 Introduction

MLLMs face high computation and storage costs from large models and abundant visual tokens, motivating token pruning. The paper highlights that naive baselines can outperform established pruning methods and identifies overlooked design and evaluation questions.

  • MLLM inference is costly because large inherited language-model parameters combine with many visual tokens from high-resolution images and videos.
  • Token pruning measures visual-token importance, then removes or merges redundant tokens to reduce subsequent computation and storage.
  • The paper investigates overlooked questions about token importance, attention scoring, language information, importance–uniqueness trade-offs, and evaluation protocols.
  • Random token selection and average pooling outperform FastV and SparseVLM in most benchmarks by a clear margin.
  • The authors aim to inform future token-pruning design and correct long-neglected evaluation issues.

2 Related Work

VLMs connect visual encoders to language models through alignment modules, but higher-resolution inputs substantially increase visual-token counts and inference demands. Existing efficiency methods include trained architectural changes and training-free pruning strategies whose attention-based importance may not match actual visual relevance.

  • VLMs serialize visual representations with an encoder and use an alignment module, such as an MLP projector or query-based network, to connect them to an LLM.
  • Higher-resolution images increase visual-token counts substantially, with LLaVA-1.5 producing 576 tokens and LLaVA-NeXT producing up to 2880.
  • Visual tokens are more numerous, spatially redundant, and lower-density than text tokens, motivating token-count reduction methods.
  • Some token-reduction approaches modify model components or use adaptive pooling, but typically require additional training and increase computational costs.
  • FastV and SparseVLM use text–visual attention to estimate token importance, which may not align well with actual visual-token relevance.

3 Benchmarking

The study benchmarks representative MLLMs, pruning methods, and diverse visual tasks, including understanding, grounding, and retrieval. Its setup also examines token distributions and attention scores across visual-token positions.

  • The study describes its datasets, models, pruning methods, experimental setup, and interpretation guidance before analyzing results.
  • Representative models include LLaVA-1.5-7B and 13B, LLaVA-Next-7B, and Qwen2-VL 7B-Instruct and 72B-Instruct.
  • The benchmark suite covers visual understanding, RefCOCO grounding, and Visual Haystack object retrieval tasks.
  • The evaluation examines FastV, SparseVLM, and MustDrop as representative visual-token pruning methods.
  • Figure 2 analyzes token distributions and attention scores by token position, with larger indexes corresponding to image-bottom locations.

4 Token Pruning Revisited: Are Simple Methods Better?

The paper revisits token pruning by comparing simple baselines with attention-based methods and examining whether retained-token position affects performance. It finds that random or pooled selection can outperform designed pruning methods, while spatially uniform retention improves results under aggressive reduction.

  • Simple Baselines: Random and Pooling outperformed carefully designed methods on nearly 2/3 benchmarks.The comparison used pruning ratios of 75% and 87.5% across multiple widely used benchmarks.
  • Position Bias: FastV assigns visual-token importance using attention scores from the last token, but later-position tokens receive higher scores and are retained more often.The position-bias analysis used 8,910 POPE samples.
  • Spatial Uniformity: Window FastV uses fixed reduction within sliding windows to ensure spatially uniform retained-token distributions.Its implementation varies the reduction ratio and window size to select a fixed number of visual tokens.
  • Spatial Uniformity: Across eight benchmarks, Window FastV’s average performance drop was 3.4% less than Vanilla FastV’s when reducing 75% of visual tokens.At a more aggressive reduction ratio of ↓88.9%, the performance-drop gap widened to 9%.
  • Spatial Understanding: On RefCOCO grounding, token-pruning methods degraded substantially, with spatially non-uniform strategies showing particular limitations in precise object localization.RefCOCO counts a prediction as correct when box IoU exceeds 0.5; the table uses Precision@1 with a 77.8% reduction ratio.

5 Language in Visual Token Pruning: When and Why Does Language Matter?

Language guidance matters most when pruning serves strongly text-driven tasks, but its benefit is task-dependent rather than universal. On Visual Haystack, removing textual information from FastV significantly reduced performance, while text-guided methods performed better overall.

  • Motivation: Token-pruning methods using textual information and methods using only visual information achieve comparable performance on common benchmarks.This motivates testing whether language guidance becomes more valuable on tasks that strongly depend on text.
  • Task setting: Visual Haystack requires selecting an image using an anchor phrase and checking whether it contains an object matching a target textual description.The task therefore demands precise alignment between visual and textual modalities.
  • Evidence for language guidance: FastVVIS, which removes textual guidance from FastV by using the last visual token, shows a significant performance drop on Visual Haystack.FastV originally scores visual-token importance using attention to the last text token, whereas FastVVIS uses the last visual token.
  • Evidence for language guidance: Text-guided pruning methods exhibit significantly better overall performance on strongly text-guided tasks.SparseVLM reaches a 77.8% compression rate while maintaining nearly identical accuracy to the uncompressed model in the reported scenarios.
  • Implication: Pruning strategies should incorporate textual guidance for tasks that rely heavily on language, while balancing linguistic information remains an open research issue.The paper concludes that pruning should adapt to task requirements rather than apply language guidance uniformly.

6 The α Dilemma: Importance vs. Redundancy in Token Pruning

Token pruning must balance removing redundant tokens against retaining task-critical predictive information. Experiments show that the preferred balance depends strongly on the task.

  • 6 The α Dilemma: Importance vs. Redundancy in Token Pruning: Token pruning can be framed as a tension between preserving structural patterns by removing redundancy and maintaining predictive capacity by retaining important tokens.The task-agnostic criterion preserves input structure, while the task-oriented criterion preserves information relevant to the target output.
  • Redundancy Criteria: Mutual-information formulations seek a short retained sequence X′ that preserves dependence on the original tokens X under a length constraint.The redundancy criterion treats pruning as compression while requiring mutual information to remain above a minimum threshold γ.
  • Importance Criteria: The importance criterion treats discarded tokens as acceptable only when they provide at most ϵ additional information about the target Y after conditioning on retained tokens.This defines importance through non-decomposable predictive information needed for task accuracy.
  • Empirical Validation of Adaptive Criteria Balancing: Adaptive scoring combines a token’s unique predictive value with its pattern distinctiveness using a tunable parameter α.The predictive term measures unique value for Y, whereas the distinctiveness term measures how unlike the other tokens the token is.
  • Empirical Validation of Adaptive Criteria Balancing: Perception-Dominant Tasks peak at α = 0.1 and 0.0, whereas Knowledge-Intensive Tasks peak at α = 0.8 ∼ 0.9.The results favor redundancy-first pruning for perception tasks and importance-first pruning for knowledge reasoning.

7 Limitations and Challenges in Token Pruning Evaluation

Token-pruning evaluation can misrepresent efficiency when it relies on FLOPs or token counts and ignores compression performed during model training. Actual latency and training-aware compression provide more informative evaluation boundaries.

  • 7.1 Beyond FLOPs: Shifting the Focus to Actual Latency Gains: FLOPs and token-reduction ratios do not reliably represent the acceleration achieved by token pruning.Methods with similar computational reductions can exhibit substantially different measured speeds.
  • 7.1 Beyond FLOPs: Shifting the Focus to Actual Latency Gains: Sparse-VLM has FLOPs only 2.8% higher than FastV but latency 26.8% greater under the same setting.The comparison supports evaluating speed gains with actual latency rather than FLOPs alone.
  • 7.1 Beyond FLOPs: Shifting the Focus to Actual Latency Gains: FastV, SparseVLM, and MustDrop do not support efficient Flash Attention because they rely on complete attention maps, while FastV prunes in one language-model layer and the other methods prune across four.The analysis motivates shallow-layer pruning with simpler operations to avoid runtime overhead.
  • 7.2 The Overlooked Role of Training-Aware Compression in MLLMs: Training-aware compression changes the effective information density of visual tokens, so removing the same number of tokens can cause greater information loss than in traditional settings.Qwen2-VL compresses visual tokens during training, making this compression relevant to inference-stage evaluation.
  • 7.2 The Overlooked Role of Training-Aware Compression in MLLMs: When training-aware compression is accounted for, the same pruning method reaches performance on par with the vanilla model across multiple benchmarks under varying reduction ratios.The result motivates further research into training-aware token compression techniques.

8 Conclusion

The paper identifies overlooked design and evaluation issues in MLLM token pruning. It argues for simpler, task-adaptive methods that balance structural integrity, predictive power, and practical latency.

  • 8 Conclusion: Naive spatial uniformity from random selection or pooling often outperforms complex pruning designs because of positional biases in visual tokens.The conclusion frames this as a challenge to attention-based scoring and language-guided pruning.
  • 8 Conclusion: Linguistic guidance helps in text-driven scenarios but can degrade performance in vision-centric tasks when task alignment is weak.Its effectiveness therefore depends on the relationship between the task and language information.
  • 8 Conclusion: Latency is a more practical and meaningful efficiency metric than FLOPs for evaluating token-pruning speed gains.The conclusion links evaluation quality to the practical measurement of runtime.
  • 8 Conclusion: Future token-pruning methods should balance simplicity, effectiveness, and task-specific adaptability.This framework follows the paper’s combined findings on structural preservation, predictive accuracy, and evaluation.

9 Limitations

The experiments primarily cover LLaVA, LLaVA-Next, and Qwen2-VL, so the analysis should be extended across broader architectures and model scales. Future work also includes comparing token pruning with token merging and evaluating more challenging rich-text OCR benchmarks.

  • Experiments primarily use LLaVA, LLaVA-Next, and Qwen2-VL, leaving broader model architectures for future study.The authors also propose evaluating models of varying sizes to test whether conclusions generalize across architectural scales.
  • Future work will examine whether token pruning or token merging should be prioritized for token reduction.
  • Future evaluations will target more challenging OCR benchmarks, especially datasets containing rich-text OCR images.

B Algorithms

The appendix presents core algorithms for several token-reduction strategies, including attention-based, reverse-attention, spatial-window, and pooling variants.

  • Vanilla FastV retains vision tokens with the highest attention scores.
  • Reverse FastV instead retains tokens with the lowest attention scores.
  • Window FastV divides image tokens into local windows and selects tokens within each window using spatial awareness.
  • Pooling is included as another core token-compression method.

C Dataset

This appendix introduces the benchmarking datasets and provides algorithm listings for token-selection, window-based selection, reverse selection, and pooling procedures.

  • C Dataset: The appendix identifies the datasets used for benchmarking.
  • B Algorithms: Vanilla FastV computes attention, selects retained indices, constructs a compressed sequence, and applies regular Transformer computation.
  • B Algorithms: Reverse FastV uses the same retention pipeline while selecting tokens according to its reverse attention strategy.
  • B Algorithms: Window-based selection divides the image-token grid into window patches, selects local indices, maps them globally, and builds the retained sequence.
  • B Algorithms: Pooling performs window pooling, constructs an index mapping, builds retained indices, and generates a compressed sequence before regular Transformer computation.
Loading 2502.11501v2…