PaperlayerView exact revision on arXiv

Source-linked AI summary

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

Weihao Ye, Qiong Wu, Wenhao Lin, Yiyi Zhou

arXiv:2409.10197v2cs.CVcs.CLcs.MM

TL;DR

MLLMs use large, redundant visual-token sequences that raise computation, while the appropriate tokens and layerwise ratios to prune under a fixed budget remain difficult to determine. FitPrune derives a training-free pruning recipe by fitting attention distributions before and after pruning. It reports a 54.9% TFLOPs reduction for LLaVA-NEXT with 0.5% performance degradation.

  • Problem

    Large, often redundant visual-token sequences raise MLLM computation, while determining which tokens to prune and how much under a predefined budget remains challenging and costly to validate.

    Source: Introduction · p. 1

  • Method

    FitPrune uses a small batch of attention statistics to find pruning ratios that minimize divergence between pre- and post-pruning cross- and intra-attention distributions.

    Source: Introduction · p. 2, Method · p. 3

  • Results

    FitPrune reduces 54.9% TFLOPs for LLaVA-NEXT with only 0.5% performance degradation while retaining high performance across VL benchmarks.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    FitPrune offers a practical, efficient approach that can be integrated into existing models for vision-language tasks.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Recent progress in Multimodal Large Language Models(MLLMs) often use large image tokens to compensate the visual shortcoming of MLLMs, which not only exhibits obvious redundancy but also greatly exacerbates the already high computation. Token pruning is an effective solution for speeding up MLLMs, but when and how to drop tokens still remains a challenge. In this paper, we propose a novel and training-free approach for the effective visual token pruning of MLLMs, termed FitPrune, which can quickly produce a complete pruning recipe for MLLMs according to a pre-defined budget. Specifically, FitPrune considers token pruning as a statistical problem of MLLM and its objective is to find out an optimal pruning scheme that can minimize the divergence of the attention distributions before and after pruning. In practice, FitPrune can be quickly accomplished based on the attention statistics from a small batch of inference data, avoiding the expensive trials of MLLMs. According to the pruning recipe, an MLLM can directly remove the redundant visual tokens of different examples during inference. To validate FitPrune, we apply it to a set of recent MLLMs, including LLaVA-1.5, LLaVA-HR and LLaVA-NEXT, and conduct extensive experiments on a set of benchmarks. The experimental results show that our FitPrune can not only reduce the computational complexity to a large extent, while retaining high performance, e.g., -54.9% FLOPs for LLaVA-NEXT with only 0.5% accuracy drop. Notably, the pruning recipe can be obtained in about 5 minutes. Our code is available at https://github.com/ywh187/FitPrune.

Introduction

MLLMs use large, often redundant visual-token sequences that increase computation, but choosing what to prune under a fixed budget is difficult and expensive to validate. FitPrune derives a pruning recipe by fitting attention distributions before and after pruning, and reports substantial compute reduction with limited performance loss.

Introduction

  • Large visual-token sequences exacerbate MLLMs’ high computation, while increasing image resolution to address visual shortcomings further raises that cost.LLaVA uses 576 image patches, requiring 6.2 times the computation of text-only inference on ScienceQA.Source: Introduction · p. 1
  • Because image tokens precede text in unidirectional attention, many are inactive; at layer 12 of 32, image-to-text attention is focused on few tokens.Experiments report that removing visual tokens barely declines accuracy on most benchmarks.Source: Introduction · pp. 1–2
  • Under a predefined computation budget, selecting tokens and layerwise pruning ratios remains difficult: manual trials are expensive, and a single attention view can miss tokens important to the other modeling path.The paper frames the question as finding an optimal pruning recipe directly for MLLMs.Source: Introduction · p. 2
  • FitPrune is training-free and treats pruning as statistical fitting, minimizing divergence between pre- and post-pruning attention distributions using both cross- and intra-attention statistics.A small data batch supports the statistics used to search for token slots and set each layer’s pruning ratio for a predefined budget.Source: Introduction · p. 2
  • FitPrune reduces 54.9% TFLOPs for LLaVA-NEXT with only 0.5% performance degradation while retaining high performance across VL benchmarks.The pruning recipe can be obtained in about 5 minutes.Source: Introduction · p. 2

Related Work

Related work describes MLLMs as language models that handle language modeling and multimodal interaction directly, while higher-resolution visual inputs intensify their computational demands. Prior token-pruning approaches use varied importance criteria, but FitPrune targets the costly manual selection of pruning ratios with a training-free recipe.

Vision-Language Models

  • MLLMs use language models for language modeling and multimodal interaction, connected to visual encoders through cross-modal projectors.Projectors range from deep query-based branches to simple projection networks.Source: Vision-Language Models · p. 2
  • Increasing image resolution can improve fine-grained vision-language tasks such as TextVQA, but exacerbates MLLMs’ already high computation and leaves redundant visual tokens to prune.The passage presents effective, adaptive pruning of less important tokens as a need for MLLM applications.Source: Vision-Language Models · p. 3

Token Pruning

  • Prior pruning methods use importance metrics and often require post-pruning tuning; training-based approaches can be costly, while training-free methods include sampling, merging, and PageRank-based selection.DiffRate relies on prediction loss and a large optimization dataset; FitPrune instead uses a small batch for attention-distribution fitting.Source: Token Pruning · p. 3
  • Existing visual-language pruning methods still require manual choices of model and layer pruning ratios, whereas FitPrune aims to derive a recipe quickly from a predefined target.The illustration depicts per-layer token reduction, recipe generation by binary search over attention statistics, and pruning during inference.Source: Token Pruning · p. 3, Figure 3 · p. 3

Method

FitPrune derives a visual-token pruning strategy by minimizing the divergence between attention distributions before and after pruning under a computation budget. It estimates attention statistics from a batch, searches for removable tokens, and applies the resulting layer-wise strategy during inference.

Method

  • FitPrune seeks a pruning strategy that minimizes divergence between original and pruned attention distributions under a computation budget.The paper primarily measures computation overhead in FLOPs.Source: Method · p. 4
  • The method jointly considers visual-token self-attention and cross-attention, using attention statistics averaged over a batch of examples.The layer attention matrix covers N visual tokens and M text tokens.Source: Method · p. 4
  • The final pruning strategy specifies token-removal counts by layer, with cumulative removals increasing across layers.The per-layer count is determined from tokens selected by both self- and cross-attention criteria.Source: Method · pp. 4–5
  • FitPrune uses binary search over a divergence bound, greedily selecting low-attention tokens whose removal keeps divergence near the candidate threshold.It adjusts the search according to whether the resulting strategy meets the target computation budget.Source: Method · pp. 4–5
  • During inference, tokens are ranked by combined self- and cross-attention importance, and the lowest-ranked tokens are pruned according to the strategy.FitPrune requires no additional modules or gradient computations.Source: Method · p. 5

Experiments

FitPrune is evaluated across ten vision-language benchmarks and several LLaVA models, with experiments examining compute reduction, comparisons with other pruning methods, and the effects of design choices. The results report reduced computation with limited performance loss, including under more aggressive pruning.

Datasets and Metrics

  • Experiments cover ten benchmarks and LLaVA-1.5, LLaVA-HR, and LLaVA-NEXT; the pruning strategy is generated from 655 samples, or 0.1% of LLaVA-655k.LLaVA-1.5 has 576 visual tokens, LLaVA-HR has 1024, and LLaVA-NEXT dynamically uses up to 2880; the reported setup uses one A100 40G GPU.Source: Datasets and Metrics · p. 5

Comparison with SOTA pruning methods

  • At a 60% pruning ratio on VQAv2, FitPrune gains 2.5% over ToMe and 4.5% over FastV; for a 60% FLOPs reduction, its performance drop is 0.9% versus FastV’s 4.5%.At 20% pruning on GQA, FitPrune has no performance degradation, compared with drops of 0.2% for ToMe and 0.4% for FastV.Source: Figure 5 · p. 6, Experimental Results · p. 7

Effects of FitPrune on different MLLMs

  • 60% visual-token pruning reduces overall computation by 52.4% with only a 0.1% average performance drop on MLLM benchmarks.For LLaVA-NEXT, a 60% FLOPs pruning target reduces overall computation by 55%, with a 0.3% drop on VQAv2; a 40% target reduces computation by 33.2% on average without performance loss.Source: Experimental Results · p. 6

Ablation study

  • At an 80% pruning ratio, average divergence in self- and cross-attention distributions reaches approximately 27%, alongside a 3.6% performance drop.Self-attention is more unstable than cross-attention, and increasing pruning shifts FitPrune toward pruning tokens in shallower layers.Source: Experimental Results · p. 7
  • With only 0.01% of the dataset (65 samples), FitPrune maintains stable performance, achieving 61.6% accuracy, comparable to using 1% of the dataset (6.6K samples).Source: Experimental Results · p. 7

Visualization results

  • Visualizations show FitPrune retaining question-relevant tokens, including poster text and cat features, while text-related tokens remain intact in a text-focused example.As more tokens are deleted, tokens from the primary object are gradually pruned.Source: Figure 7 · p. 7, Experimental Results · p. 7

Conclusion

FitPrune frames visual token pruning as a statistical problem, seeking to minimize attention-distribution divergence using a small data batch. The paper reports that it reduces computational overhead while preserving performance across evaluated MLLMs.

Conclusion

  • FitPrune is a training-free visual-token pruning approach that frames pruning as a statistical problem of minimizing attention-distribution divergence.It generates a pruning strategy from a small batch of data, avoiding costly manual trials.Source: Conclusion · p. 7
  • The paper reports that FitPrune reduces computational overhead while preserving performance across evaluated MLLMs.The approach is described as integrable into existing models for vision-language tasks.Source: Conclusion · p. 7
  • FitPrune’s pruning strategy is generated from a small batch of data rather than costly manual trials.The conclusion presents this as a practical and efficient approach to optimizing MLLMs.Source: Conclusion · p. 7

Appendix

The appendix materials cover the relationship between pruning ratio and divergence, ablations of pruning ratios and attention distributions, and generalization across models. The supplied captions establish the scope of these visuals but do not report their cell values or comparative outcomes.

Appendix

  • Figure 8 relates pruning ratio to the average divergence α used in the statistical analysis.Source: Figure 8 · p. 10
  • Table 4 reports ablation results on GQA under different pruning ratios.Source: Table 4 · p. 10
  • Table 5 reports attention-distribution ablations at a 60% pruning ratio on TextVQA, GQA, and MME.Source: Table 5 · p. 10
  • Table 6 reports FitPrune’s generalization results across MLLMs of different scales and families, including results at a 60% pruning ratio.Source: Table 6 · p. 10

Impact of pruning ratio on divergence and performance

The average divergence α rises as the pruning ratio increases, while performance declines correspondingly. The passage presents this pattern as a balance between computational savings and accuracy.

Impact of pruning ratio on divergence and performance

Detailed ablation results of pruning ratio

FitPrune maintains competitive GQA performance across pruning ratios, with larger reductions trading greater throughput and compute gains for larger accuracy drops.

Analysis of attention map changes

The attention-map analysis contrasts FitPrune’s retained regions with random token removal, alongside examples of robust performance under substantial pruning.

Ablation of attention distribution

The attention ablation finds cross-attention important but insufficient alone, while visual self-attention complements it and improves pruning performance across benchmarks.

Ablation of FitPrune with Flash Attention

FitPrune accelerates inference alone and with Flash Attention, generalizes across model families, and retains strong performance in high-detail dialogue examples.

Ablation of FitPrune with Flash Attention

Generalization across different models

Examples of multiturn dialogues

Related papers

Loading 2409.10197v2…