PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs

Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, Shanghang Zhang

arXiv:2412.01818v2cs.CVcs.AI

TL;DR

VLMs contain many more visual than textual tokens, increasing inference cost, while text-visual attention is not an ideal basis for pruning. VisPruner selects important tokens using visual-encoder attention and removes similar tokens from the remainder. Across benchmarks it outperforms text-visual-attention methods; on LLaVA-1.5-7B it prunes 94.4% of visual tokens and maintains 91.5% of original performance across 10 tasks.

  • Problem

    Visual tokens make up most of VLM input sequences and increase inference cost, while text-visual attention can be an unsuitable indicator for pruning.

    Source: §1 · pp. 1–2

  • Method

    VisPruner uses visual-encoder attention to select important tokens, then removes similar remaining tokens to retain diverse visual information.

    Source: §1 · p. 2

  • Results

    VisPruner consistently outperforms text-visual-attention methods across benchmarks; on LLaVA-1.5-7B it prunes 94.4% of visual tokens and maintains 91.5% of original performance across 10 tasks without additional training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method sustains strong performance across different VLM architectures and reduction ratios, including longer visual sequences in high-resolution images and video inputs.

    Source: §1 · p. 2

    Improving VLM performance on multimodal understanding tasks remains future work.

    Source: §8.2 · p. 14

Abstract

from arXiv · show

Large vision-language models (LVLMs) generally contain significantly more visual tokens than their textual counterparts, resulting in a considerable computational burden. Recent efforts have been made to tackle this issue by pruning visual tokens early within the language model. Most existing works use attention scores between text and visual tokens to assess the importance of visual tokens. However, in this study, we first analyze the text-visual attention in the language model and find that this score is not an ideal indicator for token pruning. Based on the analysis, We propose VisPruner, a plug-and-play method that utilizes visual cues for more effective token pruning in LVLMs. Specifically, we first use visual attention to select a limited number of significant tokens. Then, we remove duplicate tokens from the remaining ones based on their similarity. By retaining diverse tokens alongside the initially selected important tokens, we maximally preserve the visual information of the input image. Experimental results demonstrate that our VisPruner sustains strong performance across various VLM architectures and reduction ratios, significantly outperforming existing methods based on text-visual attention. Notably, without any training, VisPruner can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%, while maintaining comparable performance. Our code is available at https://github.com/Theia-4869/VisPruner.

1. Introduction

Visual tokens substantially increase VLM inference cost, while text-visual attention can favor image positions that are less informative and miss crucial content. VisPruner instead uses visual cues to retain important and diverse tokens, and experiments report strong results across benchmarks and VLM settings.

1. Introduction

  • LLaVA-1.5 converts a 336 × 336 image into 576 visual tokens, while textual inputs typically contain fewer than 100, increasing inference cost.The visual sequences are longer still in LLaVA-NeXT and Video-LLaVA.Source: §1 · p. 1
  • Text-visual attention exhibits positional shift and diffuse, high-entropy distributions, making it a poor pruning indicator and degrading performance especially at higher reduction ratios.The shift favors lower image patches while potentially discarding crucial central content.Source: §1 · p. 2
  • VisPruner selects important tokens using visual-encoder attention, then removes similar tokens from the remainder to retain diverse visual information.Pruning before the language model also avoids requiring full language-model attention and is compatible with attention optimizations such as FlashAttention.Source: §1 · p. 2
  • VisPruner is a plug-and-play approach applied to the LLaVA series and Qwen-VL, and the paper reports consistent gains over text-visual-attention methods at different reduction ratios.The experiments cover diverse vision-language benchmarks.Source: §1 · p. 2
  • VisPruner prunes 94.4% of LLaVA-1.5-7B visual tokens, reduces inference FLOPs by more than 95%, and maintains 91.5% of original performance across 10 tasks without additional training.It also continues to achieve superior performance on longer visual sequences in LLaVA-NeXT and Video-LLaVA.Source: §1 · p. 2

2. Related Work

Related work frames visual-token reduction as a way to optimize VLM inference, while distinguishing training-based compression from training-free pruning. The paper focuses on the latter and challenges the use of language-model text-visual attention as an importance signal.

2. Related Work

  • VLMs use visual encoders and alignment modules to feed visual representations to language models, but long visual sequences increase inference burden.Increasing image resolution can further raise that burden.Source: §2 · p. 2
  • Visual-token reduction is motivated by image information's greater redundancy than text, but several compression methods require additional training and costs.The paper instead focuses on training-free inference optimization by removing less important tokens.Source: §2 · pp. 2–3
  • Prior training-free methods prune tokens using language-model attention, whereas VisPruner exploits visual cues because text-visual attention may be an unsuitable pruning signal.The paper also notes concurrent work with similar conclusions [53] [59] [66].Source: §2 · p. 3

3. Text-Visual Attention Investigation

The investigation finds that text-visual attention is a flawed basis for pruning: positional bias can favor later patches even when central patches preserve more performance, and dispersed scores make important tokens difficult to identify. Across these analyses, pruning based on [CLS] attention consistently performs best, motivating attention to visual cues instead.

3.1. Preliminaries

  • The language model uses causal self-attention with relative rotary position embeddings; “last attention” refers to attention received by visual tokens from the final text token, averaged across heads for analysis.The analysis samples 1,000 LLaVA-mix665k image-text pairs using LLaVA-1.5-7B.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. Text-visual attention shift

  • In shallow layers, central-region tokens preserve higher performance than later-positioned tokens that receive more text attention, and the difference diminishes beyond layer 10.Positional bias appears from the first layer and is more pronounced in shallower layers.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Text-visual attention dispersion

  • [CLS] attention concentrates about 80% of attention on the top 20% of visual tokens, whereas last attention is more dispersed and has a high-entropy, low-peak density.The dispersion remains when position-embedding decay is removed; the authors interpret more concentrated attention as indicating greater certainty about important tokens.Source: §3.3 · p. 4, Figure 3 · p. 4
  • [CLS] attention consistently achieves the best pruning results, while last attention can lag behind random pruning on some benchmarks even after positional-decay removal.The authors attribute the remaining gap partly to attention dispersion, which makes important visual tokens difficult to identify from attention scores.Source: §3.3 · pp. 4–5

4. Exploiting Visual Cues for Token Pruning

VisPruner combines visual-encoder attention to select important tokens with similarity-based selection of diverse tokens, aiming to preserve image information while reducing visual tokens.

4.1. Attention-based important tokens

  • Visual-encoder attention selects important tokens using a dynamic threshold based on the computation budget.For CLIP-like encoders, the method uses [CLS]-to-patch attention; for encoders without [CLS], it averages the attention each patch receives.Source: §4.1 · p. 5

4.2. Similarity-based diverse tokens

  • Similarity-based pruning removes the most similar remaining tokens until the diverse-token count reaches R · (1 − r), complementing important tokens with broader image coverage.The method uses cosine similarity and retains diverse tokens to help prevent degradation on background-related questions.Source: §4.2 · p. 5

4.3. Inference with visual pruning

  • VisPruner concatenates important and diverse tokens, restores their original image-patch order, then projects them and feeds them with the instruction to the language model.The complementary design maintains considerable performance at high reduction ratios while enhancing inference efficiency.Source: §4.3 · p. 5

5. Experiments

Experiments evaluate VisPruner across image, high-resolution, and video benchmarks, multiple VLM architectures, and token-reduction settings. Results cover benchmark performance, component ablations, and inference efficiency.

5.1. Experimental setup

  • Experiments span 10 image-based and four video question-answering benchmarks, with tests on LLaVA, Qwen-VL, InternVL, and CogVLM architectures.The comparison methods include ToMe, FastV, SparseVLM, LLaVA-PruMerge, and VisionZip.Source: §5 · p. 5, §5.1 · p. 6

5.2. Main results

  • At about 5% of tokens retained, VisPruner maintains 91.5% of original performance and outperforms SparseVLM by as much as 10.2% when retaining 64 tokens.At that 64-token setting, performance decreases by 3.4%; VisPruner also beats VisionZip, which merges redundant tokens.Source: §5.2 · p. 6, Table 1 · p. 7

5.3. VisPruner with higher resolution

  • At a 94.4% reduction ratio, VisPruner preserves 86.7% of original performance on LLaVA-NeXT, outperforming FastV (74.7%), SparseVLM (74.5%), and VisionZip (82.3%).LLaVA-NeXT handles up to 2,880 visual tokens.Source: §5.3 · p. 6, Table 2 · p. 7

5.4. VisPruner with video understanding

  • On Video-LLaVA, VisPruner maintains 95.1% of original performance with an 88.9% reduction and 90.3% with a 94.4% reduction.The evaluation uses the first 1,000 samples from each of three video QA benchmarks; the authors attribute strong reduction performance to video-frame redundancy and [CLS] attention identifying key tokens.Source: Table 3 · p. 7, §5.4 · p. 7

5.5. VisPruner with other VLM architecture

  • Across Qwen-VL, InternVL, and CogVLM, VisPruner consistently outperforms the text-visual attention-based FastV across reduction ratios.For Qwen-VL, visual attention combines visual-encoder patch attention with each Q-Former query’s attention to patches.Source: §5.5 · pp. 7–8, Table 4 · p. 8

5.6. Ablation study

  • Adding diverse tokens to important-token selection yields further gains of 2.3% on TextVQA and 3.6% on POPE.Important-token selection alone improves performance by 5% and 1.5%, respectively; it can omit background information because visual-encoder attention tends to focus on the foreground.Source: Figure 5 · p. 8, §5.6 · p. 8

5.7. Efficiency analysis

  • With 160 retained tokens on LLaVA-NeXT, VisPruner achieves a ×4 reduction in inference latency and uses less memory while running faster than FastV at the same reduction ratio.Pruning before the language model also allows compatibility with FlashAttention, unlike methods requiring text-visual attention within the language model.Source: Table 5 · p. 8, §5.7 · p. 8

6. Discussion

VisPruner selects important tokens using attention, then supplements them with diverse tokens by removing redundancy. This can be viewed as dynamic pruning across regions with different information densities.

6. Discussion

  • Foreground regions contain denser semantic information than backgrounds, while some prior methods lose background information or introduce noise through token fusion.The authors identify dynamically partitioning and finely pruning regions by information density as a future direction.Source: §6 · p. 8
  • VisPruner selects important tokens with attention, then removes redundancy by similarity to retain diverse tokens as a supplement.The paper frames this as dynamic pruning: retaining more tokens in information-dense foreground regions and fewer in sparser backgrounds.Source: §6 · p. 8

7. Details of experimental setup

The experiments cover image and video benchmarks and evaluate VisPruner across multiple VLM architectures. The setup includes varied image tasks, video QA settings, and models with different visual-input designs.

7.1. Datasets

  • The setup describes 10 image benchmarks and 3 video benchmarks, using the benchmarks’ default settings and evaluation metrics.The image suite includes visual question answering and multimodal reasoning tasks; the video suite is used to test settings with higher visual redundancy.Source: §7.1 · p. 12, §7.1.1 · p. 12, §7.1.2 · p. 13

7.1.1. Image benchmarks

  • Image benchmarks span scene-graph reasoning, real-world images from blind users, science questions, OCR, object hallucination, multimodal capabilities, and integrated vision-language skills.The supplied descriptions include GQA, VizWiz, ScienceQA, TextVQA, POPE, MME, MMBench, and MM-Vet.Source: §7.1.1 · pp. 12–13

7.1.2. Video benchmarks

  • The video benchmark setup uses gpt-3.5-turbo scoring and the first 1K samples per benchmark because of commercial API usage limits.TGIF-QA uses its Frame QA task; MSVD-QA and MSRVTT-QA are also described.Source: §7.1.2 · p. 13

7.2. Model architectures

  • The evaluated architectures include LLaVA-1.5, LLaVA-NeXT, Video-LLaVA, and Qwen-VL, which differ in resolution handling, video support, and visual-text alignment modules.LLaVA-NeXT uses dynamic high-resolution sub-images, Video-LLaVA handles video frames, and Qwen-VL uses a cross-attention adapter.Source: §7.2 · p. 13

8. Detailed Analysis of Attention in VLMs

Visual-encoder attention is concentrated differently from language-model visual-text attention, which shifts toward later token positions and can be dispersed. The analysis motivates using visual attention for pruning and identifies modality misalignment as an open direction.

8.1. Attention distribution

  • The [CLS] attention in CLIP’s penultimate layer concentrates more than other visual tokens, so the method uses it for pruning; final-layer attention from other tokens is uniform.The penultimate layer retains more local features and image details; Figure 6 plots attention distributions and marks image-row length.Source: §8.1 · p. 13, Figure 6 · p. 14
  • Across LLaMA’s 32 layers, visual-text attention increases toward higher token indices, and pruning on it degrades performance especially at high reduction ratios.Text-token attention is weaker than visual-part attention, particularly after layer 2; the first two layers also differ from the remaining layers.Source: §8.1 · p. 14

8.2. Attention intensity

  • The visual encoder’s [CLS] attention concentrates on foreground objects and global cues, while the language model’s last-token attention is more dispersed and includes more noise.The authors identify visual-language misalignment as a direction for future work to improve multimodal understanding.Source: §8.2 · p. 14

9. Efficiency Analysis with FlashAttention

VisPruner prunes visual tokens before the language model, enabling FlashAttention compatibility and higher efficiency than FastV. The efficiency analyses compare the methods on three model architectures under a stated hardware and benchmark setting.

9. Efficiency Analysis with FlashAttention

  • VisPruner prunes before the LLM, enabling FlashAttention compatibility and significantly higher efficiency than FastV, which prunes within the LLM.Tables 6–8 give efficiency comparisons for LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-NeXT-7B.Source: §9 · p. 14, Table 6 · p. 17, Table 7 · p. 18
  • Efficiency analyses use a single NVIDIA A100-80GB GPU and the POPE benchmark.The passage notes that original SDPA also includes FlashAttention and has efficiency comparable to FlashAttention2, with slight differences.Source: §9 · p. 14
  • The reported comparison covers FastV and VisPruner under LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-NeXT-7B.The corresponding table captions identify the three model-specific efficiency comparisons.Source: §9 · p. 14, Table 6 · p. 17, Table 7 · p. 18
Loading 2412.01818v2…