PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Shanghang Zhang

arXiv:2410.04417v4cs.CV

TL;DR

VLMs face substantial computational overhead from visual tokens even though visual information is sparser than text. SparseVLM provides a text-guided, training-free mechanism that selects relevant text raters, adaptively prunes visual tokens, and recycles pruned tokens. Across image and video understanding tasks, it improves efficiency and consistently outperforms prior methods while preserving most accuracy.

  • Problem

    Visual tokens impose substantial computational overhead in VLMs despite containing sparser information than text, motivating efficient redundancy reduction.

  • Method

    SparseVLM uses decoder self-attention to select relevant text tokens as visual-token raters, adaptively prune redundant visual tokens using attention rank, and recycle pruned tokens without extra parameters or fine-tuning.

  • Results

    SparseVLM consistently outperforms prior state-of-the-art methods across image and video understanding benchmarks while reducing VLM computational overhead.

  • Takeaways & Limitations

    SparseVLM provides a plug-and-play training-free approach for improving VLM efficiency across diverse image and video understanding tasks.

Abstract

from arXiv · show

In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM that eliminates the need of extra parameters or fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that SparseVLM increases the efficiency of various VLMs in a number of image and video understanding tasks. For example, LLaVA when equipped with SparseVLM achieves 54% reduction in FLOPs, 37% decrease in CUDA latency while maintaining 97% of its original accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs.

1. Introduction

VLMs gain multimodal capabilities from visual tokens but incur substantial computational overhead because images contain sparser information than text. SparseVLM addresses this with text-guided, training-free visual-token sparsification and reports strong efficiency and accuracy results across image and video tasks.

  • Motivation: 2304 visual tokens from a 672×672 LLaVA image can occupy more than half the context length, creating substantial computational overhead.Visual information is typically sparser than natural language, making naive processing inefficient.
  • Approach: SparseVLM is a text-guided, training-free framework that uses decoder self-attention without extra sparsification parameters or fine-tuning.It is presented as a text-aware alternative to prior text-agnostic visual sparsification methods.
  • Approach: SparseVLM selects relevant text tokens as raters, scores visual-token significance, prunes redundant tokens, and recycles them into compact representations.The method also uses attention-matrix rank to adapt the sparsification ratio to information density.
  • Results: 37% lower CUDA latency is achieved by LLaVA with SparseVLM while retaining 97% of its original performance.The same result is also reported as a 4.5× compression rate with 97% of original performance.
  • Results: SparseVLM consistently outperforms prior state-of-the-art methods across various image and video understanding benchmarks.The introduction states this cross-benchmark pattern without restricting it to a single VLM or task.

2. Related Work

Related work motivates visual-token compression because VLMs process much longer visual sequences as resolution and multimodal capabilities increase. Existing approaches include modifying the vision tower or projection of visual representations.

  • Vision-Language Models: Higher-resolution images increase visual-sequence length, while VLMs increasingly process longer visual token sequences for multimodal comprehension and generation.LLaVA is described as encoding 336 × 336 images into 576 tokens, with higher maximum resolution also noted.
  • Visual Compression for VLMs: Visual tokens can outnumber language tokens by tens to hundreds of times and contain sparser information than human-produced text.These properties motivate compression of visual tokens in VLMs.
  • Visual Compression for VLMs: One line of prior work compresses visual representations by modifying the vision tower or using an efficient projection of the vision modality.The supplied related-work passage identifies this as the first of two categories.

3. Method

SparseVLM estimates visual-token significance from text-relevant self-attention, then adaptively sparsifies decoder layers and recycles selected pruned tokens into compact representations. Its pipeline combines rater selection, rank-based deletion, and token aggregation to reduce redundancy while preserving visual information.

  • Visual significance estimation: SparseVLM reuses decoder self-attention to estimate visual-token significance without extra training parameters.The method uses language-to-vision attention interactions as its sparsification reference.
  • Visual significance estimation: The priority matrix P compares language queries with vision keys, and its aggregated vector assigns higher significance to visual tokens with larger values.The interaction matrix has dimensions Lt × Lv, while aggregation costs Lt × Lv FLOPs once the attention matrix is available.
  • Relevant text token selection: Text tokens with strong image correlation are selected as raters, excluding visually irrelevant prompt tokens from visual-token scoring.The selected raters guide significance estimation for visual tokens.
  • Sparsification level adaptation: A rank-based strategy adapts sparsification per decoder layer by treating dimension minus rank as redundancy and removing the N lowest-scoring visual tokens.Layers with N = 0 skip sparsification, while rank computation costs Lt × Lv × min(Lt, Lv) FLOPs.
  • Visual token recycling: Pruned visual tokens are recycled by selecting high-priority deleted tokens, grouping them with density-peak k-nearest-neighbor aggregation, and reconstructing compact tokens.Tokens receive ρi × δi scores for cluster-center selection, then each group is reconstructed through element-wise summation.
  • Computational complexity: Theoretical savings account for reduced attention and FFN computation after recycling, subtracting the method’s additional overhead.For one Transformer layer, the reduction is 6(N − C)D^2 + 2(N − C)^2D.

4. Experiments

SparseVLM is evaluated across multiple vision-language architectures and image/video understanding benchmarks, using substantially fewer visual tokens. Results show strong accuracy retention and improved efficiency relative to baseline and competing methods.

  • Evaluation Setup: SparseVLM is evaluated on LLaVA, Mini-Gemini, and Qwen2-VL across image and video understanding benchmarks.The image evaluation covers eight benchmarks, while the video evaluation uses four video question answering benchmarks.
  • Image Understanding Tasks: 0.9% average accuracy decrease occurs when SparseLLaVA is pruned from 576 to 192 vision tokens.Under the same image-token configurations, SparseLLaVA also exceeds ToMe by 10.2%.
  • Image Understanding Tasks: 17.3% higher accuracy than FastV is achieved by SparseLLaVA when only 64 vision tokens are retained.The comparison evaluates aggressively reduced visual-token configurations, and SparseVLM also outperforms PDrop in accuracy and latency.
  • Image Understanding Tasks: 98.0% accuracy is maintained by Qwen2-VL after removing 54.5% of vision tokens.The reported accuracy decreases by approximately 0.8% for every 100 tokens pruned.
  • Video Understanding Tasks: 95.0% total average accuracy is achieved by SparseVideoLLaVA with 194 video tokens, compared with 80.3% for FastV.Both methods preserve 194 vision tokens, while the Video-LLaVA upper bound uses 2048 tokens and is normalized to 100.0%.

5. Analysis

The analysis examines text-rater selection, token recycling, computational efficiency, and layerwise visual-token pruning. These studies show that the proposed mechanisms improve retained information and reduce inference resources.

  • Text Rater Selection: 0.8% improvement over using all prompt tokens is obtained on TextVQA when selected text raters guide visual-token assessment.The ablation compares all text tokens, only text tokens, and only the selected text raters at 64 vision tokens.
  • Token Recycling: 1.2% and 7.2% average performance improvements are obtained on TextVQA and POPE, respectively, with token recycling.As pruning becomes more aggressive, recycling becomes more beneficial; on POPE, pruning from 192 to 64 tokens raises accuracy from 1.5% to 17.7%.
  • Computational Efficiency: 67% cache memory savings are reported relative to vanilla LLaVA while retaining 99.1% accuracy.The cache decreases from 302.4MB to 100.8MB.
  • Layerwise Visualization: More visual tokens are pruned at deeper layers while the retained regions of interest become progressively refined.Visualization shows less relevant image information being reduced while question-related tokens are retained.

6. Conclusion

SparseVLM is a text-aware, training-free token optimization method that reduces VLM test-time computation without extra parameters or fine-tuning. It combines attention-rank pruning with token recycling, retaining accuracy while improving deployment efficiency.

  • SparseVLM reduces test-time computations without extra parameters or fine-tuning costs.
  • Attention-matrix rank determines pruning ratios, while reconstructed token clusters form more compact visual representations.
  • 37.0% latency reduction and 77.8% compression preserve 97% of LLaVA’s original accuracy.
  • SparseVLM exceeded FastV accuracy by 14.7% in video understanding tasks.
  • The method supports deployment of off-the-shelf VLMs on edge devices and cloud platforms.

Impact Statement

The supplied material connects SparseVLM’s efficiency benefits to practical deployment while detailing its block-wise attention implementation and computational costs. Its design preserves efficient attention processing through incremental normalization and targeted visual-token selection.

  • SparseVLM provides practical advantages for deploying off-the-shelf large vision-language models on edge devices and cloud platforms.
  • The FlashAttention-compatible procedure computes attention scores block-wise and combines outputs using incremental normalization.
  • Selected text raters are represented in a special V matrix to obtain their mean attention scores directly through FlashAttention.
  • Top-k selection ranks visual-token outputs in descending order before retaining the corresponding indices.
  • The method’s efficiency comes with additional costs for visual-token significance estimation, rank adaptation, aggregation, and reconstruction.

D. Dataset

The evaluation uses established image, video, multimodal, hallucination, science, text-reading, and general vision-language benchmarks. The video evaluation covers four question-answering datasets spanning diverse spatio-temporal reasoning settings.

  • The experiments use GQA, MMBench, MME, POPE, ScienceQA, VQA-v2, TextVQA, and MMVet for image and multimodal evaluation.
  • The benchmarks assess capabilities including scene reasoning, perception, cognition, object hallucination, scientific knowledge, text understanding, and integrated vision-language skills.
  • The video benchmarks are TGIF-QA, MSVD-QA, MSRVTT-QA, and ActivityNet-QA.
  • TGIF-QA contains 165,000 question-answer pairs and tests repetition, action, state-transition, and frame-based video questions.
  • ActivityNet-QA contains 58,000 annotated question-answer pairs across 5,800 videos for motion, spatial, and temporal reasoning.

E. Implementation Details

SparseVLM is evaluated across three VLM frameworks using established inference configurations and a controlled software and hardware environment. CUDA-time analysis focuses on image encoding, KV-cache loading, and transformer forward computation.

  • Experiments run on a single NVIDIA A100-80G GPU with Python 3.10, PyTorch 2.1.2, CUDA 11.8, and Transformers 4.31.0.
  • Inference follows the original settings for LLaVA-1.5-7/13B, Mini-Gemini, Qwen-VL, and Video-LLaVA.
  • Reported CUDA time includes image encoding, KV-cache loading, and transformer forward time while excluding other computational components.

G. More Detailed Efficiency Analysis

SparseVLM’s efficiency is evaluated through latency–accuracy and compute–accuracy trade-offs on LLaVA across three benchmarks, with additional Video-LLaVA analysis described separately.

  • Latency–accuracy and FLOPs–accuracy trade-offs for SparseVLM on LLaVA are reported across POPE, TextVQA, and MME.These comparisons appear in Figure 8 alongside random sparse and baseline models.
  • Video-LLaVA efficiency is analyzed separately on TGIF and MSVD using latency–accuracy and token-budget–accuracy comparisons.The supplied passage identifies this analysis as Figure 10 rather than part of the LLaVA study.
  • Figure 8 compares random sparse, SparseVLM, and baseline models using latency versus accuracy and FLOPs versus accuracy.

H. More Sparsification Visualization

The paper provides visualization examples showing SparseVLM applied to diverse visual question-answering prompts and presents additional trade-off figures for MGM and Video-LLaVA.

  • Figure 11 presents visualization examples of SparseVLM processing diverse visual question-answering prompts.The examples illustrate responses to different types of queries in a visual context.
  • Figure 9 compares MGM using latency–accuracy and FLOPs–accuracy trade-offs among random sparse, SparseVLM, and baseline models.
  • Figure 10 compares Video-LLaVA using latency–accuracy and token-budget–accuracy trade-offs among random sparse, SparseVLM, and baseline models.
Loading 2410.04417v4…