PaperlayerView exact revision on arXiv

Source-linked AI summary

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs

Xu Li, Yuxuan Liang, Yi Zheng, Zhe Liu, Xiaolei Chen, Haotian Chen, Rui Zhu, Fan Shi, Xiangyang Xue

arXiv:2609.34558v1cs.CV

TL;DR

LVLMs incur substantial computational overhead from large visual-token sequences, while existing pruning approaches focus on individual importance or diversity rather than joint preservation. ACPruner combines importance estimates with vision-encoder attention coverage and greedily selects a compact subset; experiments report strong performance retention and efficiency gains across diverse LVLM families.

  • Problem

    Large visual-token sequences increase LVLM computational overhead, while existing pruning approaches focus on token importance or diversity rather than jointly preserving the original visual content.

    Source: §1 · pp. 1–2

  • Method

    ACPruner is a training-free framework that combines visual saliency and textual relevance with vision-encoder attention coverage, then greedily selects tokens to maximize importance-biased coverage.

    Source: §1 · p. 2

  • Results

    Across diverse LVLM families and task categories, ACPruner achieves strong performance retention with substantial end-to-end efficiency gains.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The evaluation spans image and video understanding across LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision, and Qwen2.5-VL backbones.

    Source: §1 · p. 2

    ACPruner relies on manually selected hyperparameters α and β, whose optimal values may vary across tasks and backbones.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.

1 Introduction

Visual token growth makes LVLM inference costly, while importance- and diversity-based pruning each leave a different preservation gap. ACPruner reframes pruning as importance-biased coverage and selects tokens to jointly preserve visual content; experiments span diverse LVLM backbones and tasks.

  • LVLMs process many visual tokens, especially with high-resolution images, multiple images, or long videos, increasing computational overhead and limiting resource- or latency-sensitive use.Source: §1 · p. 1
  • Importance-based ranking can over-retain similar content from salient regions, while diversity-based selection can preserve distinct but task-uninformative tokens.The introduction identifies these as complementary limitations of the two existing pruning paradigms.Source: §1 · p. 2
  • ACPruner formulates pruning as biased coverage maximization, prioritizing important content while selecting a compact subset that collectively preserves visual information.It estimates priority from visual saliency and textual relevance, derives token coverage from vision-encoder patch attention, then greedily maximizes marginal gain.Source: §1 · p. 2
  • ACPruner is training-free and pre-LLM, and its evaluation covers image and video tasks across LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision, and Qwen2.5-VL backbones.Source: §1 · p. 2

2 Method

The method replaces full visual-token processing with a compact subset intended to preserve model behavior. It combines token importance with directional, attention-derived coverage and greedily selects tokens for the resulting biased coverage objective.

2.1 Problem Definition

  • Visual token pruning seeks a k-token subset S of the full visual-token set V that preserves the original model behavior as much as possible.The LVLM pipeline maps vision-encoder features through a projector before the LLM processes them with textual tokens.Source: §2.1 · p. 3

2.2 Reformulating Visual Token Pruning as Biased Coverage Maximization

  • The biased coverage formulation scores a subset by how well it preserves original tokens, weighting coverage according to each token's content importance.It addresses the limitation of importance and diversity objectives that evaluate the selected subset without explicitly modeling preservation of the original token set.Source: §2.2 · p. 3

2.3 Content Importance: Which Tokens Deserve Stronger Preservation?

  • Token importance combines intra-modal visual saliency and inter-modal relevance to the input text, with a fallback to saliency alone when noun extraction returns no nouns.Saliency uses CLS-to-patch attention, or average incoming patch attention when the vision encoder lacks a CLS token; textual relevance uses visual-token similarity to extracted noun embeddings.Source: §2.3 · pp. 3–4

2.4 Attention-Induced Coverage: How Do Visual Tokens Preserve Each Other?

  • Coverage derives from patch-to-patch attention in the vision encoder, and is generally directional rather than a symmetric feature-similarity relation.The method interprets encoder attention as how one token preserves another's information.Source: §2.4 · p. 4

2.5 Optimizing the Biased Attention Coverage Objective

  • Because directly solving the objective is NP-hard, ACPruner greedily selects the candidate with the largest marginal importance-biased coverage gain until the token budget is reached.The objective is also reported to be monotone submodular and equivalent to minimizing importance-weighted representative distortion, yielding a standard greedy approximation guarantee.Source: §2.5 · pp. 4–5

3 Experiments

Across image and video benchmarks, ACPruner retains strong performance across pruning ratios and model scales, while reducing inference costs. Component ablations support combining importance signals with attention-based coverage. [p08, p11, p14, p17, p20, p22, p24]

3.1 Experimental Setup

  • The evaluation covers image and video understanding benchmarks across LVLM families and scales, with experiments conducted on eight NVIDIA L20 GPUs.Image benchmarks include GQA, VizWiz, VQA-v2, SQA, TextVQA, POPE, MME, MMB, and MM-Vet; video benchmarks include MVBench, LongVideoBench, NextQA, and VideoMME.Source: §3.1 · pp. 5–6

3.2 Main Results

  • 100.2% at 66.7%, 99.8% at 77.8%, and 98.2% at 88.9% pruning are ACPruner's reported LLaVA-1.5-7B performance retention rates.The first two are best overall; the last is the highest average retention reported at that ratio.Source: Table 1 · p. 5, §3.2 · p. 6
  • 99.4% at 77.8% and 97.8% at 88.9% pruning are ACPruner's reported LLaVA-NeXT-7B average performance retention rates.ACPruner is reported as best-performing at both ratios.Source: §3.2 · p. 6, Table 2 · p. 6
  • 99.0% at 75% and 98.3% at 85% pruning are ACPruner's reported video-understanding performance retention rates.The evaluation uses LLaVA-OneVision-7B across four video benchmarks.Source: §3.2 · p. 7, Table 3 · p. 7
  • On LLaVA-1.5-13B and LLaVA-NeXT-13B, ACPruner remains best-performing across pruning ratios, extending results across model scales.The paper reports these additional results in Appendix E.Source: §3.2 · p. 7

3.3 Practical Efficiency Analysis

  • ACPruner yields substantial efficiency gains; its modest pruning overhead is outweighed by reduced downstream LLM inference time.High-resolution models benefit more from pruning, and larger models achieve greater practical speedups; the table reports benchmark-level statistics and sample-level runtime breakdowns.Source: §3.3 · p. 8, Table 5 · p. 8

3.4 Ablation Study

  • Removing either importance signal, the importance bias, or attention-based coverage degrades performance; feature-similarity coverage also performs worse.The ablations indicate complementary importance cues and support attention-based coverage over pure importance TopK or feature similarity.Source: §3.4 · pp. 8–9, Table 6 · p. 8
  • Best α differs by backbone—about 0.6 for LLaVA-1.5-7B and 0.2 for Qwen2.5-VL-7B—while both perform strongest around β = 1.0.The paper links the α difference to the distinct intra-modality importance signals used by the backbones.Source: Figure 2 · p. 8, §3.4 · p. 9

4 Related Work: Visual Token Pruning in LVLMs

Prior LVLM pruning methods use textual relevance, visual saliency, or embedding-space diversity, but each approach has limitations. ACPruner combines pre-LLM pruning with redundancy and importance awareness, including text-aware importance estimation. [p27, p28]

4 Related Work: Visual Token Pruning in LVLMs

  • Textual-relevance methods prune after early LLM layers and may conflict with FlashAttention; early crossmodal attention can also exhibit positional bias [50] [8] [37].Vision-encoder saliency methods prune before the LLM but are usually text-agnostic and may preserve redundant salient-region tokens [3] [49] [31] [45].Source: §4 · p. 9
  • Feature-similarity methods encourage diversity but may spend token budgets on uninformative regions and are typically text-agnostic [2] [37].The contrast is with methods that select tokens using embedding-space similarity.Source: §4 · p. 9
  • ACPruner combines pre-LLM pruning, redundancy awareness, and importance-aware coverage; its textual relevance signal supports query-adaptive pruning.Tokens already well preserved by the selected subset receive smaller marginal gains during greedy selection.Source: §4 · p. 9

5 Conclusion and Limitations

ACPruner is a training-free, pre-LLM pruning framework that jointly models content importance and attention-induced coverage, with strong performance retention and substantial efficiency gains. The paper also identifies hyperparameter and vision-encoder constraints and outlines supplementary analyses and experiments.

5 Conclusion and Limitations

  • ACPruner jointly models content importance and attention-induced coverage in a training-free, pre-LLM framework, achieving strong performance retention with substantial efficiency gains.The conclusion describes experiments across diverse model families.Source: §5 · p. 9
  • ACPruner uses manually selected α and β that may need retuning for new scenarios, and its reliance on ViT-based vision encoders may limit applicability to other architectures.The authors identify extending the formulation to architectures without an explicit vision encoder as future work.Source: §5 · p. 9

A Appendix Overview

  • The appendix covers theoretical analysis, NP-hardness, pseudocode, larger-scale and ablation experiments, runtime distribution, benchmarks, comparison methods, and qualitative visualizations.Its listed larger-scale experiments include LLaVA-1.5-13B and LLaVA-NeXT-13B.Source: A Appendix Overview · p. 13

B Theoretical Analysis of ACPruner

The appendix analyzes ACPruner’s objective as a weighted distortion problem and establishes structural properties that support greedy optimization. It also examines generalized coverage utilities and shows limits of importance-only and diversity-only selection.

B.1 Preliminaries and Notation

  • The objective is monotone non-decreasing, so using a token-budget constraint |S| ≤ k yields the same optimum as |S| = k.The appendix uses the ≤ constraint for convenience.Source: §B.1 · p. 13

B.2 ACPruner as Weighted Distortion Minimization

  • Maximizing ACPruner’s objective is equivalent to minimizing importance-weighted representation distortion over the original token set, with representatives induced by encoder attention.This interprets the selected subset as a compact representative of the full visual token set.Source: §B.2 · pp. 13–14

B.3 Normalization, Monotonicity, and Submodularity

  • The ACPruner set objective is normalized, monotone, and submodular.The appendix proves these properties for the objective.Source: §B.3 · pp. 14–15

B.4 Approximation Guarantee of Greedy Optimization

  • The greedy algorithm is a principled approximation algorithm for maximizing the proposed importance-weighted attention-induced coverage objective under a token budget.The appendix establishes a provable approximation guarantee for greedy optimization.Source: §B.4 · p. 16

B.5 Generalized Coverage Transforms

  • Non-decreasing transforms of token coverage preserve a normalized, monotone, submodular objective, although this result relies on ACPruner’s max-coverage structure.The appendix extends the theoretical framework beyond its linear coverage utility.Source: §B.5 · p. 17

B.6 Lack of Uniform Guarantee for Importance-Only Selection

  • No deterministic token-wise selection rule independent of coverage has a uniform positive constant-factor guarantee for the ACPruner objective.A token overlooked by the score ranking can still be indispensable as the unique representative of valued content.Source: §B.6 · p. 18

B.7 Lack of Uniform Guarantee for Diversity-Only Selection

  • No deterministic diversity-only rule independent of importance weights and coverage has a uniform positive constant-factor guarantee across ACPruner instances.Internal dispersion alone may fail to cover content that carries value under the objective.Source: §B.7 · p. 19

C NP-hardness of the Optimization Problem

The ACPruner optimization problem is NP-hard, established by a polynomial-time reduction from MAXIMUM COVERAGE. The appendix also summarizes the pruning pipeline and its greedy selection procedure.

C NP-hardness of the Optimization Problem

  • The ACPruner optimization problem is NP-hard, as shown by a polynomial-time reduction from MAXIMUM COVERAGE.The constructed objective counts how many element tokens are covered by selected selector tokens.Source: C NP-hardness of the Optimization… · pp. 20–21

D Pseudocode of ACPruner

  • The pseudocode takes visual and text tokens, vision-encoder attention maps, a budget, and coefficients, then outputs a selected token subset.Its pipeline computes fused importance weights, derives an attention-induced coverage matrix, and performs greedy selection.Source: D Pseudocode of ACPruner · pp. 21–22

else

  • The algorithm computes token importance, derives attention-induced coverage, and iteratively selects tokens by marginal gain until the budget is reached.The pseudocode initializes coverage state and updates it after each selected token.Source: else · p. 22

E Results on Larger Model Scales

On larger 13B models, ACPruner retains performance across aggressive pruning ratios. Additional ablations support its choices of CLS attention, noun extraction, and word-level max matching for token importance.

E Results on Larger Model Scales

F.1 Ablation of Intra-modality Importance Variants

  • 98.9% performance retention makes CLS attention the strongest intra-modality importance variant; feature magnitude and similarity with the global mean perform worse.The comparison is on LLaVA-1.5-7B at 77.8% pruning.Source: §F.1 · p. 24, Table 9 · p. 24

F.2 Ablation of Noun Extraction

  • Using nouns alone performs best among the tested query-text strategies, while adding linguistic categories progressively lowers performance and full text performs worst.The comparison tests nouns, nouns with adjectives, nouns with adjectives and verbs, and full text on LLaVA-1.5-7B at 77.8% pruning.Source: §F.2 · pp. 24–25, Table 10 · p. 25

F.3 Ablation of Inter-modality Importance Variants

  • Word-level max matching performs best for inter-modality importance; word-level matching beats token-level matching, and max pooling beats mean pooling in both settings.The comparison is on LLaVA-1.5-7B at 77.8% pruning.Source: §F.3 · p. 25, Table 11 · p. 25

F.4 Ablation of Encoder Attention Layer Choices

The best layer choice differs between the two attention roles: penultimate-layer CLS attention for importance and all-layer patch attention for coverage.

F.4 Ablation of Encoder Attention Layer Choices

  • Penultimate-layer CLS attention (layer 22) gives the best intra-modality importance performance; earlier, middle, or all-layer alternatives perform worse.This matches ACPruner’s default design.Source: §F.4 · p. 25, Table 12 · p. 26
  • Averaging patch-to-patch attention across all layers gives the best token-coverage performance; using any single early, middle, or late layer underperforms.This also matches ACPruner’s default design.Source: §F.4 · p. 25, Table 13 · p. 26

G Step-wise Runtime Breakdown of ACPruner

Greedy selection dominates ACPruner’s internal runtime across the evaluated backbones, while inter-modality importance takes the smallest share.

G Step-wise Runtime Breakdown of ACPruner

  • Greedy selection accounts for more than half of ACPruner’s pruning time on every evaluated backbone, with a larger share on LLaVA-NeXT models.Inter-modality importance contributes the smallest runtime share; noun extraction and intra-modality importance take moderate shares.Source: Figure 3 · p. 26, G Step-wise Runtime Breakdown of… · p. 27

H Details of Evaluation Benchmarks

The evaluation spans image and video benchmarks covering multimodal perception, reasoning, text, hallucination, and long- or short-context video understanding.

H.1 Image Benchmarks

  • MME assesses perception and cognition across 14 subtasks, while GQA tests compositional visual reasoning with scene-graph-generated questions and dimensions including grounding and consistency.Source: §H.1 · p. 27
  • ScienceQA evaluates multimodal science reasoning, POPE tests object hallucination, and TextVQA requires reading and reasoning about scene text.Source: §H.1 · p. 27
  • VizWiz uses real questions from blind people and often poor-quality or unanswerable images; VQA-v2 uses complementary image pairs to reduce language-prior effects.Source: §H.1 · p. 27
  • MMBench-EN supports holistic multimodal comparison, while MM-Vet evaluates combinations of six vision-language capabilities using open-ended outputs.Source: §H.1 · p. 28

H.2 Video Benchmarks

  • MVBench tests video understanding across 20 tasks and four ability dimensions; LongVideoBench targets long-context reasoning over videos averaging about 39 minutes.Source: §H.2 · p. 28
  • NextQA emphasizes causal and temporal video reasoning, whereas VideoMME covers video understanding across clips ranging from short to very long.Source: §H.2 · p. 28

I Details of Comparison Methods

The comparison methods prune visual tokens through attention or textual relevance ranking, diversity objectives, redundancy removal, and combinations of these strategies.

Attention-based selection

  • FastV ranks tokens by textual attention, while FasterVLM and HiRED [3] use vision-encoder attention to rank tokens or allocate regional budgets.HiRED further ranks tokens within each region using final-layer CLS attention.Source: I Details of Comparison Methods · pp. 28–29

Token merging

Text relevance and token recycling

Diversity and redundancy

Progressive and staged pruning

  • PDrop progressively drops tokens across LLM stages, whereas LearnPruner [34] combines post-encoder redundancy pruning with later query-aware pruning.PDrop uses a predefined drop ratio; LearnPruner adds diversity tokens before its text-to-vision-attention pruning stage.Source: I Details of Comparison Methods · p. 29

Saliency with complementary content

Complexity-adaptive pruning

  • AdaptPrune and AgilePruner [5] adjust similarity thresholds using image complexity, favoring concentrated high-attention tokens for simple images and more diverse tokens for complex images.Both use effective rank and attention entropy to analyze complexity.Source: I Details of Comparison Methods · p. 29

J Case Study

The case study illustrates how ACPruner’s retained patches vary with the query and token budget, while checklist responses report experimental details, reproducibility information, and evaluation conditions.

J Case Study

  • For the same image, different questions yield different retention patterns, and under very small budgets ACPruner concentrates tokens on query-critical evidence.Figure 4 shows pruning masks at budgets of 128, 64, 32, and 16 tokens.Source: J Case Study · p. 30, Figure 4 · p. 31
  • With a relatively sufficient budget, ACPruner retains query-relevant and visually salient content while covering other informative regions rather than clustering tokens around a few salient areas.Source: J Case Study · p. 30
  • The authors report that the visualizations support selecting a compact subset that is both important and collectively informative, rather than choosing only salient tokens or distributing tokens uniformly.Source: J Case Study · p. 30

NeurIPS Paper Checklist

  • The checklist says detailed pseudocode and hyperparameter settings are provided, and that relevant datasets, backbones, and hyperparameter choices are specified.The checklist also states that code will be released once the paper is accepted.Source: NeurIPS Paper Checklist · pp. 33–34
  • The authors report deterministic decoding with temperature 0 and say error bars or significance measures are not applicable because evaluation has no randomness.Source: NeurIPS Paper Checklist · p. 34
  • The checklist reports that compute resources are specified and that practical runtime analysis is provided.Source: NeurIPS Paper Checklist · p. 35

Related papers

Loading 2609.34558v1…