PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Token Pruning via Query-Vision Interaction Decomposition

Harshithanjani Athi, Sravan Kumar Ankireddy, Jianzhong Charlie Zhang, Hyeji Kim

arXiv:2610.33215v1cs.LGcs.CV

TL;DR

Large visual-token sequences make VLM inference costly, and existing pruning signals do not explicitly model the structure of query–vision interactions. QViD uses dominant low-rank interaction modes to select query-relevant visual tokens without training. Across image and video benchmarks and multiple VLMs, it improves performance under matched token budgets, with larger gains under aggressive compression.

  • Problem

    Visual-token counts make VLM inference expensive, while attention-pattern and token-dynamics heuristics do not explicitly exploit the structure of query–vision interactions.

    Source: §1 · pp. 1–2

  • Method

    QViD is a training-free pruning method that scores visual tokens using dominant low-rank modes of the query–vision interaction matrix derived from transformer query–key projections.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Across image and video benchmarks and multiple VLMs, QViD improves performance under matched token budgets; at 64 tokens on LLaVA-1.5-7B, it improves over V2Drop by 3.5% in normalized average.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The results show that a few dominant SVD components capture much of the task-relevant query–vision structure, providing a simple, interpretable, training-free basis for efficient VLM pruning.

    Source: §5 · p. 9

    Evaluation is limited to public VLMs and standard image/video benchmarks, may not capture closed-source models, proprietary inference stacks, or specialized domains, and practical speedups may vary with hardware and inference settings.

    Source: A Limitations and Future Directions · p. 12

Abstract

from arXiv · show

Large vision-language models (VLMs), which process both visual inputs and text queries, incur high inference costs due to the large number of visual tokens, making visual token pruning a natural approach for improving efficiency. Recent work has leveraged query information to guide token selection, often through heuristics derived from attention patterns or token dynamics. However, these approaches do not explicitly exploit the underlying structure of query-vision interactions. We address this gap by observing that the query-vision interaction matrix, naturally induced by the query and key projections already present in transformer attention, is effectively low-rank, with a small number of dominant latent interaction modes capturing query-relevant semantics. Motivated by this observation, we propose Query-Vision Decomposition (QViD), a novel training-free query-aware visual token pruning method that exploits this structure. Extensive experiments on both image and video understanding benchmarks show that QViD consistently improves performance under matched token budgets, with the largest gains appearing under aggressive compression regimes.

1 Introduction

Visual-token counts make VLM inference expensive, while existing attention-guided and variation-based pruning signals do not reliably capture query relevance. QViD instead scores tokens using dominant modes of an explicitly modeled query–vision interaction matrix.

Motivation

  • VLMs process visual tokens together with a text query, but images can produce hundreds to thousands of tokens, increasing inference latency and memory costs.The introduction notes applications including visual question answering, optical character recognition, diagram understanding, and video comprehension [14] [19] [9].Source: §1 · p. 1
  • Attention-guided methods can favor tokens near text tokens in the sequence, while variation-based pruning captures token activity only indirectly rather than explicitly measuring query–vision relevance.FastV and SparseVLM [21] use early-layer attention patterns; V2Drop [3] ranks tokens by hidden-state changes across consecutive layers.Source: §1 · pp. 1–2
  • Because the query determines which image regions matter, pruning should condition on query semantics rather than merely generic token activity or attention patterns.The paper illustrates that an animal may matter for an identity question but be less relevant to questions about the background or surrounding objects.Source: §1 · p. 2

Our Approach

  • QViD is training-free and uses dominant low-rank modes of query–vision interactions to score visual tokens, explicitly conditioning on the query without relying on attention weights.The interaction matrix is derived from standard transformer query–key projections.Source: §1 · p. 2

Main contributions

  • Across multiple VLMs and image and video benchmarks, QViD consistently outperforms recent pruning methods under matched token budgets, especially under aggressive compression.The introduction also reports that isolating dominant interaction modes is crucial for effective visual-token selection.Source: §1 · p. 2

2 Related Work

Related work reduces visual-token costs through pruning or merging, including attention-guided and task-agnostic approaches. QViD is positioned as training-free pruning that explicitly uses query–vision interactions.

Visual-Token Compression

  • Attention-guided pruning scores tokens from LLM attention patterns, while PDrop progressively reduces token count and ToMe and LLaVA-PruMerge merge similar tokens.Attention-based scoring can require attention matrices and may suffer positional bias in autoregressive LLMs.Source: §2 · p. 3

Task-Agnostic Versus Query-Aware Pruning

  • V2Drop [3] avoids explicit attention scores by pruning tokens with low hidden-state variation, but its signal captures generic token dynamics rather than explicitly modeling query–vision relevance.The approach is compatible with FlashAttention and avoids attention-position bias.Source: §2 · p. 3
  • QViD ranks visual tokens by their participation in the dominant low-rank structure of a query–vision interaction matrix, providing a training-free rule conditioned on the input query.The paper describes training-free visual-token pruning as complementary to improvements in the vision encoder, projector, or LLM backbone.Source: §2 · p. 3

3 Method

QViD prunes visual tokens under a target budget by modeling query–vision interactions and using their low-rank structure to score tokens. It retains the top-scoring tokens for the remaining language-model layers without training or architectural modification.

3.1 Problem Setup

  • QViD targets training-free, query-aware pruning by retaining visual tokens most relevant to the input query under a token budget.The method can be applied at any transformer layer, a design choice studied empirically.Source: §3.1 · p. 4

3.2 Query–Vision Interaction Matrix

  • The query–vision interaction matrix measures query–visual compatibility using frozen transformer query and key projections, before positional encoding and without softmax normalization.This removes positional effects from the signal and preserves raw interactions without requiring explicit attention maps, which may be unavailable or costly in memory-efficient implementations.Source: §3.2 · p. 4

3.3 Low-Rank Structure of Query–Vision Interactions and Dominant Interaction Modes

  • The interaction matrix has a rapidly decaying singular spectrum, and its dominant modes capture task-relevant query–vision structure.Informative query tokens tend to have high dominant contribution and low residual contribution, unlike less informative words; Figure 3 illustrates the spectrum and token scores.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 QViD Token Scoring and Pruning

  • QViD scores each visual token by participation in the top r right singular directions, averages scores across attention heads, and retains the top K tokens for subsequent LLM layers.For short queries, the scoring rule uses reff = min(r, Nq, Nv); the method uses frozen projections and requires no training or architectural modification.Source: §3.4 · pp. 5–6

4 Experiments

QViD is evaluated across image, video, multimodal reasoning, hallucination-sensitive, and captioning tasks, with the main-paper results focused on image and video understanding. It consistently outperforms V2Drop under matched token budgets, while ablations test the role of SVD scoring, component count, and pruning layer.

4.1 Experimental Setup

  • Experiments cover four VLMs, image benchmarks including GQA, ScienceQA, TextVQA, POPE, MME, and MMBench, and video benchmarks VideoMME and MVBench.V2Drop is the primary baseline; additional LLaVA-1.5-7B comparisons include ToMe, FastV, HiRED, LLaVA-PruMerge, SparseVLM, and PDrop.Source: §4.1 · p. 6

4.2 QViD: Main Results

  • Across evaluated VLMs and benchmarks, QViD consistently performs better under matched token budgets, with particularly pronounced gains under aggressive compression.The main results also report improved downstream accuracy on image and video understanding tasks.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 8
  • At 64 tokens, QViD improves over V2Drop by 3.5% in normalized average on LLaVA-1.5-7B, with gains of 3.7% on TextVQA, 5.7% on POPE, and 83 points on MME.QViD achieves the best average performance at all three token budgets.Source: §4.2 · p. 7, Table 1 · p. 7
  • At 11.1% retention, Qwen2-VL-7B average normalized performance rises from 62.7% with V2Drop to 81.7% with QViD.QViD improves average normalized performance over V2Drop across all retention ratios.Source: §4.2 · p. 7, Table 2 · p. 8
  • At 10% retention on LLaVA-OV-7B, QViD preserves 91.2% of full-token performance versus 69.4% for V2Drop.The video results report consistent outperformance under matched token budgets.Source: §4.2 · p. 7, Table 3 · p. 8
  • On LLaVA-1.5-7B, QViD uses 2.53, 3.42, and 4.32 TFLOPs at 64, 128, and 192 tokens, reducing compute from the 8.86-TFLOP full-token baseline by 71.5%, 61.4%, and 51.3%.Additional evaluations report strong performance beyond standard perception tasks, including multimodal reasoning, hallucination-sensitive evaluation, and captioning.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Ablation Studies

  • Across token budgets, SVD scoring outperforms column-norm scoring on the same query–vision interaction matrix, especially under aggressive compression.The ablation attributes the difference to SVD isolating low-rank interaction modes for token selection.Source: §4.3 · p. 9, Table 4 · p. 9
  • At 128 retained tokens, performance improves sharply from 1 or 2 SVD components to 4 or 8, while 16 components gives the best average performance.The default pruning layer is ℓ=3, selected for a strong accuracy–efficiency tradeoff in the layer-selection study.Source: §4.3 · p. 9, Table 4 · p. 9

5 Conclusion

QViD uses dominant low-rank query–vision interactions to select visual tokens without attention weights, improving performance across image and video benchmarks under matched token budgets. Its evaluation scope and practical speedups remain bounded by the tested models, benchmarks, and deployment conditions.

5 Conclusion

  • Across image and video benchmarks and multiple VLMs, QViD improves performance under matched token budgets, with the largest gains under aggressive compression.The conclusion describes dominant SVD components as sufficient to capture much of the task-relevant query–vision structure.Source: §5 · p. 9
  • QViD constructs the query–vision interaction matrix from standard transformer query–key projections and scores tokens using its dominant low-rank subspace.It retains tokens participating in the main query-relevant interaction modes without relying on attention weights.Source: §5 · p. 9

A Limitations and Future Directions

  • Evaluation covers public VLMs and standard image/video benchmarks, so behavior on closed-source models, proprietary inference stacks, and specialized domains may differ.Practical speedups may also vary with hardware, batching, sequence length, and attention-kernel implementations.Source: A Limitations and Future Directions · p. 12
  • As a training-free pruning method, QViD inherits biases and failure modes of the underlying VLM.Source: A Limitations and Future Directions · p. 12

B Layer Selection Results

Layer sweeps find that early scoring layers work well, with layer 3 a useful default, while results vary more across layers for Qwen2-VL at low retention. Component ablations show that multiple dominant SVD modes improve robustness over single-mode scoring.

B Layer Selection Results

  • For LLaVA-1.5-7B, performance is relatively stable across early layers; layer 3 is best on average, while layers 1–5 differ only slightly.Performance drops mildly at layer 6, where task-relevant information may already have been mixed across tokens.Source: B Layer Selection Results · p. 12
  • Layer 3 gives Qwen2-VL-7B’s best average performance at 22.2% and 11.1% retention, and is within 0.2 percentage points of the best at 33.3%.The results support layer 3 as the default scoring layer in these experiments.Source: B Layer Selection Results · p. 12

C Effect of the Number of SVD Components

D Scoring Rule Ablation

The scoring-rule ablation finds that preserving head-specific low-rank query–vision structure outperforms scoring by raw interaction magnitude. The efficient scoring procedure computes the same right-singular subspace through QR reduction and retains top-scoring visual tokens.

D Scoring Rule Ablation

  • The ablation compares scoring rules derived from the same query–vision interaction matrix, including column norm, sigma-weighted, and per-head max variants.Column norm scores raw interaction magnitude; sigma-weighted uses singular values, while per-head max aggregates head-specific SVD scores by taking the maximum across heads.Source: D Scoring Rule Ablation · p. 13
  • Per-head SVD-based scoring performs substantially better than raw interaction-energy scoring, indicating that head-specific low-rank structure matters for token selection.Column-norm and sigma-weighted scores induce the same token ordering and identical evaluation results, and both perform poorly.Source: D Scoring Rule Ablation · pp. 13–14
  • The per-head max variant is competitive with default QViD aggregation and performs slightly better on some metrics, especially at smaller token budgets.The authors suggest different attention heads may specialize in different query-relevant visual cues, but retain fixed aggregation for consistency.Source: D Scoring Rule Ablation · p. 14

E Efficient QViD Scoring

  • Because N_q is much smaller than N_v, QR reduction computes the same right-singular subspace more efficiently than applying SVD to the full rectangular matrix.The scoring step forms per-head visual-token leverage scores, averages them across heads, then retains the top-K tokens for the remaining LLM layers.Source: E Efficient QViD Scoring · pp. 14–16
  • In LLaVA-1.5-7B, the full scoring step, including interaction-matrix construction, costs less than 50 MFLOPs.The passage attributes the small cost relative to processing all visual tokens to N_q typically being much smaller than N_v.Source: E Efficient QViD Scoring · pp. 14, 16

F Detailed Efficiency Results

QViD’s query-aware scoring adds a small cost before pruning, while pruning reduces the remaining LLM computation and improves throughput over the full-token baseline. It has slightly higher latency than V2Drop at matched retained-token budgets, alongside a stronger accuracy–compute tradeoff, especially under aggressive pruning.

F Detailed Efficiency Results

  • Reported TFLOPs include QViD’s early query-aware scoring pass and subsequent LLM computation on retained tokens; the full-token baseline processes all 576 visual tokens.V2Drop’s accounting includes its per-layer schedule, retaining the full visual sequence before pruning in early layers.Source: F Detailed Efficiency Results · p. 16
  • Removing one visual token saves roughly 0.01 TFLOPs in the remaining LLM computation; retaining 64, 128, and 192 tokens saves approximately 7.17, 6.27, and 5.38 TFLOPs, respectively.The passage states that scoring overhead is much smaller than the compute saved by pruning.Source: F Detailed Efficiency Results · p. 16
  • QViD achieves a stronger accuracy–compute tradeoff than V2Drop at 64, 128, and 192 retained tokens on POPE, GQA, and TextVQA, especially under aggressive pruning.The figure compares these methods against a full-token baseline with 576 visual tokens.Source: Figure 4 · p. 17
  • QViD runs at 10.40, 10.27, and 10.26 images/s for 64, 128, and 192 retained tokens, respectively, versus 8.60 images/s for the full-token baseline.It has slightly higher latency than V2Drop at matched budgets because it scores query-aware tokens before pruning.Source: F Detailed Efficiency Results · p. 17

G Additional Multimodal Benchmark Results

QViD improves over V2Drop across additional multimodal benchmarks, with particularly strong gains under aggressive compression on MMMU-Pro Standard. The results extend evaluation beyond the main suite to diverse image and reasoning tasks.

G Additional Multimodal Benchmark Results

  • Table 10 shows QViD consistently improves over V2Drop in average normalized performance across additional LLaVA-v1.5-7B benchmarks, with especially large gains on OCRBench and ChartQA.The evaluated benchmarks also include AI2D, MMMU, and SEEDBench.Source: G Additional Multimodal Benchmark… · p. 18
  • 28.8% versus 23.6% at 11.1% retention: QViD substantially improves over V2Drop on Qwen2-VL-7B's MMMU-Pro Standard split.At 33.3% and 22.2% retention, QViD remains close to V2Drop.Source: G Additional Multimodal Benchmark… · p. 18

H Additional Video Understanding Results

On Video-LLaVA-7B, QViD outperforms V2Drop across video retention ratios, with its largest gain at the most aggressive compression setting. The results are reported for MVBench and VideoMME.

H Additional Video Understanding Results

  • QViD outperforms V2Drop at every retention ratio on Video-LLaVA-7B, with the largest gain under the most aggressive compression setting.Table 12 compares performance on MVBench and VideoMME; Avg. is mean normalized performance relative to the full-token baseline.Source: H Additional Video Understanding Results · p. 18, Table 12 · p. 19

I Hallucination and Captioning Results

QViD improves on hallucination-sensitive and caption-generation benchmarks under token compression, while analysis of the query–vision interaction spectrum supports using a small number of dominant SVD directions. The low-rank pattern appears across datasets, although its decay rate varies.

I Hallucination and Captioning Results

  • HallusionBench reports all-pair, figure, and question accuracy, while COCO Captioning uses CIDEr, a caption-quality metric rather than an accuracy metric.In the reported lmms-eval setup, CIDEr values around 1.0 roughly correspond to a traditional CIDEr score around 100.Source: I Hallucination and Captioning Results · p. 19
  • QViD improves over V2Drop across all HallusionBench metrics and CIDEr at every token budget on LLaVA-v1.5-7B, and also improves HallusionBench performance and CIDEr on Qwen2-VL-7B.Tables 13 and 14 report HallusionBench metrics and caption quality at matched token budgets or retention ratios where available.Source: I Hallucination and Captioning Results · p. 19, Table 14 · p. 20

J.1 Low-Rank Structure of Query–Vision Interactions

  • The query–vision interaction spectrum decays rapidly after its first few components across datasets, supporting token scoring with only the dominant SVD directions.The decay rate varies across datasets, but dominant components consistently capture the strongest query-conditioned interaction modes.Source: §J.1 · pp. 20–21, Figure 5 · p. 20, Figure 6 · p. 21

J.2 Qualitative Analysis of SVD Components

The dominant SVD components emphasize task-relevant query words and visual regions, unlike residual components or often-diffuse baseline score maps. QViD also retains advantages over baselines across retention ratios in the reported confidence-interval evaluation.

J.2 Qualitative Analysis of SVD Components

  • Dominant SVD components assign higher scores to task-critical query words, while residual components emphasize less informative or function words.The dominant components include words directly tied to the answer, supporting their use for visual-token scoring.Source: §J.2 · p. 21, Figure 8 · p. 24

K Confidence Intervals

  • At 5.56% retention, QViD achieves substantial gains across all six Qwen2-VL-7B benchmarks compared with FastV and V2Drop.The results include 95% confidence intervals computed over three independent random seeds; QViD also consistently outperforms both baselines on GQA and TextVQA across retention ratios.Source: K Confidence Intervals · p. 21, Table 15 · p. 23

K.1 Additional Token-Score Visualizations

  • Across diverse image-query examples, QViD assigns high scores to localized, query-relevant regions, whereas V2Drop and SparseVLM often produce more diffuse or less relevant maps.Examples include the snowboarder, bat, scoreboard, license plate, and player jersey.Source: §K.1 · p. 21, Figure 7 · p. 22

K.2 Query-Token Scores from Dominant and Residual Components

  • Dominant SVD directions tend to emphasize task-relevant query words, while residual directions often give larger scores to function words or generic question tokens.Examples include country, state, lions, score, batter, liquor, and jersey.Source: §K.2 · p. 23, Figure 8 · p. 24

Related papers

Loading 2610.33215v1…