PaperlayerView exact revision on arXiv

Source-linked AI summary

TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning

Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li

arXiv:2609.37581v1cs.CVcs.AI

TL;DR

Query-agnostic pre-LLM pruning can discard query-relevant evidence before later text-guided pruning. TReVS combines textual relevance with visual saliency before the LLM and uses high-variance attention heads for in-LLM pruning. On LLaVA-1.5-7B, it prunes 94.4% of visual tokens while retaining 92.8% of original performance.

  • Problem

    Query-agnostic pre-LLM pruning can remove visual evidence essential to the text query, leaving it unavailable to subsequent query-aware pruning.

    Source: §1 · p. 1

  • Method

    TReVS combines vision-encoder saliency, textual relevance, and token diversity before the LLM, then uses high-variance attention heads for in-LLM pruning.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    On LLaVA-1.5-7B, TReVS prunes 94.4% of visual tokens while retaining 92.8% of original performance and outperforms state-of-the-art training-free methods across reduction ratios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective visual-token pruning preserves query-relevant evidence before the LLM and uses query-sensitive attention heads to select task-relevant tokens within it.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visual tokens. Recent visual token pruning methods increasingly follow a two-stage paradigm: they first remove visually redundant tokens after the vision encoder and then discard tokens irrelevant to the textual query within the Large Language Model (LLM). However, since the first stage typically relies solely on vision-encoder saliency, it may prematurely eliminate query-relevant tokens, depriving the subsequent text-guided stage of critical visual evidence. Our empirical analysis shows that incorporating query guidance into first-stage pruning better preserves task-relevant evidence and consistently improves performance over vision-only saliency-based pruning. We further find that high-variance attention heads are more sensitive to the textual query and yield more discriminative text-to-vision attention signals for second-stage pruning. Motivated by these findings, we propose TReVS, a training-free framework that combines textual relevance with vision-encoder saliency for pre-LLM pruning and leverages high-variance attention heads to remove task-irrelevant tokens at shallow-to-intermediate layers of the LLM. On LLaVA-1.5-7B, TReVS retains 92.8% of the unpruned baseline performance while pruning 94.4% of visual tokens, outperforming prior state-of-the-art methods.

1 Introduction

Visual-token sequences make VLM inference costly, while query-agnostic pre-LLM pruning can discard evidence needed by later query-aware pruning. TReVS coordinates query guidance across both pruning stages and reports strong performance at high token reduction.

1 Introduction

  • Query-agnostic pre-LLM pruning can discard evidence essential to the text query, which is then unavailable to subsequent query-aware pruning.Long visual-token sequences also increase inference computation, latency, and memory use.Source: §1 · p. 1
  • Textual relevance complements vision-encoder saliency and consistently improves performance by preserving query-relevant evidence before irreversible token reduction.High-variance heads provide more discriminative, query-sensitive attention signals for in-LLM pruning.Source: §1 · p. 2
  • TReVS fuses vision-encoder saliency with textual relevance, adds a small diversity set before the LLM, then prunes with high-variance attention heads during LLM prefilling.The second stage removes residual task-irrelevant tokens after cross-modal interaction develops.Source: §1 · p. 2
  • TReVS outperforms state-of-the-art training-free methods across reduction ratios; on LLaVA-1.5-7B, it prunes 94.4% of visual tokens while retaining 92.8% of original performance.It also accelerates prefilling by 2.2× and end-to-end inference by 1.4× without additional training.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

As VLMs extend to high-resolution images and long videos, visual sequences can substantially exceed text sequences. Visual-token reduction methods differ in where pruning occurs and when query information is used.

2.1 Vision-Language Models

  • As image resolution and video duration increase, visual sequences can substantially exceed corresponding text sequences, making token reduction important for efficient VLM inference.Source: §2.1 · p. 3

2.2 Visual Token Reduction for VLMs

  • Visual-guided methods prune after the vision encoder, query-aware methods prune within the LLM, and two-stage methods combine pre-LLM reduction with in-LLM refinement.These approaches use visual signals, text-to-vision attention, or both at different points in the pipeline.Source: §2.2 · p. 3
  • Unlike two-stage methods that separate visual-only pre-LLM reduction from query-aware in-LLM pruning, TReVS uses textual relevance before the LLM and query-sensitive attention within it.This coordination reduces visual redundancy without depriving the second stage of query-relevant context.Source: §2.2 · p. 3

3 Preliminary Analysis

The preliminary analyses find that textual relevance complements vision-encoder saliency during pre-LLM pruning, while attention variance identifies more query-sensitive heads. Positional attention patterns also motivate delaying attention-based pruning beyond the earliest layers.

3.1 Does textual guidance improve performance?

  • Allocating even a modest token-budget fraction to textual relevance consistently outperforms visual-only pruning, and the signals have negative Spearman correlations across datasets and budgets.Textual relevance reorders even top-K saliency-selected candidates, so it complements rather than replaces vision-encoder saliency.Source: §3.1 · p. 3

3.2 Are High-Variance Attention Heads More Query-Sensitive?

  • The highest-variance head subsets are generally more text-sensitive, with the clearest separation in middle layers, supporting attention variance as a criterion for selecting query-sensitive heads.The analysis pairs 1,000 TextVQA images with two distinct questions and evaluates cumulative head subsets.Source: §3.2 · p. 4
  • Attention generally favors later visual-token positions in the earliest layers, while later layers have flatter interior-position profiles, motivating pruning beyond the earliest layers.Source: §3.2 · p. 4

4 Method

TReVS progressively prunes visual tokens before and within the LLM, combining vision saliency with textual relevance before the LLM and using high-variance attention heads for later query-driven pruning.

4.1 Query-Aware Visual Redundancy Reduction

  • The visual saliency signal averages [CLS]-to-patch attention across all ViT heads, using attention from the penultimate ViT layer.Source: §4.1 · p. 4
  • For each visual token, TReVS computes a query-relevance score by RMS-aggregating rectified cosine similarities between projected visual features and text-token embeddings.The paper describes this aggregation as suppressing noise from weakly related text tokens while amplifying tokens broadly attended to by the query.Source: §4.1 · p. 4
  • TReVS keeps the top-Kr tokens under a fused score that favors tokens rated highly by either visual saliency or textual relevance, with a reward when both are high.It adds Kd diverse tokens sampled from the remaining candidates to form the K1-token input to the LLM.Source: §4.1 · pp. 4–5

4.2 Query-Driven Visual Token Compression

  • After Stage 1, TReVS prunes again at a shallow-to-middle LLM layer, retaining K2 visual tokens for the remaining layers.As text tokens absorb task-relevant visual information through cross-modal attention, the visual tokens become increasingly redundant.Source: §4 · p. 4, §4.2 · p. 5
  • At the second pruning layer, TReVS uses attention variance as a training-free criterion for selecting query-sensitive heads, retaining the top half ranked by variance.It scores each visual token by maximum text-to-vision attention across text positions, averaged over selected heads, then keeps the top K2 tokens.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate TReVS across image, high-resolution, and video VLMs, then examine accuracy, pruning-stage contributions, and inference efficiency. Results show strong retained performance under compression, including on POPE and high-resolution inputs, with efficiency gains and complementary benefits from both pruning stages.

5.1 Experimental Setup

  • The evaluation spans LLaVA-1.5-7B, LLaVA-NeXT-7B, and Video-LLaVA-7B, processing 576, 2,880, and 2,048 visual tokens, respectively.The reported evaluation includes image, high-resolution, and video understanding, followed by an inference-efficiency analysis.Source: §5 · p. 5, §5.1 · pp. 5–6

5.2 Main Results

  • RelAcc. retains 98.8%, 96.5%, and 92.8% of unpruned performance on LLaVA-1.5-7B, with the highest score at all three budgets.TReVS also obtains the best MME scores across the budgets; Table 1 compares performance under matched average token budgets.Source: §5.2 · p. 6, Table 1 · p. 7
  • At 32 tokens, POPE accuracy reaches 82.7%, exceeding DivPrune and VScan by 1.2% and 2.8%, respectively.Figure 5 shows that TReVS degrades more slowly under increasingly aggressive compression and performs best at 32 tokens; the dashed line is the unpruned model.Source: §5.2 · p. 6, Figure 5 · p. 6
  • On LLaVA-NeXT-7B, RelAcc. retains 96.6% and 93.0% of unpruned performance with 320 and 160 tokens, while GQA trails the best by 0.5% and 0.6%.TReVS also has the best MME scores at both budgets and the best MMB score at 160 tokens; the authors hypothesize that query-sensitive heads may retain less global context at higher resolutions.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Ablation Study

  • At the 32-token preset, RelAcc. rises from 91.8% to 92.7% with TReVS pre-LLM pruning, then to 93.2% with high-variance-head selection.Table 4 isolates the two stages; together they improve RelAcc. by 1.4%.Source: §5.3 · p. 6

5.4 Efficiency Analysis

  • At 32 tokens, TReVS achieves a 2.2× prefill speedup, a 1.4× end-to-end speedup, and 6.5× lower KV-cache consumption.Table 5 reports POPE inference efficiency for LLaVA-1.5-7B on an NVIDIA RTX 4090 GPU and measures latency per sample in milliseconds.Source: §5.4 · p. 6, Table 5 · p. 6

6 Conclusion

The conclusion identifies premature loss of task-relevant evidence as a risk of query-agnostic pre-LLM pruning. TReVS coordinates saliency, textual relevance, diversity, and high-variance heads across two pruning stages; future work may adapt token allocation to input complexity and query demands.

6 Conclusion

  • Query-agnostic pre-LLM reduction can discard task-relevant evidence before query-aware reasoning begins.The paper identifies this as an irreversible information bottleneck in two-stage visual token pruning.Source: §6 · pp. 7–8
  • TReVS coordinates vision-encoder saliency, textual relevance, token diversity, and high-variance attention heads across pre-LLM and in-LLM pruning.The conclusion highlights preserving query-relevant evidence before the LLM and using query-sensitive heads within it.Source: §6 · p. 8
  • Future work may explore adaptive allocation between pruning stages based on input complexity and query demands.Source: §6 · p. 8

Related papers

Loading 2609.37581v1…