PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization

Qingchan Zhu, Weihang You, Hanqi Jiang, Changdi Yang, Tianming Liu, Geng Yuan

arXiv:2609.03158v1cs.CVcs.CLcs.LG

TL;DR

Visual token pruning can retain redundant salient tokens while dropping evidence without close representatives, motivating a demand-side alternative. CoverPruner formulates pruning as query-weighted Representational Coverage Maximization using projector-space coverage and a first-layer attention probe. Across VLM architectures and compression rates, it achieves the best average accuracy among compared methods, with larger gains under aggressive compression.

  • Problem

    Visual token pruning must reduce inference cost without leaving discarded visual evidence unrepresented, a risk in token-local retention and merging approaches.

  • Method

    CoverPruner is a training-free selector that maximizes query-weighted coverage of the full projected visual-token set using calibrated projector-space similarity and a first-layer attention probe.

  • Results

    CoverPruner achieves the best average accuracy among compared methods across VLM architectures and compression rates, with its margin widening as token budgets tighten.

  • Takeaways & Limitations

    Under aggressive pruning, selecting representatives for disappearing evidence is more effective than ranking the evidence that remains.

  • Takeaways & Limitations

    Coverage is weaker for small objects, OCR details, precise counting, spatial relations, and other local evidence that may not be replaceable despite high projected-token similarity.

Abstract

from arXiv · show

Visual token pruning reduces the inference cost of vision-language models (VLMs), but most methods only ask which tokens to keep. This retained-token view can keep redundant high-scoring tokens while leaving discarded evidence without a close representative. We propose CoverPruner, a training-free pruner that asks the complementary demand-side question: after a token is removed, which surviving original token represents it for the target VLM? CoverPruner formulates pruning as Representational Coverage Maximization (RCM), covering the full projected visual-token set with query-weighted demand. It instantiates RCM with projector-space coverage and a lightweight first-layer attention probe. Across multiple VLM architectures and compression rates, CoverPruner achieves the best average accuracy among all compared methods, with the largest gains usually appearing under aggressive compression.

1 Introduction

Visual token pruning must preserve dispersed visual evidence, not merely retain individually salient tokens. CoverPruner reframes pruning as query-weighted coverage of discarded evidence and reports strongest gains under aggressive compression.

  • Motivation: VLM visual-token sequences can dominate inference cost because attention scales quadratically with sequence length.Modern systems produce hundreds to thousands of visual tokens, while video inputs can reach tens of thousands.
  • Motivation: Scoring can retain redundant salient patches, while merging can dilute fine-grained evidence such as small text, boundaries, or chart marks.These approaches may leave discarded evidence without a close representative.
  • Core idea: Representational Coverage Maximization covers the full visual-token set with retained representatives weighted by query relevance.A redundant sky region may need one representative, whereas distinct sign-text patches may require separate representatives.
  • CoverPruner: CoverPruner is training-free, measures coverage in projector output space, and estimates query relevance with a lightweight first-layer attention probe.It retains original representations and is used unchanged across models, benchmarks, and token budgets.
  • Results: CoverPruner achieves the best average accuracy among compared methods, with especially clear gains at aggressive token budgets.Figure 1(b) reports performance retention relative to full-model accuracy at 64 retained tokens.

2 Related Work

Related pruning methods reduce visual-token sequences through architectural compression, token-local scoring, redundancy reduction, merging, or diversity. CoverPruner instead represents the full projected-token set in the consuming VLM’s input space.

  • Visual token pruning in VLMs: Pre-fusion compression changes architectures or adds training, whereas inference-time methods remove tokens during the forward pass.Inference-time pruning commonly uses attention signals or visual-feature redundancy.
  • Visual token pruning in VLMs: Token-scoring, local-redundancy, and merging methods establish compressibility but generally lack an explicit requirement for representing discarded tokens.Merging methods reduce redundancy by fusing similar tokens into modified representations.
  • Recent methods: Recent methods combine instruction-conditioned diversity, projection-layer sensitivity, or image-adaptive attention–diversity selection.Examples include CDPruner, ZOO-Prune, and AgilePruner.
  • Representative selection and coverage: Representative-selection methods differ in their represented space: CoverPruner uses retained original projected tokens to cover the full set in the consuming VLM’s input space.This distinguishes it from diversity selection, shared-encoder coverage, and query-conditioned localization.

3 Method

CoverPruner selects a fixed-size subset of original projected visual tokens by maximizing query-weighted representational coverage in the decoder’s input space. It uses calibrated projector-space similarity, instruction-conditioned demand, and greedy marginal gains.

  • Pipeline: Given projected visual tokens, CoverPruner retains an ordered subset of size m and feeds it with instruction tokens to the unchanged decoder.It neither merges tokens nor alters their representations.
  • 3.2 Pruning as Coverage: The method defines coverage as how well retained tokens represent every pruned token, then maximizes weighted total coverage under a fixed budget.The full set supplies demand, the retained subset supplies representatives, and demand weights encode query relevance.
  • 3.2 Pruning as Coverage: Representational Coverage Maximization is monotone submodular with non-negative utilities and weights, motivating a greedy selector with a standard (1−1/e) approximation.The objective selects representatives by marginal contribution to uncovered evidence rather than by individual score.
  • 3.3 Anchoring the Pruner in the LLM: Similarity is computed in projector output space, where cosine proximity represents substitutable evidence for the downstream decoder.This defines redundancy after multimodal projection rather than in the vision encoder’s feature space.
  • 3.3 Anchoring the Pruner in the LLM: Row-wise mean-centering and non-negative clipping make coverage utilities more discriminative when raw similarities have limited dynamic range.Without dynamic range, uniformly inflated gains make redundant and novel tokens difficult to distinguish.
  • 3.3 Anchoring the Pruner in the LLM: A first-layer attention probe estimates instruction-conditioned demand before pruning, and temperature-scaled weights steer budget toward query-relevant regions.The probe is computed before pruning rather than extracted from a full decoder pass.
  • 3.4 Demand-Weighted Coverage Selection: Greedy selection adds the candidate with the largest weighted uncovered-coverage gain, updates current coverage, and repeats until the budget is filled.Once a region is represented, another similar token contributes little; tokens covering missed evidence remain valuable.

4 Experiments

CoverPruner is evaluated across multiple VLMs, pruning budgets, benchmarks, and efficiency settings, consistently preserving strong performance while reducing visual-token computation. Its strongest gains appear under aggressive compression and on dense, high-resolution visual sequences.

  • Experimental Setup: Experiments cover LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B across matched backbones, resolutions, protocols, and token budgets.Benchmarks span general VQA, reasoning, OCR, hallucination, chart understanding, and multilingual or culturally grounded settings.
  • LLaVA-1.5-7B Results: 98.4%, 96.4%, and 93.7% of full-model Acc are retained at 128, 64, and 32 tokens on LLaVA-1.5-7B.CoverPruner achieves the best overall Acc among compared methods at every budget; scoring methods drop sharply on POPE and merging methods lose more on TextVQA under aggressive compression.
  • LLaVA-1.5-7B Results: CoverPruner leads on VQAv2, TextVQA, and MME across pruning ratios from 66.7% to 94.4%.Figure 3 traces the comparison across increasingly aggressive pruning.
  • High-Resolution Results: 99.7%, 98.4%, and 96.6% of full-model performance are retained at 640, 320, and 160 tokens on LLaVA-NeXT-7B.The margin over the strongest baseline grows from 0.3 to 2.3 Acc points as the budget shrinks, with the largest gains on POPE; CoverPruner remains the top method on TextVQA.
  • Architecture Generalization: CoverPruner achieves the best overall Rel. on Qwen2.5-VL-7B, with its lead growing from 1.3 to 4.1 Rel. points across pruning levels.This evaluation uses fixed 1008×1008 resolution and 1,296 visual tokens despite Qwen2.5-VL’s internal token compression.
  • Ablations: Coverage-weighted relevance recovers full-model-level performance, while raw-attention ranking substantially reduces POPE performance.The first-layer attention probe is the best relevance source on both TextVQA and POPE, and combining it with coverage yields the strongest configuration.
  • Efficiency: 7.5× lower prefill latency, 2.6× lower decode latency, and 10.7× lower FLOPs accompany an F1 increase from 86.8 →87.3 at 320 retained tokens.On POPE with LLaVA-NeXT-7B, CoverPruner outperforms SparseVLM and DivPrune on both latency and F1.
  • Qualitative Analysis: Selection concentrates tokens on text-bearing regions for TextVQA, informative objects and relationships for GQA, and the whole scene for POPE.Diverse regions receive more representatives while redundant regions share one, without explicit spatial supervision.

5 Analysis and Discussion

The analysis explains CoverPruner’s advantage as a set-level coverage effect: relevance guides demand, while marginal coverage preserves representatives for diverse visual evidence. This advantage is most pronounced when budgets are tight and signals are read from the consuming LLM.

  • Ranking Versus Coverage: First-layer attention is more effective as a demand weight inside coverage than as an isolated ranker.Ranking concentrates tokens around query-salient patches, whereas demand weighting shifts budget toward relevant regions while preserving representatives across the image.
  • Where Coverage Helps Most: Under aggressive compression, coverage selects tokens by marginal contribution to uncovered evidence, preserving accuracy where existing methods over-concentrate, remain query-blind, or alter representations.At moderate budgets, several subsets may preserve enough evidence, so method gaps are smaller.
  • Signals From the Consuming LLM: Projector-space similarity measures redundancy in the consuming VLM’s input space, while first-layer attention supplies query demand before pruning.Mean-centering makes similarity useful for best-representative selection and avoids reliance on external proxies.

6 Conclusion

CoverPruner reframes visual-token selection as Representational Coverage Maximization and implements this demand-side view with a training-free selector. Across evaluated VLMs and compression rates, it achieves the highest average accuracy among compared baselines, with wider gains under tighter budgets.

  • CoverPruner formulates visual-token selection as Representational Coverage Maximization, selecting representatives for evidence that disappears rather than ranking only surviving evidence.The approach shifts pruning from survivor importance to demand coverage in the target VLM’s representation space.
  • CoverPruner combines projector-space coverage, mean-centered non-negative utilities, and a first-layer attention probe without modifying the model or adding a trained component.
  • Across evaluated VLMs and compression rates, CoverPruner achieves the highest average accuracy among compared baselines.Its margin widens as the token budget tightens, the regime where existing rankers and merging methods break down.
  • Query relevance contributes more by reshaping demand than by ordering tokens.

7 Limitations

CoverPruner’s applicability is limited by required model access, local-evidence failure modes, quadratic similarity costs, and settings where visual demand changes over time or requires finer-grained evaluation.

  • CoverPruner requires projector visual tokens and early decoder signals, limiting direct use with closed API-only MLLMs.
  • Coverage is weaker for small objects, OCR details, precise counting, spatial relations, and other local evidence that may not be replaceable despite high projected-token similarity.This risk matters particularly in accessibility, safety-sensitive, multilingual, and culturally grounded settings.
  • Pairwise similarity is quadratic in visual-token count, requiring more careful engineering for higher-resolution inputs.
  • Image-level coverage applied directly to video could over-represent persistent backgrounds while under-representing short-lived actions, transient objects, or frame changes.
  • One-time pre-decoding selection does not capture visual-demand shifts during multi-step reasoning, dialogue, or long-form generation.
  • Finer-grained analysis across OCR, counting, grounding, spatial reasoning, hallucination, and temporal localization is needed to characterize true evidence substitution.

A Experimental Setup and Benchmark Details

The evaluation uses multiple VLM benchmarks spanning visual question answering, diagrams, text, hallucination, spatial understanding, cultural contexts, and multimodal reasoning. The inference-time pipeline builds projected-token coverage and query demand, then selects a fixed-budget subset by marginal coverage gain.

  • Experiment setup: Accuracy is reported as benchmark average, while Rel. denotes the percentage of full-model accuracy retained.Latency uses one NVIDIA RTX A6000 GPU with FP16 precision and averages 100 forward passes after 10 warm-up iterations, excluding tokenization, vision encoding, and projection.
  • Benchmarks: The benchmark suite covers natural-image VQA, compositional scene understanding, scientific diagrams, text-rich images, charts, hallucination, spatial perception, cultural contexts, and multimodal reasoning.Examples include VQAv2, GQA, AI2D, ScienceQA-IMG, OCRVQA, TextVQA, ChartQA, HallBench, RealWorldQA, CCBench, POPE, MME, and MMBench variants.
  • Benchmarks: POPE tests whether pruning preserves whole-scene contextual evidence needed for object-presence judgments.
  • Benchmarks: MMBench and MMBench-CN measure whether pruning behavior transfers across English and Chinese language settings.
  • Inference pipeline: CoverPruner’s inference pipeline computes projected visual-token similarities, converts them into non-negative utilities, estimates instruction-conditioned demand, and selects tokens by marginal coverage gain.The output retains the selected visual tokens in original order for decoding.

C Additional Experimental Results

Additional experiments on 13B LLaVA models preserve CoverPruner’s best-overall-accuracy pattern across token budgets. Performance retention remains high at moderate compression, while the advantage over competing methods grows under aggressive compression.

  • LLaVA-1.5-13B: CoverPruner achieves the best overall accuracy among compared methods at every token budget for LLaVA-1.5-13B.With 77.8% of visual tokens removed, it retains 98.4% of full-model performance.
  • LLaVA-1.5-13B: Under the most aggressive 94.4% reduction for LLaVA-1.5-13B, the Rel. gap over the strongest competing method widens to 2.9 points.
  • LLaVA-NeXT-13B: CoverPruner again achieves the best overall accuracy at every budget for LLaVA-NeXT-13B.It retains 99.9%, 98.3%, and 96.8% of full-model performance as the retained-token count decreases.
  • LLaVA-NeXT-13B: For LLaVA-NeXT-13B, the margin over the strongest baseline increases under heavier compression.

D Additional Qualitative Visualizations

The qualitative studies examine CoverPruner’s behavior under aggressive pruning, its retained-token patterns, overhead, calibration, query conditioning, and representation-space choice. Together, they show how coverage preserves diverse or contextual evidence while projector-space similarity and calibrated relevance support the method.

  • Qualitative case studies: Figures 5 and 6 present complementary cases where CoverPruner succeeds while the full model fails, and vice versa, under aggressive pruning.Figure 5 uses GQA, while Figure 6 uses POPE; check and cross marks indicate correct and incorrect responses.
  • Retained-token patterns: CoverPruner retains fewer tokens in repetitive regions and more tokens in visually diverse regions such as people, vehicles, and signs.Figure 7 compares FastV, SparseVLM, and CoverPruner at pruning ratios of 77.8%, 88.9%, and 94.4%.
  • Overhead: Similarity and attention probing are computed once over the original sequence, while coverage selection grows with the retained budget.Table 8 reports pruning overhead before decoder prefill, whereas Table 5 reports decoder latency in ms/token.
  • Calibration and conditioning: 54.79 to 55.23 TextVQA: mean-centering improves performance with LLM-embed relevance at τ = 0.3.The appendix describes the gain as small but consistent with the deployed mean-centered non-negative utilities.
  • Calibration and conditioning: The default τ = 0.3 relevance conditioning yields modest, consistent gains on both benchmarks, while performance is robust to τ within this range.The authors conclude that relevance-signal quality matters more than conditioning strength.
  • Representation space: Using pre-projector encoder features instead of projector-output features causes a substantial drop on both benchmarks.The comparison holds the RCM objective, demand weights, and greedy selection protocol fixed, changing only the similarity features.

E.5 Variants Considered

The appendix evaluates aggregation alternatives to the default sum formulation. Top-K mean offers a task-dependent benefit, whereas the concave-over-modular alternative substantially underperforms because it loses best-representative semantics.

  • Aggregation variants: Top-K mean with K =20 marginally outperforms the unweighted sum baseline on TextVQA but does not improve POPE.The result suggests density bias is task-dependent and most pronounced on OCR-heavy benchmarks.
  • Aggregation variants: Top-K mean is not deployed by default because it marginally helps TextVQA but does not improve POPE.The appendix therefore keeps sum aggregation with τ =0.3 as the default configuration.
  • Aggregation variants: The concave-over-modular reformulation substantially underperforms because cumulative similarity can obscure whether one nearby token provides the best representation.The authors report this as a negative result and retain the max-based representative semantics instead.
Loading 2609.03158v1…