PaperlayerView exact revision on arXiv

Source-linked AI summary

Centering before Pruning: Lightweight Geometry Correction for Diversity-Based Visual Token Pruning in LVLMs

Shunjie Wen, Jaeyeon Lee, Dong-Wan Choi

arXiv:2608.30263v1cs.CVcs.AI

TL;DR

LVLM visual-token sequences are long and redundant, while raw cosine geometry poorly resolves diversity despite implicitly favoring distinctive tokens. Cen-Prune combines centered cosine similarity for subset diversity with raw-space distinctiveness for token preference, preserving the selector and yielding robust gains across image and video benchmarks.

  • Problem

    Raw visual-token similarities concentrate in the positive range, limiting diversity resolution while entangling pairwise diversity with an implicit preference for distinctive tokens.

  • Method

    Cen-Prune uses centered cosine similarity to measure subset diversity and raw-space distinctiveness as a complementary token-wise preference without changing the underlying selection mechanism.

  • Results

    Cen-Prune provides robust improvements across image- and video-understanding benchmarks, architectures, compression levels, and existing diversity-based pruners.

  • Takeaways & Limitations

    Separating diversity geometry from raw-space distinctiveness preserves the importance signal while improving the representation of residual diversity.

Abstract

from arXiv · show

Large vision-language models (LVLMs) incur substantial inference costs due to their long and highly redundant visual-token sequences. Diversity-based pruning mitigates this cost by selecting token subsets based on pairwise cosine similarity. We find, however, that similarities between raw visual tokens are strongly concentrated in the positive range, limiting their ability to distinguish non-redundant tokens. A natural way to improve this resolution is to center token features before computing cosine similarity. Centering indeed reveals a substantially richer pairwise structure, yet unexpectedly degrades pruning performance when used alone. We show that this apparent contradiction arises because the raw geometry does more than represent pairwise diversity: it also implicitly favors globally distinctive tokens, which tend to contain semantically informative content. Centering better resolves subset diversity but loses this useful token-wise preference, revealing that diversity and distinctiveness are entangled in the raw geometry. Based on this analysis, we propose the \textbf{Cen}tered Geometry \textbf{Prune}r (Cen-Prune), which measures subset diversity using centered cosine similarity while retaining raw-space distinctiveness as a complementary token-wise preference. This lightweight, plug-and-play correction leaves the underlying selection mechanism unchanged and incurs negligible computational overhead. Extensive experiments across multiple image- and video-understanding benchmarks and LVLM architectures demonstrate that Cen-Prune provides robust improvements in overall performance across existing diversity-based pruners.

1 Introduction

LVLMs process long, redundant visual-token sequences, making visual tokens a major source of inference latency and memory costs. The paper identifies a tension between raw geometry’s implicit preference for distinctive tokens and centered geometry’s improved resolution of diversity, motivating their combination.

  • Dynamic-resolution images and multi-frame videos expand visual-token sequences, making them a substantial part of LVLM inputs.
  • Because attention cost grows quadratically with sequence length, visual tokens become a major source of latency and memory consumption.
  • Diversity-based pruning retains mutually dissimilar tokens as a training-free way to remove redundant visual tokens with limited performance degradation.
  • Raw visual-token similarities concentrate in the positive range, limiting pairwise resolution while implicitly favoring globally distinctive, often semantically important tokens.
  • Centering broadens pairwise structure and exposes residual diversity, but applying existing selectors to centered features reduces downstream performance.
  • The resulting tension is that raw geometry preserves token importance but poorly resolves diversity, whereas centered geometry resolves diversity but loses useful importance preference.
  • Cen-Prune combines centered-geometry diversity with raw-space distinctiveness while preserving the original selection procedure and adding negligible computational overhead.

2 Related Works

Visual token pruning methods use importance or diversity signals to select compact visual subsets. The paper emphasizes that diversity and importance are entangled because raw similarity geometry can favor distinctive, informative tokens even without an explicit importance term.

  • Visual token pruning reduces LVLM inference cost by retaining a compact set of visual tokens, using token importance and subset diversity as complementary principles.
  • Importance-based methods estimate token contribution from attention, instruction relevance, or semantic coverage before removing or merging redundant tokens.
  • Diversity-based methods retain complementary subsets through relationships among tokens, including clustering, max–min diversity, or determinantal objectives.
  • Even methods without explicit importance terms can favor tokens distinctive relative to the image, which are often visually distinctive and informative.
  • Prior work has not clearly separated this token-wise preference from subset-level diversity, motivating the paper’s analysis.

3 Preliminaries

The preliminaries formulate visual token pruning as selecting a small subset under cosine-based diversity objectives. They introduce max–min and determinantal selection and motivate examining the encoder geometry shared by both.

  • Visual Token Pruning Problem: The paper formulates pruning as selecting n ≪ N visual tokens whose retention minimizes downstream performance degradation.
  • Visual Token Pruning Problem: Each token is represented in normalized form, while its centered representation subtracts the mean token of the corresponding image.
  • Diversity-Based Selection: Diversity-based selection maximizes pairwise dissimilarity so the retained subset preserves non-redundant visual information.
  • Max–Min Diversity: Max–min diversity selects tokens by maximizing their minimum pairwise distance, typically through greedy farthest-point selection.
  • Geometry Intuition: Figure 1 contrasts raw-space max–min selection, centered residual directions, and distinctiveness-aware selection of a token that is both diverse and distinctive.
  • Determinantal Diversity: Determinantal diversity favors subsets spanning large volume and can incorporate token quality through a quality score.
  • Diversity-Based Selection: Both selection mechanisms rely on cosine geometry induced by the similarity matrix, motivating analysis of whether encoder features provide suitable diversity geometry.

4 Geometry Analysis and Cen-Prune

Raw visual-token geometry is dominated by positive similarities, so existing diversity selectors mostly remove near-duplicates without forming broadly dissimilar subsets. Centering exposes residual angular diversity but loses token distinctiveness; Cen-Prune separates these roles by combining centered subset diversity with raw-space distinctiveness.

  • Visual Similarity Is Highly Concentrated: Raw visual-token similarities are concentrated in a narrow, predominantly positive range, limiting separation of redundant and non-redundant tokens.The analysis uses visual tokens from LLaVA-1.5-7B on 1,000 COCO val2017 images.
  • Selectors Only Deduplicate: DPP similarly fails to favor negative correlations, producing only 2.5% negative-cosine pairs and leaving the underlying similarity geometry as the primary bottleneck.For two unit vectors, the Gram determinant 1 − cos^2 θ is insensitive to the sign of pairwise similarity.
  • Mean Centering for Pure Diversification: Centering removes the image-wise mean feature, broadens pairwise similarities into the negative range, and exposes residual directions for stronger diversification.After centering, 32% of all token pairs have cosine similarity below −0.5, with the distribution peaking near −0.72.
  • Mean Centering for Pure Diversification: Centered selection favors opposing residual directions more strongly as budgets tighten, yet underperforms raw selection because separation alone does not identify which tokens should be retained.Under DivPrune, negative-cosine pairs rise from 10.0% to 51.7% for centered selection as n decreases from 128 to 8, versus 2.1% to 18.8% for raw selection.
  • Method: Cen-Prune: Cen-Prune measures subset diversity with centered cosine geometry while preserving raw-space angular distinctiveness as a soft token-selection preference, without changing the selector.The method removes the shared direction to build the selection matrix and retains raw geometry to compute token distinctiveness.

5 Experiments

Experiments across image and video LVLMs show that Cen-Prune improves diversity-based pruning across architectures, compression levels, and objectives while adding little computational overhead.

  • Experimental Setup: Cen-Prune is evaluated on LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, and Qwen2.5-VL using image and video benchmarks.The image evaluation covers 11 multimodal benchmarks, while video evaluation uses four video-understanding benchmarks.
  • Image-Language Understanding: 93.4% and 93.6% performance are achieved with 32 retained tokens on LLaVA-1.5-7B using Cen-Prune with DivPrune and ZOO-Prune.These results match the query-aware MMTok result of 93.4%.
  • Image-Language Understanding: 88.9% performance is retained at 90% token reduction on Qwen2.5-VL-7B, versus 85.1% for DivPrune, 83.2% for MMTok, and 79.5% for CDPruner.The result is reported for dynamic-resolution inputs.
  • Video-Language Understanding: 92.2% performance is retained at 90.5% token reduction on LLaVA-Video-7B, exceeding DivPrune at 87.9% and CDPruner at 85.8%.Cen-Prune improves overall performance across four video-understanding benchmarks.
  • Component Ablation: Ablations show that centering alone can reduce performance, distinctiveness scoring alone improves it, and combining both is best in concentrated embedding spaces.The complementary effects are observed across LLaVA-1.5-7B and Qwen2.5-VL-7B.
  • Disentangling Diversity and Distinctiveness: Cen-Prune restores comparable [CLS] attention mass while preserving lower mean cosine similarity and more negative pairs than the raw selector.The reported comparisons are 0.311 vs. 0.312 attention mass, 0.306 vs. 0.414 mean cosine similarity, and 13.4% vs. 7.1% negative pairs.
  • Efficiency Analysis: With 64 retained tokens, Cen-Prune yields a 1.99× prefill speedup, a 1.67× end-to-end speedup, and a 76.3% FLOPs reduction.Its computational cost remains close to DivPrune and CDPruner.

6 Conclusion

The paper concludes that raw visual-token geometry entangles subset diversity with token-wise distinctiveness, motivating Cen-Prune’s complementary use of centered and raw geometries.

  • Conclusion: Raw cosine geometry provides limited diversity resolution while implicitly favoring globally distinctive tokens.Centering resolves residual diversity but weakens this importance preference.
  • Conclusion: Cen-Prune measures subset diversity with centered geometry while retaining raw-space distinctiveness for token selection.The method is training-free, preserves the underlying selector, and adds little computational overhead.
  • Conclusion: Experiments across LVLM architectures and image and video benchmarks demonstrate robust improvements.The paper presents complementary geometries as a general approach for diversity-based pruning.

Supplementary Material

The supplementary material provides additional empirical analyses, implementation details, benchmark results, ablations, efficiency analyses, and qualitative visualizations.

  • Supplementary Organization: Section A provides details of the empirical analysis.
  • Supplementary Organization: Section B presents additional implementation details for benchmark evaluations.
  • Supplementary Organization: Section C reports larger-scale model comparisons, additional ablations, and further efficiency analysis.
  • Supplementary Organization: Section D presents qualitative visualizations of DivPrune and Cen-Prune.

A.1 Data and Protocol for Geometry Analysis

The geometry analysis uses fixed visual-token probes and evaluates how feature-space concentration changes across model components. It motivates backbone-specific centering or standardized centering before constructing diversity geometry.

  • Protocol: The analysis uses a frozen set of 1,000 COCO val2017 probe images shared across backbones and feature spaces.Statistics are computed per image and summarized by unweighted means and standard deviations across the probe set.
  • Geometry across backbones: In LLaVA-1.5, the projector raises mean cosine from 0.309 to 0.366, lowers negative-pair fraction from 3.57% to 2.27%, and reduces effective rank from 21.79 to 13.20.The post-projector representation therefore provides less directional contrast than the pre-projector representation.
  • Geometry across backbones: Qwen2.5-VL’s merger-aligned encoder representation has mean cosine 0.993, no negative pairs, and effective rank 1.06, while merging lowers mean cosine to 0.300 and raises effective rank to 31.31.The two backbones therefore exhibit opposite geometry changes across their visual-processing stages.
  • Feature-space selection: Cen-Prune is applied after the LLaVA projector and before the Qwen visual merger, targeting the most concentrated feature spaces consumed by each pruning pipeline.The selected feature spaces are backbone-specific rather than universally fixed.
  • Coordinate conditioning: Qwen’s plain-centered geometry is ill-conditioned: 74.6% of pairs have |cos(x_i, x_j)| > 0.9 because the top dimension explains 98.9% of centered variance.LLaVA’s corresponding top-one and top-ten variance shares are 0.28% and 1.86%, motivating different centering treatments.
  • Coordinate conditioning: Standardized centering reduces Qwen’s endpoint-concentrated pairs from 74.6% to 0.008% and increases effective rank from 1.73 to 53.65, motivating plain centering for LLaVA and standardized centering for Qwen.Both variants remove the image-wise mean before constructing the pairwise diversity matrix.

B.2 Datasets

The evaluation covers broad image and video understanding settings while integrating Cen-Prune into existing diversity-based selectors. Experiments use frozen model weights and backbone-specific pruning protocols.

  • Image datasets: Image evaluation spans 11 benchmarks covering VQA, compositional reasoning, OCR, hallucination, scientific reasoning, assistive VQA, and expert-level multimodal understanding.The benchmark suite is intended to test diverse visual and reasoning abilities.
  • Video datasets: Video evaluation uses four benchmarks covering temporal reasoning, long-form understanding, egocentric perception, and objective video-based multimodal evaluation.These benchmarks test whether pruning preserves temporal video comprehension.
  • Evaluation protocol: Cen-Prune is applied after the multimodal projector in LLaVA models and before the visual merger in Qwen2.5-VL.The placement follows the geometry analysis for each architecture.
  • Pruning protocol: For LLaVA-NeXT, pruning preserves the same token proportion in each crop, including inputs with up to 2,880 visual tokens.Qwen2.5-VL is evaluated at 80%, 90%, and 95% pruning ratios under dynamic resolution.
  • Pruner integration: The integrations replace the original diversity geometry with centered cosine similarity and multiply selection scores by normalized raw-space distinctiveness.This design is implemented for DivPrune, ZOO-Prune, and CDPruner.

C.1 Additional Quantitative Results

Additional experiments show that Cen-Prune’s gains persist across compression settings, comparison protocols, and larger model scales. The reported improvements remain robust against diversity-based baselines.

  • LLaVA-NeXT-7B: 0.7% to 2.0%: Cen-Prune improves overall relative performance retention across compression ratios on LLaVA-NeXT-7B compared with diversity-based pruners.The improvement holds across the different budget settings reported in the complete results.
  • Comparison with HoloV: Under a fixed 320-token budget and a protocol aligned for comparison with HoloV, Cen-Prune maintains a robust overall gain and surpasses HoloV.The comparison uses the high-resolution LLaVA-NeXT setting.
  • Larger-scale models: On LLaVA-NeXT-13B and LLaVA-1.5-13B, Cen-Prune still delivers robust overall improvement against diversity-based pruners.These additional results extend the comparison beyond the 7B-scale models reported in the main paper.

C.2 Additional Ablation Studies

Ablations examine how centering, distinctiveness scoring, normalization, and efficiency affect Cen-Prune. The results support preserving relative variation, using raw cosine distinctiveness, and retaining the lightweight design.

  • Centering strategy: Mean-direction projection can discard token-specific variation, whereas mean centering removes the shared offset while preserving relative variation among visual tokens.This comparison supports mean centering as the preferred correction in the ablation.
  • Distinctiveness score: Raw-ℓ2 distinctiveness improves some benchmarks, especially POPE, but causes a noticeable drop on TextVQA; [CLS] attention is less compatible with residual diversity.The ablation favors the cosine-based distinctiveness score for the overall objective.
  • Score normalization: Z-score normalization with a sigmoid performs comparably to min–max normalization, with only minor benchmark differences.Min–max normalization is retained for its simplicity and slightly stronger average performance.
  • Efficiency: Across all token budgets, integrating Cen-Prune introduces negligible additional computational overhead.Small runtime differences fall within normal measurement variation and do not indicate a consistent change.

D Visualizations

The visualizations compare Cen-Prune with DivPrune on retained visual cues for small objects and text-oriented images. Cen-Prune retains more task-relevant information under low-budget settings, while DivPrune can lose textual integrity even with 128 tokens.

  • Small-object coverage: With only 32 retained visual tokens, Cen-Prune often covers small targets such as bicycles and trucks better than DivPrune.The comparison uses green outlines for correct answers and red outlines for incorrect answers.
  • Text-oriented Q&A: On text-oriented images, DivPrune struggles to preserve textual content even with 128 retained visual tokens.This can lead to incomplete text recognition.
  • Evaluation views: The visual comparisons evaluate retained-token quality through small-object coverage and text-oriented question answering.Figure A2 covers POPE small-object cases, while Figure A3 covers TextVQA text-oriented images.
Loading 2608.30263v1…