PaperlayerView exact revision on arXiv

Source-linked AI summary

AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models

Changwoo Baek, Jouwon Song, Sohyeon Kim, Kyeongbo Kong

arXiv:2603.01236v1cs.CVcs.LG

TL;DR

LVLM visual-token pruning reduces the quadratic cost of long visual-token sequences, but prior attention-based and diversity-based strategies were not systematically characterized. The paper uses effective rank and attention entropy to analyze their behavior, then develops image-aware and adaptive pruning strategies. It finds that pruning preferences depend on image complexity and that the resulting adjustments improve performance across benchmarks while also addressing hallucination tendencies.

  • Problem

    The paper addresses the lack of systematic evidence about how pruning methods preserve feature diversity, influence hallucinations, and perform across different image complexities.

  • Method

    The study combines erank-based diversity analysis, attention-entropy analysis, hallucination evaluation, and image-aware adjustments to existing and adaptive pruning strategies.

  • Results

    Attention-based pruning favors simple images and suppresses hallucinations, diversity-based pruning favors complex images, and image-aware adaptive strategies consistently improve performance across benchmarks.

  • Takeaways & Limitations

    The findings provide an actionable, model-agnostic basis for choosing or adapting pruning strategies according to image complexity.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have adopted visual token pruning strategies to mitigate substantial computational overhead incurred by extensive visual token sequences. While prior works primarily focus on either attention-based or diversity-based pruning methods, in-depth analysis of these approaches' characteristics and limitations remains largely unexplored. In this work, we conduct thorough empirical analysis using effective rank (erank) as a measure of feature diversity and attention score entropy to investigate visual token processing mechanisms and analyze the strengths and weaknesses of each approach. Our analysis reveals two insights: (1) Our erank-based quantitative analysis shows that many diversity-oriented pruning methods preserve substantially less feature diversity than intended; moreover, analysis using the CHAIR dataset reveals that the diversity they do retain is closely tied to increased hallucination frequency compared to attention-based pruning. (2) We further observe that attention-based approaches are more effective on simple images where visual evidence is concentrated, while diversity-based methods better handle complex images with distributed features. Building on these empirical insights, we show that incorporating image-aware adjustments into existing hybrid pruning strategies consistently improves their performance. We also provide a minimal instantiation of our empirical findings through a simple adaptive pruning mechanism, which achieves strong and reliable performance across standard benchmarks as well as hallucination-specific evaluations. Our project page available at https://cvsp-lab.github.io/AgilePruner.

1 INTRODUCTION

This section identifies unresolved questions about how visual token-pruning strategies preserve diversity, relate to hallucinations, and vary with image complexity. The study analyzes these behaviors and translates them into image-aware pruning improvements and an adaptive procedure.

  • LVLM visual-token sequences create quadratic attention costs that motivate pruning unnecessary or redundant tokens.
  • Existing pruning methods use attention scores, feature similarity, or hybrids, but their retained diversity, hallucination effects, and image-dependent preferences remain insufficiently characterized.
  • The study quantifies retained diversity with effective rank and examines hallucination patterns and pruning effectiveness across image complexities.
  • Many diversity-aware methods preserve less diversity than intended, while higher retained diversity is strongly associated with increased CHAIR hallucination frequency; attention-based pruning yields more conservative outputs.
  • Attention-based pruning works better for simple images with concentrated cues, whereas diversity-based pruning excels on complex images with distributed semantic information.
  • Image-aware adjustments consistently improve existing pruning strategies, and a minimal adaptive procedure performs strongly across nine datasets, CHAIR, and multiple LVLM architectures.

2 RELATED WORKS

Prior work reduces LVLM computation by pruning visual tokens, primarily through attention-based selection or diversity-based redundancy reduction. These approaches differ in whether they prioritize concentrated salient information or broader feature diversity.

  • LVLMs convert images into visual tokens aligned with language-model embeddings for multimodal reasoning.
  • Visual-token reduction decreases computation and memory usage, improving inference efficiency without requiring additional training.
  • Attention-based methods select tokens using vision-encoder attention scores, exploiting concentration on a small set of key tokens but risking regional redundancy and insufficient diversity.
  • Diversity-based methods use inter-token similarity to reduce redundancy and enhance the diversity of selected tokens.

3 PRELIMINARIES

The paper evaluates visual-token processing through attention concentration and embedding diversity. Attention entropy measures how broadly attention is distributed, while effective rank measures how many embedding dimensions are effectively used.

  • Visual-token pruning retains K ≪ N tokens through a selection function and forms the basis for analyzing attention concentration and embedding diversity.
  • Attention entropy is computed from renormalized, head-averaged class-token attention after excluding the class token’s self-attention.
  • Lower attention entropy indicates concentration on a few regions, whereas higher entropy indicates more uniform distribution across visual tokens.
  • Effective rank is an entropy-based measure of the number of dimensions effectively utilized by a token-embedding matrix.
  • The erank computation derives singular values from the embedding matrix and yields values between 1 and L.
  • Low erank indicates concentration in a few dominant dimensions, while high erank indicates more even distribution across dimensions.

4 EMPIRICAL STUDIES

The empirical studies characterize how pruning strategies preserve diversity, relate to hallucination, and vary with image complexity. These findings motivate adaptive, image-aware pruning that improves hybrid strategies and remains effective under aggressive token reduction.

  • Analysis Overview: The study measures retained visual-feature diversity with effective rank and examines its relationship with hallucination across pruning paradigms.The analysis focuses on token sets selected by existing methods and connects their diversity properties to hallucination behavior.
  • Diversity Preservation: DivPrune records the highest mean erank at 21.84, while PruMerge+ records the lowest at 10.91 on retained 64-token sets from POPE.The comparison shows substantial variation in preserved diversity across pruning approaches.
  • Diversity and Hallucination: Higher retained diversity is associated with more hallucinations, whereas attention-based pruning produces more conservative outputs with suppressed hallucinations.On CHAIR, diversity-based methods have higher hallucination metrics CS and CI but higher recall than attention-based methods.
  • Diversity and Hallucination: Diversity-based pruning yields broader descriptions and captures more ground-truth objects, but it also introduces speculative expressions, hallucinated objects, and incorrect phrases.FasterVLM instead focuses on main objects and produces more conservative descriptions.
  • Diversity and Hallucination: Increasing the attention-based selection ratio at a fixed 64-token budget gradually decreases CHAIR hallucination metrics CS and CI, while reducing recall.This exposes a trade-off between conservative token selection and object coverage.
  • Image Complexity: Attention-based pruning performs better on simple images with concentrated evidence, while diversity-based pruning performs better on complex images with distributed semantic information.Simple images have lower attention entropy and erank, whereas complex images have higher values; OCR has entropy 4.61 and erank 78, compared with POPE at 4.87 and 106.
  • Adaptive Pruning: Image-aware adjustments consistently improve existing hybrid and mixed pruning strategies across token budgets and benchmarks.The adaptive rule assigns more diversity to high-erank complex images and avoids the performance drop observed with inverse adaptation.
  • Adaptive Pruning: The adaptive threshold selects tokens in descending attention order, removes similar neighbors, and adjusts token diversity according to image complexity.A low threshold preserves focused tokens, whereas a high threshold removes more similar tokens and increases diversity; the dynamic threshold is capped by τmax.

5 EXPERIMENTS

Experiments show that adaptive thresholding balances attention and diversity across image types, preserving accuracy under aggressive pruning while reducing hallucination tendencies.

  • Experimental Setup: The method is evaluated on LLaVA-1.5-7B against attention-based, vision-encoder, and diversity-based pruning baselines across nine multimodal benchmarks.Experiments use fixed pretrained weights and temperature 0 for deterministic evaluation.
  • Main Results: With 64 tokens, the method incurs only a 3.24% performance drop, while attention-based methods suffer more than 25% degradation.It also exceeds VisionZip and DivPrune by 2.2% and 1.74%, respectively.
  • Main Results: Adaptive thresholding provides a principled alternative to fixed or non-adaptive pruning by balancing information preservation and diversity.The conclusion follows from the empirical analysis of pruning tendencies and the method’s benchmark behavior.
  • Hallucination Analysis: Diversity-based methods generally achieve higher CHAIR hallucination scores with higher recall, whereas attention-based methods show the opposite trade-off.CHAIR evaluation reports both CS and CI for hallucination analysis.
  • Analysis: Diversity-oriented pruning can weaken overall benchmark performance by missing concentrated reliable information, while attention-based pruning struggles with questions involving multiple objects.The two strategies therefore exhibit complementary weaknesses.
  • Hallucination Analysis: The adaptive method achieves 52.2 CS, 15.9 CI, and 75.7 recall, close to full-token values.These CHAIR results support balancing attention and diversity according to image characteristics.

6 CONCLUSION

The paper concludes that pruning behavior depends on retained diversity and image complexity, and that image-aware adaptive adjustments improve pruning across settings.

  • Conclusion: The study characterizes how retained feature diversity relates to hallucination tendencies and how image complexity shifts pruning effectiveness.These patterns are analyzed using effective rank and attention entropy.
  • Conclusion: Simple images favor attention-based selection, whereas complex images benefit from diversity-based retention.The conclusion presents this as a consistent behavioral pattern across pruning approaches.
  • Conclusion: Image-aware adjustments to hybrid and mixed pruning methods yield consistent benchmark improvements.The paper describes these principles as broadly applicable and model-agnostic.
  • Conclusion: The adaptive thresholding procedure provides a minimal instantiation that achieves strong performance and reduces hallucination.The supplied conclusion passage ends before giving further quantitative detail.

A EFFICIENCY ANALYSIS

The efficiency analysis makes erank practical through a covariance-based formulation and shows that adaptive pruning substantially reduces computation while retaining performance.

  • Metric Computation: Erank measures feature-matrix representational complexity from the distribution of its singular spectrum.The implementation uses an N × N covariance matrix when the token count is much smaller than the feature dimension.
  • Metric Computation: The covariance formulation reduces complexity to O(N^2D + N^3) when N ≪ D, compared with O(ND^2) for naive SVD.This exploits the smaller covariance matrix C = XX^⊤.
  • Runtime: Erank computation takes 3.4 ms per image versus 115 ms for full inference, accounting for approximately 3.2% of total inference time.The cost scales approximately linearly with batch size because samples are processed independently.
  • Efficiency–Performance Trade-off: At 64 tokens, the method reduces FLOPs by 89% while preserving 96.2% of vanilla LLaVA-1.5-7B performance.It outperforms SparseVLM and PyramidDrop in accuracy under this efficiency comparison.
  • Efficiency–Performance Trade-off: Pre-pruning reduces computation across all subsequent language-model layers, yielding savings in FLOPs, memory, and latency.The method is also compatible with FlashAttention for further efficiency gains.

B.1 EVALUATION ON OTHERS MODEL

Additional analyses confirm complexity-dependent pruning preferences, show that threshold increases raise retained diversity, and demonstrate stable performance across models and token budgets.

  • Cross-Model Evaluation: Across LLaVA-1.5-13B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B, the method demonstrates stable and strong performance.The evaluation extends beyond the primary LLaVA-1.5-7B setting.
  • Additional Analysis: Attention entropy and erank have a Pearson correlation of 0.63 across MME, indicating related measures of visual information dispersion.Entropy reflects attention-weight spread, while erank additionally captures feature-space geometry.
  • Additional Analysis: Entropy-based threshold adaptation produces performance trends closely aligned with erank-based adaptation, with variations of about 0.13.This supports entropy as an alternative signal for adaptive thresholding.
  • Threshold Analysis: As similarity threshold τ increases, the erank of selected tokens consistently rises across datasets.Higher τ removes more similar tokens and produces a more diverse final set.
  • Threshold Analysis: Larger τ improves performance for complex images but can remove fine-grained important details from simple images.The effect of threshold therefore depends on image complexity.
  • Adaptive Token Counts: High-erank images benefit from retaining more diverse tokens, whereas low-erank images are better represented by fewer focused tokens.This motivates adaptive token counts in addition to adaptive thresholds.

D ATTENTION ENTROPY AND ERANK SCALE

Entropy varies within a narrow range, whereas erank spans a much broader scale, making the two measures sensitive to different feature-structure variations.

  • Mean entropy is 4.80, with values predominantly concentrated between approximately 4.0 and 5.4.The median is 4.78, with Q1 at 4.63 and Q3 at 4.96.
  • Mean erank is 94.87, spanning a substantially broader distribution than entropy.The median is 95.40, with Q1 at 81.59 and Q3 at 108.80.
  • Erank captures coarse-grained variation in token correlation structure and spectral dispersion, complementing entropy’s fine-grained sensitivity.

E ROBUSTNESS ANALYSIS OF ERANK UNDER INPUT CORRUPTIONS

Erank remains stable under common image corruptions, while qualitative reasoning examples illustrate complementary strengths and failure modes of attention-, diversity-, and adaptive pruning.

  • Robustness of erank: Erank was evaluated under 15 COCO-C corruption types across 2,374 MME images at severities 1 and 3.The analysis compared corrupted-image erank values with clean-image values.
  • Robustness of erank: Mean absolute erank deviation was 2.78 at severity 1 and 4.11 at severity 3, a small fraction of natural variation.Clean MME erank ranged from 44.08 to 147.49, with standard deviation 19.46.
  • Robustness of erank: Global spatial corruptions caused moderately larger erank deviations, whereas local appearance distortions had minimal effects.Zoom blur, frost, snow, and elastic transform typically changed erank by 4–7 points, while brightness, pixelation, and JPEG compression changed it by 1–2.5 points.
  • Qualitative reasoning behavior: The adaptive method retained tokens needed to detect small objects and showed robust behavior across diverse reasoning tasks.The evidence includes Existence-task examples and quantitative and qualitative evaluations.
  • Qualitative reasoning behavior: Diversity-based pruning preserves broader spatial cues but can hallucinate, attention-based pruning is stable but narrow, and adaptive pruning balances both mechanisms.These tendencies appear in counting and spatial reasoning examples.
  • Qualitative reasoning behavior: Adaptive pruning succeeds or fails according to whether image complexity and evidence distribution align with its token-selection behavior.Examples include concentrated and distributed semantic cues across fine-grained reasoning cases.

F.2 FAILURE MODES OF THE ADAPTIVE METHOD IN FINE-GRAINED REASONING ACROSS IMAGE COMPLEXITY

The adaptive method changes its pruning emphasis with image complexity, succeeding when selected token coverage matches evidence distribution but failing when crucial cues are mismatched.

  • Low-erank images with many objects: On low-erank images, adaptive pruning emphasizes high-attention tokens and resembles attention-based pruning.
  • Low-erank images with many objects: When few objects are localized, this focused selection supports accurate counting and spatial reasoning.
  • Low-erank images with many objects: When many objects are spread across a low-erank image, concentrated selection misses broader layout information, causing counting errors and incomplete reasoning.
  • High-erank images with locally concentrated key evidence: On high-erank images, adaptive pruning resembles diversity-based pruning and selects more widely dispersed tokens.
  • High-erank images with locally concentrated key evidence: Dispersed selection supports accurate counting and relational reasoning when objects or semantic cues are broadly distributed.
  • High-erank images with locally concentrated key evidence: When crucial evidence is localized in a high-erank scene, dispersed selection dilutes attention around it and causes failures in counting and spatial inference.

G MORE EXAMPLES ON CHAIR

Additional CHAIR examples contrast broader but more hallucination-prone diversity-based captions with attention-based captions, while increasing attention-selected tokens suppresses hallucinations.

  • CHAIR qualitative comparisons: DivPrune often generates broader, more descriptive captions but introduces hallucinated objects compared with FasterVLM.
  • Attention-diversity ratio experiment: With a fixed 64-token budget, hallucinated objects diminish as the DivPrune-to-Attention ratio R increases and disappear when R ≥50%.Response style shifts from speculative and exploratory toward more factual descriptions.
  • CHAIR qualitative comparisons: The CHAIR annotations distinguish ground-truth and hallucinated object words, with incorrect phrases marked in red.
Loading 2603.01236v1…