PaperlayerView exact revision on arXiv

Source-linked AI summary

E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models

Taoyu Qian, Qi Wang, Daqian Shi, Yuanhao Jiang, Shang Gao, Hualong Yu

arXiv:2608.23253v1cs.CVcs.AI

TL;DR

Long visual-token sequences make VLM inference costly, and direct attention aggregation does not adequately represent uncertainty or conflict across heads and layers. E2S-Pruner progressively fuses this evidence with D–S theory and spatial novelty, retaining strong aggregate performance while improving throughput across model families. Its main boundaries are higher peak memory and weaker TextVQA performance under extreme compression.

  • Problem

    Existing visual-token pruning relies heavily on attention scores and direct aggregation across heads and layers, making uncertainty, complementary evidence, and conflict difficult to represent.

  • Method

    E2S-Pruner estimates attention-head reliability, fuses three-state token evidence within layers, recursively fuses layers with D–S theory, and applies spatial novelty regularization.

  • Results

    On LLaVA-1.5-7B, E2S-Pruner retains 98.0%, 96.8%, and 90.6% of aggregate performance at 192, 128, and 64 visual tokens, with 1.96× and 2.09× baseline throughput at 128 and 64 tokens; Qwen2-VL-7B results show cross-model generalization.

  • Takeaways & Limitations

    The method provides a performance–efficiency trade-off for visual-token compression while modeling conflicting evidence and promoting coverage of distinct image regions.

  • Takeaways & Limitations

    Peak GPU memory remains approximately 21% above the dense baseline, and at 64 tokens TextVQA scores 49.5 versus 54.0 for LLaVA-PruMerge.

Abstract

from arXiv · show

Vision-language models typically encode an image into hundreds of visual tokens, incurring substantial inference latency and GPU memory overhead. Existing pruning methods largely rely on attention scores and directly aggregate outputs across attention heads and network layers, making it difficult to characterize evidential uncertainty and conflict. We propose E2S-Pruner, a progressive two-stage evidence-fusion framework for visual token pruning that requires no auxiliary model, trainable parameters, or fine-tuning. In the first stage, E2S-Pruner treats each attention head as an independent evidence source, estimates its reliability from evidence clarity and inter-head consistency, and represents each visual token using three states: important, unimportant, and uncertain. In the second stage, Dempster--Shafer evidence theory is used to quantify inter-layer conflict and fuse complementary evidence from multiple network layers. We further introduce a spatial novelty constraint that promotes coverage of distinct image regions and prevents the retained tokens from concentrating in a few locally salient areas. On LLaVA-1.5-7B, E2S-Pruner retains 98.0%, 96.8%, and 90.6% of the aggregate performance when the average numbers of retained visual tokens are 192, 128, and 64, respectively, while improving throughput by 1.96x and 2.09x under the 128-token and 64-token settings. Experiments on Qwen2-VL-7B further demonstrate cross-model generalization. Code is available at https://github.com/taoyu-qian/E2S-Pruner.git.

1. Introduction

VLM deployment is constrained by the computation and memory costs of long visual-token sequences, while direct attention aggregation can obscure complementary or conflicting evidence. E2S-Pruner addresses this with progressive D–S evidence fusion and spatially diverse token selection.

  • VLMs require substantial computation and parameters to represent diverse real-world information, limiting deployment in resource-constrained edge environments.
  • Attention-based pruning dominates existing compression methods, but direct aggregation across heads and layers can weaken tokens supported by complementary evidence.
  • E2S-Pruner fuses attention-head evidence into important, unimportant, and uncertain states, then recursively fuses layer-wise evidence while quantifying conflict.
  • E2S-Pruner defines head reliability from evidence clarity and inter-head agreement, then uses belief and plausibility to prioritize visual-token retention.
  • A spatial novelty constraint assigns a soft bonus to the highest-priority token in each spatial cell, improving coverage and preventing concentration in a small image region.

2. Related Work

Visual-token compression methods use merging, attention saliency, representation dynamics, redundancy, or contextual dependence, but these signals can vary across heads and layers. E2S-Pruner combines evidence modeling with spatial coverage to target a more stable performance–efficiency trade-off.

  • Visual-token compression methods broadly use token merging or token pruning to reduce inference computation as visual context length increases.
  • Similarity-based merging can combine visually similar but semantically distinct regions, while LLaVA-PruMerge uses sparse class-to-patch attention and feature similarity.
  • Attention- and relevance-based methods include shallow one-shot pruning, partitioned token allocation, and text-related visual-token selection.
  • Other approaches retain complementary information through pivot-token redundancy, remove slowly changing tokens, or use contextual dependence and representation dynamics.
  • Evidence quality varies across attention heads and layers, so simple aggregation can obscure local details, complementary semantics, and high-conflict visual evidence.
  • E2S-Pruner jointly models multi-head and multi-layer information with D–S theory and adds a spatial constraint for regional coverage.

3. Preliminary

The preliminary material defines attention-based visual-token scoring and the D–S concepts used to represent, compare, and fuse uncertain evidence. Conventional pruning aggregates head and layer scores before retaining the top K tokens.

  • A VLM encodes images and text as visual and text tokens, and visual-token importance is measured using text-to-visual attention.
  • For each LLM layer and attention head, NT and NV denote text and visual-token counts, with NV much larger than NT.
  • The attention score uses text-token queries, visual-token keys, and head dimension d to measure visual-token importance.
  • Conventional pruning aggregates single-head scores across heads and layers, ranks visual tokens, and retains the top K.
  • D–S evidence theory represents possible outcomes in a frame of discernment and provides a mechanism for fusing multiple evidence sources.
  • Basic probability assignment distributes support over propositions or subsets, allowing evidence to remain uncertain about individual hypotheses.
  • The conflict degree sums products of masses assigned to mutually exclusive propositions, with values closer to 1 indicating stronger disagreement.
  • Dempster’s rule combines evidence sources, while belief and plausibility provide lower and upper bounds on support for a proposition.

4. Methodology

E2S-Pruner progressively fuses visual-token evidence within attention heads and across selected layers, then applies spatial novelty regularization before Top-K pruning.

  • Progressive pruning: The method performs three pruning operations in shallow, middle, and deep LLM layers, producing a coarse-to-fine compression schedule.Visual-token counts decrease stepwise as pruning progresses.
  • Attention-head fusion: Each attention head supplies evidence from text-to-visual attention, which is normalized and mapped to a support score for each visual token.Average normalized attention maps to a support score of 0.5, indicating no clear selection preference.
  • Attention-head fusion: Evidence reliability is the product of decision clarity and support consistency, so clear and mutually consistent head responses receive greater weight.The consistency term measures inter-head agreement through deviations from the mean score.
  • Attention-head fusion: D–S evidence assigns each token mass over important, unimportant, and uncertain states, representing disagreement among heads as uncertainty.This avoids premature pruning and supports subsequent multi-source fusion.
  • Layer-wise evidence fusion: Evidence is recursively fused across designated pruning layers, and plausibility rather than belief determines retention priority to protect tokens with insufficient evidence.The fusion order exploits shallow edge and texture cues, middle-layer object structure, and deep global semantics.
  • Spatial novelty regularization: Spatial novelty adds a soft bonus to the highest-priority candidate in each occupied spatial cell before global Top-K selection.The constraint improves regional coverage without altering the evidence-fusion result and does not require every cell to retain a token.

5. Experiments and Analysis

Experiments evaluate E2S-Pruner under matched average-token budgets on LLaVA-1.5-7B and in cross-model tests on Qwen2-VL-7B using complementary multimodal benchmarks.

  • Evaluation protocol: Seven benchmarks evaluate pruning from complementary perspectives across LLaVA-1.5-7B and Qwen2-VL-7B.LLaVA-1.5-7B uses GQA, SQA, TextVQA, POPE, MME, and MMBench; Qwen2-VL-7B uses SQA, POPE, MME, and AI2D.
  • Pruning settings: Methods are compared under the same average-token budget despite differing pruning locations and token-removal schedules.The average number of retained tokens is used as the common comparison basis.
  • Pruning settings: The progressive strategy evaluates average-token budgets of 192, 128, and 64.LLaVA-1.5-7B progressively prunes after layers 3, 17, and 22.
  • Pruning settings: At those budgets, LLaVA-1.5-7B retains 194 →148 →96, 122 →48 →42, and 20 →8 →0 visual tokens across the three pruning operations.The sequences correspond respectively to average-token budgets of 192, 128, and 64.
  • Evaluation protocol: Performance is measured with official benchmark protocols and summarized using normalized performance retention relative to the dense baseline.Each numerator is the pruned-model score and its corresponding denominator is the dense-baseline score.

5.2. Comparative Experiment

E2S-Pruner achieves the highest aggregate performance among compared pruning methods across 192-, 128-, and 64-token budgets, while retaining strong multimodal capability under increasingly aggressive compression. Its main limitation appears on fine-grained TextVQA at 64 tokens.

  • 98.0%, 96.8%, and 90.6% of dense LLaVA-1.5-7B aggregate performance are retained at average budgets of 192, 128, and 64 visual tokens.E2S-Pruner achieves the highest aggregate performance at every evaluated budget.
  • At 192 tokens, E2S-Pruner exceeds SparseVLM and V2Drop by 2.1 and 0.4 percentage points in aggregate retention.It also performs best on SQA, TextVQA, and MMBench, although V2Drop is slightly better on POPE and MME.
  • At 128 tokens, E2S-Pruner ranks first on all six benchmarks and improves aggregate retention over V2Drop and SparseVLM by 2.8 and 3.6 percentage points.Its benchmark gains over V2Drop are 1.5, 1.1, 2.2, 2.2, 67, and 1.7 points on GQA, SQA, TextVQA, POPE, MME, and MMBench.
  • At 64 tokens, E2S-Pruner retains 90.6% of dense-model performance, exceeding V2Drop and SparseVLM by 3.7 and 4.1 percentage points.The gains over V2Drop are especially large on GQA and MME, at 5.0 and 186 points.
  • At 64 tokens, E2S-Pruner scores 49.5 on TextVQA, below LLaVA-PruMerge’s 54.0.The reported limitation is attributed to removing fine-grained text regions under severe compression.
  • Aggregate retention decreases from 98.0% at 192 tokens to 96.8% at 128 tokens and 90.6% at 64 tokens.The reported trend indicates slower degradation as pruning becomes more aggressive than for most competing methods.

5.3. Ablation Experiment

Ablations show that complete evidence reasoning and its supporting components improve aggregate retention and task stability at a 128-token budget. Spatial novelty and conflict protection specifically preserve complementary or regionally distributed evidence.

  • 97.7% aggregate performance is retained while processing approximately 22.2% of the original visual-token budget with the complete method.All ablation variants use identical models, datasets, prompts, protocols, and pruning locations.
  • Replacing evidence reasoning with attention-based Top-K selection reduces performance retention from 97.7% to 92.9%.The complete method improves SQA, TextVQA, MME, and MMBench by 5.0, 0.2, 87, and 4.3 points, respectively.
  • Removing head-reliability estimation reduces aggregate retention to 95.8%, 1.9 percentage points below the complete method.The reported decreases on SQA, MME, and MMBench are 1.9, 10, and 2.4 points.
  • Using only the current evidence layer reduces retention to 96.2%, while cross-layer fusion improves SQA, MME, and MMBench by 1.0, 16, and 2.1 points.The ablation attributes this to complementarity among shallow details, intermediate structures, and deep semantics.
  • Removing conflict protection reduces retention to 96.8%, while the complete method improves SQA and MMBench by 1.0 and 1.2 points.The reported interpretation is that inter-layer disagreement can still accompany important tokens.
  • Removing spatial novelty reduces retention to 96.3%, with MME and MMBench decreasing by 5 and 2.3 points.Spatial novelty improves regional coverage by counteracting concentration in a few high-response areas.

5.4. Analysis of Visual Token Retention Strategies

Under a fixed average budget of 128 visual tokens, E2S-Pruner is robust to the precise allocation across pruning stages, while different tasks favor different layer allocations. The selected default schedule balances aggregate performance, TextVQA accuracy, and inference efficiency.

  • Schedule robustness: All five retention schedules achieve 97.7%–97.8% aggregate performance under the same average budget of 128 visual tokens.The schedules differ only in token counts assigned to the three pruning stages.
  • Schedule robustness: The 576 →132 →40 →32 schedule achieves the highest aggregate retention at 97.8%, only 0.1 percentage points above the alternatives.This indicates robustness to the precise allocation of tokens across layers.
  • Task-specific allocation: The 576 →122 →48 →42 schedule gives the best TextVQA score of 56.0 when more tokens are retained in deep layers.The reported result links continued deep-layer visual participation with scene-text recognition.
  • Task-specific allocation: Allocating more tokens to intermediate layers increases MME from 1779 to 1785.SQA and MMBench vary only slightly across schedules.
  • Default schedule: The default schedule is 576 →122 →48 →42 because it preserves more deep-layer visual information, performs better on text-intensive tasks, and maintains strong overall efficiency.Its aggregate retention is 0.1 percentage points below the maximum.

5.5. Inference Efficiency Analysis

E2S-Pruner converts visual-token reduction into practical acceleration, achieving lower latency and higher throughput at 128- and 64-token budgets. The method preserves strong task performance but incurs additional GPU-memory overhead from evidence modeling and fusion.

  • 1.96× throughput at 128 tokens accompanies 48.5% lower end-to-end latency than the dense LLaVA-1.5-7B baseline.Throughput rises from 5.28 to 10.37 items/s, while end-to-end latency falls from 820.15 s to 422.28 s.
  • 2.09× throughput at 64 tokens accompanies 51.5% lower end-to-end latency while retaining 88.7% of MMBench performance.Throughput reaches 11.02 items/s under the 64-token budget.
  • 10.37 to 11.02 items/s: throughput increases as the average visual-token budget decreases from 128 to 64.The reported acceleration is not offset by the additional evidence computation.
  • 18,847 and 18,807 MB: peak GPU memory is approximately 21% above the dense baseline under the two evaluated budgets.The overhead mainly comes from intermediate states used for evidence modeling and cross-layer fusion.

5.6. Cross-Model Generalization Analysis

E2S-Pruner generalizes from LLaVA-1.5-7B to Qwen2-VL-7B across substantial visual-token reductions. It achieves the highest average aggregate-performance retention in both Qwen2-VL-7B settings while preserving balanced capabilities.

  • 98.9% aggregate-performance retention at a 66.7% reduction ratio is the highest among the compared Qwen2-VL-7B methods.This exceeds FastV, DART, and V2Drop by 4.8, 2.1, and 1.3 percentage points, respectively.
  • 83.8 SQA and 80.2 AI2D are the best scores achieved by E2S-Pruner at the 66.7% reduction ratio.Its aggregate result remains strongest despite slightly lower POPE and MME scores than the best individual methods.
  • 95.4% aggregate-performance retention at a 77.8% reduction ratio remains superior to all competing methods.The margins over FastV, DART, and V2Drop are 4.5, 1.5, and 0.5 percentage points.
  • 82.9 SQA at the 77.8% reduction ratio is 3.3 points above the second-best method, indicating preservation of information for complex visual reasoning.Aggregate performance remains superior even though E2S-Pruner does not lead POPE, MME, or AI2D individually.
  • Cross-model transfer to Qwen2-VL-7B preserves stable aggregate performance across reduction ratios.The results are described as demonstrating that E2S-Pruner is not tied to a particular architecture or model family.

6. Conclusion

E2S-Pruner combines progressive evidence fusion with spatial protection to reduce visual-token inference cost while preserving important and geographically diverse image information. Its remaining deployment boundaries are memory overhead and vulnerability to fine-grained textual information loss at extremely low budgets.

  • E2S-Pruner recursively fuses attention-head evidence within layers and information across pruning layers using D–S evidence theory.Conflict protection reduces the risk of removing potentially important tokens associated with strong disagreement.
  • Spatial novelty regularization promotes coverage of distinct image regions and prevents excessive concentration in locally salient areas.This constraint complements evidence-based token prioritization during selection.
  • 98.0%, 96.8%, and 90.6% aggregate-performance retention is achieved on LLaVA-1.5-7B at average budgets of 192, 128, and 64 visual tokens.At 128 and 64 tokens, end-to-end latency falls by 48.5% and 51.5%, while throughput reaches 1.96× and 2.09× the baseline.
  • Extremely low token budgets can lose fine-grained textual-region information, while evidence fusion and spatial novelty computation increase peak GPU memory.Future work targets lower-overhead caching and adaptive protection for text and fine-grained objects.
Loading 2608.23253v1…