Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

Zhengyao Fang; Pengyuan Lyu; Chengquan Zhang; Guangming Lu; Jun Yu; Wenjie Pei

Source-linked AI summary

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang, Guangming Lu, Jun Yu, Wenjie Pei

arXiv:2603.09480v2Mar 10, 2026cs.CV

TL;DR

VLMs generate excessive visual tokens, while existing compression methods struggle to preserve both semantic importance and information diversity. PruneSID uses semantic grouping, intra-group redundancy pruning, and image-adaptive compression budgets. It achieves strong image and video results under extreme token compression, while remaining less effective for fine-grained instruction-specific reasoning.

Problem
VLMs generate excessive visual tokens, and existing compression methods struggle to balance importance preservation with information diversity.
Method
PruneSID is a training-free framework combining PSCA for semantically coherent token grouping, intra-group NMS for redundancy pruning, and dynamic compression ratios based on image complexity.
Results
PruneSID achieves state-of-the-art results across image and video benchmarks, including 92.8% accuracy on LLaVA-NeXT and 95.5% on Video-LLaVA while retaining about 5% of visual tokens.
Takeaways & Limitations
The framework generalizes across image and video VLMs while preserving performance under high visual-token compression.
Takeaways & Limitations
The task-agnostic method may be less effective for fine-grained or instruction-specific reasoning under extreme compression because relevant tokens can be overlooked.

Abstract

from arXiv · show

Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8 $\times$ faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.

1 INTRODUCTION

VLMs generate more visual tokens than needed, creating computational inefficiency and a trade-off between preserving semantic importance and information diversity. PruneSID addresses this with semantic grouping, redundancy pruning, and image-adaptive token budgets, achieving strong results across image and video VLMs.

Motivation: LLaVA-1.5 and LLaVA-NeXT typically generate 576 and 2880 visual tokens per image, respectively, exceeding essential semantic-content requirements.These token sequences create substantial computational inefficiency during VLM processing.
Motivation: Attention-guided methods preserve salient regions but neglect contextual background, while duplication-aware methods may remove semantically critical high-attention regions.The two paradigms therefore expose complementary limitations in scene comprehension and semantic preservation.
Results: 92.8% accuracy is maintained on LLaVA-NeXT at 5.6% token retention, a 2.5 percentage point improvement over prior approaches.The method also reports new state-of-the-art results on Video-LLaVA with 6.6% token retention.
Approach: PruneSID combines PSCA-based semantic clustering with intra-group NMS to balance semantic importance and information diversity without training.Its contributions also include an information-aware dynamic compression ratio that allocates more tokens to complex images and compresses simpler scenes more strongly.

2 VISUAL TOKEN REDUCTION IN VLMS

Visual token reduction methods primarily compress tokens either within LLM layers or before LLM interfacing in the vision encoder. These approaches use attention, similarity, clustering, or adaptive selection, but represent distinct compression strategies.

LLM-stage compression: Training-free token-pruning methods reduce visual tokens within the layers of large language models to improve inference efficiency.The related-work discussion identifies this as one broad family of visual token reduction approaches.
Comparison: The literature therefore spans both LLM-stage pruning and early vision-encoder compression, with different mechanisms for selecting or grouping visual tokens.The supplied related-work passages distinguish these families by where compression occurs and how tokens are selected.
Vision-encoder compression: Vision-encoder compression methods reduce tokens before interfacing with the LLM, including adaptive attention-based selection and similarity clustering.LLaVA-PruMerge is described as selecting high-attention tokens and clustering them using key similarity.

3 OUR METHOD

PruneSID compresses visual tokens through semantic grouping followed by redundancy-aware pruning, balancing importance preservation with information diversity. Its dynamic token budget adapts compression to image complexity, while the overall objective retains informative, diverse representations for downstream VLM processing.

Overview: The method reduces X ∈ R^T×D to a compact representation with N ≪ T while preserving salient visual patterns and downstream information integrity.The input contains T visual token embeddings of dimension D, and the output contains N retained tokens.
Semantic-aware token grouping via PSCA: PSCA uses low-rank PCA over the token dimension to identify semantic directions and partition tokens into K coherent groups.Tokens are assigned according to their contribution to principal directions, producing groups with shared semantic information.
Intra-group redundancy removal via NMS: Intra-group NMS ranks tokens by principal-direction contribution and suppresses candidates whose similarity to selected tokens exceeds an adaptive threshold.The threshold is scaled from the image’s average pairwise token similarity, encouraging stronger suppression in more redundant images.
Intra-group redundancy removal via NMS: Group-wise quotas allocate the global token budget across filtered groups, and the selected top-n_k tokens are concatenated into the final compact set.Quotas are proportional to filtered group sizes and rounded to integers before concatenation.
Information-aware dynamic compression ratio: Fixed compression can lose information in complex scenes and retain redundancy in simple scenes, motivating an information-aware budget that varies with image content.The method uses an image information score derived from global redundancy; more informative images receive more tokens, while simpler images are compressed more aggressively.
Theoretical overview of PSCA–NMS pruning: The pruning objective maximizes retained semantic information while limiting pairwise redundancy, with PSCA and NMS addressing the two terms jointly.PSCA favors large semantic projections, whereas NMS enforces a similarity constraint to support diverse semantic coverage.

4 EXPERIMENTS

Experiments evaluate PRUNESID across multiple VLM architectures, image and video tasks, compression settings, grouping strategies, and dynamic token budgets. The method maintains strong performance under aggressive compression, while ablations and latency measurements examine its components, adaptability, and efficiency.

Main results on image understanding tasks: PRUNESID achieves approximately 96% average accuracy on LLaVA-1.5 while retaining only 64 of 576 visual tokens.This setting surpasses VisionZip by 1.9% across benchmarks.
Dynamic compression ratio: Dynamic compression consistently improves performance over fixed-token baselines under matched average token budgets, with benchmark effects varying across datasets.The strategy allocates token budgets according to image-level information scores and content complexity.
Main results on image understanding tasks: PRUNESID improves over VisionZip by 0.9%, 1.5%, and 2.5% on LLaVA-NeXT at retention ratios of 22.2%, 11.1%, and 5.6%, respectively.At 5.6% retention, it preserves 92.8% of full-model performance.
Cross-architecture and video generalization: PRUNESID generalizes to Mini-Gemini and Video-LLaVA, achieving 95.5% average accuracy on video benchmarks with only 6.6% token retention.For Video-LLaVA, 2048 tokens are reduced to 136 by compressing each frame from 256 to 17 tokens.
Ablation study: PSCA outperforms random and KMeans grouping across four benchmarks, supporting semantically coherent groups for more effective redundancy reduction.The number of groups shows a bell-shaped performance trend, with moderate K values balancing redundancy modeling and pruning stability.
Efficiency and limitation: At 5.6% retention, PRUNESID reduces prefilling time from 218ms to 27.8ms, a 7.8× improvement, while raising POPE F1 from 86.6% to 89.0% versus VisionZip.Overall inference time is 89ms per sample, and the method maintains similar efficiency to VisionZip with higher reported performance.

5 CONCLUSION

PRUNESID is a training-free, task-agnostic framework that combines semantic grouping, redundancy pruning, and image-adaptive token retention. It achieves strong results across image and video VLM benchmarks while retaining about 5% of visual tokens.

Framework: PRUNESID combines PSCA-based semantically coherent grouping with intra-group NMS to balance importance-aware selection and information diversity.The framework also adapts retained token counts according to image complexity.
Results: PRUNESID achieves state-of-the-art results across image and video VLM benchmarks while retaining ∼5% of visual tokens.The reported results span both image and video modalities.
Results: 92.8% accuracy is achieved on LLaVA-NeXT with ∼5% visual-token retention.
Results: 95.5% accuracy is achieved on Video-LLaVA with ∼5% visual-token retention.

A.1.1 DETAILED THEORETICAL ANALYSIS OF PSCA–NMS PRUNING

The theoretical analysis defines effective information as semantic contribution minus redundancy among retained tokens. PSCA targets semantic importance, while NMS limits overlap to preserve diverse information.

Effective information: Effective information combines the semantic information carried by retained tokens with penalties for overlapping semantic content.I(si) denotes token information, while R(si, sj) denotes shared semantic information between tokens.
PSCA: PSCA approximates semantic-information maximization by identifying dominant global semantic directions and ranking tokens by their projections.Tokens with larger projections onto principal directions are treated as having greater semantic contribution.
NMS: NMS reduces redundancy by suppressing tokens whose pairwise similarity exceeds threshold ϵ, promoting diverse semantic coverage.The procedure addresses similarity among spatially adjacent or visually similar tokens.
Joint optimization: Together, PSCA and NMS approximate maximizing effective information under the retained-token constraint.
Guarantee: 15 guarantees redundancy is upper-bounded by a constant proportional to ϵ while PSCA maximizes semantic contribution.
Efficiency: PruneSID maintains comparable efficiency to VisionZip while achieving better performance in the runtime comparison.

A.1.2 PSEUDOCODE OF OUR METHOD

The pseudocode implements PruneSID in two stages: PSCA groups visual tokens into semantically coherent sets, then intra-group NMS removes redundant tokens within the token budget.

Pipeline: The method first applies PSCA-based token grouping, then performs intra-group NMS for redundancy removal.This two-stage design targets semantic preservation and token diversity across compression ratios.
PSCA grouping: PSCA projects T visual tokens into K = floor(N / 4) components and assigns each token to the group with the largest absolute projection.The resulting groups are returned as semantically coherent token sets.
Intra-group NMS: NMS sorts tokens within each group by projection score and retains a token only when its similarity with every selected token is below τ.The threshold is set as τ = λ · ρ, with λ = N / 32.

A.1.3 RUNTIME COMPARISON AND EFFICIENCY ANALYSIS

PruneSID’s implementation accelerates execution through parallel processing and reduces prefilling latency while preserving a favorable speed–accuracy trade-off against VisionZip. Its reported preprocessing time is comparable to VisionZip’s, while downstream performance is higher.

Implementation: Parallel processing significantly accelerates the implementation of the two-stage compression pipeline.The released codebase contains the full implementation details.
Runtime: 22.4ms per sample is PruneSID’s processing time, compared with 17.4ms for VisionZip.The benchmark uses the same hardware and software configuration as the main paper’s Efficient Analysis section.
Prefilling latency: 218ms to 27.8ms is the reported reduction in total prefilling latency from early compression.
Accuracy: 2.4% is PruneSID’s performance advantage over VisionZip on POPE under this configuration.The method also achieves 92.8% average accuracy across multiple benchmarks.

A.2 ADDITIONAL EXPERIMENTS

The additional experiments assess PruneSID’s implementation, generalization across model scales, and applicability to more advanced VLMs.

The experiments begin by detailing the implementation setup in Sec. A.2.1.
They evaluate generalization across model scales using LLaVA-1.5 and LLaVA-NeXT in Sec. A.2.2.
They further assess applicability to more advanced architectures and models.

A.2.1 EXPERIMENT DETAILS

The extended experiments evaluate implementation fairness, model-scale and architecture generalization, component contributions, threshold sensitivity, and qualitative token selection. Across these analyses, PruneSID maintains strong compressed performance, with PSCA and NMS jointly supporting importance and diversity preservation.

Experiment details: Experiments use a single NVIDIA L20 GPU and the lmms-eval framework, with VisionZip and PruneSID benchmarked under identical hardware and settings.
Model-scale generalization: On LLaVA-1.5 13B, average accuracy reaches 98.0%, 97.4%, and 95.2% of the uncompressed baseline when retaining 192, 128, and 64 tokens.LLaVA-1.5 encodes each image into 576 visual tokens.
Architecture generalization: Across Qwen2-VL retention levels of 33.3%, 22.2%, and 11.1%, PruneSID consistently achieves higher average performance than PACT.
High-resolution evaluation: +1.6%, +2.2%, and +1.8% are PruneSID’s average improvements over VisionZip at 33.3%, 22.2%, and 11.1% retention on Qwen2.5-VL.The comparison includes HRBench-8k and XLRS high-resolution benchmarks, where the method remains effective.
Component ablations: 1.7–2.9% is the full model’s average advantage over ablations that remove PSCA or NMS, confirming their complementary contributions.The full model consistently outperforms both variants across token-retention settings.
Information preservation: 0.9–1.5% and 2.7–12.1% are the full model’s advantages over Ascend and Descend, respectively, showing joint preservation of importance and diversity.Descend reaches 84.2% at 64 tokens, while Ascend notably drops on POPE.
Threshold sensitivity: Performance remains stable for α values from 24 to 40, while larger α shifts retained tokens toward higher-numbered, less redundant semantic groups.Group 0 generally retains fewer tokens because it often represents redundant background regions.
Qualitative analysis: At 5.6% retention, visualizations compare PruneSID’s retained tokens with VisionZip and show selective preservation of representative tokens.Additional visualizations examine grouping and selection at 32 and 64 retained tokens.