PaperlayerView exact revision on arXiv

Source-linked AI summary

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

Shiyu Li, Zi-Yuan Hu, Shijia Huang, Yanyang Li, Yiwu Zhong, Liwei Wang

arXiv:2609.01004v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Long visual-token sequences make MLLM inference computationally and memory intensive, while existing pruning can be misled by redundant high-norm outliers. SinkPruner uses a training-free visual sanitizer followed by text-guided pruning, achieving aggressive reduction with strong performance retention and cross-architecture transfer.

  • Problem

    Long visual-token sequences impose costly MLLM inference, and existing pruning methods can prioritize redundant high-norm outliers that attract abnormally high attention.

  • Method

    SinkPruner is a training-free coarse-to-fine framework that filters high-norm outliers before using early-layer text-guided attention to retain semantically aligned tokens.

  • Results

    SinkPruner achieves aggressive token reduction while preserving strong perception and reasoning performance across diverse image and video benchmarks and MLLM architectures.

  • Takeaways & Limitations

    The visual sanitizer transfers as a plug-and-play module that can enhance existing pruning methods across fixed-grid and dynamic-resolution MLLMs.

  • Takeaways & Limitations

    Evaluation focuses on offline inference with pre-recorded, fixed-length inputs, leaving online streaming with evolving temporal contexts unresolved.

Abstract

from arXiv · show

Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods. Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query. Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.

1 Introduction

MLLMs process long visual-token sequences with costly attention, limiting context availability and deployment in resource-constrained settings. SinkPruner addresses pruning errors caused by high-norm outliers through a coarse-to-fine visual sanitizer and text-guided pruner.

  • Motivation: Quadratic attention over extensive visual-token sequences creates prohibitive computational and memory costs during MLLM inference.These costs arise when images or videos are projected into long visual-token sequences jointly processed by the LLM decoder.
  • Motivation: This inefficiency restricts textual-reasoning context and hinders deployment in edge computing and robotics.
  • Prior Approaches: Existing pruning methods use vision-centric redundancy cues or text-guided semantic relevance to reduce visual-token sequences.Vision-centric methods use CLS attention or feature similarity, whereas text-guided methods select tokens appearing relevant to the query.
  • Pruning Challenge: High-norm outliers attract abnormally high attention despite originating from non-informative background regions and being redundant in feature and spatial dimensions.This systematically misleads attention-based pruning criteria that rely on CLS attention scores.
  • SinkPruner: SinkPruner first filters high-norm outliers with a visual sanitizer, then applies early-layer text-guided pruning to retain tokens aligned with the textual instruction.The pre-filtering is designed to reduce attention sink and attention dispersion before cross-modal token selection.
  • Results: 88.9% visual-token reduction preserves 96.5% performance on LLaVA-1.5 while yielding superior accuracy-latency trade-offs across image and video tasks.The framework also transfers across fixed-grid and dynamic-resolution architectures and enhances existing vision-centric pruning methods as a plug-and-play module.

2 Related Work

Related work addresses visual-token inefficiency across language and vision domains, with MLLM methods broadly divided into vision-centric and text-guided strategies. Vision-centric approaches merge, cluster, or rank tokens using visual similarity and CLS-token correlations.

  • MLLMs: MLLMs bridge powerful language models with visual understanding across diverse architectures and tasks.
  • Token Compression: Token compression research spans natural language processing and computer vision, where methods reduce sequence length or visual redundancy.
  • MLLM Pruning: MLLM visual-token pruning methods are broadly categorized as vision-centric or text-guided.
  • Vision-Centric Methods: Early vision-centric methods such as ToMe and PruMerge merge or cluster tokens using feature similarity without training.
  • Vision-Centric Methods: Importance-based vision-centric methods including VisionZip, Vispruner, and FasterVLM prioritize tokens with high correlations to the [CLS] token.

3 Methodology

SinkPruner addresses visual-token inefficiency by identifying high-norm outliers as redundant attention sinks and applying coarse-to-fine pruning before language decoding. Its visual sanitizer combines norm-based filtering, aggregation, attention selection, and similarity-based diversification, followed by text-guided relevance pruning.

  • Efficiency Bottleneck: Transformer attention incurs quadratic cost in sequence length, while visual tokens often dominate MLLM input sequences, making visual-token reduction central to inference efficiency.The total sequence length is n = nsys + nvis + ntext, and visual tokens often exceed text prompts by an order of magnitude.
  • Redundancy in High-Norm Tokens: High-norm outlier tokens are spatially and feature-wise redundant, typically corresponding to homogeneous background patches with high neighbor and intra-set similarity.Low-norm tokens remain more diverse and typically carry richer semantic details, whereas high-norm tokens exhibit representational collapse.
  • Attention Sink Trap: Because high-norm outliers receive the highest [CLS] attention despite background content, attention-based vision-centric pruning can mistakenly preserve redundant tokens as important.These outliers act as attention sinks that absorb a disproportionate amount of global attention.
  • SinkPruner Framework: SinkPruner is a training-free, coarse-to-fine framework that first sanitizes visual tokens and then applies text-guided pruning in early LLM decoder layers.The visual sanitizer conditions the visual stream before language decoding, yielding a purified and diverse token set for subsequent selection.
  • Visual Sanitizer: The visual sanitizer ranks token norms, aggregates high-norm outliers into one sink token, retains high-[CLS]-attention low-norm tokens, and diversifies remaining tokens by similarity.The resulting representation is Z = [xsink, Xres, Xdiv], which is projected and fed into the LLM.
  • Text-Guided Pruner: The text-guided pruner aggregates text-to-vision attention across textual queries and retains the top-K visual tokens by global relevance score.This query-aware selection focuses the LLM's computational budget on regions required for reasoning.

4 Experiments

Experiments across image-language, video-language, reasoning, efficiency, ablation, and transfer settings show that SinkPruner remains effective under aggressive token reduction and generalizes across architectures and pruning methods.

  • Main Results: SinkPruner retains 96.5% average performance with 64 visual tokens, an 88.9% reduction, on LLaVA-1.5-7B.It exceeds VisionZip by 3.3 percentage points and HoloV by 1.8 percentage points.
  • Main Results: SinkPruner retains 93.7% of the full-model average under a 32-token budget on harder reasoning benchmarks, versus 88.7% for VisionZip and 85.9% for FastV.The largest gaps occur on MM-Vet and AI2D, which require several spatially distinct pieces of evidence.
  • Main Results: At an 80% pruning ratio on video tasks, SinkPruner achieves 98.0% average performance retention and outperforms DART and DivPrune without retraining.This evaluates transfer to video-language understanding and spatiotemporal cues.
  • Ablation Studies: Removing the visual sanitizer causes a 10.2% drop on MMB, while retaining high-norm outliers reduces MME from 1754.1 to 1705.6.The ablation identifies high-norm removal as the dominant sanitizer component and confirms the importance of upstream purification.
  • Transferability: Adding high-norm filtration to VisionZip improves performance by 0.9% to 2.6% across MMBench and POPE at identical 32- and 64-token budgets.The filtration module is applied before VisionZip’s standard dominant-token selection.

5 Conclusion

SinkPruner is a training-free cascading visual token pruning framework that reduces visual redundancy before language decoding. Experiments across image and video benchmarks show strong performance preservation, architectural generalization, and transferability of its visual sanitizer, while online streaming remains open.

  • SinkPruner filters high-norm outliers before applying text-guided pruning, suppressing massive activations and attention dispersion in the LLM decoder.This coarse-to-fine design enables more precise text-guided pruning.
  • Extensive experiments show aggressive token reduction while preserving strong perception and reasoning performance across diverse image and video benchmarks.The framework generalizes from fixed-grid to dynamic-resolution MLLM architectures.
  • The visual sanitizer functions as a plug-and-play module with potential to improve existing pruning methods more broadly.
  • Current evaluation covers offline inference on static images and finite, fixed-length video clips, leaving online pruning for continuously evolving video streams unresolved.Online streaming would require updating visual history without future knowledge and without revisiting pruning decisions.

A Experiments on Non-CLIP Visual Encoders

High-norm outlier tokens occur in both DINO and DINOv2, indicating that the phenomenon extends beyond CLIP-based encoders. Because norm scales differ across encoders, SinkPruner uses within-encoder ranking rather than an absolute cutoff.

  • Both DINO and DINOv2 exhibit a small group of high-norm outliers separated from most regular patch tokens.The finding suggests abnormal high-norm tokens are a broader property of modern vision encoders rather than CLIP-specific artifacts.
  • 2.37% of DINOv2 patch tokens have ℓ2 norms larger than 150, although most lie roughly within [0, 100].These outliers remain sparse but prominent enough to distort feature distributions and potentially interfere with token compression.
  • DINOv2 has a shifted norm range relative to CLIP-based features, with its outlier mode appearing at much larger values.
  • The top-ρ rule ranks tokens within each encoder, adapting to shifted norm ranges without per-encoder calibration.This design supports transfer across model families.

B Sensitivity Analysis

The sensitivity analysis examines high-norm threshold selection and evaluates SinkPruner across diverse image-language and video-language benchmarks. Performance is stable across a broad threshold range but declines with overly aggressive filtering.

  • Non-CLIP visual encoders: Figure 5 compares patch-feature norm distributions for DINO and DINOv2, highlighting separated high-norm outliers and differing norm scales.For DINOv2, 2.37% of tokens exceed norm 150, motivating scale-free ranking instead of an absolute threshold.
  • Sensitivity analysis: Performance remains stable for high-norm thresholds τ in [45, 90] on both POPE and MME.An excessively small threshold, τ = 30, causes a clear performance drop by potentially removing useful visual information.
  • Benchmark coverage: The evaluation covers twelve image-language benchmarks and four video-language benchmarks spanning perception, OCR, multimodal reasoning, and temporal understanding.Reported tasks include GQA, MMBench, MME, POPE, ScienceQA, VQA-v2, TextVQA, MMStar, MMMU, AI2D, MM-Vet, NextQA, MVBench, SEED-Bench, and VideoMME.
  • Benchmark coverage: Video benchmarks assess temporal actions, event tracking, object interactions, temporal dependencies, and broader compositional video understanding.NextQA, MVBench, SEED-Bench, and VideoMME provide complementary temporal and multimodal evaluation settings.
  • Evaluation protocol: Benchmark metrics follow official protocols, including accuracy, VQA score or accuracy, MME score, and POPE Accuracy with Precision, Recall, and F1 when required.

C.2 Comparison methods.

SinkPruner is compared with state-of-the-art visual token reduction methods across LLaVA-1.5 and Qwen2.5-VL. The comparison controls the input token sequence for dynamic-resolution experiments and reports pruning hyperparameters across target budgets.

  • Comparison methods: The comparison spans LLaVA-1.5 and Qwen2.5-VL architectures and includes token merging, attention-based pruning, adaptive allocation, and hierarchical retention methods.Baselines include ToMe, MustDrop, FastV, PDrop, and PruMerge, among others.
  • Pruning hyperparameters: LLaVA-1.5-7B pruning hyperparameters are reported across target token budgets, with 576 original visual tokens.The high-norm fraction ρ is shared across budgets and backbones, while salience and diversity pools equal the target retained budget.
  • Comparison methods: For dynamic-resolution Qwen2.5-VL experiments, reimplemented baselines use the same evaluation harness and max_pixels setting as SinkPruner.All methods therefore observe an identical visual token sequence before pruning.

D.1 LLaVA Implementation

The implementation adapts SinkPruner to LLaVA and Qwen backbones with sanitizer-based filtering, progressive text-guided pruning, and encoder-specific salience scoring. It also uses a batched approximation to accelerate diversity selection.

  • D.1 LLaVA Implementation: LLaVA experiments use LLaVA-1.5-7B with 576 original visual tokens and official benchmark evaluation protocols.
  • D.1 LLaVA Implementation: The visual sanitizer identifies high-norm outliers with a scale-free top-ρ rule using ρ = 1% of visual tokens.For LLaVA, salience and diversity pools match the retained-token budget: 128, 64, or 32 tokens.
  • D.1 LLaVA Implementation: Progressive text-guided pruning acts at LLaVA layers 2, 6, and 15, reducing tokens through three successive retention stages.The retained-token schedules are (230, 130, 92), (110, 74, 42), and (54, 36, 22) for the three target budgets.
  • D.1 LLaVA Implementation: For encoders without a CLS token, salience uses average received visual self-attention in place of CLS attention.The score is computed at the penultimate encoder layer, averaged across heads, and substituted into top-kres selection without changing the rest of the pipeline.
  • D.1 LLaVA Implementation: Under 88.9% pruning, the CLS-free score retains 91.8% of Qwen2.5-VL full-model performance, exceeding HoloV by 5.6 points and VisionZip by 4.1 points.
  • D.1 LLaVA Implementation: Batched diversity selection scores candidates simultaneously and appends b least-similar tokens, reducing sequential rounds from kdiv to ⌈kdiv/16⌉ with b = 16.The batched rule approximates, rather than exactly reproduces, the serial farthest-point rule when b > 1.

E Additional Experiments

Additional experiments evaluate SinkPruner across model architectures, image and video benchmarks, and real inference settings. They report consistently stronger accuracy-efficiency trade-offs and robustness beyond the main experiments.

  • E Additional Experiments: SinkPruner consistently achieves a stronger accuracy-efficiency trade-off across different architectures and evaluation settings.
  • E Additional Experiments: The experiments are designed to support SinkPruner’s effectiveness, efficiency, generalizability, and robustness across image-language and video-language tasks.
  • E Additional Experiments: Additional results cover image benchmarks on LLaVA-1.5, video benchmarks on Qwen2.5-VL, and real POPE inference measurements.The measurements include runtime, prefill cost, decoding latency, and relative accuracy under different pruning ratios.
  • E Additional Experiments: The extended settings vary model capacity and visual-sequence length while keeping the vision encoder, top-ρ rule, and other hyperparameters unchanged.
  • E Additional Experiments: With 64 retained tokens on LLaVA-1.5-13B, SinkPruner preserves 96.2% of full-model performance and outperforms VisionZip by 2.0 relative-performance points.With 192 retained tokens, it preserves 99.2% of the full-model average.

F.2 Transfer to High-Resolution Inputs

On high-resolution LLaVA-NeXT inputs with roughly 2,880 visual tokens, SinkPruner maintains the strongest average retention at both tested budgets. Its advantage is largest on POPE and GQA, but TextVQA remains a weakness under tiled encoding.

  • F.2 Transfer to High-Resolution Inputs: LLaVA-NeXT expands each image to roughly 2,880 visual tokens, five times the 576-token LLaVA-1.5 setting.Reported averages normalize each row against the Upper Bound measured in the same run.
  • F.2 Transfer to High-Resolution Inputs: 94.2% retention with 320 tokens and 92.2% with 160 tokens give SinkPruner the best average results at both high-resolution budgets.It leads the strongest baseline VisPruner by 2.0 and 5.7 points, respectively.
  • F.2 Transfer to High-Resolution Inputs: On POPE, SinkPruner retains 99.8% of full-model performance at 320 tokens, while every baseline degrades by at least 7 points.It also leads VisPruner on GQA by 1.8 and 3.9 points at the two budgets.
  • F.2 Transfer to High-Resolution Inputs: TextVQA is the only benchmark where SinkPruner does not lead, retaining 89.0% and 88.8% versus VisPruner’s 93.9% and 91.2%.
  • F.2 Transfer to High-Resolution Inputs: Under tiled high-resolution encoding, small glyph patches can have large norms, causing norm-based sanitization to aggregate genuine text tokens with background outliers.The authors leave text-aware sanitizer modifications for future work.

G Ablations of the Remaining Hyperparameters

Ablations indicate that several implementation choices are not sharply optimized: pruning-layer perturbations produce a flat response, while salience-diversity splits yield smooth but metric-dependent variation. Additional analysis links sanitizer preprocessing to reduced attention sinks and dispersion.

  • G Ablations of the Remaining Hyperparameters: The paper treats pruning locations, progressive schedules, and pool allocation as empirical implementation defaults rather than core contributions, without grid-search optimization.
  • G Ablations of the Remaining Hyperparameters: Pruning layers (2, 6, 15) implement coarse-to-fine removal of visual redundancy, cross-modal irrelevance, and later text-relevant evidence.
  • G Ablations of the Remaining Hyperparameters: Perturbing each pruning layer by ±1 keeps POPE accuracy within 80.17–81.38 and MME Total within about 35 points, with no configuration collapse.The reported configuration is not the best-performing setting, consistent with using it without benchmark-specific optimization.
  • G Ablations of the Remaining Hyperparameters: Varying the salience-diversity allocation at fixed total budget produces smooth ranges of 80.49–81.91 for POPE and 1629.05–1682.61 for MME Total.Diversity-heavy splits favor POPE, whereas salience-heavy splits favor MME Total.
  • H.1 Mitigating Massive Activations: Standard text-guided pruning retains a 14.23% sink ratio, whereas visual sanitization reduces the ratio to 3.The sanitizer filters redundant visual content before decoder pruning, suppressing massive activations associated with attention sinks.
  • H.1 Mitigating Massive Activations: SinkPruner reduces text-visual attention entropy from SparseVLM’s 6.36 to 4.85, indicating sharper token-selection patterns after redundant visual content is filtered.

I.2 Visualizations of high-norm patches and low-norm patches

The visualizations show that high-norm outlier tokens concentrate in repetitive, non-semantic regions, whereas informative low-norm tokens capture distinctive semantic content. High attention peaks also spatially align with abnormally large feature norms, supporting the redundancy of these outliers.

  • High-norm outliers predominantly occupy locally repetitive background regions, while retained low-norm tokens lie in visually distinctive, semantically meaningful areas.These observations support redundancy in both spatial and representational dimensions.
  • CLS attention maps contain sparse, peaky responses that frequently appear in non-semantic background regions.
  • High attention peaks spatially align with abnormally large feature norms, indicating that high attention is largely driven by high-norm outliers.
Loading 2609.01004v1…