PaperlayerView exact revision on arXiv

Source-linked AI summary

S$^2$Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

Yuanyuan Jia, Shunpu Tang, Qianqian Yang

arXiv:2609.01224v1cs.CV

TL;DR

Long visual-token sequences increase MLLM inference costs, and existing pruning criteria can produce stable spatial biases without consistently outperforming Uniform Grid sampling. S2Prune preserves broad spatial coverage, adapts token density to local structure, and uses ERC for local selection; across ten benchmarks and two architectures, it shows consistent gains over training-free baselines, retaining 79.3% of full-model performance with 32 tokens.

  • Problem

    Long visual-token sequences increase Transformer prefill computation, KV-cache usage, and inference latency, while existing criteria do not always outperform Uniform Grid sampling.

  • Method

    S2Prune assigns at least one token to each region, distributes remaining budget using Laplacian variation, and selects representative tokens within regions using ERC.

  • Results

    Across ten benchmarks and two MLLM architectures, S2Prune shows consistent gains over training-free baselines; with 32 of 576 visual tokens, it reaches 55.9 average accuracy and retains 79.3% of full-model performance.

  • Takeaways & Limitations

    Effective visual token pruning should consider both which tokens are selected and where the token budget is spent.

  • Takeaways & Limitations

    Uniform density can over-represent simple regions and under-represent detail-rich regions when local structure is highly uneven, especially at smaller budgets.

Abstract

from arXiv · show

Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S$^2$Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S$^2$Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.

1. Introduction

Visual token pruning reduces the cost of processing long visual sequences, but token scores can create stable spatial biases. S2Prune addresses this by preserving broad coverage while allocating density to structurally rich regions and selecting representatives with ERC.

  • Long visual-token sequences increase Transformer prefill computation, KV-cache usage, and inference latency.
  • Importance- and diversity-based pruning criteria do not always outperform Uniform Grid sampling, which preserves broader spatial coverage.
  • Different token scores induce distinct, stable spatial biases, determining where the limited token budget is spent.
  • S2Prune preserves broad coverage while adapting token density to local image structure, assigning more capacity to detail-rich regions.The method frames pruning as spatial rate allocation under a fixed token budget.
  • S2Prune assigns at least one token per region, distributes remaining tokens using Laplacian variation, and selects representatives with ERC within each region.Regional structure determines token counts, while ERC determines which tokens are selected.
  • On Qwen2.5-VL-7B-Instruct, S2Prune achieves the highest average accuracy among evaluated training-free pruning methods at 128, 64, and 32 tokens.At 32 tokens, it reaches 55.9 average accuracy and retains 79.3% of full-model performance.

2. Related Work

VLM token pruning methods reduce inference costs by removing redundant visual representations, while recent work increasingly treats spatial organization as an important selection factor. S2Prune differs by allocating representation capacity according to intrinsic structural variation rather than semantic relevance.

  • High-resolution VLM inputs produce longer visual-token sequences, increasing prefill latency and KV-cache memory while attention cost grows quadratically with sequence length.
  • Visual token pruning reduces VLM inference cost by removing redundant visual representations without modifying the underlying architecture.
  • Existing VLM pruning methods use attention, cross-modal relevance, redundancy, feature relations, diversity, or combinations of these criteria.
  • Recent approaches increasingly incorporate spatial organization, including uniform sampling, spatial distance, adaptive pruning, and regional budget allocation.
  • S2Prune allocates representation capacity according to image-intrinsic structural variation rather than treating regional budgets as indicators of task-relevant evidence.The paper distinguishes semantic relevance from the token density needed to preserve visual structure.

3. Preliminary and Motivation

Visual token pruning can create stable spatial biases, making broad spatial coverage valuable under aggressive budgets. These findings motivate allocating token density according to regional structure rather than using uniform density everywhere.

  • Token-score spatial bias: Different token-scoring criteria produce persistent spatial preferences across image–question pairs, independent of current content or task semantics.The patterns include recurring horizontal and vertical bands and periodic row-and-column structures.
  • Token-score spatial bias: Token importance alone does not determine an adequate spatial distribution of retained visual tokens.The resulting representation can favor positions without reflecting the current image’s spatial structure.
  • Broad spatial coverage: Random sampling remains competitive with carefully designed criteria, showing that sophisticated scoring does not automatically yield a better sparse representation.Random uses neither token-level importance, attention, nor similarity.
  • Broad spatial coverage: At B = 64, Uniform Grid outperforms Random on four of five benchmarks while remaining competitive with complex pruning criteria.Uniform Grid distributes retained tokens consistently across image regions, providing broad spatial support.
  • Adaptive spatial density: Fixed sampling density becomes restrictive when fine-grained structure is concentrated in only a few regions, because detail-rich areas may be underrepresented.The controlled comparison varies regional budgets by structural variation while keeping the token budget and local representative rule identical.
  • Adaptive spatial density: Adaptive density gains are modest for homogeneous structure but substantially larger for heterogeneous images, especially at B = 32.Averaged across MMBench, TextVQA, and POPE, high-heterogeneity images gain 1.39 points at B = 64 and 2.79 points at B = 32.

4. Method

S2Prune separates regional budget allocation from local representative selection, preserving broad spatial coverage while concentrating tokens in structurally richer regions.

  • Adaptive Spatial Scaffold: S2Prune converts a global budget into region-specific capacities, assigning at least one token to every coarse region.Remaining capacity is allocated subject to regional limits, preserving spatial support under a fixed budget.
  • Adaptive Spatial Scaffold: Laplacian variation measures regional structural complexity, allocating higher token density to regions with richer local variation.The Laplacian responds weakly in smooth regions and strongly around rapid local changes.
  • Local Representative Selection: Each region is divided into approximately equal-area cells, with one representative token selected independently per cell.The allocation scaffold determines where capacity is spent before local selection chooses representatives.
  • Local Representative Selection: Early Representation Change (ERC) is the decoder update magnitude computed after the first decoder block, rather than the incoming visual-feature magnitude.Pruning then removes unselected tokens while preserving the order and positional information of the retained tokens.
  • Local Representative Selection: ERC selects the representative with the largest score within each cell, without comparing scores across distant regions.This keeps regional allocation and local token selection as separate decisions.

5. Experiments

The evaluation covers ten multimodal benchmarks, multiple token budgets, and two MLLM architectures, with S2Prune compared against training-free pruning baselines.

  • Evaluation Setup: S2Prune is evaluated on ten image benchmarks spanning reasoning, question answering, OCR, hallucination, perception, and general multimodal understanding.The benchmarks include MMMU, GQA, SQAIMG, VQAText, VizWiz, POPE, MME, MMBEN, MMBCN, and MMVet.
  • Evaluation Setup: The main comparisons use Qwen2.5-VL-7B-Instruct, with LLaVA-OneVision-7B used for cross-architecture evaluation.Unless stated otherwise, analyses use Qwen2.5-VL-7B-Instruct.
  • Main Results: S2Prune achieves the highest aggregate accuracy among evaluated training-free methods at 128, 64, and 32 tokens on Qwen2.5-VL-7B.At 32 tokens, it reaches 55.9 Acc. and retains 79.3% of full-model performance.
  • Cross-Architecture Results: S2Prune leads all four benchmarks and attains the highest Acc. and Rel. at every tested budget on LLaVA-OneVision-7B.These results support transferability beyond Qwen2.5-VL.

FLOPs

Efficiency and ablation results examine computation, memory, regional allocation, and local representative selection under visual-token pruning.

  • Efficiency Analysis: Nearly 2.0× prefill speedup, from 56.97 ms to 28.66 ms, accompanies the reduction from 576 to 64 visual tokens.The efficiency evaluation uses one prefill stage and one decoding stage per sample.
  • Efficiency Analysis: KV-cache consumption falls from 31.50 MB to 4.50 MB when visual tokens are reduced from 576 to 64.These measurements were performed on a single NVIDIA GeForce RTX 5090 GPU.
  • Structural-Prior Ablation: Laplacian allocation gives the highest scores on all three benchmarks when spatial partition, budget, and local ERC selection are fixed.The ablation compares Uniform, Random, FFT, Effective Rank, and Laplacian allocation signals.
  • Local-Selector Ablation: ERC leads all three local-selector tasks with 84.6/81.0/55.2 and produces the best Acc./Rel. of 73.6/85.4%.This comparison fixes spatial allocation and pruning, isolating the local representative selector.

6. Conclusion

The paper reframes visual token pruning around spatial allocation, showing that token-selection criteria can create stable spatial biases and that effective pruning must account for budget placement.

  • Conclusion: Different token-selection criteria induce stable spatial biases and do not always outperform Uniform Grid sampling under aggressive pruning.This motivates examining spatial organization alongside token-wise scoring.
  • Conclusion: S2Prune separates regional budget allocation from local token selection, using Laplacian variation for capacity and ERC for representative choice.The method is training-free and uses the first decoder response for ERC.
  • Conclusion: Experiments across ten benchmarks and two MLLM architectures show consistent gains over training-free baselines.The results support considering both which tokens are selected and where the token budget is spent.

Supplementary Material

S2Prune partitions the 576-token visual grid, allocates a fixed budget across regions using structural variation, and selects local representatives after decoder Layer 0 while preserving sequence order and positional IDs.

  • Coarse Spatial Partition: The 24×24 visual-token grid is partitioned into budget-dependent coarse grids, with complete, non-overlapping coverage.For budgets 32, 64, 128, and 192, the coarse grids are 4×4, 5×5, 8×8, and 9×9, respectively.
  • Regional Token-Budget Allocation: Each region receives at least one token, and remaining capacity is allocated according to its normalized Laplacian-based structural score.Largest-remainder allocation redistributes capacity when regions saturate and continues until exactly B tokens are assigned.
  • Recursive Cell Partition: Each region is recursively split into exactly its assigned number of non-empty, disjoint rectangular cells that fully cover the region.The largest splittable rectangle is divided along its longer dimension, with deterministic tie handling.
  • ERC-Based Local Selection: ERC is computed from visual-token representations before and after decoder Layer 0, and the highest-ERC token is retained from each local cell.Structural complexity determines regional capacity, whereas ERC determines the representative within each cell.
  • Token Ordering and Pruning: After selection, retained tokens are sorted by their original decoder-visible indices, physically pruned after Layer 0, and continue inference with original M-RoPE IDs.S2Prune does not reorder the decoder-visible sequence before pruning, and text and special tokens remain unchanged.

A.7. Training-Free Setting and Evaluation Consistency

S2Prune is a training-free method evaluated under consistent experimental conditions, using deterministic regional allocation and recursive partitioning without trainable or dataset-specific selection components.

  • Training-Free Setting: S2Prune introduces no trainable parameters and excludes query attention, text relevance, global semantic Top-K selection, and dataset-specific tuning.The method therefore operates without training-based or dataset-specific pruning components.
  • Evaluation Consistency: All method comparisons use the same checkpoint, preprocessing, prompts, generation configuration, sample ordering, and evaluation pipeline.These controls are held constant across compared methods.
  • Regional Token-Budget Allocation: The regional allocator initializes every region with one token, distributes the remainder by normalized structural scores, and falls back to uniform weights when scores are effectively zero.Unsaturated regions receive capacity through iterative largest-remainder allocation until the target budget is met.
  • Recursive Cell Partition: Recursive cell partitioning produces Bg non-overlapping local cells from each region and splits the largest rectangle along rows or columns according to its shape.The procedure returns exactly Bg cells for a regional budget satisfying 1 ≤ Bg ≤ |R|.

B. Additional Spatial-Bias Results

Additional analyses show that selection-frequency spatial patterns remain stable across token budgets, including substantial correlations between adjacent and widely separated budgets.

  • Budget Stability: The supplementary spatial-bias measurements repeat the main analysis at B = 32 and B = 128 using the same 1,000 MMBEN samples and score definitions.Only the retained-token budget changes relative to the main-paper analysis.
  • Budget Stability: Pearson correlations exceed 0.946 for adjacent budgets, while 32-versus-128 comparisons remain at or above 0.819 and 0.889 for Pearson and Spearman correlations.These correlations indicate stable positional patterns despite changes in edge and center mass.

C. Additional Qwen2.5-VL-7B Results

Supplementary Qwen2.5-VL-7B results show strong aggregate performance across budgets and robustness to coarse-grid changes, while the best method varies across individual datasets.

  • B = 192 Results: At B = 192, the 9×9 configuration achieves 65.9 Acc. and 95.5% Rel., compared with 65.7 Acc. and 94.9% Rel. for FastV.The 9×9 configuration has the best aggregate performance, while the 10×10 configuration ranks second on both metrics.
  • Per-Benchmark Results: Across datasets, FastV remains strongest on VQAText and VizWiz, while GridPrune leads on MMMU and MMBCN and HoloV leads on POPE.The aggregate leader can therefore differ from the leading method on individual benchmarks.
  • Grid-Size Sensitivity: At B = 64, the 4×4, 5×5, and 6×6 grids differ by at most 0.1 aggregate Acc.; at B = 32, the 3×3, 4×4, and 5×5 grids remain within 0.8 Acc.The authors use fixed budget-dependent configurations across benchmarks rather than dataset-specific grid settings.
  • Qualitative POPE Example: At B = 32, S2Prune answers both POPE queries correctly, whereas FastV and GridPrune answer incorrectly.The queries concern a fork and a couch.

E. Qualitative Results

Qualitative POPE cases compare retained-token locations and answers for FastV, GridPrune, and S2Prune at budgets of 32 and 128 tokens out of 576. Across the shown cases, S2Prune answers correctly while the other methods answer incorrectly, with token placement preserving local evidence around queried objects.

  • Qualitative Results: At B = 32/576 and B = 128/576, S2Prune answers every shown POPE query correctly, whereas FastV and GridPrune answer incorrectly.The examples cover objects including a snowboard, bench, potted plant, chair, backpack, and tennis racket.
  • Qualitative Results: The queried evidence spans thin objects, visually similar surroundings, and cluttered scenes containing people or other content.Examples include a thin tennis racket, a snowboard surrounded by similar snow, and chairs or backpacks amid people and scene content.
  • Qualitative Results: FastV often concentrates tokens near image boundaries, while GridPrune covers the image more broadly but can leave limited local evidence on the queried object.These patterns are visible through overlays of retained visual-token positions.
  • Qualitative Results: S2Prune retains a more spatially distributed token set with local evidence around the target object without using query-conditioned scoring.Regional image structure allocates capacity, and ERC selects a local representative within each cell.
  • Qualitative Results: The selected cases illustrate spatial behavior and complement, rather than replace, aggregate POPE evaluation.The overlays support comparison of both final answers and the spatial evidence preserved by each method.
Loading 2609.01224v1…