PaperlayerView exact revision on arXiv

Source-linked AI summary

IPCV: Information-Preserving Compression for MLLM Visual Encoders

Yuan Chen, Zichen Wen, Yuzhou Wu, Xuyang Liu, Shuang Chen, Junpeng Ma, Weijia Li, Conghui He, Linfeng Zhang

arXiv:2512.18747v1cs.CVcs.AI

TL;DR

Long visual-token sequences make MLLM vision encoding costly, while pruning in the ViT risks discarding language-relevant cues. IPCV combines early ViT pruning with reconstruction and attention stabilization, achieving strong accuracy–efficiency trade-offs across image and video benchmarks.

  • Problem

    Long visual-token sequences increase MLLM computation, and ViT pruning without language guidance risks removing cues important for downstream reasoning.

    Source: §1 · pp. 1–2

  • Method

    IPCV prunes tokens early in the ViT, reconstructs them using similar retained tokens, and uses Attention Stabilization to approximate pruned-token keys and values.

    Source: §1 · p. 2

  • Results

    Across image and video benchmarks, IPCV achieves superior accuracy–efficiency trade-offs; on image tasks, at 60.8% runtime it retains 94.9% Avg. Acc.

    Source: §4.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    After compression, the visual-token sequence passed to the LLM retains its original length and ordering.

    Source: §3.5 · p. 5

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) deliver strong vision-language performance but at high computational cost, driven by numerous visual tokens processed by the Vision Transformer (ViT) encoder. Existing token pruning strategies are inadequate: LLM-stage token pruning overlooks the ViT's overhead, while conventional ViT token pruning, without language guidance, risks discarding textually critical visual cues and introduces feature distortions amplified by the ViT's bidirectional attention. To meet these challenges, we propose IPCV, a training-free, information-preserving compression framework for MLLM visual encoders. IPCV enables aggressive token pruning inside the ViT via Neighbor-Guided Reconstruction (NGR) that temporarily reconstructs pruned tokens to participate in attention with minimal overhead, then fully restores them before passing to the LLM. Besides, we introduce Attention Stabilization (AS) to further alleviate the negative influence from token pruning by approximating the K/V of pruned tokens. It can be directly applied to previous LLM-side token pruning methods to enhance their performance. Extensive experiments show that IPCV substantially reduces end-to-end computation and outperforms state-of-the-art training-free token compression methods across diverse image and video benchmarks. Our code is available at https://github.com/Perkzi/IPCV.

1. Introduction

MLLMs process long visual-token sequences, making vision encoding a significant computational cost, yet pruning in the ViT risks removing cues important for language reasoning. IPCV combines early ViT pruning with reconstruction and attention stabilization, and is evaluated across image and video benchmarks.

1. Introduction

  • Visual-token redundancy and quadratic self-attention make long image and video sequences costly, hindering MLLM adoption and edge deployment.Source: §1 · p. 1
  • LLM-only pruning overlooks growing ViT costs, while vision-only pruning methods can underperform on multimodal tasks and lack language guidance.Source: §1 · pp. 1–2
  • IPCV prunes tokens in shallow ViT layers and reconstructs them using updates from similar retained tokens, whose changes resemble pruned-token changes.Source: §1 · p. 2, Figure 2 · p. 2
  • Attention Stabilization approximates pruned-token keys and values, and IPCV is training-free, compatible with LLM-stage compression, and evaluated across diverse benchmarks.Source: §1 · p. 2

2. Related Work

Prior visual-token compression methods operate at projectors, within LLMs, or inside visual encoders, but each placement has different scope and costs. IPCV targets the visual encoder while restoring a complete token set for the LLM.

2. Related Work

  • Projector methods shorten sequences before the LLM, while LLM-stage methods prune tokens during language-model processing.Source: §2 · p. 2
  • LLM-only compression leaves ViT computation intact, limiting its acceleration when substantial computation occurs in the vision encoder.Source: §2 · p. 3
  • Visual encoders use structural or dynamic compression, but many methods target single ViTs or require extra training; IPCV prunes early and reconstructs tokens for the LLM.Source: §2 · p. 3, Figure 3 · p. 3

3. Methodology

IPCV reduces computation by pruning tokens early in the ViT, reconstructing their updates from similar retained tokens, and restoring the complete sequence for the LLM. Attention Stabilization lets reconstructed tokens contribute to attention while skipping feed-forward layers, and the restored sequence remains compatible with LLM-side pruning.

3. Methodology

  • IPCV prunes redundant tokens in early ViT layers and uses Neighbor-Guided Reconstruction to maintain a semantically intact token set for the LLM.Source: §3 · p. 3

3.1. Token Pruning in the Vision Encoder

  • At pruning layer lp, tokens are scored using feature differences, and the top-K tokens with the largest scores are retained.Source: §3.1 · p. 3

3.2. Neighbor-Guided Reconstruction (NGR)

  • NGR reconstructs removed tokens by adding the average updates of their k nearest retained neighbors to their original embeddings; similar tokens exhibit similar layer-to-layer changes.Source: §3.2 · pp. 3–4

3.3. Attention Stabilization (AS)

  • Attention Stabilization temporarily includes reconstructed pruned tokens as keys and values in attention, while they skip costly feed-forward layers.Source: §3.3 · p. 4

3.4. Reintegration at the Final ViT Layer

  • At the final ViT layer, Reintegration restores removed tokens so the LLM receives the original sequence length and ordering without extra computation in earlier layers.Source: §3.4 · p. 5

3.5. Compatibility with LLM-side Pruning

  • Because IPCV preserves token length and ordering, it is compatible with existing architectures and can be combined with LLM-side pruning methods such as FastV and DART [49].Source: §3.5 · p. 5

4. Experiments

Experiments compare IPCV with vision-side and LLM-side pruning across image and video benchmarks, then test it on a larger MLLM architecture. Across these settings, IPCV reports favorable performance–efficiency trade-offs.

4.1. Experimental Setup

  • Experiments use Qwen2-VL-7B-Instruct and InternVL3-38B, comparing training-free vision-side compression with LLM-side pruning methods.Vision-side methods retain symmetric token fractions across the ViT and LLM, while LLM-side methods retain all ViT tokens and prune LLM tokens more aggressively.Source: §4.1 · pp. 5–6

4.2. Main Results

  • At 60.8% vanilla runtime, IPCV retains 94.9% Avg. Acc. on image tasks, exceeding the next-best method by at least 12 points; on video it reaches 95.7% of vanilla performance with lower latency.At 70.9% vanilla runtime, IPCV preserves 97.8% of vanilla performance on image tasks and outperforms baselines at comparable or higher runtime.Source: §4.2 · pp. 6–7

4.3. Experiments on InternVL3

  • On InternVL3-38B, IPCV achieves higher accuracy at similar or lower runtime than alternative baselines.IPCV is paired with FastV in this architecture comparison, with relative latency measured on MMBench-EN.Source: Table 3 · p. 6, §4.3 · p. 7

5. Analysis

The analysis examines inference efficiency, compatibility with LLM-side pruning, component ablations, and sensitivity to IPCV’s hyperparameters. It reports efficiency gains alongside accuracy trade-offs and stable performance across parameter choices.

5.1. Inference Efficiency Analysis

  • IPCV latency falls to 70.9%, 60.8%, and 51.7% of vanilla while maintaining strong accuracy; ViT-stage pruning reduces latency further but degrades accuracy more sharply.LLM-stage methods plateau around 69% runtime and do not gain further speedup at higher pruning ratios.Source: §5.1 · p. 7, Table 4 · p. 7
  • FLOPs alone do not reliably predict latency: methods with similar FLOPs can yield different GPU times.The efficiency analysis also measures total and prefilling latency, FLOPs, and KV cache usage on MMBench-EN.Source: §5.1 · p. 7

5.2. Compatibility in LLM-stage

  • IPCV paired with FastV, DART, or SparseVLM maintains over 93% of vanilla performance, but SparseVLM often has higher latency and V2Drop fluctuates more across benchmarks.The analysis identifies FastV and DART as the preferred LLM-stage methods to pair with IPCV.Source: §5.2 · p. 7, Table 5 · p. 8

5.3. Ablation Study

  • Removing either Attention Stabilization or Reintegration slightly reduces accuracy, while removing both causes a more noticeable drop; both mechanisms add only marginal runtime.Pruned tokens participate in lightweight attention while skipping more costly feed-forward layers, and the two mechanisms are complementary.Source: §5.3 · p. 8, Table 6 · p. 8

5.4. Sensitivity Analysis

  • Performance remains stable across Attention Stabilization depths of 3–9 and a wide range of NGR neighbor sizes; earlier pruning is faster, while later pruning is more accurate.Very small neighbor counts can increase variance, whereas large counts may add noise from less similar tokens.Source: §5.4 · p. 8, Figure 5 · p. 8

6. Conclusion

IPCV is a training-free framework for compressing MLLM visual encoders, with reported accuracy–efficiency gains over existing training-free methods across image and video benchmarks. The conclusion also identifies architecture generalization and future refinement of reconstruction and Attention Stabilization as directions and scope.

6. Conclusion

  • IPCV achieves superior accuracy–efficiency trade-offs to existing training-free methods across diverse image and video benchmarks and generalizes across model architectures.The conclusion also identifies IPCV as a flexible foundation for future work.Source: §6 · p. 8
  • The framework combines early-stage token pruning in the ViT with Attention Stabilization and Reintegration modules based on Neighbor-Guided Reconstruction.The conclusion characterizes this design as enabling aggressive token reduction while preserving semantic information needed for downstream reasoning.Source: §6 · p. 8
  • Future work may refine the reconstruction mechanism and further analyze Attention Stabilization to improve semantic preservation and computational efficiency.Source: §6 · p. 8

A. Baseline Details

  • The evaluation compares IPCV with representative baselines from both MLLM language-side pruning and ViT compression.The stated purpose is fair evaluation of the framework.Source: A. Baseline Details · p. 11

A.1. MLLM Compression Methods

The described MLLM-side compression methods prune or sparsify visual tokens within the language model, using attention, text guidance, variation, or token similarity to identify tokens for removal.

A.1. MLLM Compression Methods

  • FastV removes low-attention-score vision tokens early in large vision-language models to accelerate inference.Source: §A.1 · p. 11
  • SparseVLM [60] uses text tokens to rate visual-token significance and prunes redundant tokens with a recycling mechanism intended to minimize information loss.Source: §A.1 · p. 11
  • V2Drop [2] prunes tokens according to variation across layers, while DART [49] removes duplicates by similarity and is compatible with FlashAttention without large GPU memory overhead.Source: §A.1 · p. 12

A.2. ViT Compression Methods

ViT compression methods shorten visual-token sequences before the LLM, while MLLMs’ image and video inputs can produce long sequences that strain encoder and cross-modal computation.

A.2. ViT Compression Methods

  • ToMe [1] accelerates vision transformers by merging similar tokens across layers using bipartite matching based on attention keys.Source: §A.2 · p. 12
  • ToFu [16] adaptively chooses pruning or merging by layer and uses MLERP merging to better preserve feature norms.Source: §A.2 · p. 12

B. More Related Works

  • MLLMs typically pair a CLIP- or SigLIP-pretrained ViT, or Qwen2-VL’s dynamic-resolution ViT, with a lightweight projector and an LLM.Source: B. More Related Works · p. 12
  • Higher-resolution images and longer temporal windows yield long visual sequences that strain the vision encoder and cross-modal layers, increasing compute and latency.Examples include 5,220 ViT-stage tokens for a 308 × 196 Qwen2-VL image and 2,048 tokens for eight Video-LLaVA frames.Source: B. More Related Works · p. 12

C. Implementation

Experiments use different Nvidia GPU setups for Qwen2-VL-7B-Instruct and InternVL3-38B, and ToMe and ToFu are run with proportional token reduction across layers.

C. Implementation

  • Qwen2-VL-7B-Instruct image benchmarks use RTX 4090 (48GB), while its video benchmarks use A100-80G.Source: C. Implementation · p. 12
  • InternVL3-38B image benchmarks use A100-80G.Source: C. Implementation · p. 12
  • ToMe and ToFu reduce tokens proportionally across layers until reaching target sparsity, with a minor modification to their original implementations.Other baseline settings follow the original papers.Source: C. Implementation · p. 12

D. Datasets

The evaluation spans image and video understanding benchmarks, with image datasets covering varied visual reasoning and video datasets testing temporal and long-form understanding.

D. Datasets

  • The evaluation spans a diverse set of benchmarks for both image and video understanding.The section covers eight image datasets and four video datasets.Source: D. Datasets · p. 12

D.1. Image Datasets

  • Image benchmarks cover compositional reasoning, fine-grained abilities, perception and cognition, object hallucination, OCR, and assistive scenarios.Examples include GQA’s multi-step questions, TextVQA’s embedded-text questions, and VizWiz’s real-world assistive setting.Source: §D.1 · p. 12

D.2. Video Datasets

  • Video benchmarks assess temporal understanding across short and long clips, including long-form reasoning tasks.MVBench spans 20 temporal tasks, while EgoSchema tests very long-form video-language understanding and Video-MME includes videos from 11 seconds to 1 hour.Source: §D.2 · pp. 12–13

E. Theoretical Analysis

The theoretical analysis bounds NGR reconstruction error under stated assumptions and derives a bound on the distance between original and reconstructed token sets. Visual examples show compression tending to remove less informative regions while alleviating effects on fine-grained content.

E. Theoretical Analysis

  • The analysis assumes the ViT mapping is LViT-Lipschitz with respect to Hausdorff distance and that token embeddings are bounded.These assumptions frame the reconstruction-error bound.Source: E. Theoretical Analysis · p. 13
  • For a full-set token i and retained token r, the update deviation is bounded by their input-state distance plus LViT times the Hausdorff distance between token sets.The pointwise bound is ∥∆i − ∆r∥2 ≤ ∥hi,lp − hr,lp∥2 + LViT dH(X, Y).Source: E. Theoretical Analysis · p. 13
  • dH(X′, bX′) ≤ 2B(LViT + 1) bounds the Hausdorff distance between the uncompressed and reconstructed final-layer token sets.The theorem concerns the set-level distance after NGR reconstruction.Source: E. Theoretical Analysis · p. 13

F. Visual Examples of Token Compression

Loading 2512.18747v1…