PaperlayerView exact revision on arXiv

Source-linked AI summary

Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM

Penghao Wu, Lewei Lu, Ziwei Liu

arXiv:2505.15816v1cs.CV

TL;DR

Long vision-token sequences make decoder-only LMMs computationally expensive, while reducing token counts risks losing fine-grained information. The paper studies computation-level redundancy and proposes ProxyV, which uses proxy vision tokens for heavy operations. It reports reduced prefilling cost with maintained or improved fine-grained benchmark performance, and a non-spatial variant that can combine with token reduction.

  • Problem

    Long vision-token sequences raise computational costs in LMMs, while pruning or merging tokens risks losing fine-grained information.

    Source: §1 · pp. 1–2

  • Method

    The paper studies computation-level redundancy and proposes ProxyV, which uses proxy vision tokens for heavy computation while lightweight modules update original vision tokens.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    101% performance with prefilling FLOPs and time reduced by 43% and 40%, respectively, and 102.4% performance with reductions of 36% and 33% on fine-grained benchmarks using Vicuna1.5-7B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A non-spatial ProxyV variant can be combined with token-reduction methods to further enhance efficiency.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large multimodal models excel in multimodal tasks but face significant computational challenges due to excessive computation on visual tokens. Unlike token reduction methods that focus on token-level redundancy, we identify and study the computation-level redundancy on vision tokens to ensure no information loss. Our key insight is that vision tokens from the pretrained vision encoder do not necessarily require all the heavy operations (e.g., self-attention, FFNs) in decoder-only LMMs and could be processed more lightly with proper designs. We designed a series of experiments to discover and progressively squeeze out the vision-related computation redundancy. Based on our findings, we propose ProxyV, a novel approach that utilizes proxy vision tokens to alleviate the computational burden on original vision tokens. ProxyV enhances efficiency without compromising performance and can even yield notable performance gains in scenarios with more moderate efficiency improvements. Furthermore, the flexibility of ProxyV is demonstrated through its combination with token reduction methods to boost efficiency further. The code will be made public at this https://github.com/penghao-wu/ProxyV URL.

1. Introduction

Long vision-token sequences make decoder-only LMMs computationally costly, while token pruning or merging risks losing fine-grained information. The paper studies computation-level redundancy and proposes ProxyV, which shifts heavy operations to proxy vision tokens.

1. Introduction

  • Experiments identify computation-level redundancy in vision-token processing, and ProxyV assigns heavy computation to proxy tokens while lightweight modules update the original tokens.The paper also explores reducing computation progressively and reports that lightweight vision-specific MLPs can compensate for performance drops from skipped operations.Source: §1 · pp. 1–2
  • Token pruning and merging risk discarding fine-grained or contextual information, whereas computation-level reduction aims to preserve all vision tokens.Question-guided selection can also miss information needed in later turns, and many such methods rely on text-to-image attention scores.Source: §1 · p. 2
  • 101% performance with prefilling FLOPs and time reduced by 43% and 40%, respectively, and 102.4% performance with reductions of 36% and 33% on fine-grained benchmarks using Vicuna1.5-7B.The two outcomes correspond to ProxyV applied from layers 12 and 16, respectively.Source: Figure 1 · p. 1, §1 · p. 2
  • A non-spatial ProxyV variant can be combined with token-reduction methods to further enhance efficiency.The paper also validates ProxyV with different LLMs.Source: §1 · p. 2

2. Discover Computation-level Redundancy

Masking vision-token attention shows that computation redundancy varies across decoder-only LMMs and is concentrated in middle or later layers. Skipping attention and FFNs can reduce computation, but lightweight MLP replacements help limit performance loss.

2. Discover Computation-level Redundancy

  • Masking vision-token attention in middle or later layers has minimal or no effect on performance, while masking it throughout the LLM causes a significant drop.The redundancy pattern varies across LLMs; for example, Qwen2-7B retains 100% performance when masking only the latter 25% of layers.Source: §2 · p. 3, Figure 2 · p. 3
  • Fine-tuning with vision-to-all attention skipped mitigates performance loss, but FLOPs reduction remains limited because heavy FFNs still run on vision tokens.The experiment evaluates skipping from layers 0, 12, and 16 on Vicuna1.5-7B.Source: §2 · p. 3, Table 1 · p. 3
  • Replacing vision-token attention and FFNs with lightweight MLPs reduces FLOPs, with performance versus attention-only skipping varying by start layer.The replacement degrades performance for layer 0, is similar at layer 12, and improves it at layer 16; the added vision-specific modules have 9.44M parameters per layer.Source: §2 · pp. 3–4, Table 2 · p. 3

3. A Better Solution with Proxy Vision Tokens

ProxyV shifts compute-intensive decoder operations from full vision tokens to a smaller proxy set, which then guides updates to the original tokens. The section reports performance gains across layer placements and LLM backbones, contrasts ProxyV with token reduction, and introduces a non-spatial variant that can be combined with token reduction.

3.1. Proxy Tokens with Spatial Priors

  • Proxy tokens replace full vision tokens in compute-intensive attention and FFN operations, then guide spatially corresponding original tokens through a lightweight update module.Each proxy corresponds to an r × r group of original tokens; the module uses a lightweight two-layer MLP in the described implementation.Source: §3.1 · p. 4, §3 · p. 4
  • Middle-to-rear-layer ProxyV achieves 101%–102% performance with a smaller efficiency gain, while middle-layer application achieves 100%–101% with moderate efficiency gains.These trends are reported across different LLM backbones.Source: §3.1 · p. 4, Table 4 · p. 6
  • On 100 DetailCaps-4870 examples, vision-specific MLPs reduce MIR from 3.62 to 3.10, which the authors interpret as better visual-text alignment associated with performance improvement.The paper describes this as a preliminary mechanism study and leaves further investigation for future work.Source: §3.1 · p. 4

3.2. Comparison with Token Reduction Methods

  • VisionZip and PyramidDrop show notable grounding degradation and much worse document-parsing performance, whereas ProxyV retains visual information in dense or structured cases.The comparison is configured for similar efficiency improvement; the supplied passage gives no exact table values.Source: §3.2 · p. 5

3.3. A ProxyV Variant without Spatial Constraints

  • The non-spatial ProxyV variant performs similarly to the spatial version, and combining it with VisionZip increases efficiency while maintaining performance.The variant removes the spatial-prior requirement, allowing combination with token reduction methods.Source: §3.3 · pp. 5–6

4. Experiment Details

The experiments use a two-stage training pipeline, AnyRes image encoding, and fixed prefilling measurements. The supplied passages also describe baseline configurations and summarize the token-reduction comparisons in Tables 5 and 6.

Training

  • Training uses one epoch of projector and vision-module pretraining on 1.2M ShareGPT4V captioning examples, followed by one epoch of LLaVA-Next instruction tuning on 779K examples with the LLM unfrozen.Source: §4 · p. 6

Image encoding

  • Each image grid is encoded at 336×336 into 24×24 features with CLIP-ViT-L-336px, using at most five AnyRes grids and a two-layer MLP projector.Features are flattened in raster order, concatenated grid by grid, and separated by a token.Source: §4 · p. 6

Implementation details

  • The spatial ProxyV setup reduces 576 full vision tokens to 36 proxies, each guiding 16 full tokens; its guided-update module adds 14.68M parameters per Vicuna1.5-7B layer.The VisionZip baseline uses 360 dominant and 40 contextual tokens; PyramidDrop reduces tokens by 50% after layers 12, 20, and 26.Source: §4 · pp. 6–7

Comparison tables

  • Table 5 compares ProxyV with token-reduction methods, while Table 6 reports the non-spatial variant and its combination with VisionZip.The captions characterize token reduction as underperforming on document parsing and the combination as improving efficiency while maintaining performance.Source: Table 5 · p. 7, Table 6 · p. 7

Measurement setup

  • Prefilling FLOPs and time are measured with five image grids, 50 text tokens, eager attention, and one H100 GPU.Full evaluation results are provided in the appendix.Source: §4 · p. 7

5. Related Works

Related work contrasts decoder-only LMMs, which process visual and text tokens together, with cross-attention designs. It also surveys token-reduction methods and notes their potential information loss when images contain dense information.

5.1. Large Multimodal Models

  • Decoder-only LMMs concatenate visual and text tokens and process them through self-attention, while cross-attention designs avoid intensive vision-to-vision attention but add training complexity and parameters.The related-work discussion says cross-attention methods can underperform decoder-only models, especially on fine-grained visual tasks.Source: §5.1 · pp. 7–8
  • Higher-resolution, multi-grid, video, and multi-image inputs increase visual-token counts and computational burden in LMMs.Source: §5.1 · p. 8

5.2. Token Reduction in LMMs

  • Token reduction is applied before the LLM, during prefilling, or through KV-cache compression during decoding; pruning and merging can lose information in dense images and complex or multi-turn questions.The discussion presents computation-level redundancy reduction as a complementary approach that can be combined with token reduction.Source: §5.2 · pp. 8–9

6. Conclusion

The conclusion presents ProxyV as a way to reduce computation-level redundancy on vision tokens. It reports that proxy tokens and lightweight vision-specific MLPs support efficiency and performance, and that a non-spatial variant can be combined with token reduction.

6. Conclusion

  • ProxyV uses proxy vision tokens to reduce the burden of heavy attention and FFN operations, with experiments across LLMs validating its effectiveness and a non-spatial variant enabling combination with token reduction.The conclusion also reports that lightweight vision-specific MLPs can compensate for performance drops when heavy operations are skipped.Source: §6 · p. 9

Impact Statement

The work aims to reduce LMM computational costs while maintaining or improving performance, with potential accessibility and energy benefits.

Impact Statement

  • ProxyV aims to reduce computational costs while maintaining or enhancing LMM performance.The stated goals include improving accessibility and environmental sustainability by decreasing energy consumption.Source: Impact Statement · p. 9
  • ProxyV may inherit bias, misuse, and privacy concerns associated with LMMs and LLMs, depending on datasets and tasks.The paper calls for responsible and equitable deployment that considers these concerns.Source: Impact Statement · p. 9
  • The paper recommends focusing on ethical considerations when deploying ProxyV.Source: Impact Statement · p. 9

A. Evaluation on More Benchmarks

On general multimodal benchmarks, ProxyV consistently avoids performance loss or improves results; the evaluation spans eleven benchmarks and specified dataset splits.

A. Evaluation on More Benchmarks

  • The paper evaluates models on eleven general multimodal benchmarks, including MMBench, SEED-Bench, RefCOCO, and MMMU.Source: A. Evaluation on More Benchmarks · p. 13
  • For some benchmarks, evaluation uses specified splits, including MMBench en-dev, the SEED-Bench image split, and RefCOCO test-A and test-B.The passage also specifies splits for GQA, MME, MMMU, and ScienceQA.Source: A. Evaluation on More Benchmarks · p. 13
  • ProxyV consistently achieves no performance loss or even improvements on general multimodal benchmarks.The evaluation results are presented in Table 7.Source: A. Evaluation on More Benchmarks · p. 13, Table 7 · p. 13
  • On benchmarks requiring less fine-grained visual information, redundancy appears earlier and skipping operations causes smaller performance losses.Source: A. Evaluation on More Benchmarks · p. 13
  • Applying ProxyV from middle-to-later layers achieves similar or better overall performance and better efficiency than applying it only in later layers.The paper attributes the larger gain to introducing vision-specific modules in more layers.Source: A. Evaluation on More Benchmarks · p. 13

B. Fine-grained Benchmark Details and Full Results

This section specifies evaluation splits and directs readers to full-result tables for fine-grained benchmarks and exploratory experiments.

B. Fine-grained Benchmark Details and Full Results

  • Evaluations use validation splits of DocVQA, InfoVQA, and TextVQA, plus the MMBench English dev and MME perception splits.RefCOCO scores average testA and testB; evaluations use the lmms-eval framework.Source: B. Fine-grained Benchmark Details and… · p. 13
  • Tables 8 and 10 provide full results for exploratory experiments and ProxyV on fine-grained benchmarks.Table 8 covers the full results of Tables 1–3, while Table 10 contains the full results of Table 4.Source: B. Fine-grained Benchmark Details and… · p. 13, Table 8 · p. 14
  • Table 9 covers token-reduction methods, non-spatial ProxyV, and combining Proxvy-ns with VisionZip.Source: Table 9 · p. 14
  • Table 10 provides full ProxyV results with different LLMs on fine-grained benchmarks.Source: Table 10 · p. 15

Related papers

Loading 2505.15816v1…