PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration

Hanxun Yu, Wentong Li, Xuan Qu, Song Wang, Junbo Chen, Jianke Zhu

arXiv:2601.22674v3cs.CV

TL;DR

Excessive visual tokens make MLLM inference costly, while component-specific compression can neglect textual context and degrade decoding. VisionTrim provides training-free, full-pipeline compression through DVTS and TGVC, achieving near-original performance under substantial token reduction across image and video benchmarks.

  • Problem

    Excessive visual tokens substantially increase MLLM inference cost, while existing reductions may focus on isolated stages and neglect textual context needed for accurate decoding.

  • Method

    VisionTrim jointly compresses visual tokens across vision encoding and LLM decoding using global-local DVTS selection and text-guided TGVC complement and merging.

  • Results

    98.8% of original performance is achieved with an 88.9% token-count reduction without additional training costs.

  • Takeaways & Limitations

    VisionTrim consistently surpasses previous methods across reduction ratios in image and video understanding tasks.

  • Takeaways & Limitations

    When prompts are irrelevant or misleading, textual initialization becomes effectively random, although visual-token redundancy allows unsupervised-like clustering behavior.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) suffer from high computational costs due to excessive visual tokens, particularly in high-resolution and video-based scenarios. Existing token reduction methods typically focus on isolated pipeline components and often neglect textual alignment, leading to performance degradation. In this paper, we propose VisionTrim, a unified framework for training-free MLLM acceleration, integrating two effective plug-and-play modules: 1) the Dominant Vision Token Selection (DVTS) module, which preserves essential visual tokens via a global-local view, and 2) the Text-Guided Vision Complement (TGVC) module, which facilitates context-aware token merging guided by textual cues. Extensive experiments across diverse image and video multimodal benchmarks demonstrate the performance superiority of our VisionTrim, advancing practical MLLM deployment in real-world applications. The code is available at: https://github.com/hanxunyu/VisionTrim.

1 INTRODUCTION

MLLMs rely on visual tokens that substantially increase inference cost, especially in high-resolution and video settings. VisionTrim addresses this by compressing tokens across the full pipeline with DVTS and TGVC while retaining visual and textual context.

  • Visual tokens dominate MLLM input sequences, substantially increasing inference complexity and restricting practical deployment, particularly for high-resolution and video models.
  • Existing acceleration methods typically compress tokens in either vision encoding or LLM decoding rather than across the full MLLM pipeline.
  • Neglecting textual information can discard context essential for accurate LLM decoding and substantially degrade performance.
  • VisionTrim is a unified, training-free framework whose plug-and-play modules accelerate both vision encoding and LLM decoding.
  • DVTS selects essential visual tokens using global semantic importance and local spatial continuity, while TGVC merges pruned tokens according to textual cues.
  • Experiments span standard and high-resolution settings as well as image- and video-based MLLMs, demonstrating VisionTrim’s superiority over prior methods.

2 RELATED WORK

MLLMs combine LLMs with visual encoders for multimodal tasks, but their visual-token sequences create a Transformer complexity challenge. Related approaches therefore seek compact visual representations before LLM processing.

  • MLLMs integrate LLMs with visual encoders to improve performance on multimodal tasks.
  • Typical image- and video-based MLLMs project Vision Transformer features into an LLM-interpretable space through an MLP.
  • Transformer complexity scales quadratically with input sequence length, making visual-token count a major efficiency challenge for MLLMs.
  • Efficient visual projectors address this challenge by producing compact visual representations with fewer tokens before LLM processing.

3 METHODOLOGY

VisionTrim compresses visual tokens across the MLLM pipeline through two plug-and-play modules: DVTS selects dominant tokens using global semantics and local spatial continuity, while TGVC complements them with text-guided aggregation.

  • Multi-Stage Pruning Strategy: DVTS and TGVC can reduce tokens during vision encoding or between transformer layers during LLM decoding.At decoding, the first generated token supplies global semantic attention, while cross-modal attention guides TGVC.
  • Dominant Vision Token Selection: DVTS selects dominant visual tokens by combining global semantic importance with local spatial continuity.It uses [CLS] attention for global importance and LTAM to model feature similarity and positional proximity.
  • Dominant Vision Token Selection: LTAM computes local token importance from affinity with neighboring tokens using feature and positional similarities.Affinity is evaluated within a k × k neighborhood and aggregated across neighboring tokens.
  • Text-Guided Vision Complement: TGVC uses textual context to identify, assign, and merge discarded visual tokens that complement the selected dominant tokens.Remaining tokens are scored against text features, selected as clustering centers, assigned by text-guided similarity, and aggregated iteratively.
  • Text-Guided Vision Complement: The final representation concatenates top-K dominant tokens with R text-guided complement tokens.The complement tokens are produced after iterative clustering and aggregation of remaining visual tokens.

4 EXPERIMENT

Experiments across normal- and high-resolution images, video, multiple MLLMs, ablations, and efficiency metrics show that VisionTrim preserves performance while substantially reducing visual tokens and computational costs.

  • Main Results: VisionTrim consistently maintains or improves performance across normal-resolution benchmarks and token configurations.The evaluation uses LLaVA-1.5-7B and reports improvements on POPE, SQA, and TextVQA without degradation.
  • Main Results: 99.9% original performance is retained using 22.2% visual tokens on high-resolution LLaVA-NeXT-7B.This corresponds to nearly 95% token reduction.
  • Main Results: 98.0% original performance is achieved with a 93.4% pruning ratio on Video-LLaVA-7B, outperforming other methods across four benchmarks.VisionTrim consistently exceeds 96.0% performance in the video evaluation.
  • Broader Validation: Using approximately one-third of the original input tokens, VisionTrim incurs about 0.1% performance loss across Qwen2-VL and Qwen2.5-VL cases and improves MMBench by 2.1% over vanilla Qwen2-VL.The broader validation includes both competitive performance and occasional gains over the baseline MLLMs.
  • Component-wise Analysis: Applying DVTS and TGVC across both vision encoding and LLM decoding yields higher performance with 90.1% lower memory usage than stage-specific approaches.The ablation also reports 91.6% KV-cache memory reduction for encoder-only application and gains over SparseVLM for decoder-only application.
  • TGVC Analysis: TGVC becomes more beneficial at higher compression ratios, producing performance gains exceeding 4% while retaining visual tokens related to textual instructions.The module can be applied multiple times to improve alignment between textual tokens and corresponding visual information.
  • Efficiency Analysis: At an 88.9% reduction ratio, VisionTrim reduces CUDA time by 61.4%, FLOPs by 91.7%, and storage memory by 93.3% while maintaining 99.1% SQA accuracy.At equal token counts, it is 44.3% faster than SparseVLM and uses 50.0% less computational budget than VisionZip.

5 CONCLUSION AND LIMITATIONS

VisionTrim is presented as a training-free framework combining DVTS and TGVC to compress visual tokens across vision encoding and LLM decoding. The authors report strong image and video performance, while acknowledging that compression is not entirely lossless.

  • Conclusion: VisionTrim combines DVTS’s global-semantic and local-spatial token selection with TGVC’s text-guided visual token complement.Together, the plug-and-play modules accelerate both vision encoding and LLM decoding.
  • Conclusion: 98.8% of original performance is achieved with an 88.9% token-count reduction and no additional training costs.The authors state that VisionTrim consistently surpasses previous methods across reduction ratios in image and video understanding tasks.
  • Limitations: The method is not entirely without loss, motivating future work on lossless visual-token compression.The authors identify further exploration of visual-token redundancy as a research direction.

TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL

The supplementary material provides additional analyses, implementation details, experiments, broader impacts, and asset-license information.

  • Supplementary Organization: The supplementary material includes individual-case performance analysis, additional implementation details, and more experimental results.It also contains sections on broader impacts and asset license and consent.

A INDIVIDUAL CASE PERFORMANCE ANALYSIS

Case studies under an 88.9% token reduction examine how DVTS and TGVC affect factual responses and knowledge boundary drift. They show that TGVC can recover errors caused by pruning key visual cues, while deployment reliability remains a future evaluation concern.

  • Case Analysis: Under an 88.9% reduction ratio, DVTS alone can cause knowledge boundary drift, turning correct original answers into incorrect responses.The analysis progressively applies DVTS and TGVC on LLaVA-1.5-7B.
  • Case Analysis: Combining DVTS with TGVC recovers correct responses in cases where VisionZip or DVTS alone loses key visual cues.The authors attribute the recovery to TGVC restoring essential text-related visual tokens.
  • Mechanism: The two-stage design first selects dominant tokens using global semantics and local spatial continuity, then merges discarded tokens using text guidance.This design is intended to mitigate loss of critical visual information and minimize knowledge boundary drift.
  • Deployment Boundary: Future work will test critical instances before deployment and conduct stability and reliability assessments for applications such as AI-based disease diagnosis.The stated goal is to support reliable and trustworthy real-world deployment.

C ADDITIONAL IMPLEMENTATION DETAILS

VisionTrim’s computational analysis motivates visual-token compression through the quadratic cost of decoder processing and details token configurations for different MLLMs. Under extreme compression, the method retains substantial original performance without additional training.

  • Computational Budget Estimation: Causal self-attention models global dependencies among input tokens, while decoder-layer computation includes multi-head attention and feed-forward operations.The input sequence length is L, and the decoder’s computational cost includes both MHA and FFN modules.
  • Computational Motivation: Quadratic scaling with input-token count makes reducing visual tokens important, especially because they are normally 20 times more numerous than prompt tokens.The analysis emphasizes reducing visual tokens while preserving model performance.
  • Theoretical Computation Reduction: VisionTrim retains K main visual tokens and R complement tokens after TGVC, with token reduction analyzed relative to the initial visual-token count N.The retained-token count is K + R, and the reduction ratio is expressed using γ = (K + R)/N.
  • Token Configurations: The method specifies multiple token budgets for LLaVA-1.5, high-resolution LLaVA-NeXT, and video-based Video-LLaVA configurations.For Video-LLaVA, pruning reduces the budget to 17 tokens per frame across eight frames, or 136 tokens total.
  • Extreme Token Counts: Even with 16 tokens, VisionTrim retains 90% of original performance without additional training.At one token, the method preserves approximately 82.8% of original performance training-free, while fine-tuning with one-tenth of the LLaVA-1.5 dataset reaches 86.5%.
  • Additional Evaluations: VisionTrim’s performance comparison is evaluated on InternVL2-2B at a 70.0% token reduction ratio.The supplied table caption identifies the model and reduction setting but does not provide the comparison values.

D.2.2 ADDITIONAL TASK GENERALIZATION

Additional evaluations examine VisionTrim across MLLMs, image and video settings, captioning, classification, retrieval, and hardware-oriented efficiency metrics. The reported results describe consistent benchmark performance and acceleration across these broader tests.

  • Ablation Setup: Ablation results compare different inserted LLM layers across datasets on LLaVA-1.5-7B while fixing α1 at 50%.The supplied table caption identifies the ablation setting but does not provide the numerical outcomes.
  • Additional Task Generalization: Additional generalization tests cover generative image captioning, fine-grained visual classification, and multimodal retrieval.The reported results state that VisionTrim performs consistently better across these benchmarks.
  • Efficiency Evaluation: Hardware-oriented experiments on an NVIDIA A40 measure prefilling time, total inference time, and memory usage.The results report substantial acceleration across these metrics relative to VisionZip.
  • Cross-Model Evaluation: VisionTrim is evaluated on LLaVA-OneVision-7B and Qwen2-VL-7B across single-image, multi-image, and video benchmarks.LLaVA-OneVision uses 243 tokens per image and 66 tokens per video frame, while Qwen2-VL uses approximately one-third of its original tokens.

D.3.1 ABLATION STUDIES ON LAYER SELECTION.

VisionTrim’s default modules are inserted between the second and third LLM layers, and layer-selection ablations indicate that shallow insertion generally performs better. The experiments maintain an overall token-retention ratio of 75.0%.

  • Layer Selection: The default VisionTrim insertion point is between the second and third LLM layers.The modules are inserted only at this location in the default settings and provided experiments.
  • Layer Selection: To preserve a 75.0% overall token-retention ratio, α is adaptively adjusted according to the inserted layer k.The ablation varies the insertion layer while adapting the retention rate.
  • Efficiency Context: The efficiency analysis is additionally reported for LLaVA-NeXT-7B/13B.The supplied table caption identifies the models and analysis but does not provide numerical results.
  • Layer Selection: Shallow-layer insertion typically produces better performance than insertion into deeper LLM layers.This observation is reported from the layer-selection experiments.

D.3.2 TRADE-OFFS FOR OCR-HEAVY TASKS

VisionTrim is assessed on OCR-heavy, video, random-pruning, and non-CLIP-encoder settings, with results emphasizing accuracy–efficiency trade-offs and the role of text-guided complementarity. The method is also illustrated on detailed image and video understanding examples.

  • Video Compression: In video MLLMs, VisionTrim targets inter-frame redundancy rather than relying only on uniform fixed-frame sampling.The method uses visual-feature similarity across frames to address temporal redundancy.
  • Text Guidance: VisionTrim’s two-stage design outperforms direct text-guided methods, particularly on hallucination- and OCR-related benchmarks.DVTS first preserves core visual tokens, followed by TGVC aggregation of text-related tokens.
  • Random Pruning: VisionTrim achieves significant performance improvements across all benchmarks compared with indiscriminate random pruning.The comparison is reported for LLaVA-1.5-7B with 32 tokens.
  • Encoder Generalizability: VisionTrim is also evaluated after replacing CLIP with DINOv2 as the vision encoder and fine-tuning the resulting LLaVA model.This experiment tests generalizability to a non-CLIP vision encoder.
  • Module Trade-offs: Using only DVTS improves inference efficiency but causes a larger performance drop, whereas TGVC preserves performance more effectively with smaller efficiency gains.TGVC incurs additional token-level similarity computations, reducing its efficiency improvement relative to DVTS alone.
  • Qualitative Examples: VisionTrim’s examples show recognition of fine visual details, including distinguishing characters and identifying a bus license plate.The examples span image- and video-based understanding tasks.

D.5.3 ANALYSIS OF LAYER-WISE ATTENTION DISTRIBUTION

VisionTrim analyzes layer-wise attention to identify redundancy in visual tokens across the vision encoder and LLM decoding stages. Its visualizations compare attention behavior with and without token compression and connect retained tokens to efficiency and alignment.

  • Cross-Stage Analysis: The analysis treats redundancy as occurring in both vision encoders and LLMs, motivating layer-wise attention inspection across the full multimodal pipeline.The paper explicitly frames this analysis around the causes of visual-token redundancy.
  • Vision-Encoder Attention: Attention spreads broadly across the image in shallow vision-encoder layers, then concentrates on fewer dominant tokens in deeper layers, peaking by layer 23.This concentration motivates extracting visual tokens from later encoder representations.
  • LLM Decoding Comparison: With VisionTrim, the vanilla 576 visual tokens are reduced to 128 before LLM input, while the comparison baseline randomly selects 128 tokens.The visualization compares attention maps across all 32 LLM layers for VisionTrim and the vanilla LLaVA-1.5-7B model.
  • Qualitative Evidence: Figure 8 illustrates detailed image- and video-understanding examples, marking factual errors in red.The figure provides qualitative evidence alongside the layer-wise attention visualizations.
  • LLM Decoding Comparison: VisionTrim improves alignment between visual and textual tokens, helping the LLM focus on retained image tokens while preserving salient visual information.The reported effect is associated with maintained model performance and improved inference speed and efficiency.
Loading 2601.22674v3…