PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionZip: Longer is Better but Not Necessary in Vision Language Models

Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, Jiaya Jia

arXiv:2412.04467v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Vision-language models use excessively long visual-token sequences even though common vision encoders produce substantial token redundancy, increasing computational demands. VisionZip extracts informative tokens and merges retained tokens to reduce this redundancy, and evaluations report preserved performance with faster inference across image and video understanding tasks. The paper concludes that reducing redundant visual tokens is broadly useful and motivates vision encoders with lower redundancy.

  • Problem

    Vision-language models use far more visual tokens than text tokens, consuming substantial computational resources despite significant redundancy in tokens generated by common vision encoders.

  • Method

    VisionZip is a text-agnostic method that selects dominant visual tokens and merges retained tokens by similarity before sending them to the language model.

  • Results

    VisionZip substantially reduces visual-token counts while preserving model performance and improves inference efficiency across image and video understanding evaluations.

  • Takeaways & Limitations

    The results support reducing redundant visual tokens for practical VLM deployment and motivate developing vision encoders with lower redundancy.

  • Takeaways & Limitations

    Dropping visual tokens can cause misalignment because models trained on full token sequences may struggle to adapt to sharply reduced token counts.

Abstract

from arXiv · show

Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .

1. Introduction

Vision-language models rely on long visual-token sequences that impose substantial computational costs, despite considerable redundancy in tokens from common vision encoders. VisionZip addresses this by selecting informative tokens, improving efficiency while preserving performance across evaluated settings.

  • Motivation: LLaVA-1.5 uses 576 visual tokens, while a 672x672 image in LLaVA-NeXT yields more than 2,880, far exceeding typical text-token counts.These long visual sequences consume substantial memory and computation, limiting practical deployment in edge computing, autonomous driving, and robotics.
  • Motivation: Attention analyses of CLIP and SigLIP show that only a few visual tokens receive high attention and contain substantial information, while most receive little attention.The study identifies this concentration as evidence of significant redundancy in visual tokens.
  • Method: VisionZip selects dominant tokens and merges retained tokens by similarity to preserve potentially important small details while reducing visual-token redundancy.The text-agnostic method can operate training-free, with fine-tuning, or from scratch.

2. VisionZip

VisionZip addresses redundant visual tokens by selecting informative tokens and merging remaining tokens before they enter the LLM. This training-free, text-agnostic framework reduces token length while preserving performance and supports efficient tuning for improved alignment.

  • 2. VisionZip: VisionZip reduces visual-token redundancy before LLM input, avoiding heavy early-layer attention computation and supporting text-agnostic compatibility with existing acceleration algorithms.The framework is described as training-free, with projector fine-tuning available to improve adaptation.
  • 2.2. Redundancy Observation: Visual tokens from CLIP and SigLIP are highly redundant: most receive minimal attention, while a few aggregate substantial information.The method identifies these highly attended tokens as dominant visual tokens.
  • 2.3. Informative Visual Token Zip: VisionZip selects dominant tokens using attention scores from the vision encoder, retaining tokens most attended to by the CLS token or by other sequence tokens.For models without CLS tokens, token importance is based on average received attention.
  • 2.3. Informative Visual Token Zip: Remaining non-dominant tokens are merged by semantic similarity to preserve small but potentially important information.The merging procedure operates on keys that summarize token information during self-attention.
  • 2.4. Efficient Tuning: The method can reduce visual-token length by up to tenfold, but this reduction may cause vision-language misalignment because the model was trained with full visual tokens.Efficient projector tuning uses minimal instruction data while keeping other components frozen.
  • 2.5. Usage of VisionZip: VisionZip applies to image, video, and multi-turn understanding tasks, preserving over 90% of original performance while saving 3 times runtime and memory.The method is presented as a plug-and-play vision-encoder approach.

3. Experiments

Experiments show that VisionZip preserves or improves image and video understanding while using far fewer visual tokens. It also delivers substantial inference-efficiency gains across model architectures and tasks.

  • 3.1. Effectiveness on Image Understanding: When LLaVA-1.5 tokens fall from 576 to 192, VisionZip loses only 1.5% average accuracy and surpasses FastV and SparseVLM by 10.3% and 2.1%.At 64 tokens, it exceeds FastV and SparseVLM by 18.4% and 8.2%, respectively.
  • 3.1. Effectiveness on Image Understanding: With LLaVA-NeXT, VisionZip reaches 92.0% performance at about 5% of the visual tokens without training and 95.2% after tuning.These results surpass SparseVLM by 5.8% and 9%, respectively.
  • 3.3. Efficiency Analysis: On Mini-Gemini, the performance gap between VisionZip and the previous state-of-the-art method increases sharply as the retained token count decreases.The comparison covers POPE, TextVQA, and GQA across different visual token counts.
  • 3.2. Effectiveness on Video Understanding: VisionZip achieves 93.2% accuracy across four video benchmarks using 136 tokens, outperforming SparseVLM by 6.7%.On MSRVTT, the improvement over SparseVLM reaches 37.2%, and performance exceeds 90% on every benchmark.
  • 3.3. Efficiency Analysis: On POPE with LLaVA-NeXT 7B, VisionZip improves overall time efficiency 3× and prefilling time 7.8× versus the vanilla model.Prefilling time is the latency required to generate the first token.

4. Analysis and Discussion

The analysis attributes visual-token redundancy to attention concentrating information into a few proxy tokens, which can misalign text-relevant selection. VisionZip addresses this while reducing computation and supporting practical dialogue settings.

  • Reasons for Redundancy: Attention spreads broadly in early encoder layers, then concentrates on a few dominant tokens, peaking at the 23rd layer used for extraction.The final layer shows more dispersed attention.
  • Reasons for Redundancy: Transformer self-attention and softmax gradients may amplify high-attention regions while suppressing low-attention regions, concentrating information into proxy tokens.The paper relates this effect to Attention Sink and global-token observations.
  • Feature Misalignment: Text-relevant methods can select peripheral proxy tokens rather than tokens containing the visual details they are meant to represent.The paper gives an example where information about a person is assigned to a proxy token on the road.
  • Feature Misalignment: 51.1 to 46.4, a drop of approximately 9%, occurred when SparseVLM selected 64 tokens after 50 high-attention tokens were masked.Using the top 128 VisionZip tokens before SparseVLM filtering instead improved performance to 52.5, an increase of about 2.6%.
  • Efficiency and Deployment: VisionZip reduces visual tokens before the LLM, avoiding extensive computation and memory use in shallow LLM layers.It is compatible with quantization and enables the 13B model to be faster and better than the 7B model.
  • Multi-turn Conversations: Previous text-relevant selections can become poorly matched to current questions in multi-turn dialogue because cached visual tokens reflect earlier questions.VisionZip is presented as better suited to this setting.

5. Related Work

Vision-language models integrate visual and textual information by processing images, videos, or multiple images as extensive visual-token sequences. Increasing resolution and multimodal inputs substantially raises token requirements.

  • Vision-Language Models: Vision-language models process extensive visual-token sequences to support multimodal generation.The visual-token sequence is produced from visual inputs and handled alongside language.
  • Vision-Language Models: Higher resolutions and inputs containing videos or multiple images increase the number of visual tokens required.LLaVA-NeXT processes 672 × 672 images into 2304 tokens, while video models face additional token demands.

6. Conclusion

The paper identifies substantial redundancy in visual tokens and proposes VisionZip to reduce token counts while preserving performance and improving efficiency across practical multimodal settings.

  • Conclusion: Increasing visual-token length can improve performance, but current visual tokens contain significant redundancy.The conclusion frames this as an observation across popular VLMs.
  • Conclusion: VisionZip substantially reduces visual tokens while preserving model performance and enhancing computational efficiency.The method is described as broadly applicable to image and video understanding.
  • Conclusion: VisionZip is suitable for multi-turn dialogue and practical applications, while motivating vision encoders with lower redundancy for longer video sequences.The latter is presented as a future direction.

A.1. Comparison with Text-relevant Efficient VLM

VisionZip removes redundant visual tokens before LLM processing, contrasting with methods that select tokens during the LLM forward pass. The appendix describes its efficiency, compatibility, selection procedure, and evaluation settings.

  • Comparison with Text-relevant Efficient VLM: Most efficient VLMs use text–visual attention during the LLM forward pass to decide which visual tokens to retain.VisionZip instead removes redundancy before the tokens enter the LLM.
  • Better Performance: VisionZip achieves better training-free performance because vision encoders pre-group information into dominant tokens that text-based selection may overlook.The paper states that text-selected tokens often carry less information than dominant tokens.
  • More Efficient: VisionZip avoids heavy early-layer attention and intermediate token-selection computations, reducing memory use relative to prior approaches.The appendix notes that prior methods can sometimes use more memory than the vanilla model.
  • More Application Scenarios: VisionZip operates outside the LLM, making it compatible with existing LLMs and LLM acceleration algorithms.The method is also described as better suited to multi-turn conversations and practical scenarios.
  • Non-CLS Vision Encoders: For non-CLS encoders, VisionZip averages attention across heads and identifies important tokens by the attention received from other sequence tokens.Higher-average-attention tokens are retained.
  • Evaluation: The appendix reports experiments on widely used visual-understanding benchmarks, including SEEDBench, MMMU, MMVet, and multiple other evaluation sets.The listed benchmarks cover image, video, reasoning, OCR, hallucination, and multimodal understanding tasks.

B.1.3. Additional Experiments for LLaVA-1.5

Additional LLaVA-1.5 experiments show that VisionZip preserves nearly all performance with substantially fewer visual tokens, including during training and across LLaVA-NeXT evaluations.

  • 93.7% performance is achieved with only 64 visual tokens in LLaVA-1.5 13B without additional training time.Efficient tuning raises this to 94.8%.
  • VisionZip is evaluated across 11 benchmarks for LLaVA-1.5 13B using 192, 128, and 64 visual-token configurations.LLaVA-Bench and MMVet are excluded from the average because of their small size and unstable results.
  • VisionZip is applied during fine-tuning with 192, 128, and 64 visual tokens under otherwise vanilla training settings.Experiments use eight Nvidia 3090 24G GPUs with batch size 4.
  • 99.6% of performance is retained with 128 tokens during fine-tuning, while retaining 192 tokens improves performance by 0.6%.The authors attribute this to reducing interference from less informative visual tokens during training.
  • Across stable LLaVA-NeXT 7B benchmarks, VisionZip consistently delivers strong performance.The reported evaluations use a vanilla visual-token count of 2880.
  • With 640 tokens, eliminating 77.8% of tokens, VisionZip lets LLaVA-NeXT 13B outperform the 7B model in training-free mode.The 13B model also achieves faster generation speed.

B.1.5. Additional Experiments for Mini-Gemini

Mini-Gemini experiments show that VisionZip retains most performance even after aggressive visual-token removal, with fine-tuning improving the retained performance.

  • VisionZip is compared with approaches including SparseVLM and FastV, with its performance advantage increasing as fewer tokens are retained.The detailed analysis follows the main-paper comparison.
  • Over 90% of performance remains after removing 88.9% of visual tokens in training-free mode.Fine-tuning raises performance to as high as 95%.
  • Performance remains virtually unaffected when 66.7% of visual tokens are discarded.The authors interpret these results as evidence of substantial visual-token redundancy.

B.1.6. Ablation Study

The ablation study examines fine-tuning data compatibility and finds that limited compatible data is sufficient, indicating gains come from adapting to reduced visual-token inputs rather than added knowledge.

  • Using one-tenth of the LLaVA 1.5 dataset is sufficient for fine-tuning the multimodal projector.The comparison varies token counts of 640, 320, and 160 when adapting LLaVA-NeXT.
  • VisionZip’s gains are attributed to adapting to the sudden reduction in tokens, which helps bridge the visual and LLM spaces.The passage distinguishes this adaptation from acquiring additional knowledge through continued training.
  • The evaluation context includes video question-answering benchmarks covering repetition, action, state transition, frame, and long-term spatio-temporal reasoning.The listed datasets include TGIF-QA, MSVD-QA, MSRVTT-QA, and ActivityNet-QA.

B.2.2. Future Direction

VisionZip reduces visual-token costs to support longer video inputs and faster training and inference, while attention analyses motivate extracting informative features instead of merely increasing token length.

  • B.2.2. Future Direction: 5-10× fewer video tokens allow models to process 5-10× more frames within the same memory constraints.Existing methods use at least 256 tokens per frame, limiting the number of frames that can be input.
  • B.2.2. Future Direction: VisionZip encodes 10× more video frames than VideoLLaVA without increasing the token count in the Zootopia example.VideoLLaVA can encode only an 8-frame video and tends to describe a single frame in detail.
  • B.2.2. Future Direction: 2× training-time reduction is reported when VisionZip is applied during training on LLaVA-NeXT 7B with 640 retained visual tokens.This configuration also achieves better performance than applying VisionZip only during inference.
  • B.2.2. Future Direction: With 640 retained tokens, LLaVA-NeXT 13B achieves faster inference than the 7B model while maintaining superior performance.Inference measurements record prefilling and total testing time on TextVQA across 640, 320, and 160 token settings.
  • B.2.2. Future Direction: VisionZip addresses the high token requirements of high-resolution images, videos, and multiple-image inputs by extracting information more efficiently.The paper contrasts this goal with merely increasing visual-token length.
  • B.2.2. Future Direction: Prior efficient-VLM methods commonly identify redundancy from low LLM attention to visual tokens during the forward process, whereas VisionZip focuses on redundancy in vision-encoder outputs.The distinction motivates a vision-encoder-centered approach.
  • B.2.2. Future Direction: COCO visualizations show that only a few tokens receive high attention and substantial information, while most receive minimal attention and limited information.This pattern is presented as evidence of significant redundancy in popular vision encoders.
  • B.2.2. Future Direction: Attention is broad in early CLIP layers, converges onto a few tokens in middle layers, and is most concentrated by the 23rd layer used for visual-token extraction.Attention disperses again in the final layer during alignment with the CLIP text branch.

D.3. Visualization of Feature Misalignment

The visualization demonstrates that feature misalignment occurs across additional COCO images: selected tokens attend to dominant rather than semantically similar tokens. This helps explain why semantic token-selection methods may choose less informative tokens, and the authors provide a Gradio demo for analysis.

  • Feature misalignment appears across additional COCO images, indicating that the phenomenon is widespread.
  • Selected tokens from image subjects attend to dominant tokens rather than semantically relevant tokens.The visualization compares attention to a selected token with the attention map for the whole image.
  • Text-relevant token selection can identify semantically relevant tokens that contain less information than dominant tokens aggregating information.
  • A Gradio demo was developed to support visualization and analysis of visual redundancy and feature misalignment.The corresponding code is provided on the GitHub page.
Loading 2412.04467v2…