Source-linked AI summary
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, Dahua Lin
TL;DR
LVLMs incur high costs because image-token counts grow rapidly with resolution, while earlier compression can lose useful visual information. PyramidDrop studies layer-wise redundancy, retains more tokens in shallow layers, and progressively drops attention-ranked tokens in deeper stages. It reports comparable performance with substantial training and inference acceleration, including plug-and-play inference gains over FastV.
Problem
High-resolution LVLM inputs create excessive vision tokens and rapidly rising transformer costs, while prior compression strategies can lose crucial image information.
Method
PyramidDrop partitions the LVLM into stages and progressively drops image tokens using lightweight attention-based importance ranking, while retaining all tokens in shallow layers.
Results
PyramidDrop achieves up to 1.82× training and 2.22× inference acceleration, with comparable performance and stronger inference results than FastV.
Takeaways & Limitations
Layer-aware visual-token reduction supports efficient LVLM training and inference without the performance loss associated with premature compression.
Takeaways & Limitations
Video redundancy differs substantially from image redundancy, and further work is needed for more complex video visual-question-answering tasks.
Abstract
from arXiv · showhide
In large vision-language models (LVLMs), images serve as inputs that carry a wealth of information. As the idiom "A picture is worth a thousand words" implies, representing a single image in current LVLMs can require hundreds or even thousands of tokens. This results in significant computational costs, which grow quadratically as input image resolution increases, thereby severely impacting the efficiency of both training and inference. Previous approaches have attempted to reduce the number of image tokens either before or within the early layers of LVLMs. However, these strategies inevitably result in the loss of crucial image information, ultimately diminishing model performance. To address this challenge, we conduct an empirical study revealing that all visual tokens are necessary for LVLMs in the shallow layers, and token redundancy progressively increases in the deeper layers of the model. To this end, we propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost their efficiency in both training and inference with neglectable performance loss. Specifically, we partition the LVLM into several stages and drop part of the image tokens at the end of each stage with a pre-defined ratio, creating pyramid-like visual tokens across model layers. The dropping is based on a lightweight similarity calculation with a negligible time overhead. Extensive experiments demonstrate that PyramidDrop can achieve a 40% training time and 55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance. Besides, the PyramidDrop could also serve as a plug-and-play strategy for inference acceleration without training, with better performance and lower inference cost than counterparts. Code is available at https://github.com/Cooperx521/PyramidDrop.
1. Introduction
LVLMs face rapidly increasing computational costs because high-resolution images produce many redundant visual tokens. PyramidDrop preserves tokens in shallow layers, progressively removes them in deeper layers, and accelerates training and inference with comparable performance.
- Motivation: High-resolution images create quadratically increasing vision-token sequences, making transformer training and inference prohibitively expensive.The resulting sequences can reach tens of thousands of tokens.
- Motivation: Earlier compression methods either remove tokens before the base LLM or drop them in shallow layers, causing information loss and performance degradation.Shallow-layer dropping occurs before the model fully understands the image.
- Visual redundancy: LVLMs need most image tokens in shallow layers, while token redundancy increases and fewer instruction-relevant tokens remain necessary in deeper layers.Attention is broad and relatively uniform early, then becomes sparse and focused on question-related image regions.
- PyramidDrop: PyramidDrop partitions the LVLM into stages, retains all shallow-layer image tokens, and progressively drops tokens using lightweight attention ranking.The strategy is designed to avoid early information loss while reducing deeper-layer computation.
- Results: 40% training time reduction was achieved for LLaVA-NeXT-7B across 16 vision-language tasks without sacrificing performance.PyramidDrop also supports doubled input resolution using 70% of vanilla LLaVA-NeXT training time.
- Results: PyramidDrop provides inference acceleration as a plug-and-play strategy, with enhanced performance and fewer FLOPs than FastV.It can be applied without additional training.
2. Related Work
Prior token-reduction work targets language-model inference or compresses visual tokens before or near the beginning of LVLM processing. PyramidDrop instead studies visual redundancy across layers and progressively reduces tokens for both training and inference.
- Token reduction: Language-model token-reduction methods prune or retain key-value cache entries to accelerate inference and reduce memory use.Examples include attention-sink retention, adaptive cache management, and heavy-hitter pruning.
- PyramidDrop: Unlike prior token-reduction approaches, PyramidDrop provides a progressive visual-token reduction solution for both LVLM training and inference.Its distinction is a broader study of visual redundancy in LVLMs.
- LVLMs: LVLMs have increased image-token counts for high-resolution understanding, intensifying the need for efficient visual processing.The passage situates this development alongside models such as LLaVA, InstructBLIP, and MiniGPT-4.
3. Method
PyramidDrop models visual redundancy as layer-dependent: it keeps more tokens early and progressively drops less relevant tokens later. Staged attention-based selection reduces sequence length and computation while preserving performance across inference settings.
- Visual redundancy study: The redundancy study removes varying image-token ratios at selected layers of LLaVA-v1.5-7B and evaluates performance on TextVQA.Retained tokens are those receiving the highest attention from text tokens.
- Visual redundancy study: At layer 16, retaining only 10% of image tokens causes no obvious performance decline, while layer-24 performance is nearly independent of image tokens.These results indicate that necessary image information has already been captured in deeper layers.
- Visual redundancy study: Attention maps change from broad shallow-layer attention over most image tokens to sparse middle-layer attention focused on question-related local regions.This supports progressively increasing redundancy with depth.
- Design motivation: Uniform compression across layers can discard valuable shallow-layer information while retaining unnecessary deeper-layer redundancy.PyramidDrop is motivated by this mismatch between fixed compression and layer-dependent redundancy.
- Progressive visual redundancy reduction: PyramidDrop partitions the language model into stages and drops a predefined fraction of image tokens at each stage boundary.The forward pass retains a higher proportion of tokens in shallow layers and reduces them progressively afterward.
- LVLM formulation: The model input combines projected vision tokens with tokenized instruction and answer text, which are transformed jointly through decoder layers.The formulation denotes the pretrained LVLM as M = (L, V, P).
- Token selection: Image-token importance is ranked using attention between all image tokens and the instruction’s last token, reusing query and key states from the stage’s self-attention block.The calculation is intended to remain efficient and consistent between training and inference.
- Token schedule: With retention ratio λ, the image-token count decreases geometrically across stages as V_s = V_0 · λ^(s−1).The method therefore produces rapidly shrinking visual sequences in deeper layers.
4. Experiment
Experiments across image, high-resolution, and video settings show that PyramidDrop reduces training or inference cost while generally preserving multimodal performance. Its multi-stage token reduction also remains effective across architectures, resolutions, and inference-only use.
- Inference acceleration: PyramidDrop outperformed FastV during inference, including 0.5% higher TextVQA and 0.7% higher SEED-Bench (Image) results.On LLaVA-NeXT, it also scored 1533.0 on MME, 1.5% above FastV, and exceeded FastV by 0.4% on GQA.
- High-resolution benchmarks: Comparable performance was maintained on fine-grained benchmarks such as TextVQA, DocVQA, and OCRVQA despite image-token compression.The results indicate that redundant information was compressed while critical image content was preserved.
- Higher-resolution settings: 55% of vanilla LLaVA-NeXT-p9 training time was required with more image patches, while average performance across 16 benchmarks was slightly higher.Training also used 70% of the original LLaVA-NeXT-p5 time while achieving better performance on diverse tasks.
- Model properties: PyramidDrop-trained models preserve more image information at a given retained-token ratio, or allow more compression at equivalent performance.The multi-stage training strategy progressively prunes tokens and encourages essential information to consolidate into fewer, denser representations.
- Video LVLMs: PyramidDrop reduced Video-LLaVA training time by 27.8% while obtaining comparable results on TGIF, MSVD, and MSRVTT.Inference-only use on video tasks also achieved accuracy and scores comparable to vanilla Video-LLaVA with lower inference FLOPs.
5. Conclusion
PyramidDrop reduces visual token redundancy in LVLMs while boosting training and inference efficiency without performance loss. Its empirical basis is that redundancy increases progressively from shallow to deeper layers.
- PyramidDrop reduces visual token redundancy in LVLMs while targeting efficient deployment without performance loss.The strategy is designed to concentrate computation on valuable visual information.
- 1.82× training and 2.22× inference acceleration are achieved with PyramidDrop.
B. Ablation Study about Stage S
The stage count controls when and how often PyramidDrop removes image tokens. More stages prune earlier and leave fewer tokens later, while fewer stages retain more redundancy.
- More stages remove more image tokens at earlier layers, leaving fewer tokens for later language-model layers.
- Fewer stages reduce the number of token-compression steps, resulting in more redundancy later in the forward process.
- The stage count therefore determines the balance between earlier token compression and later visual-token retention.
B.1. Results Analysis
PyramidDrop remains robust across stage counts 3 to 5, but excessive staging harms performance on challenging benchmarks. The authors select four stages as a balance between preserving performance and pruning redundancy.
- Results Analysis: Performance remains robust when the number of stages varies from 3 to 5, indicating limited sensitivity to this hyperparameter.
- Results Analysis: A noticeable performance decline occurs on SEED Bench and TextVQA when the number of stages increases to 5.
- Results Analysis: With 3 or 4 stages, no significant performance drop is observed, so the authors select S = 4.This setting balances performance preservation with effective redundancy pruning.
- Results Analysis: Increasing the stage count prunes more tokens earlier, whereas fewer stages increase redundancy through fewer compression steps.