Source-linked AI summary
Multi-Image Visual Token Pruning in Large Visual Language Models
Rongyang Zhang, Chengqiang Lu, Cong Li, Hongchao Gu, Tingjia Shen, Xuyang Zhi, Qimeng Wang, Yan Gao, Yi Wu, Yao Hu, Hao Wang, Enhong Chen
TL;DR
LVLM visual token pruning faces computational costs, attention-mechanism incompatibility, limited architectural generalizability, and multi-image context constraints. AVTP is a training-free framework using hidden-state variations, architecture-aware layer selection, and image-aware pruning rates; experiments show it consistently outperforms existing pruning methods across multi-image benchmarks.
Problem
Existing visual token pruning methods face computational costs from redundant visual tokens, dependence on attention computations incompatible with FlashAttention, limited architectural generalizability, and weak adaptation to multi-image scenarios.
Method
AVTP is a training-free framework that estimates token importance from hidden-state variations, selects pruning layers across architectures, and allocates pruning budgets adaptively across multiple images.
Results
AVTP consistently outperforms existing visual token pruning methods across a range of multi-image benchmarks.
Takeaways & Limitations
Image-aware adaptive pruning preserves more tokens for important images while pruning less important images more aggressively in multi-image contexts.
Takeaways & Limitations
Several critical AVTP hyperparameters and the relationship between hidden-state modifications and token significance lack formal mathematical substantiation.
Abstract
from arXiv · showhide
With the growing demand for processing multiple image sequences in real-world applications, various visual token pruning methods have emerged to mitigate the computational and context length constraints faced by Large Vision Language Models (LVLMs). However, most existing pruning approaches rely on static strategies that struggle to adapt across different architectural LVLMs and multi-image scenarios, and are additionally constrained by their dependence on attention computations that are incompatible with efficient techniques like FlashAttention. To address these limitations, we propose a training-free, Adaptive Visual Token Pruning (AVTP) framework, applicable to diverse LVLM architectures. We strategically determine pruning layers based on empirical analysis of visual attention distributions across various LVLMs, and implement adaptive pruning ratios in multi-image contexts where images of higher importance retain proportionally more tokens. We conduct extensive experiments across different LVLMs to demonstrate the effectiveness and robustness of AVTP. Specifically, Qwen3VL-8B achieves 2 times inference speedup while maintaining 96.1\% of its original accuracy on multiple multi-image benchmarks, InternVL3.5-8B retains 94.1\% accuracy, and LLaVA-OV-7B even exceeds its original baseline performance. Our code is available at \href{https://github.com/zry13/AVTP}{this link}.
1 Introduction
AVTP targets computational and architectural limitations in visual token pruning for LVLMs, especially in multi-image settings. It combines architecture-aware layer selection with image-aware pruning and reports broad benchmark effectiveness.
- Visual tokens grow quadratically with image resolution, making redundant-token removal essential for practical LVLM deployment.
- Existing pruning methods are limited by attention-computation requirements, poor generalizability across LVLM architectures, and weaker suitability for multi-image tasks.
- AVTP uses training-free, architecture-aware pruning-layer selection based on model-specific visual attention patterns.
- AVTP dynamically adjusts image-level keep ratios so less relevant images are pruned more aggressively while important images retain more visual tokens.
- Extensive experiments across three mainstream LVLMs and five multi-image benchmarks demonstrate AVTP's effectiveness and robustness.
2 Related Work
Related work covers LVLM architectures and four broad families of visual-token pruning methods. The supplied framework passage illustrates AVTP's distinct adaptive design for selecting layers and allocating pruning across images.
- Large Vision-Language Models: LVLMs combine a vision encoder, visual projection module, and large language model for multimodal comprehension.
- Token Pruning for LVLMs: Visual-token pruning methods include transformation-based, similarity-based, attention-based, and query-based approaches.
- AVTP Framework: AVTP identifies high-attention layers through sampling-based analysis and performs image-aware pruning with dynamically assigned keep ratios.
3 Method
AVTP combines hidden-state-based token importance, architecture-aware pruning-layer selection, and image-aware adaptive keep ratios for efficient multi-image LVLM inference. The method avoids intermediate attention storage while adapting pruning to model architecture and image relevance.
- Architecture-Aware Pruning Layer Selection: Attention-critical layers vary across LVLM architectures, so AVTP selects pruning layers from architecture-specific visual-attention distributions instead of always using shallow layers.LLaVA models show stronger shallow-layer visual attention, whereas this pattern is not universal across LVLMs.
- Method Overview: The unified design targets attention-mechanism incompatibility, model generalizability, and context limitations in multi-image pruning.It combines token importance assessment, architecture-aware layer selection, and differentiated image-level pruning.
- Token Importance Formulation: AVTP assesses visual-token importance from hidden-state variations across forward-propagation layers, requiring only forward passes and remaining compatible with FlashAttention.The approach uses token embedding changes rather than intermediate attention weights.
- Architecture-Aware Pruning Layer Selection: AVTP identifies pruning layers by ranking cumulative visual-token attention across sampled inputs, averaging layer rankings, and selecting the top-k layers.The selected layers are used with a progressive pruning schedule.
- Image-Aware Adaptive Pruning: For multi-image inputs, AVTP assigns each image an adaptive keep ratio so more important images retain more visual tokens and less important images undergo more aggressive pruning.The ratio uses a base keep ratio, an importance-dependent variance parameter, and the average image importance.
4 Experiments
AVTP is evaluated across three LVLM architectures and five multi-image benchmarks, with experiments examining efficiency, robustness, adaptive components, and pruning-layer choices. The results show consistent acceleration and competitive or improved accuracy, while ablations support dynamic layer selection and image-aware pruning.
- Experimental Setting: Experiments cover InternVL3.5-8B, LLaVA-OneVision-7B, and Qwen3VL-8B across five multi-image benchmarks.The benchmarks are MuirBench, MIRB, BLINK, Qbench2, and NLVR2.
- Main Results: AVTP consistently reduces inference latency across evaluated models and can improve multi-image performance when unpruned sequences exceed the model’s training-time context range.The authors attribute LLaVA-OneVision-7B’s gains to shortening sequences back into the model’s context-length comfort zone.
- Main Results: AVTP offers a more favorable accuracy–latency trade-off across InternVL3-8B, LLaVA-OneVision-7B, and Qwen3VL-8B than prior pruning methods.It reaches the highest or near-highest accuracy on several benchmarks while maintaining comparable or lower latency, with strongest results on selected MuirBench and Qbench2 evaluations.
- Main Results: AVTP remains effective across heterogeneous LVLM architectures, whereas competing methods can lose their gains when transferred between models.The evaluated models differ in backbone design, vision–language fusion, and pretraining data.
- Ablation Study: Removing dynamic layer selection and adaptive pruning rates degrades performance, while restoring either component improves accuracy and the complete configuration performs best.The ablations evaluate InternVL3.5-8B and Qwen3VL-8B across MuirBench, MIRB, and BLINK; adaptive pruning shows particularly pronounced gains on BLINK.
- Ablation Study: AVTP maintains robust performance as the number of input images increases, outperforming other methods across varying image quantities and achieving the highest scores in three of four configurations.On Qwen3VL-8B’s MuirBench evaluation, FAST-V shows a marked drop for cases with at least eight images, whereas AVTP remains robust.
5 Conclusion
AVTP is a training-free framework addressing architecture dependence and weak multi-image performance in visual token pruning. It combines adaptive layer selection, hidden-state-based token importance, and image-aware pruning rates, with experiments showing consistent improvements and complementary component benefits.
- AVTP addresses existing pruning methods’ dependence on specific model architectures and suboptimal performance in multi-image scenarios.
- AVTP adaptively selects pruning layers across diverse architectures through Dynamic Pruning Layer Selection.
- The framework estimates visual token importance from hidden-state variations and allocates pruning budgets across images using Image-Aware Adaptive Pruning Rates.
- Extensive multi-image benchmark experiments show AVTP consistently outperforms existing visual token pruning methods.
- Ablation studies validate each component and highlight their complementary benefits.
6 Limitations
The authors identify insufficient theoretical grounding for several AVTP hyperparameters, limited evaluation against complementary acceleration methods, and evolutionary rather than fundamentally paradigm-shifting algorithmic novelty.
- Several AVTP hyperparameters are established through exhaustive empirical experimentation because they lack sufficient theoretical grounding.This includes top-k for layer selection, α for keep-ratio variation, and the correlation between embedding modifications and token significance.
- The evaluation primarily compares AVTP with visual token pruning methods rather than comprehensively analyzing quantization and knowledge distillation.The authors state that AVTP is inherently compatible with these orthogonal techniques.
- The visual token importance formulation advances V2DROP but represents an evolutionary advancement rather than a fundamentally novel algorithmic paradigm.
A Additional Ablation Experiments
Additional ablations evaluate AVTP across pruning ratios on Qwen3VL-8B, showing a trade-off between inference efficiency and performance as fewer tokens are retained.
- AVTP is evaluated on Qwen3VL-8B across various pruning ratios, with experiments conducted on a single A100 80GB GPU.
- Higher pruning ratios improve inference efficiency but produce a corresponding performance trade-off.
- 33.17% on MIRB and 50.13% on BLINK are achieved at a 5% keep ratio, down from baseline scores of 45.21% and 56.71%, respectively.
A.2 Effect of the Hyperparameter α in Image-Aware Adaptive Pruning
The α ablation examines how amplifying image-aware pruning affects pruning-rate allocation, while noting that larger values can exceed valid bounds and offer limited benefits on fewer-image benchmarks.
- α controls the variance in pruning ratios across different images in Image-Aware Adaptive Pruning.
- The main experiments set α to 1.0, while the ablation increases it to amplify image-aware pruning’s influence on pruning-rate allocation.
- Larger α values may produce pruning rates above 1 or below 0, so constraints are imposed to prevent extreme cases.
- For benchmarks with fewer input images, including BLINK, QBench2, and NLVR2, larger α values provide marginal benefits and smaller values can perform better.
B Performance on Single-Image and Video Benchmarks
The section extends AVTP evaluation to single-image VQA and video benchmarks under a uniform 50% vision-token retention setting.
- AVTP is evaluated on several single-image VQA benchmarks and video benchmarks beyond the main multi-image experiments.
- 50% of vision tokens are retained while the remaining tokens are pruned in all pruning-method comparisons.
- All other experimental settings remain identical to those used previously.
B.1 Evaluation Benchmark
The evaluation spans image and video benchmarks covering perception, reasoning, multimodal understanding, hallucination, cognition, and temporal comprehension.
- MMBench: MMBench assesses perception and reasoning through three hierarchical levels, from core abilities to 20 specific dimensions.
- ScienceQA: ScienceQA evaluates multimodal understanding, multi-step reasoning, and interpretability across hierarchically organized topics, categories, and skills.
- POPE: POPE measures object hallucination using binary object-presence questions and Accuracy, Recall, Precision, and F1 Score across three sampling strategies.
- MME: MME evaluates perceptual and cognitive abilities across 14 subtasks using manually designed instruction-answer pairs to minimize data leakage.
- MVBench: MVBench defines 20 video-understanding tasks requiring comprehension of temporal dimensions beyond single-frame analysis.
B.2 Experimental Results
Additional experiments compare pruning methods on single-image and video benchmarks, where AVTP achieves the best reported performance on four evaluated benchmarks.
- Additional comparative experiments evaluate MMBench, MMStar, POPE, MME, and MVBench, with results presented in Table 7.
- FAST-V encounters numerous out-of-memory cases on MVBench, causing significantly reduced accuracy and unmeasurable latency.
- AVTP achieves the best performance on MMBench, MMStar, MME, and MVBench.
- Image-Aware Adaptive Pruning does not provide benefits for single-image and video benchmarks.