Source-linked AI summary
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, Baobao Chang
TL;DR
LVLMs incur high inference costs because visual inputs become long token sequences, while deep layers devote little attention to visual tokens. FastV uses early-layer attention to rank and prune visual tokens later, achieving substantial FLOPs reductions without sacrificing performance across image and video tasks. The method also supports higher-resolution processing at equal token counts, though its explanation relies on an assumption about visual redundancy and anchor tokens.
Problem
LVLMs transform images into hundreds or thousands of tokens, increasing computational complexity, while deep-layer attention to image tokens is markedly lower than attention to textual tokens.
Method
FastV is a plug-and-play method that ranks visual tokens by attention at a selected layer and prunes low-attention tokens in later layers.
Results
FastV achieves about 45% FLOPs reduction at K=2 and R=50% across different LVLMs without sacrificing performance, while also outperforming lower-resolution features at equal image-token counts.
Takeaways & Limitations
FastV can make a 13B LVLM infer as fast as a 7B model while maintaining superior A-OKVQA performance and can enable higher-resolution inputs without additional inference costs.
Takeaways & Limitations
The proposed explanation assumes that redundant visual signals aggregate image-related, instruction-specific features onto non-image anchor tokens in shallow layers.
Abstract
from arXiv · showhide
In this study, we identify the inefficient attention phenomena in Large Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5, QwenVL-Chat and Video-LLaVA. We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs, suggesting a need for a sparser approach compared to textual data handling. To this end, we introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent ones. Our evaluations demonstrate FastV's ability to dramatically reduce computational costs (e.g., a 45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a wide range of image and video understanding tasks. The computational efficiency and performance trade-off of FastV are highly customizable and pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve a lower budget than that of a 7B-parameter model, while still maintaining superior performance. We believe FastV has practical values for deployment of LVLMs in edge devices and commercial models. Code is released at https://github.com/pkunlp-icler/FastV.
1 Introduction
LVLMs process hundreds or thousands of visual tokens, creating substantial computational costs, while deep-layer attention to those tokens is highly inefficient. FastV uses early-layer attention signals to prune visual tokens later, reducing inference cost while preserving performance across tasks and models.
- Motivation: Hundreds or thousands of visual tokens increase LVLM computational complexity and inference cost.This burden accompanies the use of large language-model components for vision-language capabilities.
- Motivation: After layer 2, LLaVA 1.5 image tokens receive only 0.21% of system-prompt attention, versus 50% in the initial two layers.The imbalance between visual and textual attention changes sharply from shallow to deep layers during image captioning.
- FastV: FastV dynamically ranks visual tokens at a selected layer and removes low-attention tokens in subsequent layers.Computations proceed normally before the selected layer, after which tokens are filtered according to average received attention.
- FastV: Direct token elimination reduces both self-attention and FFN computation in deeper layers, unlike methods that only optimize attention sparsity.This design provides a theoretical FLOPs reduction while retaining relatively high performance.
- Results: FastV enables higher-resolution image processing at equal image-token counts, producing better performance than lower-resolution features.This supports increasing image resolution without additional inference costs under the tested settings.
2 Related Work
LVLMs inherit growing token and attention costs from visual prompting and autoregressive generation. FastV addresses this efficiency problem by reducing visual tokens inside LVLMs using language-model attention signals without retraining.
- LVLM efficiency: Higher-resolution images require longer visual-token sequences and incur quadratic growth in token-related computation.Visual prompt generators transform image embeddings into language-model-compatible prompts, increasing the required token count.
- Inference optimization: Autoregressive generation becomes progressively slower as context length increases because attention computation grows with the context.Prior inference-optimization methods target memory consumption or long-context processing.
- Related approaches: LLaMA-VID represents each video frame with two key tokens, but its additional fine-tuning stage limits broad applicability.The cited approach addresses long-context LVLM processing through cross-attention.
- Token reduction: FastV reduces LVLM visual tokens using signals from the language model and requires no model retraining.The paper presents this as a simple approach applicable across LVLMs and evaluates it on video-QA and comprehensive benchmarks.
3 Inefficient Visual Attention in VLLMs
LVLMs process visual and textual inputs as sequential embeddings, but attention allocation becomes increasingly imbalanced across layers. Image tokens receive the lowest attention efficiency, while deep-layer attention concentrates on system prompts and a small set of anchor tokens.
- Preliminaries: Images are encoded into visual embeddings, combined with text embeddings, and processed sequentially by a transformer decoder during autoregressive generation.The analysis categorizes inputs into system prompt, image, instruction, and output tokens and measures their attention during decoding.
- Experiment Settings: The study measures total attention allocation per token type and average attention efficiency per token across layers and sampled image-text responses.Attention statistics are averaged over attention heads and 1,000 image-text pairs using LLaVA1.5-7B.
- Results: The first 2 layers are defined as shallow, while the remaining 30 layers are defined as deep layers.This shallow/deep division is used to compare attention allocation and efficiency across the model depth.
- Results: In shallow layers, attention is relatively balanced and output tokens tend to attend to preceding output tokens, whereas deep-layer attention favors system prompts.The attention imbalance increases with layer depth.
- Results: 472 times: system prompts have higher deep-layer attention efficiency than image tokens and account for 85% of total attention scores.Image tokens have the lowest attention efficiency in both shallow and deep layers.
- Results: FastV dynamically prunes a chosen percentage of image tokens after a selected layer, while the illustrated outputs retain correctness and reduce FLOPs.Image or video inputs are first converted into visual tokens and then processed by the language-model decoder.
- Results: Deep-layer attention maps show strong vertical lines on system prompts and sparse attention over image tokens, consistent with a small set of anchor tokens aggregating input information.Figure 4 compares attention distributions in shallow, middle, and final layers during one LLaVA1.5-7B response.
4 FastV
FastV is a training-free, plug-and-play method that ranks and prunes redundant image tokens at a selected transformer layer, reducing later-layer computation. Its cost reduction is controlled by filtering layer K and ratio R.
- Dynamically Prune Vision Tokens: FastV ranks image tokens at layer K and prunes the last R% according to their average received attention in subsequent layers.The method uses attention-based ranking in its experiments and requires no model training.
- Computing Cost Estimation: For one transformer layer, estimated computation is 4nd^2 + 2n^2d + 2ndm, covering multi-head attention and the feed-forward network.Here n is token count, d is hidden size, and m is FFN intermediate size.
- Computing Cost Estimation: After layer K, FastV changes the token count from n to n̂ = (1 − R%) · n across the remaining T layers when estimating total FLOPs.The resulting reduction depends on both the filtering layer and filtering ratio.
- Computing Cost Estimation: Theoretical FLOPs reduction varies with K and R, as illustrated by a heat map and a 3D parameter graph.These visualizations describe the computation trade-off across FastV configurations.
5 Experiment
Experiments evaluate FastV across image, video, fine-grained, and multimodal benchmarks using multiple open-source LVLMs. Results show substantial computation savings with maintained or competitive performance, while parameter settings and model architecture affect the trade-off.
- Evaluation Setup: FastV is evaluated on image captioning, VQA, multimodal reasoning, video QA, and fine-grained benchmarks using multiple open-source LVLMs.The models include LLaVA1.5-7B and 13B, Qwen-VL, Video-LLaVA, and InstructBLIP.
- Main Results: About 45% FLOPs reduction is achieved with K=2 and R=50% across different LVLMs without sacrificing performance.The cost-performance trade-off can be adjusted by lowering K and increasing R.
- Main Results: An 13B model with FastV can infer as fast as a 7B model while retaining superior A-OKVQA performance.The latency comparison uses a real inference budget test reported on a single A40 GPU.
- Main Results: FastV maintains PCA-Bench and OCR-VQA sub-scores while significantly reducing FLOPs.The scores cover perception, cognition, and action abilities.
- Video Understanding: FastV generally improves video-QA performance while saving more than 40% computation, especially on TGIF.Video-LLaVA processes substantially more visual tokens than LLaVA1.5 in the cited example.
- Fine-grained Benchmarks and More Models: InstructBLIP shows slightly greater degradation than LLaVA under the same configuration, but the gap closes when K is set to 5.The authors relate this sensitivity to Q-Former's prior reduction of image tokens.
- Balance between Cost and Performance: When K is small, lowering R improves performance with a smaller FLOPs reduction, whereas changing R has minimal impact when K is large.This pattern is observed on OCR-VQA and is interpreted as evidence of high deep-layer image-token redundancy.
- Training with Less Tokens: Training with 50% fewer image tokens through pooling causes diminished performance, whereas FastV reduces image tokens without compromising performance.The comparison uses retrained LLaVA1.5-7B models with average pooling after the CLIP encoder.
6 Conclusion
FastV is a plug-and-play method that reduces LVLM inference costs by pruning unnecessary visual tokens according to attention scores, while preserving performance.
- FastV reduces LVLM inference costs by pruning visual tokens with low attention scores.The method targets inefficient visual-token attention computation in deep layers.
B Evaluation Tasks Description
The evaluation covers image captioning, visual question answering, multimodal reasoning, video question answering, and fine-grained image-language benchmarks. These tasks use established datasets and task-specific metrics or evaluation procedures.
- Image Captioning: Image captioning is evaluated on Nocaps and Flickr30k using CIDEr, with the prompt “Describe the image in one sentence.”The task asks the model to generate an image description.
- Visual Question Answering: Visual question answering is evaluated on A-OKVQA and OCR-VQA using multiple-choice and Rouge-L scores, respectively.The task requires answering image-question pairs; A-OKVQA uses a multiple-choice evaluation prompt.
- Multimodal Reasoning: Multimodal reasoning is evaluated on MMMU and PCA-Bench, which target knowledge-intensive, multi-discipline reasoning and complex embodied reasoning with error localization.MMMU includes college-level subject knowledge and reasoning tasks.
- Video Question Answering: Video question answering is evaluated on TGIF-QA, MSVD-QA, and MSRVTT-QA following the Video-ChatGPT evaluation pipeline.Video inputs are formed by sampling multiple frames, producing longer image-token sequences; default Video-LLaVA questions and GPT evaluation are used.
- Fine-grained Benchmarks: Fine-grained evaluation uses MME, SEED-Bench, SciQA-IMG, and MMVet to assess perception, cognition, and related image-language abilities.MME emphasizes comprehensive, intuitive, and quantifiable analysis across diverse tasks without extensive prompt engineering.