PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

Shaolei Zhang, Qingkai Fang, Zhe Yang, Yang Feng

arXiv:2501.03895v2cs.CVcs.AIcs.CL

TL;DR

LMMs are computationally expensive because visual inputs add many vision tokens to parameter-heavy LLM contexts, while prior efficiency work often overlooks token quantity. LLaVA-Mini analyzes layer-wise vision-token use, pre-fuses visual information into text, and compresses each image to one vision token. Across image and video benchmarks, it maintains comparable performance to LLaVA-v1.5 while reducing computation and latency and supporting very long videos.

  • Problem

    LMMs incur substantial overhead from large model parameters and numerous vision tokens, while existing efficiency methods often focus on model downsizing rather than reducing token quantity.

  • Method

    LLaVA-Mini analyzes layer-wise vision-token importance, pre-fuses visual information into instruction text, and compresses the vision input to one token per image.

  • Results

    Across 11 image and 7 video benchmarks, LLaVA-Mini achieves performance comparable to LLaVA-v1.5 using 1 vision token instead of 576, with 77% FLOPs reduction and latency below 40 ms.

  • Takeaways & Limitations

    LLaVA-Mini supports efficient image, high-resolution image, and video understanding, including videos exceeding 10,000 frames on 24GB GPU hardware.

Abstract

from arXiv · show

The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LLMs), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards efficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve a high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens in advance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.

1 INTRODUCTION

LMMs incur high computational costs because visual inputs add many vision tokens to parameter-heavy LLM contexts, especially for high-resolution images and videos. LLaVA-Mini shifts visual-text fusion before the LLM and compresses each image to one vision token while maintaining comparable performance.

  • Motivation: 576 vision tokens represent one image with CLIP ViT-L/336px, increasing computational complexity when integrated into the LLM context.The added context increases computational overhead and inference latency.
  • Motivation: Existing efficiency approaches emphasize model downsizing, quantization, or token reduction, while often overlooking vision-token quantity as a route to shorten context.The paper identifies token-count reduction as a complementary efficiency avenue.
  • Analysis: Layer-wise analysis finds vision tokens are crucial in early LLM layers but receive sharply less attention in deeper layers, where instruction tokens dominate.Later-layer token removal can preserve some visual understanding, motivating earlier fusion.
  • Approach: LLaVA-Mini pre-fuses visual information into instruction text before the LLM and compresses the remaining vision input to one token per image.The design targets efficient high-resolution image and long-video processing while preserving visual understanding.
  • Results: 11 image and 7 video benchmarks show performance comparable to LLaVA-v1.5 using 1 vision token instead of 576.The reported compression rate is 0.17%.

2 RELATED WORK

Prior efficient LMMs reduce model size, quantize computation, or shorten visual context through token and frame reduction. LLaVA-Mini instead studies vision-text interaction inside the LLM and uses modality pre-fusion for extreme token compression with comparable performance.

  • Model efficiency: Earlier efficiency methods replace the LLM backbone with smaller models, but reduced parameter scale can impair visual-task capabilities.Quantization is another established efficiency strategy.
  • Image token reduction: Image-based methods merge similar vision tokens or use Q-formers to compress them to fixed lengths, but direct reduction can lose visual information.The cited approaches include token merging, PruMerge, TokenPacker, Qwen-VL, and MQT-LLaVA.
  • Video efficiency: Video-based methods select fixed frames or condense videos into fixed-length representations, potentially losing key frames or temporal information.This limitation arises when videos vary in length and content.
  • LLaVA-Mini: LLaVA-Mini explores vision-text interaction within the LLM and introduces modality pre-fusion, enabling one vision token with comparable performance.This extends token reduction beyond operations performed solely after the vision encoder.

3 HOW DOES LLAVA UNDERSTAND VISION TOKENS?

The analysis examines how vision tokens contribute across LMM layers using attention, entropy, visualization, and token-removal experiments. It finds broad, important visual-token usage early, followed by reduced reliance later, motivating LLaVA-Mini’s one-token architecture and pre-fusion design.

  • 3.1 LLAVA ARCHITECTURE: LLaVA encodes visual inputs into projected vision tokens and combines them with embedded language instructions as input to an LLM.The architecture generates language responses from visual inputs and language instructions.
  • 3.2 PRELIMINARY ANALYSES: The analysis compares attention behavior across LLaVA variants with different model sizes and training datasets to identify shared layer-wise characteristics.The evaluated models include Vicuna-, Mistral-, and LLaVA-NeXT-based systems.
  • Attention analysis: Vision tokens receive more attention in early layers, while deeper layers direct over 80% of attention toward instruction tokens.The authors interpret this shift as early visual-information fusion into instruction tokens before response generation.
  • Entropy analysis: Higher early-layer attention entropy indicates that most visual tokens are attended to relatively evenly at those layers.This supports the importance of retaining broad visual-token coverage early in processing.
  • Visualization analysis: Attention visualization shows broad visual-token attention early but concentration on only some visual tokens later.The pattern is presented as evidence that direct early token reduction can lose visual information.
  • Token-removal analysis: Removing vision tokens in early layers completely eliminates visual understanding, whereas removal in higher layers has minimal effect.The experiment evaluates LLaVA-v1.5 on GQA and MMBench across layer groups.
  • LLaVA-Mini architecture: LLaVA-Mini’s architecture represents each image with one vision token and combines query-based compression with modality pre-fusion.The visual encoder and projection layer map image features into the LLM embedding space before the compressed input is processed.

4 LLAVA-MINI

LLaVA-Mini compresses visual inputs to minimal vision tokens while preserving visual information by moving visual-text fusion before the LLM backbone. Its architecture supports standard and high-resolution images, videos, and two-stage training.

  • Architecture: LLaVA-Mini uses a compression module and modality pre-fusion module to reduce vision tokens while retaining visual information.Pre-fusion integrates visual information into text tokens before the LLM backbone, while compression reduces the remaining vision tokens.
  • Architecture: The vision encoder extracts image features, which are projected into the LLM word-embedding space as vision tokens, while instructions become text representations.The visual and textual representations share the LLM embedding dimension before subsequent processing.
  • Vision Token Compression: A query-based compression module uses learnable queries and cross-attention to selectively extract important information from all vision tokens.The compressed output contains C × C vision tokens, with similarity weights relating compressed queries to the original tokens.
  • Modality Pre-fusion: The pre-fusion module concatenates vision and text tokens, then extracts the text-token outputs as fusion tokens containing related visual information.The compressed vision tokens and fusion tokens are jointly fed into the LLM.
  • High-Resolution Image: For high-resolution images, four sub-images and the original image provide local and global visual information before compression to C2 vision tokens.The resulting LLM input contains C2 compressed vision tokens plus lq fusion tokens.
  • Video and Training: For videos, LLaVA-Mini represents each frame with minimal vision tokens, reducing the token and memory burden of processing many frames.LLaVA-Mini follows a two-stage training process: projection-only vision-language pretraining followed by end-to-end instruction tuning with compression and pre-fusion.

5 EXPERIMENTS

Experiments evaluate LLaVA-Mini on image and video understanding benchmarks against established multimodal baselines. The results show comparable or superior understanding with substantially lower computational, latency, and memory costs.

  • Experimental Setting: LLaVA-Mini is evaluated on 11 image benchmarks and 7 video benchmarks using comparisons with advanced image- and video-based LMMs.The experimental setting includes standard benchmark tables and baseline comparisons.
  • Experimental Setting: LLaVA-Mini uses the same core configuration and training data as LLaVA-v1.5, with C set to 1 so vision tokens are compressed to one token.The standard configuration uses CLIP ViT-L/336px and Vicuna-v1.5-7B.
  • Main Results: LLaVA-Mini demonstrates superior overall performance on video benchmarks while representing each frame with one vision token.Compared with methods using more tokens per frame, this supports processing more frames within the LLM context.
  • Efficiency: 77% FLOPs reduction and 2.9 times speedup are achieved versus LLaVA-v1.5, with response latency below 40 ms.At high resolution, the reduction reaches 82% FLOPs and 3.76 times speedup.
  • Efficiency: 0.6 MB per image is required by LLaVA-Mini, compared with approximately 200-358 MB for previous methods during video processing.The comparison concerns memory requirements for processing videos of varying lengths.

6 ANALYSES

The analyses identify modality pre-fusion as central to preserving performance under extreme vision-token compression, while examples show strong image and video understanding with one token per frame.

  • 6.1 SUPERIORITY OF MODALITY PRE-FUSION: Without pre-fusion, compressing vision tokens causes around a 5% performance drop, even when retaining 144 tokens.Under equal FLOPs, adding pre-fusion layers provides greater benefit than adding compressed vision tokens.
  • 6.2 EFFECT OF QUERY-BASED COMPRESSION: Query-based compression adaptively captures important image features with minimal additional computational cost and outperforms direct average pooling.The comparison is reported in the query-based compression ablation.
  • 6.3 SCALABILITY WITH VISION TOKENS: LLaVA-Mini improves performance as the number of vision tokens increases and outperforms LLaVA-v1.5 when both use 576 tokens.It uses 1 token for standard images and 64 for high-resolution images.
  • 6.4 CASE STUDY: With one vision token per frame, LLaVA-Mini can process videos at 1 fps, whereas Video-LLaVA’s fixed eight-frame sampling can miss key frames and induce hallucinations.A case study contrasts LLaVA-Mini’s correct celebration description with Video-LLaVA’s imagined soccer scene.

7 CONCLUSION

The paper concludes that LLaVA-Mini uses minimal vision tokens to support image and video understanding while improving computational efficiency, latency, and memory usage.

  • 7 CONCLUSION: LLaVA-Mini is an efficient LMM with minimal vision tokens for image and video understanding.The model is presented as a practical solution for real-time multimodal interaction.
  • 7 CONCLUSION: The attention analysis measures how instruction, vision, and response tokens attend to one another across LLM layers.Attention weights are averaged across heads and analyzed by token type.
  • 7 CONCLUSION: Attention entropy normalizes attention weights within a token type before entropy is computed.This accounts for token-type attention sums that may not equal one.
  • 7 CONCLUSION: The analysis treats global and local vision tokens collectively and retains the original resolution settings for LLaVA-v1.5 and LLaVA-NeXT.Different vision-token subtypes are not distinguished in the analysis.

B TRAINING DETAILS

LLaVA-Mini follows a two-stage training process and adds compression and modality pre-fusion modules before the LLM while leaving the main vision-language components unchanged.

  • B TRAINING DETAILS: The model adds compression and modality pre-fusion modules before the LLM, while keeping the vision encoder, LLM, and training loss unchanged.The pre-fusion module uses the same decoder-only architecture, structure, and hyperparameters as the LLM.
  • B TRAINING DETAILS: Training follows two stages: vision-language pretraining followed by instruction tuning.The detailed two-stage training settings are reported in Table 9.

C BENCHMARKS

The evaluation covers broad image and video understanding benchmarks and compares LLaVA-Mini with established, efficient, and long-video multimodal models.

  • C.1 IMAGE-BASED BENCHMARKS: The evaluation includes 11 widely adopted image-based benchmarks spanning visual question answering, OCR, hallucination, reasoning, and general multimodal understanding.The listed benchmarks include VQAv2, GQA, VisWiz, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, SEED-Bench, LLaVA-Bench-in-the-Wild, and MM-Vet.
  • C.2 VIDEO-BASED BENCHMARKS: Video evaluation includes open-ended QA benchmarks and a generative benchmark assessing correctness, detail, contextual understanding, and temporal understanding.The open-ended benchmarks include MSVD-QA, MSRVTT-QA, and ActivityNet-QA.
  • C.2 VIDEO-BASED BENCHMARKS: MVBench evaluates 20 challenging video-understanding tasks using multiple-choice accuracy.Its task categories include action, object, position, scene, count, attribute, pose, character, and cognition.
  • C.2 VIDEO-BASED BENCHMARKS: MLVU evaluates multi-task long-video understanding across topic reasoning, anomaly recognition, needle QA, ego reasoning, plot QA, action order, and action count.The benchmark uses multiple-choice accuracy.
  • C.2 VIDEO-BASED BENCHMARKS: EgoSchema assesses long-form question answering for first-person videos using multiple-choice accuracy.It is designed to measure long-video understanding capabilities.
  • C.3 COMPARISON MODELS: LLaVA-Mini is compared with image-based LMMs, efficient models using fewer vision tokens, advanced video LMMs, and long-video systems.The comparison set includes LLaVA-v1.5, token-merging approaches, and models such as Video-LLaVA and MovieChat.
  • C.3 COMPARISON MODELS: LLaVA-Mini shifts token-reduction attention from the vision encoder to vision-text interaction inside the LLM, enabling one-token compression.The proposed modality pre-fusion is the mechanism associated with this shift.
  • C.4 ABLATIONS AND EFFICIENCY: At the same compression rate, LLaVA-Mini’s compression module outperforms PruMerge, PruMerge++, and MQT-LLaVA.This comparison supports the advantage of query-based compression.

F VISUALIZATION OF COMPRESSION

LLaVA-Mini uses query-based cross-attention to compress visual tokens while preserving information, adapting attention to concentrated or distributed image content. The visualizations show this mechanism focusing on salient regions and components.

  • Learnable queries interact with all vision tokens through cross-attention to capture key visual information during compression.The method is designed to adaptively compress vision tokens while preserving essential information.
  • Brighter heat-map regions indicate locations receiving greater weight during compression across photographs, text, screenshots, and cartoons.
  • When critical information is concentrated, LLaVA-Mini focuses attention on the corresponding key locations.Examples include cases labeled (b), (d), (h), and (i) in Figure 12.
  • When the main object is unclear, its attention becomes more dispersed to preserve a broader range of visual information.This adaptive pattern is reported for cases such as (f), (j), (i), and (m).
  • For a complex image with multiple related sub-figures, the module attends to the charger shape, product name, and charging port.This example illustrates adaptive focus on several visually and logically relevant components.

G MORE CASES

The additional cases show LLaVA-Mini handling complex image reasoning, difficult cursive OCR, and longer first-person videos with one vision token. Compared with other evaluated models, it captures more of the relevant visual content in these examples.

  • Image Understanding: Only LLaVA-Mini and GPT-4o correctly capture the metaphorical and counterfactual phrases in the complex image-reasoning example.The example requires interpreting visual entities and reasoning about the scene’s humorous aspects.
  • Image Understanding: Video-LLaVA and LLaVA-v1.5 hallucinate in the complex reasoning example, misdescribing the image or failing to recognize the violin.
  • Image Understanding: Both LLaVA-Mini and GPT-4o accurately recognize unusual cursive handwriting, while Video-LLaVA and LLaVA-v1.5 make multiple recognition errors.The errors include incorrect words and an added exclamation mark.
  • Video Understanding: LLaVA-Mini provides a more comprehensive and detailed understanding of the longer first-person video, including entities such as a yoga mat.Video-LLaVA imagines that the person takes a shower, while Video-ChatGPT gives shorter responses lacking some details.

H DETAILED RESULTS ON MVBENCH

This section reports detailed results for the 20 MVBench subsets corresponding to the paper’s aggregate results in Table 3.

  • Table 15 reports detailed results on each MVBench subset corresponding to Table 3.
  • The supplied passage identifies the table as a detailed breakdown rather than reporting individual subset values.
  • The table covers 20 MVBench subsets.
Loading 2501.03895v2…