PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Yanwei Li, Chengyao Wang, Jiaya Jia

arXiv:2311.17043v1cs.CVcs.CL

TL;DR

Long-video VLMs face computational burdens because representing consecutive frames requires excessive visual tokens. LLaMA-VID uses two tokens per frame—an instruction-conditioned context token and a content token—and reports leading performance on most video- and image-based benchmarks while supporting efficient long-video processing.

  • Problem

    VLMs require excessive visual tokens for long videos, making computational demands grow with video length and constraining practical use.

  • Method

    LLaMA-VID represents each frame with an instruction-conditioned context token and a content token that captures visual cues, using adaptive token compression and cross-modality attention.

  • Results

    LLaMA-VID outperforms previous leading methods on most video- and image-based benchmarks, with Vicuna-7B reaching 69.7% on MSVD-QA and 57.7% on MSRVTT-QA.

  • Takeaways & Limitations

    The dual-token representation enables efficient encoding of long videos while preserving image details and extends VLM capacity with an additional context token.

Abstract

from arXiv · show

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

1. Introduction

LLaMA-VID targets the excessive visual-token cost of long-video VLMs by encoding each frame with context and content tokens. The method combines instruction-guided context aggregation with efficient content representation and reports leading performance across most video- and image-based benchmarks.

  • Motivation: Long videos require excessive visual tokens, causing computational demands to increase with video length and limiting practical VLM application.A 10K-frame video would require over 320K tokens in models using 32 or over 256 tokens per image.
  • LLaMA-VID: LLaMA-VID represents each frame with a context token encoding user-conditioned image context and a content token preserving frame-level visual cues.The content representation can use one token per video frame or more than 256 tokens per single image, depending on computational constraints.
  • LLaMA-VID: The method generates context tokens through instruction-guided cross-modality attention and content tokens through adaptive average pooling before projection into the LLM space.Global pooling favors video efficiency, while additional tokens preserve image details; the framework also includes a 9K-conversation movie-level instruction dataset.
  • Contribution: Each video frame can be encoded with only two tokens, while the context token aggregates informative image features to extend VLM capacity.This dual-token design is presented as reducing long-video overload while preserving critical information.
  • Results: LLaMA-VID outperforms previous leading methods on most video- and image-based benchmarks and can be trained in 2 days on one machine with 8×A100 GPUs.Figure 1 summarizes leading performance with 7B LLMs across video- and image-based benchmarks.

2. Related Work

Related work established powerful language and vision-language models for multimodal understanding, reasoning, and instruction following. LLaMA-VID builds on this foundation by collecting multimodal instruction data for long-video processing.

  • Large Language Models: LLMs advanced language generation and reasoning, while instruction tuning improved output quality in models such as Alpaca and Vicuna.These models use specially designed instruction pairs and build on pretrained language-model capabilities.
  • Multimodal Instruction Data: Unlike approaches that invoke predefined tools for visual applications, this work collects multimodality instruction data containing text, images, and videos for long-video processing.The collected data is intended to leverage pretrained LLM capabilities in a multimodal setting.
  • Vision Language Models: VLMs integrate vision and language models for cross-modality understanding and reasoning across image-text tasks.CLIP, ALIGN, Flamingo, and BLIP-2 are cited as influential developments in visual-language modeling and alignment.

3. LLaMA-VID

LLaMA-VID combines an encoder-decoder pipeline with a dual-token representation for each frame, using text-guided context features and adaptively pooled visual content. Its three-stage training strategy aligns modalities, tunes multimodal instructions, and extends support to long videos.

  • 3. LLaMA-VID: LLaMA-VID uses an encoder and decoder to produce visual embeddings and text-guided features before transforming context and content tokens for an LLM.The LLM receives the user directive together with visual tokens and generates the response.
  • 3.1. Encoder and Decoder: Text-guided queries interact with frame embeddings to highlight visual cues related to the user instruction.The queries are generated from the user instruction and visual embedding through a cross-modality interaction in the text decoder.
  • 3.2. Token Generation: Each frame is represented by one context token plus n content tokens, with n adaptively selected from 1 to N according to computational constraints.Single images can retain the original visual-embedding resolution, whereas long videos can downsample each frame to one content token.
  • 3.2. Token Generation: The context token aggregates text-related visual features into one embedding, while the content token preserves visually enriched frame information after pooling and projection.The two projected tokens are concatenated to represent each frame in the LLM language space.
  • 3.3. Training Strategy: Training proceeds through modality alignment, instruction tuning, and long video tuning.The alignment stage uses image- and video-caption pairs; instruction tuning uses multimodal QA data while freezing the visual encoder.
  • 3.3. Training Strategy: Long video tuning uses 15K long QA pairs from movie scenes and LongLoRA-related data, supporting 64K tokens with more than 3-hour videos.The movie-derived data covers video summaries, plot understanding, and detail reasoning, with visual and subtitle tokens concatenated for each frame.

4. Experiments

LLaMA-VID is evaluated on video and image benchmarks, qualitative long-video examples, and component ablations. Results show strong performance with compact tokenization, while instruction-guided context tokens and decoder choices materially affect outcomes.

  • Video-based Benchmarks: LLaMA-VID with two tokens per frame consistently outperforms prior methods across zero-shot video QA benchmarks.With Vicuna-7B, it reaches 69.7% on MSVD-QA and 57.7% on MSRVTT-QA, gains of 2.2% and 0.7% over the previous leader.
  • Image-based Benchmarks: LLaMA-VID outperforms leading methods on most image benchmarks and reaches gains of 2.3% in GQA, 10.7 in MME, and 4.2% in VizWiz.With Vicuna-13B, it leads on 6 benchmarks and ranks in the top two on the remainder.
  • Qualitative Results: Qualitative tests show LLaMA-VID handles single images, short videos, and a 3-hour movie, including plot summaries, recognition, and plot-related reasoning.The setup uses 577 tokens for single images and 2 tokens for videos.
  • Generated Token Types: Both instruction-guided context and image-content tokens are important: using both achieves the best performance across all evaluated benchmarks.Removing the context token prevents the one-token content representation from adapting to input instructions.
  • Generated Token Numbers: With one content token per frame, adding a context token reduces computational cost to 1/128 of the original setting with an approximately 2%-6% performance drop.Using 64 content tokens causes about a 1% to 2% performance decrease but improves SQAI by 1%.
  • Text Decoder: A pre-trained QFormer text decoder improves results over a simple two-layer BERT decoder, producing gains of 2.2% to 2.7% across datasets.The QFormer configuration reaches peak performance on all datasets in the comparison.
  • Response in Context Attention: Context attention focuses on image regions relevant to the input question, such as river buildings and a seagull when identifying a fishing village.The visualization uses the top 20 normalized response scores for the first two queries before Softmax.

5. Conclusion

The conclusion presents LLaMA-VID as a token-generation approach that represents images with instruction-conditioned context tokens and content tokens. Content-token compression can be adjusted to the available budget while preserving the method’s compact representation.

  • 5. Conclusion: LLaMA-VID represents each image with a context token generated from instructions and a content token produced from image content.The content token may be compressed to one token or left uncompressed depending on the budget.

A. Experimental Details

The supplementary experimental details describe staged training for LLaMA-VID and provide a table of its training settings.

  • A. Experimental Details: LLaMA-VID uses a staged training strategy, freezing the proposed text decoder during pretraining and optimizing it during fine-tuning.The settings are customized for the distinct stages shown in the main framework.
  • A. Experimental Details: Table 7 summarizes the training settings used by LLaMA-VID.

B. Instruction Tuning Details

To support hour-long video understanding, LLaMA-VID extends the language-model context and fine-tunes on a dedicated Long-VideoQA dataset. The dataset combines movie-derived and LongAlpaca question-answer pairs with generated plot, detail, and reasoning instructions.

  • Context Extension: Position interpolation expands LLaMA-VID’s language-model context from 4K to 64K tokens for hour-long video inputs.The extended model is subsequently fine-tuned using Long-VideoQA.
  • Long-VideoQA Dataset: Long-VideoQA contains 15K question-and-answer pairs, including 9K from movie scenes and 6K from LongAlpaca.It is designed for relevant and diverse instruction-following scenarios on long-duration videos.
  • Long-VideoQA Dataset: The dataset uses movies from MovieNet with synopses and scripts to create instruction-following data about plot dynamics and character profiles.Claude-2 generates five plot-related reasoning pairs and five detail-related descriptions for each film.

C. Additional Analyses

Additional analyses demonstrate LLaMA-VID’s ability to summarize and reason over hour-long movies, answer image-based questions, and adapt visual attention to user queries.

  • Long Videos: LLaMA-VID summarizes hour-long movies and supports character understanding and plot-related reasoning across diverse narrative genres.The comparisons use movie names or subtitles as inputs and report stronger performance than LLaMA 2 and LongLoRA on these tasks.
  • Images: The model identifies objects and attributes in environments, then uses environmental cues to answer questions about object functions, spatial relationships, and contextual relevance.The examples characterize this as extending beyond recognition toward integrated visual reasoning.
  • Response in Context Attention: Text-guided attention prioritizes image regions relevant to the input question and changes its focus when different questions concern the same image.The visualization highlights the top 20 attention scores while retaining the first five queries in Qt.
  • Instruction Pairs: The supplementary examples use instruction pairs for plot summaries, movie reasoning, and movie-detail questions generated from scripts or synopses.The instructions request brief and detailed summaries, definite-answer questions, and questions about plots and character relationships.
Loading 2311.17043v1…