PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Yanwei Li, Chengyao Wang, Jiaya Jia

arXiv:2311.17043v1cs.CVcs.CL

TL;DR

Long videos require many visual tokens, creating computational demands for VLMs. LLaMA-VID represents each frame with context and content tokens, and outperforms previous leading methods on most video- and image-based benchmarks.

  • Problem

    VLMs require excessive visual tokens for long videos, increasing computational demands and constraining their practical use on extensive videos.

    Source: §1 · p. 1

  • Method

    LLaMA-VID represents each frame with a context token guided by user input and a content token capturing visual details.

    Source: §1 · p. 1

  • Results

    LLaMA-VID outperformed prior leading methods on most video- and image-based benchmarks; on MSVD-QA, Vicuna-7B achieved 69.7% accuracy, a 2.2% absolute gain over [34].

    Source: §1 · p. 2, §4.2 · p. 5

  • Takeaways & Limitations

    LLaMA-VID supports long-video use with existing LLMs, and its instruction dataset includes hour-long video understanding.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

1. Introduction

LLaMA-VID reduces video token demands by representing each frame with context and content tokens, while reporting leading performance on most video- and image-based benchmarks. Its context token captures user-guided image context, and its content token preserves frame details.

Introduction

  • Prior video approaches use representative queries [30] [62] or temporal compression [38] [39], but the long-video token challenge remains unresolved.Excessive tokens per frame are the primary obstacle, and simple temporal compression can significantly damage representations over long intervals.Source: §1 · p. 1
  • With 7B LLMs, LLaMA-VID achieves leading performance on most video- and image-based benchmarks.The paper also reports completing training within 2 days on a single machine with 8×A100 GPUs.Source: Figure 1 · p. 1, §1 · p. 2
  • Long videos strain VLMs because frame tokens grow with video length: BLIP [14] [29] uses 32 tokens per image, while LLaVA [33] uses over 256.A 10K-frame video would require over 320K tokens, and simple temporal compression can damage long-term representation.Source: §1 · p. 1
  • LLaMA-VID represents each frame with a context token for user-guided overall context and a content token for finer visual cues, reducing overload while preserving critical information.The content token can extend from 1 token per video frame to beyond 256 tokens per single-image frame; the context token aggregates informative image features and extends VLMs’ upper limit.Source: §1 · pp. 1–2
  • The method uses instruction-guided queries with context attention, pooled image features for content tokens, and linear projections into LLM space.It extracts features with a pretrained vision transformer and constructs an instruction-based dataset of 9K movie-level conversations for plot reasoning and detail understanding.Source: §1 · p. 2
  • LLaMA-VID can be instantiated with various decoders and LLMs, and the authors report that it completes training within 2 days on one 8×A100 machine.The paper describes extensive empirical studies examining the effectiveness of each component.Source: §1 · p. 2

2. Related Work

Prior work advances language models and vision-language models through pretraining, instruction tuning, and multimodal alignment, while video systems remain constrained by the tokens needed for long sequences. This work addresses that limit by encoding each frame with two tokens.

2.1. Large Language Models

  • Transformer [55], GPT [6], and later models [15] [35] [64], including ChatGPT, GPT-4 [44], and LLaMA [53], mark successive advances in language modeling.Source: §2.1 · p. 2
  • Instruction tuning [45] [56] supports models such as Alpaca and Vicuna, while this work uses text-image-video instruction data for long-video processing.Prior visual applications also use LLM reasoning with predefined tools [57] [60].Source: §2.1 · p. 2

2.2. Vision Language Models

  • VLMs combine vision and language models for cross-modal understanding [11] and reasoning [19] [27] [37], with approaches including CLIP [46], ALIGN [24], Flamingo [2], BLIP-2 [29], InstructBLIP [14], MiniGPT-4 [65], and LLaVA [33].Source: §2.2 · p. 2
  • Video-LLaMA [62], VideoChat [30], and Video-ChatGPT [39] support video understanding, but frame-token demands hinder sequences over one hour; this work encodes each frame with two tokens for long-video understanding in existing LLMs.Source: §2.2 · p. 2

3. LLaMA-VID

LLaMA-VID uses instruction-guided context tokens and pooled visual content tokens to represent frames compactly, then trains the model in three stages for image and long-video understanding.

3.1. Encoder and Decoder

  • For each video frame, a transformer-based visual encoder produces visual embeddings; using these embeddings and the user instruction, a text decoder generates queries containing visual cues most related to that instruction.The text decoder can be instantiated with BERT [15] or QFormer [14].Source: §3.1 · p. 3

3.2. Token Generation

  • Context attention condenses instruction-related visual features into one context token, while adaptive pooling produces content tokens at a resolution chosen for compute constraints.The content token can retain the original visual resolution for a single image or be reduced to one token per frame for long videos; the two token types are concatenated for the LLM.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3. Training Strategy

  • Training proceeds through modality alignment, instruction tuning, and long-video tuning; alignment uses 790K image- and video-caption pairs and optimizes context attention and projectors.The alignment set comprises 558K image-caption pairs and 232K video-caption pairs, while the visual encoder and text decoder remain frozen.Source: §3.3 · p. 4
  • Instruction tuning uses 40K ShareGPT conversations, 625K visual QA pairs [19] [22] [26] [33] [40] [41] [47] [49], and 98K video QA pairs.All modules except the visual encoder are optimized, and the image token is randomly placed at the beginning or end of the user input.Source: §3.3 · p. 4
  • Long-video tuning adds 15K long QA pairs from more than 400 movies and scripts, enabling LLaMA-VID to accept 64K tokens and videos longer than 3 hours.The set includes 9K movie-scene conversions and 6K LongLoRA samples; generated movie QA covers summaries, plot, and detail reasoning.Source: §3.3 · pp. 4–5

4. Experiments

LLaMA-VID is evaluated on video and image benchmarks, where it reports strong performance with two tokens per frame and with an added context token for images. Ablations examine token types, token counts, text decoders, and instruction-guided context attention.

4.1. Experimental Setup

  • The experiments use EVA-G and QFormer by default, freeze the visual encoder, train for one epoch per stage, and sample video at 1 FPS.Evaluation covers zero-shot video QA and image benchmarks; training data is mainly drawn from [5] [21] [32] [39].Source: §4.1 · p. 5

4.2. Main Results

  • 69.7% accuracy on MSVD-QA and 57.7% on MSRVTT-QA with Vicuna-7B exceed the previous leader [34] by 2.2% and 0.7%, respectively; ActivityNet-QA reaches top accuracy and a 3.3 score.With two tokens per frame, LLaMA-VID also achieves the best performance across all metrics on the video generative benchmark, and performance improves with larger LLMs.Source: §4.2 · p. 5, Table 2 · p. 6
  • With Vicuna-7B, LLaMA-VID leads all eight image benchmarks, improving over LLaVA-1.5 [32] by 2.3% on GQA, 10.7 on MME, and 4.2% on VizWiz; Vicuna-13B leads six and ranks top-two elsewhere.Comparisons use the same training data and image resolution as LLaVA-1.5 [32].Source: Table 3 · p. 6, §4.2 · p. 6
  • LLaMA-VID handles single images, short trailers, and a three-hour movie, recognizing image details, connecting images to game plots, summarizing trailers, and reasoning about movie storylines.The examples use 577 tokens for single images and two tokens for videos.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3. Component-wise Analysis

  • Both token types contribute: the instruction-guided context token improves results over a single content token, and using both tokens gives the best performance across benchmarks.Without context, a single compressed content token cannot adjust to input instructions.Source: Table 4 · p. 7, §4.3 · p. 7
  • At 224×224, 64 content tokens lower performance about 1%–2% but raise SQAI 1%; one content token plus context cuts computation to 1/128 with about 2%–6% performance drop.The model can dynamically compress content tokens according to resource budget and content importance, and reaches peak SQAI performance with only two tokens.Source: §4.3 · pp. 7–8, Table 5 · p. 8
  • 2.2%–2.7% gains across datasets are reported with pre-trained QFormer over the two-layer BERT decoder, while even BERT-generated context tokens improve most benchmarks.BERT is randomly initialized and retains only its first two layers; QFormer uses pre-trained modules fixed for modality alignment.Source: Table 6 · p. 8, §4.3 · p. 8
  • Instruction-guided context attention focuses on image regions relevant to the question, including riverbank buildings and a seagull when asked about a fishing village.The visualization shows the top-scoring response areas for the first two context queries.Source: Figure 6 · p. 8, §4.3 · p. 8

5. Conclusion

LLaMA-VID represents images with an instruction-conditioned context token and an image-content token, enabling efficient video-frame encoding. The work also introduces an hour-long video-understanding instruction dataset and reports superior performance on several video- and image-based benchmarks.

5. Conclusion

  • LLaMA-VID represents each image with an instruction-conditioned context token and a content token produced from the image.Depending on the token budget, the content token can be compressed to one token or expressed without compression.Source: §5 · p. 8
  • The approach preserves image details and encodes each video frame with only two tokens.The content token may be compressed to one token or expressed without compression, depending on the budget.Source: §5 · p. 8
  • The work constructs an instruction dataset for hour-long video understanding and reports superiority on several video- and image-based benchmarks.The authors hope LLaMA-VID will serve as a strong benchmark for efficient visual representation.Source: §5 · p. 8

A. Experimental Details

LLaMA-VID follows a staged training strategy, freezing the text decoder during pretraining and optional long-video tuning but optimizing it during instruction tuning. The token-length limit rises to 64K for optional stage-3 tuning of hour-long movies.

A. Experimental Details

  • The training strategy freezes the text decoder during pretraining, then optimizes it during finetuning.The approach is similar to [32].Source: A. Experimental Details · p. 9
  • In stage 1, modality alignment keeps the text decoder fixed and limits the maximum token length to 2K.Source: A. Experimental Details · p. 9
  • In stage 2, instruction tuning unfreezes and optimizes the text decoder, with a maximum token length of 2K for video tuning.Source: A. Experimental Details · p. 9
  • If stage 3 is used for long-video tuning, the text decoder is frozen again and the maximum token length increases to 64K for hour-long movies.Freezing the decoder conserves memory.Source: A. Experimental Details · p. 9

B. Instruction Tuning Details

LLaMA-VID extends the language model’s context to 64K tokens using position interpolation, then fine-tunes it on Long-VideoQA, a dataset designed for hour-long video instruction following. Long-VideoQA combines movie-derived and LongAlpaca examples with generated summary, plot, reasoning, and detail-focused tasks.

Context Extension

  • Position interpolation scales rotary position encoding from 4K to 64K tokens, enabling extended sequences for hour-long video understanding.After context adaptation, the model is supervised fine-tuned on Long-VideoQA.Source: B. Instruction Tuning Details · p. 9

Long-VideoQA Dataset

  • Long-VideoQA contains 15K QA pairs: 9K derived from movie scenes and 6K sourced from LongAlpaca [12].The movie data uses MovieNet [21] films, synopses, and scripts; GPT-4 [44] and Claude-2 serve as teachers for creating instruction-following data.Source: B. Instruction Tuning Details · p. 9
  • Generated tasks cover brief and detailed video summaries, plot- and character-focused QA, plot reasoning, and fine-grained descriptions.GPT-4 generates summaries and plot-level QA from synopses; Claude-2 uses scripts to produce five plot-reasoning pairs and five detail descriptions per film.Source: B. Instruction Tuning Details · pp. 9–10

C. Additional Analyses

Additional analyses show LLaMA-VID handling hour-long movie reasoning, image-based questions, and question-dependent visual attention. Its movie comparisons favor LLaMA-VID over LLaMA 2 [54] and LongLoRA [12] on character understanding and plot reasoning.

LLaMA 2

C. Additional Analyses

  • Examples span hour-long Romance, Adventure, and Sci-Fi films, with LLaMA-VID synthesizing frames and subtitles to summarize storylines and reason about plots.Figure 8 presents examples using Vicuna-7B-based models.Source: C. Additional Analyses · p. 10, Figure 8 · p. 12
  • Image examples show LLaMA-VID identifying object details and using environmental cues to answer questions about object functions, spatial relationships, and context.These examples illustrate knowledge-based perception and reasoning beyond recognition.Source: C. Additional Analyses · p. 10, Figure 10 · p. 14
  • LLaMA-VID’s context attention shifts toward regions relevant to each question, including when different questions concern the same image.The visualization highlights the top 20 attention scores while retaining the first five queries in Qt.Source: Figure 11 · p. 15

Related papers

Loading 2311.17043v1…