Source-linked AI summary
Long Context Transfer from Language to Vision
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, Ziwei Liu
TL;DR
Extremely long videos challenge LMMs because visual-token counts and long-video supervision are limited. The paper extends the language backbone’s text context and transfers it to vision without long-video training, introducing V-NIAH to measure visual long-context retrieval. LongVA processes more than 200K visual tokens, improves with more input frames, and achieves state-of-the-art Video-MME results.
Problem
LMMs struggle with extremely long videos because visual-token counts grow with frames and existing long-video datasets provide limited dense supervision.
Method
The paper extends a language model’s context on text, aligns it with visual inputs using short image data, and introduces V-NIAH for visual long-context evaluation.
Results
LongVA processes more than 200K visual tokens, improves with more input frames on long-video question answering, and achieves state-of-the-art results on Video-MME.
Takeaways & Limitations
Long context transfer enables LMMs to handle long visual inputs without explicit long-video text-pair training.
Takeaways & Limitations
The language model’s maximum trainable context depends on hardware and vocabulary size, and video analysis was limited to 384 frames by computational and memory constraints.
Abstract
from arXiv · showhide
Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model's NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME among 7B-scale models by densely sampling more input frames. Our work is open-sourced at https://github.com/EvolvingLMMs-Lab/LongVA.
1 Introduction
Existing LMMs struggle with extremely long videos because visual-token growth and limited long-video supervision constrain processing. LongVA instead transfers an extended language context to vision, introduces V-NIAH, and processes over 200K visual tokens.
- Extremely long videos remain difficult for LMMs despite strong performance on single images and short videos.
- Excessive visual-token growth and limited high-quality long-video datasets are major obstacles to processing many frames.Existing datasets mostly contain clips under one minute or annotate only a few frames, providing sparse long-range supervision.
- LongVA extends the language model context on text, then aligns the extended model with visual inputs without long-video text pairs.This transfers the language model’s context length to the multimodal model.
- Long context transfer describes directly transferring a language model’s context to modality-aligned multimodal models.
- V-NIAH benchmarks whether LMMs can locate and retrieve visual information across extremely long contexts.
- LongVA can perceive more than 200K visual tokens and achieves state-of-the-art performance on Video-MME and MLVU.
2 Related Work
Prior work mainly connects vision and language through visual resampling or direct feature projection, while long-context research has rarely examined multimodal context extrapolation. This paper positions long-context transfer from text to vision as complementary and evaluates video understanding with established and newly needed benchmarks.
- Vision-language connectors commonly compress visual features with resamplers or project image features directly into the language model.
- Long-context transfer from text to image is presented as orthogonal to existing connector approaches and able to enable more frames.
- Long-context extrapolation has been studied mainly for language models through positional, data-curation, and system-optimization techniques.
- Multimodal context extrapolation remains limited, and closely related work does not benchmark effective visual context length.
- Video benchmarks cover perception and reasoning tasks, but most focus on short videos and lack long-context data and metrics.
3 Long Video Assistant
LongVA extends the language model’s context length on text, then transfers that capacity to vision using only short image data. UniRes provides a unified image-video encoding, enabling long-video processing without long-video training.
- 3 Long Video Assistant: LongVA first extends a language model’s text context, then uses it for modality alignment and visual instruction tuning without long video-text pairs.This transfers the language model’s context length directly to the multimodal model.
- 3 Long Video Assistant: Long-context training uses continued pretraining with a 224K context length and 900M tokens, supported by memory-efficient sequence-training optimizations.The implementation uses FlashAttention-2, Ring Attention, activation checkpointing, parameter offload, and zigzag sequence sharding.
- 3 Long Video Assistant: The language-backbone approach contrasts with prior long-vision methods that reduce visual tokens through improved resamplers.LongVA instead relies on long context transfer from the language model to vision.
- 3 Long Video Assistant: UniRes encodes images and videos consistently by treating video frames as multiple grids arranged like an extended image.Images are divided into 336 × 336 grids, separately encoded, projected, and pooled.
- 3 Long Video Assistant: LongVA is evaluated under a train-short, test-long protocol using image-text data during training and long videos during testing.Compared with AnyRes, UniRes performs better on V-NIAH and Video-MME but slightly worse on low-resolution image benchmarks.
- 3 Long Video Assistant: LongVA’s training cost totals 3.5 days on 8×A100-80G, combining two days of long-context text training with 1.5 days of image-text alignment.The extended language model achieves perfect NIAH results within its 224K training context and generalizes beyond it.
4 V-NIAH
V-NIAH evaluates whether multimodal models can locate a visual needle embedded at different positions in long video haystacks. Its design measures retrieval across increasing frame counts and needle depths.
- 4 V-NIAH: V-NIAH extends the language NIAH test to vision by inserting a needle image into a haystack video and asking a needle-related question.The benchmark is designed to measure long-range visual information retrieval.
- 4 V-NIAH: V-NIAH was introduced because existing visual benchmarks lacked a benchmark for measuring the effective visual context length of multimodal models.The test focuses on locating and retrieving long-range visual information.
- 4 V-NIAH: The benchmark uses five video question-answering problems as single-frame needles inserted into hours-long videos sampled at 1 FPS.Needle images come from VQA benchmarks or AI generation, with counterfactual questions and locating prompts.
- 4 V-NIAH: Figure 4 plots total haystack frames on the x-axis against needle frame depth on the y-axis.The black dotted line marks the backbone’s training length, assuming 144 tokens per frame.
5 Experiments
Experiments evaluate LongVA on synthetic visual retrieval, long-video, short-video, and image benchmarks, comparing context length and visual encoding choices. LongVA performs strongly on long-video evaluation, while gains are smaller or mixed on shorter benchmarks and some image tasks.
- Evaluation Setup: LongVA is evaluated primarily on V-NIAH and Video-MME, with MLVU results additionally reported in the appendix.V-NIAH tests visual information retrieval, while Video-MME covers short, medium, and long videos with an average duration of 1017 seconds.
- V-NIAH Results: LongVA can almost perfectly retrieve information for fewer than 2000 input frames in V-NIAH, whereas LLaVA-NeXT-Video-32K degrades beyond approximately 200 frames.Training-free RoPE extrapolation improves the baseline only marginally.
- V-NIAH Results: LongVA with UniRes outperforms the otherwise comparable LongVA-AnyRes model on V-NIAH retrieval.The paper attributes this to UniRes using a unified representation for images and videos and enabling training with short vision data.
- Video Evaluation: LongVA achieves state-of-the-art Video-MME performance among LMMs under 10B parameters and improves on the long subset as more frames are sampled, up to 384 frames.Its performance slightly drops when upsampling from 128 to 384 frames while remaining competitive; analysis beyond 384 frames was limited by computational and memory constraints.
- Video Evaluation: On short-video benchmarks, denser sampling helps LongVA on NeXTQA and ActivityNetQA but yields limited or no gains on other datasets.LongVA generally scores lower than LLaVA-NeXT-Video-7B on VideoChatGPT and Video-DD; DPO substantially improves its performance.
- Image Evaluation: Compared with LongVA-AnyRes, UniRes improves InfoVQA performance but reduces scores on AI2D and ChartQA, consistent with InfoVQA containing higher-resolution images.The image-size analysis is presented in Figure 5.
6 Qualitative Results
Qualitative examples show LongVA-DPO handling both short and long videos. The examples emphasize accurate descriptions and retrieval of specific visual details in long-video scenes.
- Short and Long Videos: LongVA-DPO accurately describes a short humorous video involving people playfully interacting with condiments.The example is sourced from [78].
- Short and Long Videos: In two long-video examples, LongVA-DPO retrieves details including a train’s color and the colors of umbrellas in a scene.The long-video examples are sourced from link1 and link2.
7 Conclusion
The paper extends language context and aligns the extended model with visual inputs to improve long-video understanding. LongVA combines this approach with V-NIAH evaluation and achieves strong Video-MME performance.
- Conclusion: LongVA extends the language model on text before aligning it with visual inputs, improving LMM capability on long videos through long context transfer.The model shows improved performance with more input frames.
- Conclusion: The paper introduces V-NIAH, a synthetic benchmark for measuring the visual context length of video LMMs.V-NIAH is intended to measure visual context handling over long inputs.
- Conclusion: LongVA achieves state-of-the-art results on Video-MME while the paper presents the approach as requiring no long-video training pairs.The conclusion describes the model as improving long-video capability through language-to-vision context transfer.
A Needle In A Haystack Test
The Needle In A Haystack test measures whether a language model can retrieve a city-associated random number from long textual contexts. Performance is evaluated across needle positions and context lengths, including distractor needles.
- Results: Figure 7 reports the NIAH results of Qwen-7B-Instruct after long-context training.The figure presents results for the language-model evaluation described in this section.
- Test Design: The test asks the model to identify and retrieve a random number associated with a city embedded in a long context.The evaluated variant uses randomly assigned cities and numbers.
- Test Design: A 7-digit magic number associated with Singapore is inserted into Paul Graham’s Essays as the needle.The needle follows the format “The special magic {City} number is: {XXXXXXX}.”
- Evaluation Procedure: Performance is measured across document depths and context lengths, with five trials using different needles at each setting.A harder variant adds three or five distractor needles associated with other cities.
B UniRes Encoding Scheme
UniRes and AnyRes both partition high-resolution images into 336x336 grids, but they differ in base-image handling and feature flattening.
- Both UniRes and AnyRes divide a high-resolution image into multiple 336x336 grids using CLIP-ViT-L-336px.
- AnyRes prepends a smaller version of the full image before the high-resolution image grids.
- UniRes flattens encoded image features in raster order within each grid, unlike AnyRes’s grid-combination strategy.
C MLVU Results
On MLVU, LongVA achieves state-of-the-art performance among open-source models and ranks behind only GPT-4o.
- Table 8 reports evaluation results by the authors of MLVU.
- LongVA achieves state-of-the-art results among open-source models on the MLVU benchmark.
- GPT-4o is the only model reported to outperform LongVA on MLVU.
D Visual Needle In A Haystack Test
V-NIAH uses five visual question-answering problems as needles in a synthetic benchmark for evaluating long-range visual-context retrieval.
- V-NIAH uses five visual question-answering problems as the needles.
- The five visual questions and answers are the only human-annotated components in V-NIAH’s construction.
- The benchmark is designed to test LMMs’ long-context capability.