Long Context Transfer from Language to Vision

• Extremely long videos remain difficult for LMMs despite strong performance on single images and short videos.Source: §1 · p. 1
• Excessive visual-token growth and limited high-quality long-video datasets are major obstacles to processing many frames.Existing datasets mostly contain clips under one minute or annotate only a few frames, providing sparse long-range supervision.Source: §1 · pp. 1–2
• LongVA extends the language model context on text, then aligns the extended model with visual inputs without long-video text pairs.This transfers the language model’s context length to the multimodal model.Source: §1 · p. 2
• Long context transfer describes directly transferring a language model’s context to modality-aligned multimodal models.Source: §1 · p. 2
• V-NIAH benchmarks whether LMMs can locate and retrieve visual information across extremely long contexts.Source: §1 · p. 2
• LongVA can perceive more than 200K visual tokens and achieves state-of-the-art performance on Video-MME and MLVU.Source: §1 · p. 2

• Vision-language connectors commonly compress visual features with resamplers or project image features directly into the language model.Source: §2 · p. 2
• Long-context transfer from text to image is presented as orthogonal to existing connector approaches and able to enable more frames.Source: §2 · p. 3
• Long-context extrapolation has been studied mainly for language models through positional, data-curation, and system-optimization techniques.Source: §2 · p. 3
• Multimodal context extrapolation remains limited, and closely related work does not benchmark effective visual context length.Source: §2 · p. 3
• Video benchmarks cover perception and reasoning tasks, but most focus on short videos and lack long-context data and metrics.Source: §2 · p. 3

3 Long Video Assistant

• LongVA first extends a language model’s text context, then uses it for modality alignment and visual instruction tuning without long video-text pairs.This transfers the language model’s context length directly to the multimodal model.Source: §3 · p. 3
• Long-context training uses continued pretraining with a 224K context length and 900M tokens, supported by memory-efficient sequence-training optimizations.The implementation uses FlashAttention-2, Ring Attention, activation checkpointing, parameter offload, and zigzag sequence sharding.Source: §3.1 · pp. 3–4
• The language-backbone approach contrasts with prior long-vision methods that reduce visual tokens through improved resamplers.LongVA instead relies on long context transfer from the language model to vision.Source: Figure 1 · p. 4, §3.2 · p. 5
• UniRes encodes images and videos consistently by treating video frames as multiple grids arranged like an extended image.Images are divided into 336 × 336 grids, separately encoded, projected, and pooled.Source: §3.2 · p. 4, Figure 2 · p. 5
• LongVA is evaluated under a train-short, test-long protocol using image-text data during training and long videos during testing.Compared with AnyRes, UniRes performs better on V-NIAH and Video-MME but slightly worse on low-resolution image benchmarks.Source: §3.2 · p. 4
• LongVA’s training cost totals 3.5 days on 8×A100-80G, combining two days of long-context text training with 1.5 days of image-text alignment.The extended language model achieves perfect NIAH results within its 224K training context and generalizes beyond it.Source: §3.1 · p. 4, §3.2 · p. 5

4 V-NIAH

• V-NIAH extends the language NIAH test to vision by inserting a needle image into a haystack video and asking a needle-related question.The benchmark is designed to measure long-range visual information retrieval.Source: §4 · p. 5, Figure 3 · p. 6
• V-NIAH was introduced because existing visual benchmarks lacked a benchmark for measuring the effective visual context length of multimodal models.The test focuses on locating and retrieving long-range visual information.Source: §4 · p. 5
• The benchmark uses five video question-answering problems as single-frame needles inserted into hours-long videos sampled at 1 FPS.Needle images come from VQA benchmarks or AI generation, with counterfactual questions and locating prompts.Source: §4 · p. 5
• Figure 4 plots total haystack frames on the x-axis against needle frame depth on the y-axis.The black dotted line marks the backbone’s training length, assuming 144 tokens per frame.Source: Figure 4 · p. 6

Evaluation Setup

• LongVA is evaluated primarily on V-NIAH and Video-MME, with MLVU results additionally reported in the appendix.V-NIAH tests visual information retrieval, while Video-MME covers short, medium, and long videos with an average duration of 1017 seconds.Source: §5 · p. 6

V-NIAH Results

• LongVA can almost perfectly retrieve information for fewer than 2000 input frames in V-NIAH, whereas LLaVA-NeXT-Video-32K degrades beyond approximately 200 frames.Training-free RoPE extrapolation improves the baseline only marginally.Source: §5.1 · p. 7
• LongVA with UniRes outperforms the otherwise comparable LongVA-AnyRes model on V-NIAH retrieval.The paper attributes this to UniRes using a unified representation for images and videos and enabling training with short vision data.Source: §5.1 · p. 7

Video Evaluation

• LongVA achieves state-of-the-art Video-MME performance among LMMs under 10B parameters and improves on the long subset as more frames are sampled, up to 384 frames.Its performance slightly drops when upsampling from 128 to 384 frames while remaining competitive; analysis beyond 384 frames was limited by computational and memory constraints.Source: §5.2 · p. 7
• On short-video benchmarks, denser sampling helps LongVA on NeXTQA and ActivityNetQA but yields limited or no gains on other datasets.LongVA generally scores lower than LLaVA-NeXT-Video-7B on VideoChatGPT and Video-DD; DPO substantially improves its performance.Source: §5.2 · p. 8

Image Evaluation

• Compared with LongVA-AnyRes, UniRes improves InfoVQA performance but reduces scores on AI2D and ChartQA, consistent with InfoVQA containing higher-resolution images.The image-size analysis is presented in Figure 5.Source: §5.3 · p. 8, Table 7 · p. 8, Figure 5 · p. 8

Short and Long Videos

• LongVA-DPO accurately describes a short humorous video involving people playfully interacting with condiments.The example is sourced from [78].Source: §6 · p. 9, Figure 6 · p. 9
• In two long-video examples, LongVA-DPO retrieves details including a train’s color and the colors of umbrellas in a scene.The long-video examples are sourced from link1 and link2.Source: §6 · p. 9

Conclusion

• LongVA extends the language model on text before aligning it with visual inputs, improving LMM capability on long videos through long context transfer.The model shows improved performance with more input frames.Source: §7 · p. 9
• The paper introduces V-NIAH, a synthetic benchmark for measuring the visual context length of video LMMs.V-NIAH is intended to measure visual context handling over long inputs.Source: §7 · p. 9
• LongVA achieves state-of-the-art results on Video-MME while the paper presents the approach as requiring no long-video training pairs.The conclusion describes the model as improving long-video capability through language-to-vision context transfer.Source: §7 · p. 9

Results

• Figure 7 reports the NIAH results of Qwen-7B-Instruct after long-context training.The figure presents results for the language-model evaluation described in this section.Source: Figure 7 · p. 15

Test Design

• The test asks the model to identify and retrieve a random number associated with a city embedded in a long context.The evaluated variant uses randomly assigned cities and numbers.Source: A Needle In A Haystack Test · p. 15
• A 7-digit magic number associated with Singapore is inserted into Paul Graham’s Essays as the needle.The needle follows the format “The special magic {City} number is: {XXXXXXX}.”Source: A Needle In A Haystack Test · p. 15

Evaluation Procedure

• Performance is measured across document depths and context lengths, with five trials using different needles at each setting.A harder variant adds three or five distractor needles associated with other cities.Source: A Needle In A Haystack Test · p. 15

• Both UniRes and AnyRes divide a high-resolution image into multiple 336x336 grids using CLIP-ViT-L-336px.Source: B UniRes Encoding Scheme · p. 16
• AnyRes prepends a smaller version of the full image before the high-resolution image grids.Source: B UniRes Encoding Scheme · p. 16
• UniRes flattens encoded image features in raster order within each grid, unlike AnyRes’s grid-combination strategy.Source: B UniRes Encoding Scheme · p. 16

• Table 8 reports evaluation results by the authors of MLVU.Source: Table 8 · p. 16, C MLVU Results · p. 16
• LongVA achieves state-of-the-art results among open-source models on the MLVU benchmark.Source: C MLVU Results · p. 16
• GPT-4o is the only model reported to outperform LongVA on MLVU.Source: C MLVU Results · p. 16

• V-NIAH uses five visual question-answering problems as the needles.Source: D Visual Needle In A Haystack Test · p. 16, Table 9 · p. 17
• The five visual questions and answers are the only human-annotated components in V-NIAH’s construction.Source: D Visual Needle In A Haystack Test · p. 16
• The benchmark is designed to test LMMs’ long-context capability.Source: D Visual Needle In A Haystack Test · p. 16