Source-linked AI summary
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, Bo Zhao
TL;DR
Long-video MLLMs face context-length and computational constraints, while visual-token reduction can impair fine-grained perception. Video-XL compresses interval-level visual information into VST key-values using dynamic compression and trains the module with curriculum learning and composite data. It delivers strong benchmark performance, high-fidelity compression, and cost-effective processing of thousands of frames.
Problem
Long videos exceed MLLM context limits and incur substantial processing costs, while existing visual-token reduction methods can lose information needed for fine-grained perception.
Method
Video-XL uses VSTs to summarize interval visual information into persistent KVs, with dynamic intervals, curriculum learning, and composite image and synthetic data curation.
Results
Video-XL achieves superior performance across popular long-video benchmarks, maintains performance across 2×–16× compression, and processes 2048 frames on one A100 GPU with 95% Needle-in-a-Haystack accuracy.
Takeaways & Limitations
Video-XL provides a high-fidelity and cost-effective approach for extending MLLMs to long-video understanding.
Takeaways & Limitations
Training requires substantial GPU memory and computation, and performance can decline when contexts exceed 1,000 frames.
Abstract
from arXiv · showhide
Long video understanding poses a significant challenge for current Multi-modal Large Language Models (MLLMs). Notably, the MLLMs are constrained by their limited context lengths and the substantial costs while processing long videos. Although several existing methods attempt to reduce visual tokens, their strategies encounter severe bottleneck, restricting MLLMs' ability to perceive fine-grained visual details. In this work, we propose Video-XL, a novel approach that leverages MLLMs' inherent key-value (KV) sparsification capacity to condense the visual input. Specifically, we introduce a new special token, the Visual Summarization Token (VST), for each interval of the video, which summarizes the visual information within the interval as its associated KV. The VST module is trained by instruction fine-tuning, where two optimizing strategies are offered. 1.Curriculum learning, where VST learns to make small (easy) and large compression (hard) progressively. 2. Composite data curation, which integrates single-image, multi-image, and synthetic data to overcome the scarcity of long-video instruction data. The compression quality is further improved by dynamic compression, which customizes compression granularity based on the information density of different video intervals. Video-XL's effectiveness is verified from three aspects. First, it achieves a superior long-video understanding capability, outperforming state-of-the-art models of comparable sizes across multiple popular benchmarks. Second, it effectively preserves video information, with minimal compression loss even at 16x compression ratio. Third, it realizes outstanding cost-effectiveness, enabling high-quality processing of thousands of frames on a single A100 GPU.
1. Introduction
Video-XL addresses long-video MLLM bottlenecks by compressing visual information through the model’s KV sparsification capacity rather than reducing visual-encoder tokens. It combines VST compression, dynamic intervals, curriculum learning, and composite data curation, achieving strong long-video performance and cost-effectiveness.
- Motivation: Long videos exceed MLLMs’ context limits and impose substantial computation and memory costs because each frame consumes many visual tokens.One example uses 144 visual tokens per frame.
- Motivation: Existing visual-token reduction methods enable longer inputs but substantially lose visual information, limiting fine-grained long-video perception.
- Approach: Video-XL uses LLM KV sparsification to generate compact long-video representations instead of reducing tokens from the visual encoder.The approach is motivated by sparse attention patterns in long-input processing.
- Approach: Visual Summarization Tokens summarize interval visual information into persistent KVs, while other visual-token KVs are off-loaded; dynamic compression uses smaller intervals for information-dense content.The resulting representation reduces processing costs for the entire video.
- Training: The VST module is instruction-tuned with curriculum learning, progressively moving from small to larger compression ratios.Training begins with ratios such as 2× and 4× before progressing toward 8× and 16×.
- Training: Composite data curation combines single-image, multi-image, and synthetic VICO data to address scarce long-video instruction data.The image data supports transfer from image comprehension, while VICO strengthens retrieval of useful information.
- Results: Video-XL outperforms comparable-size models across popular long-video benchmarks, maintains performance across 2×–16× compression, and handles 2048 frames on one A100 GPU with 95% Needle-in-a-Haystack accuracy.
2. Related Work
Prior video MLLMs extend image understanding to video, but long-video systems must address excessive visual-token counts and context constraints. Existing solutions use memory, token merging, cross-attention, context extension, training, or architectural changes, with information loss remaining a concern for some approaches.
- Video MLLMs: Video MLLMs transfer image-understanding advances to video, but long videos often exceed current LLM context lengths because of their token volume.
- Prior approaches: Memory modules, token merging, and cross-attention modules are among prior strategies for long-context video modeling.
- Limitations and alternatives: Some token-reduction methods suffer serious information loss that obstructs fine-grained comprehension, while other works extend context length or modify training and architecture.
3. Method
Video-XL encodes visual inputs, partitions them into semantic intervals, and compresses each interval into VST key-values that remain available as a proxy for earlier visual tokens. Instruction tuning combines curriculum compression with composite image and synthetic video data.
- Architecture: Video-XL uses a visual encoder, a visual-language projector, and an LLM backbone, with the VST module generating compressed visual representations.CLIP-ViT-L provides visual encoding and a two-layer MLP projects embeddings into visual tokens.
- VST Compression: The method partitions visual tokens into context-window-sized intervals and introduces VSTs to compress each interval’s visual information into key-values.Compression ratios determine how VSTs are interleaved with visual tokens.
- VST Compression: After each interval is encoded, VST key-values are preserved and visual-token key-values are off-loaded; later intervals condition on accumulated VSTs.
- Dynamic compression: Dynamic compression identifies semantic-change boundaries with depth scores, assigning small intervals to information-dense regions and large intervals to sparse regions.A threshold δ selects peak depth scores as interval boundaries.
- Training objective: Training optimizes ground-truth response likelihood conditioned on VST-compressed KVs and task instructions using standard autoregression.
- Curriculum learning: Curriculum learning starts with small compression ratios and progressively increases them to 8, 12, and 16.
- Composite data curation: Composite curation augments scarce long-video instruction data with unified single-image and multi-image data plus VICO synthetic QA data.VICO contains 20k QA pairs associated with videos averaging three minutes and supports key-information ordering.
4. Experiment
Video-XL is evaluated across long-video benchmarks, extra-long retrieval, efficiency, compression, and ablation settings. Results show strong benchmark performance, high-fidelity compression, reduced resource use, and benefits from dynamic compression, curriculum learning, and complementary data.
- Main Results: Video-XL consistently performs strongly across popular long-video benchmarks, including MLVU, Video-MME, VNBench, VideoVista, and LongVideoBench.It outperforms comparable open-source models across several evaluations and surpasses GPT-4o on MLVU Dev despite having 7B parameters.
- Extra-Long Evaluation: Video-XL covers longer inputs and preserves retrieval accuracy in the Needle-in-the-Haystack evaluation.It maintains 100% accuracy within 128 frames and nearly 95% accuracy on longer inputs, while processing over 2048 frames on one A100-80GB GPU.
- Compression Mechanism: At 16× compression, Video-XL outperforms previous compression methods across long-video and visual-question-answering benchmarks with minimal performance loss.Across tested ratios, its performance remains close to the upper bound, including evaluations at 2×, 8×, 16×, and 32× compression.
- Dynamic Compression Strategy: Dynamic compression substantially improves performance over fixed compression and requires training-time use to provide gains.Fixed compression uses 1440-token intervals, while dynamic compression adapts granularity to video information density.
- Curriculum Learning: Curriculum learning improves performance over randomized or fixed 16× compression baselines, supporting progressive training from smaller compression ratios.The ablation compares training without curriculum against randomized ratios within 16× and fixed 16× training.
- Composite Data Curation: Image and multi-image data provide complementary benefits, while VICO improves multi-detail understanding and can outperform larger video instruction datasets.Image data mainly improves holistic and single-detail tasks; multi-image data substantially improves fine-grained relational understanding, and VICO 5k outperforms NeXTQA 32k and CinePile 10k.
5. Conclusion
Video-XL processes long videos using compressed visual representations generated by VSTs and dynamic compression. It combines progressive curriculum learning and composite data curation, with experiments showing strong benchmark performance, compression quality, and cost-effectiveness.
- Video-XL enables long-video processing through compressed representations generated by Visual Summarization Tokens (VSTs).
- Dynamic compression adjusts compression according to video information density to better retain visual information.
- Curriculum learning progressively trains VSTs across different compression ratios.
- Composite data curation jointly uses multiple data sources to improve model training performance.
- Video-XL achieves superior performance across popular long-video benchmarks while delivering competitive compression quality and cost-effectiveness.
Overview of Supplementary Material
The supplementary material covers limitations, related work, implementation details, further analyses, experimental settings, and qualitative results. It also details VST-based KV condensation and comparisons with related compression methods.
- Overview of Supplementary Material: The supplementary material includes limitations, concurrent-work comparisons, LLM Compressor details, further discussion, dataset analysis, experiments, and qualitative results.
- Limitations and Future Works: Video-XL requires substantial training memory because CLIP, the projector, and the LLM are all unfrozen during training.
- Limitations and Future Works: Video-XL occasionally makes errors when the context exceeds 1,000 frames in Needle-in-the-haystack evaluation.
- Relations to Concurrent Works: Compared with LongVA, Video-XL jointly trains on single images, multiple images, and videos rather than image data alone.
- Relations to Concurrent Works: Video-XL splits long visual sequences into fine-grained intervals with interleaved VSTs, unlike VoCo-LLaMA’s end-appended special tokens.
- Details in LLM Compressor: For each chunk, VSTs condense raw key-value activations from L visual tokens into k activations, yielding a compression ratio α = L/k with k ≪L.
- Details in LLM Compressor: VSTs primarily reuse the LLM’s original parameters and add no more than 1B parameters to the Qwen2 7B base model.
D. Further Discussion of Video-XL
Video-XL’s further analysis examines dynamic compression, training choices, computational efficiency, context scaling, and generalization across language models.
- Dynamic compression: Dynamic compression maintains consistent performance across benchmarks by adapting compression granularity, whereas fixed intervals are sensitive to interval size.Larger fixed intervals can damage visual detail; smaller intervals improve performance but are benchmark-dependent and lengthen training.
- Training methods: The single-stage training method produces superior results, while the two-stage method offers better training efficiency.The authors attribute this trade-off partly to the projector’s pre-alignment of CLIP and LLM representations.
- Computational efficiency: Video-XL reduces KV-cache memory by the average compression ratio and lowers self-attention computation by retaining local tokens and preceding VSTs.Inserted VSTs increase computation in other modules such as the MLP, so total FLOPs involve both savings and added costs.
- Computational efficiency: The computational formulation decomposes forward FLOPs into self-attention and other-module costs, with Video-XL using a cached context shorter than the raw context.The notation defines s as input context length and s_pst as cached context length.
- Generalization: Video-XL generalizes across Qwen2, Vicuna, and LLaMA2, and context-length increases boost performance on several long-video benchmarks.The model trains on relatively short videos under two minutes but handles videos nearly an hour long during inference.
E. Analysis of VICO Dataset
VICO is an automated long-video data pipeline and dataset designed to support visual compression and long-video understanding. It combines diverse video sources and lengths with clip captions and event/action-order QA pairs, while scaling VICO data benefits retrieval performance.
- VICO is an automated long-video data production pipeline and dataset created to enhance long-video understanding and visual compression.
- Diversified Video Categories: VICO combines movies, television, cartoons, egocentric videos, documentaries, games, sports, tutorials, and surveillance footage.
- Versertile video length: Videos range from 1 minute to over 9 minutes and include clip captions plus event/action-order QA pairs.These annotations are intended to support long-video comprehension by MLLMs.
- Robust Generalization: Scaling up VICO data benefits both LongVA and VideoXL on general long-video tasks such as MLVU and Video-MME.The reported results associate larger VICO training sets with more precise and comprehensive retrieval of captured information.
F. Experimental Settings & Additional Results
Video-XL uses specified fine-tuning and benchmark context-length settings, and its evaluation extends beyond long videos to mainstream image understanding benchmarks. The reported experiments show significant advantages on those image QA benchmarks.
- Experimental Settings: Video-XL's reported hyperparameters apply to fine-tuning, while inference settings specify context lengths for different benchmarks.The method modifies the LLM workflow, and the relevant settings are listed in Tables 6 and F.
- Additional Results: Video-XL also demonstrates strong image understanding despite being designed for long-video understanding.The paper evaluates it on several image QA benchmarks.
- Additional Results: Video-XL exhibits significant advantages over previous methods on mainstream image understanding benchmarks.
G. Qualitative Results
Qualitative examples use 10–30-minute MLVU videos to illustrate Video-XL on plot reasoning and long-video summarization. The examples focus on extracting narrative events and summarizing key events.
- The qualitative examples draw on MLVU videos lasting 10 to 30 minutes.
- PlotQA: PlotQA evaluates reasoning about questions concerning the plot of a narrative video.The qualitative cases are presented as demonstrations of long-video understanding.
- Video Summarization: Video summarization requires the MLLM to summarize the key events in a long video.Figures 6 and 7 provide visualization cases for this task.
- VICO Data: The qualitative section presents visualization cases for VICO data, including clip captions and event/action-order QA pairs.