Source-linked AI summary
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, Limin Wang
TL;DR
Long-context video modeling must make extremely long video contexts efficient to process without losing essential information. The paper introduces HiCo, LongVid, short-to-long learning, and a harder multi-hop benchmark, with VideoChat-Flash achieving leading benchmark performance and 99.1% retrieval accuracy over 10,000 frames among open-source MLLMs.
Problem
Efficiently understanding extremely long video contexts remains challenging, while visual-token compression can lose detailed information and reduce performance.
Method
The paper combines two-stage HiCo compression with LongVid, multi-stage short-to-long learning, and a Multi-Hop Needle-In-A-Video-Haystack benchmark.
Results
VideoChat-Flash achieves leading performance with high efficiency across video understanding benchmarks and 99.1% retrieval accuracy over 10,000 frames in NIAH among open-source MLLMs.
Takeaways & Limitations
Hierarchical compression preserves performance while substantially reducing long-video computation, and the combined data and training strategy supports videos of various lengths.
Takeaways & Limitations
NIAH-Video can suffer from training-data leakage and is insufficiently discriminative when models already exceed 99% visual-retrieval accuracy.
Abstract
from arXiv · showhide
Long-context video modeling is critical for multimodal large language models (MLLMs), enabling them to process movies, online video streams, and so on. Despite its advances, handling long videos remains challenging due to the difficulty in efficiently understanding the extremely long video context. This paper aims to address this issue from aspects of model architecture, training data, training strategy and evaluation benchmark. First, we propose a novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details, achieving an extreme compression ratio of approximately 1/50 with almost no performance loss. Second, we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of real-world long videos named LongVid, and a challenging ``Multi-Hop Needle-In-A-Video-Haystack'' benchmark. Finally, we build a powerful video MLLM named VideoChat-Flash, which shows a leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scale. It first gets 99.1% accuracy over 10,000 frames in NIAH among open-source models.
1. Introduction
Long-context video modeling remains difficult because extremely long contexts are costly to process and visual compression can discard detail. VideoChat-Flash addresses this with hierarchical compression, short-to-long learning, LongVid, and a new reasoning-focused benchmark.
- Extremely long video contexts are difficult for MLLMs to understand efficiently, despite strong short-video performance.
- Visual-token compression can improve efficiency but may lose detailed information, creating a performance–efficiency trade-off.
- HiCo compresses video tokens first at the Clip-level using spatio-temporal redundancy, then at the Video-level using task-relevant attention sparsity.
- LongVid and short-to-long learning expand training with large-scale long-video data and staged learning across image, short-video, and long-video inputs.LongVid contains 300,000 hours of videos and 2 billion words of textual annotations.
- VideoChat-Flash achieves leading performance with high efficiency across video benchmarks and reaches 99.1% retrieval accuracy over 10,000 frames in NIAH among open-source MLLMs.
2. Related Works
Related work addresses long-video understanding through larger language-model context windows and video-token compression, while benchmarks commonly use questions about collected long videos. The paper motivates hierarchical compression and more challenging evaluation for reasoning over videos of different lengths.
- Prior long-video MLLMs mainly extend the LLM context window or compress video tokens to handle longer inputs.
- VideoChat-Flash uses Clip-level compression during encoding and Video-level compression during LLM context interaction.
- Existing long-video benchmarks typically collect long videos and ask content-based multiple-choice questions to test understanding and reasoning.
3. Method
VideoChat-Flash combines hierarchical video-token compression with duration-aware sampling, timestamp prompts, long-video training data, and multi-stage learning to process long videos efficiently.
- HiCo: Efficient Long Video Modeling: HiCo compresses long-video context at clip and video levels, using spatio-temporal encoding, similar-token merging, and attention sparsity.Clip-level compression aggregates information across frames; video-level compression further drops tokens during LLM interaction.
- HiCo: Efficient Long Video Modeling: Duration-based sampling uses dense sampling for short videos and sparse sampling for long videos, with density adjusted according to video duration.For D > Tmax, the sampling density is ϕ = Tmax/D.
- HiCo: Efficient Long Video Modeling: A timestamp prompt states the video duration and uniformly sampled frame count, supporting timestamp-sensitive tasks such as temporal grounding.The paper reports that this simple prompt is sufficient without additional timestamp modules.
- HiCo: Efficient Long Video Modeling: 16 tokens per frame is the average representation achieved while retaining key information through the video encoder.The approach enables heavy compression by exploiting the encoder’s spatio-temporal modeling.
- Large-scale Corpus for Long Video Training: LongVid provides 114,228 long videos and 3,444,849 question-answering pairs across five task types for long-video instruction tuning.The dataset spans movies, egocentric videos, news, interviews, how-to videos, and other in-the-wild long videos.
- Multi-stage Short-to-Long Learning: VideoChat-Flash uses short-to-long training, progressing from visual alignment and short-video learning to joint short- and long-video instruction tuning.Stage 3 mixes 1.1M images, 1.7M short videos, and 0.7M long videos, with 64 to 512 sampled frames.
4. Experiments
Experiments evaluate VideoChat-Flash across broad video benchmarks, needle-in-a-haystack tasks, ablations, and efficiency measurements. The results show strong performance with substantial compression and computational savings.
- General Video Understanding Evaluation: VideoChat-Flash achieves leading results on diverse video-language benchmarks at the 2B and 7B model scales.It outperforms substantially larger models and proprietary systems on the reported benchmarks.
- Long Video Context Evaluation: 99.1% retrieval accuracy is achieved on Single-Hop NIAH over 10,000 frames.The task inserts image needles at different positions in a uniformly sampled long-video sequence.
- Long Video Context Evaluation: 31.3% CAP and 25.4% QA are achieved on average in Multi-Hop NIAH, about 8 points higher than LongVA.The task requires tracing chains of indicating images and answering related questions.
- Ablation & Analysis: 16 tokens per frame reduce the computational load from 196 tokens per frame while barely compromising performance.The reported gains use hierarchical compression, with duration-based sampling and timestamp prompts also improving performance.
- Ablation & Analysis: 2% compression ratio maintains most of the performance with parameter-free similar token merging.The comparison covers 16, 49, and 196 tokens per frame and several connector strategies.
- Model Efficiency: Only VideoChat-Flash completes inference on 10,000 frames using a single A100-80G GPU.Its compute load is reported as two orders of magnitude lower than LongVILA.
5. Conclusions
The paper addresses long-context video modeling through coordinated changes to architecture, data, training, and evaluation. VideoChat-Flash combines hierarchical compression with LongVid, short-to-long learning, and a challenging benchmark, achieving strong video-understanding performance.
- 5. Conclusions: HiCo compresses long video context hierarchically with nearly no performance loss.The paper also introduces LongVid, short-to-long learning, and a more challenging long-video evaluation benchmark.
- 5. Conclusions: VideoChat-Flash demonstrates outstanding performance across various video-understanding benchmarks.The conclusion attributes this validation to the proposed architecture, corpus, training strategy, and benchmark.
Supplementary Material
Figure 8 examines visual redundancy in long-context video across the 28 layers of Qwen2-7B. The experiment tests dropping 50% of visual tokens at different depths.
- Visual redundancy: Figure 8 measures visual redundancy across the layers of Qwen2-7B.The model has 28 layers.
- Experimental setup: The experiment uses Qwen2-7B as the evaluation model.The tested model is identified as having 28 layers.
- Token dropping: 50% of visual tokens are dropped from shallow to deep layers to test the impact of token reduction.The intervention is evaluated at different depths in the model.
6. More Results & Discussions
Additional analyses examine visual-token redundancy, attention patterns, stronger video encoders, and image benchmarks. They support progressive compression and show that VideoChat-Flash remains competitive beyond video tasks.
- 6.1. Visual Dropout in LLM: Half-token removal at shallow LLM layers causes only marginal degradation, revealing redundancy between compressed clips.This redundancy remains even after each frame is encoded into 16 tokens.
- 6.1. Visual Dropout in LLM: Deep-layer attention is more reliable because long-video attention shifts from dispersed shallow-layer patterns to focused deep-layer regions.The paper warns that relying on shallow-layer attention may introduce bias.
- 6.2. Video Encoder: A stronger video encoder produces better compressed representations.The comparison includes UMT and InternVideo2-1B.
- 6.3. Results on Image Understanding Benchmarks: 45.2 MMMU and 1843.4 MME are reported for VideoChat-Flash, versus 35.3 and 1603.7 for LLaVA-NeXT.The video-focused model achieves these results with significantly lower computational cost.
7. Implementation Details
The implementation compares video-token compression strategies and trains VideoChat-Flash through multiple stages spanning image, short-video, and long-video data.
- Token compression strategies: Four popular token compression strategies are considered for compressing features from video clips.The comparison is presented in Fig. 10.
- Token compression strategies: Spatial downsampling reduces video tokens with pooling, interpolation, or convolution but lacks temporal interaction between frames.The experiments use pixel shuffle as the convolution-based implementation.
- Training stages: Stage 1 uses 558k image-text pairs and 481k short video-text pairs for video-language alignment.
- Training stages: Stage 2 performs visual pre-training with 3.5 million images and 2.5 million short video-text pairs.The data include recaptioned video descriptions and detailed descriptions from WebVid and Kinetics.
- Training stages: Stage 3 jointly instruction-tunes on 3.5 million samples comprising 1.1M images, 1.7M short videos, and 0.7M long videos.Short videos are under 60 seconds, while long videos span 60–3600 seconds.
8. Dataset Details of LongVid
LongVid combines four open-source video datasets and constructs hierarchical captioning, event-relation, and temporal-grounding tasks across ego-centric, instructional, movie, and in-the-wild videos.
- Source datasets: LongVid curates videos from Ego4D, HowTo100M, HD-VILA, and MiraData.
- Ego-centric videos: LongVid uses 3,662 Ego4D long videos with hierarchical captions for short, medium, and long video segments.Lower-level captions are concatenated to create dense captions for longer segments.
- Ego-centric videos: Ego4D captions support event-relation recognition and temporal grounding in addition to short and dense video captioning.Event-relation recognition orders events, while temporal grounding uses short-video captions with timestamps.
- Movie videos: MiraData supplies multi-level movie captions covering short, dense, and fine-grained perspectives such as objects, backgrounds, camera movements, and style.Event and background labels are extracted for relation-recognition tasks, and timestamped event labels support temporal grounding.
- Instructional videos: HowTo100M contributes refined interleaved captions and temporal annotations of instructional steps through HowToInterlink7M and HTStep.
- In-the-wild videos: HD-VILA adds in-the-wild videos to broaden long-video coverage beyond domain-specific datasets.The selected segments can be reconstructed into videos longer than 60 seconds and use Panda-70M captions.
9. Qualitative Results
Qualitative comparisons evaluate VideoChat-Flash against Gemini-1.5 Pro, LongVU, and VideoLLaMA2 on fine-grained short-video and long-video understanding tasks.
- Evaluated tasks: The qualitative comparison covers fast-motion understanding, action counting, surveillance anomaly detection, and movie understanding.These tasks correspond to Figs. 11–14.
- Compared models: The comparison includes the newest Gemini-1.5 Pro-002 evaluation version alongside open-source LongVU and VideoLLaMA2.
- Short-video understanding: Dense sampling enables the model to capture fast motion and determine an object’s final position under a cup.
- Short-video understanding: The model captures short-video actions while recording their occurrence frequency.
- Long-video understanding: For long videos, the model detects anomalous surveillance behaviors and provides corresponding inferences.
- Long-video understanding: The model understands movie plots while retaining detailed visual information.