Source-linked AI summary
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, Vikas Chandra
TL;DR
Long-video understanding is constrained by the large token cost of visual frames and fixed LLM context lengths. LongVU addresses this with adaptive temporal and spatial compression guided by DINOv2, cross-modal queries, and inter-frame dependencies. It consistently outperforms prior video LLMs across benchmarks and also yields a strong lightweight model.
Problem
Long-video MLLMs face a context-length bottleneck because visual frames require many tokens, making hour-long videos difficult to process.
Method
LongVU adaptively removes redundant frames, selects query-relevant frames for higher-resolution features, and compresses spatial tokens using temporal dependencies.
Results
LongVU consistently outperforms recent open-source video LLMs across multiple benchmarks, including approximately 5% higher average accuracy than LLaVA-OneVision.
Takeaways & Limitations
The approach supports long-video processing within commonly used context lengths while preserving much visual detail and scaling to a quality Llama3.2-3B model.
Takeaways & Limitations
The spatial compression strategy assumes DINOv2 has effectively reduced inter-frame redundancy and uses a first-frame comparison within each sliding window.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.
1 Introduction
LongVU addresses the context-length bottleneck in long video understanding by adaptively compressing video tokens while preserving visual details. It reports stronger performance than recent video LLMs across multiple benchmarks, including long-video settings.
- Motivation: Long videos are difficult to process because image representations consume hundreds or thousands of tokens, while an 8k context limits the number of usable frames.LLaVA-1.6 uses 576–2,880 tokens per image and LLaVA-OneVision uses 7,290; an hour-long video may require over 200k tokens.
- Motivation: Uniform sampling can miss critical frames, whereas dense sampling can exceed the context limit and truncate targeted-frame tokens.These strategies expose a trade-off between temporal coverage and the amount of visual detail retained.
- Approach: LongVU uses temporal and spatial adaptive compression to accommodate lengthy videos without exceeding commonly used LLM context lengths.Its design leverages inter-frame similarity and cross-modal queries to reduce redundancy while retaining visual information.
- Results: LongVU significantly outperforms recent open-source video LLMs across EgoSchema, MVBench, VideoMME, and MLVU.The reported comparisons include VideoChat2, LongVA, and LLaVA-OneVision.
- Results: Approximately 5% higher average accuracy than LLaVA-OneVision is reported for LongVU.The paper also reports a 3.4% improvement on the VideoMME Long subset for its Llama3.2-3B version over Phi-3.5-vision-instruct-4B.
2 Related Work
Prior video-language systems commonly trade frame coverage against per-frame detail under fixed context limits. LongVU instead proposes adaptive token reduction using cross-modal queries and inter-frame dependencies to accommodate long videos.
- Video-language models: Video LMMs typically encode frames and rearrange their visual features, using either Q-Former fusion or direct feature concatenation.These approaches provide standard ways to combine visual and linguistic information for video understanding.
- Long-video processing: Fixed context length creates a trade-off between the number of tokens per frame and the number of input frames.This trade-off becomes especially restrictive for lengthy videos.
- Long-video processing: Uniform frame sampling can preserve context feasibility but causes significant loss of visual details.Other approaches use pooling, text decoders, or dynamic token merging to reduce video dimensions or preserve more frames.
- LongVU: LongVU’s architecture uses DINOv2 for redundant-frame removal, combines DINOv2 and SigLIP features, then applies query-guided and temporal-dependency-based compression.The three stages correspond to temporal reduction, selective feature reduction, and spatial token compression.
- LongVU: LongVU combines cross-modal query selection with inter-frame dependencies for spatiotemporal adaptive token reduction.The strategy is designed to mitigate temporal redundancy and fit long videos within the available context length.
3 Method
LongVU performs adaptive compression in stages: it removes redundant frames, preserves query-relevant frames at higher resolution, and compresses spatial tokens using temporal dependencies.
- Temporal reduction: DINOv2 features identify highly similar frames within non-overlapping eight-frame windows for temporal reduction.The method compresses an original sequence of N frames to T frames by removing redundant frames.
- Temporal reduction: Temporal reduction removes approximately half of the video frames.This result is reported in the method’s evaluation discussion.
- Feature fusion: Remaining frames are fused using SigLIP and DINOv2 features through a Spatial Vision Aggregator with learnable queries.The fused representation retains features from both vision encoders for subsequent compression.
- Selective feature reduction: When fused features exceed the context length, text-guided cross-modal attention preserves selected frames at full resolution and pools the others to lower resolution.The strategy reduces spatial tokens from Hh × Wh to Hl × Wl while retaining long-range context.
- Spatial token compression: If low-resolution features still exceed the context limit, spatial token compression compares frames within sliding windows and removes temporally redundant spatial tokens.The first frame in each window retains full resolution, while cosine similarity guides token reduction in later frames.
- Spatial token compression: The first-frame comparison strategy is selected for simplicity and effectiveness after alternatives produced similar performance and compression rates.The assumption is that prior DINOv2 reduction has already lowered inter-frame redundancy.
4 Experiments
LongVU is evaluated across diverse video-understanding benchmarks and ablations, showing strong performance, effective scaling to a smaller LLM, and substantial adaptive token compression.
- Benchmarks and metrics: LongVU is evaluated on EgoSchema, MVBench, VideoMME, and MLVU, including long-video subsets spanning minutes to hours.VideoMME includes 1 minute–1 hour videos and a 30-minute–1-hour long subset; MLVU spans 3 minutes–2 hours.
- Quantitative results: 6.0% and 2.4% accuracy improvements over VideoChat2 and LLaVA-OneVision, respectively, are reported on VideoMME.On the VideoMME Long subset, LongVU surpasses LLaVA-OneVision by 12.8%.
- Small-size models: 3.4% accuracy is the margin by which lightweight LongVU surpasses Phi-3.5-vision-instruct on VideoMME (Long).The experiments report consistent improvement over baselines when LongVU uses Llama3.2-3B.
- Qualitative results: LongVU demonstrates orientation recognition, detailed description, needle-frame identification, action counting, and hour-long frame-specific question answering.These capabilities are illustrated through qualitative examples in Figure 3.
- Ablation studies: 144 tokens per frame outperform 64 tokens on VideoMME and MLVU under uniform sampling, but perform worse on EgoSchema within an 8k context.Using 144 tokens preserves more visual detail while allowing fewer than 60 frames, motivating adaptive token allocation.
- Ablation studies: DINOv2 features outperform SigLIP features for temporal frame reduction, while query-guided selection improves results across all evaluated benchmarks.Cross-modal queries particularly improve MLVU frame-retrieval subtasks such as counting and needle detection.
- Ablation studies: Spatial token compression improves performance within 8k context and reaches results comparable to or slightly better than 16k context.Across spatial-compression strategies, selecting the first frame in each sliding window gives slightly better performance with similar reduction rates.
- Compression analysis: 45.9% of frames are retained after temporal reduction and 40.4% of tokens are reduced after spatial compression on average.The rates are estimated from hundreds of randomly sampled videos.
5 Conclusion
LongVU addresses long-video understanding within a commonly used context length through spatiotemporal adaptive compression. Experiments validate its advantages and support scaling to a lightweight video-language model.
- LongVU reduces video tokens while preserving much visual detail by leveraging cross-modal queries and inter-frame similarities.
- Experiments across various video understanding benchmarks consistently validate LongVU's advantages.
- LongVU supports a quality lightweight video-language understanding model based on Llama3.2-3B.
A Training Datasets
LongVU uses combined image-language alignment and finetuning, then trains on large-scale video-text pairs from publicly accessible databases, with MovieChat providing long-video data.
- Image-language training combines alignment and finetuning in one stage using single-image LLaVA-OneVision data.
- Video-language training uses a subset of VideoChat2-IT containing video-text data from several publicly accessible databases.
- MovieChat is added as complementary long-video training data.
- Training data statistics are presented in Table 6, while Table 7 compares video LMMs on VideoMME.
B Frame-level Position Encoding
LongVU adds frame-level positional embeddings based on absolute frame timesteps to preserve temporal boundaries after frame reduction and dynamic token sampling. Ablations show little overall performance effect, so FPE is excluded by default.
- FPE preserves temporal boundaries and relative inter-frame timing when reduced frames are concatenated with dynamically sampled tokens.
- Adding FPE does not affect overall performance much across several benchmarks.
- Because its performance effect is small, FPE is omitted from the default setting.
- Table 8 studies FPE inclusion, while Table 9 reports strategy ablations across MLVU subtasks.
C DINOv2 v.s. SigLIP
DINOv2 features capture subtle frame differences more effectively than SigLIP features because DINOv2 is vision-centric whereas SigLIP is aligned to semantic space.
- DINOv2 captures subtle frame differences and low-level visual features more effectively than vision-language contrastive methods.
- Figure 6 compares similarity between the first frame and remaining frames using SigLIP and DINOv2 features.
- DINOv2's vision-centric focus enables more effective capture of subtle frame differences than SigLIP's semantic-space alignment.
D Needle-In-A-Video-Haystack
The experiments evaluate LongVU’s adaptive compression under an 8k context length, including variants without spatial token compression or query-guided reduction. Cross-modal querying improves key-frame identification, while adaptive compression further improves VQA accuracy.
- D Needle-In-A-Video-Haystack: Under an 8k context length, the default adaptive-compression setting is compared with variants without spatial token compression or query-guided reduction.The comparison is shown in Figure 7.
- D Needle-In-A-Video-Haystack: Cross-modal querying significantly improves accurate identification of key frames when videos contain fewer than 1.4k frames.The query selectively retains full tokens for frames relevant to the text query.
- D Needle-In-A-Video-Haystack: Adaptive token compression further boosts VQA accuracy in the needle-in-a-video-haystack evaluation.Figure 7 reports the comparison among the default method and ablated configurations.
E Limitation
The study primarily targets video understanding and uses video-only data during video supervised fine-tuning. Image understanding performance decreases after this stage, and mixed image, multi-image, and video training is left for future work.
- E Limitation: Video-only supervised fine-tuning decreases the model’s image understanding capabilities.The authors attribute the limitation to the training focus and leave mixed-modality training for future work because of GPU-resource constraints.