Source-linked AI summary
LongVLM: Efficient Long Video Understanding via Large Language Models
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, Bohan Zhuang
TL;DR
Existing VideoLLMs can provide overall video comprehension but may overlook local information needed for detailed understanding of long videos. LongVLM addresses this by hierarchically merging features within ordered short-term segments and integrating global semantics, outperforming prior methods across reported long-video benchmarks and producing more precise responses.
Problem
Existing VideoLLMs reduce visual-token costs through pooling or query aggregation but can overlook local information needed for detailed long-video understanding.
Method
LongVLM divides long videos into short-term segments, hierarchically merges local tokens while preserving segment order, and integrates global semantics into the video representation.
Results
LongVLM surpasses previous state-of-the-art methods on the VideoChatGPT benchmark and zero-shot video question-answering datasets, producing more precise fine-grained responses.
Takeaways & Limitations
Combining local segment features with global information enables LongVLM to capture detailed information and provide consistent, accurate responses for long-term videos.
Takeaways & Limitations
The framework is specifically designed for video-to-text generation scenarios.
Abstract
from arXiv · showhide
Empowered by Large Language Models (LLMs), recent advancements in Video-based LLMs (VideoLLMs) have driven progress in various video understanding tasks. These models encode video representations through pooling or query aggregation over a vast number of visual tokens, making computational and memory costs affordable. Despite successfully providing an overall comprehension of video content, existing VideoLLMs still face challenges in achieving detailed understanding due to overlooking local information in long-term videos. To tackle this challenge, we introduce LongVLM, a simple yet powerful VideoLLM for long video understanding, building upon the observation that long videos often consist of sequential key events, complex actions, and camera movements. Our approach proposes to decompose long videos into multiple short-term segments and encode local features for each segment via a hierarchical token merging module. These features are concatenated in temporal order to maintain the storyline across sequential short-term segments. Additionally, we propose to integrate global semantics into each local feature to enhance context understanding. In this way, we encode video representations that incorporate both local and global information, enabling the LLM to generate comprehensive responses for long-term videos. Experimental results on the VideoChatGPT benchmark and zero-shot video question-answering datasets demonstrate the superior capabilities of our model over the previous state-of-the-art methods. Qualitative examples show that our model produces more precise responses for long video understanding. Code is available at https://github.com/ziplab/LongVLM.
1 Introduction
LongVLM addresses the difficulty of fine-grained long-video understanding by preserving local information across ordered short-term segments while adding global semantics. It reports stronger benchmark performance and more precise long-video responses than prior VideoLLMs.
- Motivation: Existing VideoLLMs reduce visual-token costs through pooling or query aggregation but can miss local information needed for detailed long-video understanding.A 100-frame clip with CLIP-ViT-L/14 can produce 25.6K visual tokens, motivating compression methods.
- Motivation: Long videos contain sequential key actions, complex activities, and camera movements that can be represented as ordered short-term segments.The paper treats segment-level features as local information for temporal-structural understanding of fine-grained events.
- Approach: LongVLM decomposes videos into short segments, extracts local features, preserves their temporal order, and integrates global semantics into each segment feature.The approach uses hierarchical token merging for compact segment representations and combines local and global information before LLM generation.
- Contributions: LongVLM is designed for fine-grained long-video understanding while maintaining affordable computational cost.This contribution combines local segment representations with efficient visual-token aggregation.
- Results: LongVLM surpasses previous state-of-the-art methods on the VideoChatGPT benchmark and zero-shot video question-answering datasets.The reported gains are accompanied by more precise and accurate responses at a fine-grained level for long-term videos.
2 Related Work
VideoLLMs commonly connect pretrained visual encoders to LLMs through projection and visual-token aggregation. Long-video methods additionally target temporal redundancy and complex spatial-temporal dependencies through efficient architectures, aggregation, clip selection, and temporal alignment.
- VideoLLMs: Pooling and query aggregation reduce visual-token volume in VideoLLMs, but the common representation pipeline must still support video-language dialogue interactions.VideoChatGPT and Valley use pooling, while VideoChat and Video-LLaMA use query-based aggregation variants.
- VideoLLMs: VideoLLMs typically use a pretrained visual encoder, a projection layer into the LLM text space, and a pretrained LLM for response generation.Representative systems use pooling, Q-Former aggregation, or video-specific Q-Formers to compact visual representations.
- Long-video understanding: Long-term video understanding requires modeling complicated spatial-temporal dependencies while removing temporal redundancy over extended durations.Prior work addresses this through efficient architectures, temporal pooling or aggregation, and dynamic clip selection.
- Long-video understanding: Other approaches capture event temporality, causality, and dynamics with temporal alignment modules for long-term video-language understanding.These methods complement representation-compression strategies by explicitly targeting temporal relations.
3 Method
LongVLM samples frames, divides the video into short-term segments, and hierarchically merges tokens within each segment to form temporally ordered local features. It also aggregates global [CLS] semantics and feeds projected local-global representations with instructions into an LLM.
- Overall architecture: The architecture comprises a visual encoder, projection layer, and LLM that together transform sampled video frames and user instructions into responses.The visual encoder extracts frame-level patch and [CLS] features before projection into the LLM space.
- Local feature aggregation: The video is divided into S segments of K frames, and patch features within each segment are merged into compact segment-level features.Sequential concatenation of segment features preserves the temporal order of short-term events.
- Local feature aggregation: Hierarchical token merging reduces redundant visual tokens within each short-term segment by repeatedly matching similar token pairs and average-pooling them.The method starts from KN tokens and iteratively produces a compact local feature with M tokens.
- Local feature aggregation: The ordered local representation lets the LLM model temporal structure across multiple event occurrences while eliminating redundancy in the visual sequence.LLM positional encoding preserves the order of the segment-level representations.
- Global semantics integration: Global semantic features are formed by temporal averaging of [CLS] tokens from selected encoder layers and concatenated with local features before LLM input.The LLM attention mechanism allows local tokens to attend to the global semantic feature, enriching segment context.
4 Experiments
LongVLM is evaluated on video generation, zero-shot question answering, ablations, and qualitative examples. It consistently supports fine-grained long-video understanding through local feature aggregation and global semantic integration.
- Main Results: LongVLM outperforms all compared methods across the five VideoChatGPT evaluation aspects.The benchmark evaluates Correctness Information, Detail Orientation, Contextual Understanding, Temporal Understanding, and Consistency.
- Main Results: +0.17 in Detail Orientation and +0.65 in Consistency are reported over BT-Adapter on the video-based generation benchmark.These gains are identified as particularly notable in the benchmark comparison.
- Main Results: 47.6%, 59.8%, and 70.0% accuracy are achieved on ANET-QA, MSRVTT-QA, and MSVD-QA, respectively.These results surpass BT-Adapter by 1.9%, 2.8%, and 2.5%, respectively.
- Ablation Study: Local feature aggregation either improves or maintains comparable performance across evaluation metrics relative to global-semantic-only methods.The ablation supports preserving short-term segment information for long-video understanding.
- Ablation Study: Concatenating global features before local features yields better results than the reverse ordering.The authors connect this ordering to access to video-wide semantic information through causal attention.
- Qualitative Results: With the same input frames, LongVLM captures details such as fixing a chain rather than fixing a wheel, while Video-ChatGPT can misidentify detailed objects or locations.Global semantic integration also helps recognize actions such as long jump and objects such as an axe.
5 Conclusion
LongVLM models long videos with local short-term features while preserving the temporal structure of sequential events. Combining local and global information supports detailed, consistent, and accurate responses, but the current framework is limited to video-to-text generation.
- Conclusion: LongVLM extracts local features from short-term segments to model local dependencies while preserving temporal structure.
- Conclusion: Integrating local and global information enables detailed information capture and consistent, accurate responses for long-term videos.
- Conclusion: The framework is specifically designed for video-to-text generation, with future work extending it to video-centric multimodal generation and longer training videos.
Appendix
The appendix contains additional ablation results and further generation examples for LongVLM.
- Appendix: Additional ablation results are provided in Section 6.
- Appendix: Section 7 provides more generation results from LongVLM.
6 Additional Results
Additional experiments examine temperature effects, performance on longer-video datasets, and ablations of LongVLM’s local and global aggregation design.
- Effects of temperature: Lower temperatures produce shorter, more accurate, and more deterministic responses, whereas increasing temperature from 0.2 to 2.0 lowers performance and raises word counts.The comparison covers performance scores and averaged response lengths on the Video-ChatGPT benchmark.
- Additional ablation results on zero-shot QA: Token merging outperforms pooling on longer videos in ablations on ANET-QA and Egoschema.The comparison tests local feature aggregation strategies using accuracy (%) as the reported metric.
- Additional ablation results on zero-shot QA: Concatenating global features before local features performs better than using the opposite concatenation order.This ablation evaluates the ordering of global and local feature integration on ANET-QA and Egoschema.
7 Additional Qualitative Examples
Additional qualitative examples show LongVLM responding precisely to detailed questions across VideoChatGPT, ANET-QA, MSVD-QA, and MSRVTT-QA.
- Qualitative examples: LongVLM generates precise responses that correspond to video content and correctly answer questions about detailed information.The examples are drawn from the VideoChatGPT benchmark, ANET-QA, and MSRVTT-QA.
- VideoChatGPT: VideoChatGPT examples focus on temporal understanding, with bold text marking the correct content.The examples provide additional responses generated by LongVLM.
- Zero-shot QA datasets: Additional examples are provided for zero-shot ANET-QA, while MSVD-QA and MSRVTT-QA examples are presented together.The figure captions identify bold text as the correct content in these qualitative examples.