Source-linked AI summary
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, Limin Wang
TL;DR
Video MLLMs still face limitations in fine-grained visual perception and long-context video understanding. InternVideo2.5 models long and rich context by combining dense visual supervision through direct preference optimization with adaptive hierarchical token compression. It improves short- and long-video benchmark performance, extends video memory to at least six times the original, and supports specialized visual perception capabilities.
Problem
MLLMs have difficulty recognizing, localizing, and recalling fine-grained visual content and handling long-term multimodal context, limiting video understanding and reasoning.
Method
InternVideo2.5 combines direct preference optimization with dense visual annotations and adaptive hierarchical compression of visual and semantic tokens.
Results
InternVideo2.5 improves short- and long-video understanding benchmarks and retains video inputs at least six times longer than the original.
Takeaways & Limitations
Long and rich multimodal context provides a workable route to improving MLLM memory, focus, video understanding, and specialized visual perception.
Takeaways & Limitations
Processing extended contexts remains computationally costly, and the current implementation primarily focuses on visual context rather than reasoning-related areas.
Abstract
from arXiv · showhide
This paper aims to improve the performance of video multimodal large language models (MLLM) via long and rich context (LRC) modeling. As a result, we develop a new version of InternVideo2.5 with a focus on enhancing the original MLLMs' ability to perceive fine-grained details and capture long-form temporal structure in videos. Specifically, our approach incorporates dense vision task annotations into MLLMs using direct preference optimization and develops compact spatiotemporal representations through adaptive hierarchical token compression. Experimental results demonstrate this unique design of LRC greatly improves the results of video MLLM in mainstream video understanding benchmarks (short & long), enabling the MLLM to memorize significantly longer video inputs (at least 6x longer than the original), and master specialized vision capabilities like object tracking and segmentation. Our work highlights the importance of multimodal context richness (length and fineness) in empowering MLLM's innate abilites (focus and memory), providing new insights for future research on video MLLM. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5
1 Introduction
The paper studies whether richer multimodal context—longer temporal coverage and finer visual detail—can improve MLLMs’ vision-centric abilities without directly scaling model size or data volume. InternVideo2.5 combines dense visual supervision with adaptive token compression to improve video understanding, memory, and perception.
- Motivation: MLLMs still struggle with accurately recognizing, localizing, and recalling objects, scenes, and motions, despite progress from scaling vision-language data and model size.These limitations impede understanding and reasoning performance, while existing improvements do not establish systematic solutions.
- Motivation: The paper investigates how multimodal context length and fineness influence MLLMs’ vision-centric abilities and performance.Long context supports extended temporal dependencies, while fine-grained context supports subtle object details and spatiotemporal relationships.
- Approach: InternVideo2.5 transfers dense visual annotations into MLLMs through direct preference optimization and adaptively compresses visual and semantic tokens into compact spatiotemporal representations.Vision expert models serve as preference models for the preference-optimization process.
- Contributions: The LRC design unifies hierarchical token compression and task preference optimization to enhance MLLMs with efficient learning and rich vision annotations.The paper presents this as a comprehensive study of long and rich context for improving MLLM memory and focus.
- Results: InternVideo2.5 achieves leading performance across several short- and long-video benchmarks while retaining inputs at least six times longer than the original.The reported improvements also include expert-level visual perception abilities.
2 Related Work
Related work addresses long-video understanding through context-window expansion, token compression, agent decomposition, and increasingly broad evaluation. Fine-grained vision tasks remain challenging because they require precise predictions beyond captioning and question answering.
- Long Video Understanding: Long-video methods primarily expand MLLM context windows, compress tokens for efficiency, or decompose comprehension into agent-based subtasks.Agent-based approaches may combine expert models and LLMs with self-reflection or chain-of-thought.
- Long Video Understanding: Context-window extension enables longer video processing but introduces computational scalability challenges despite parallel distribution across temporal and tensor dimensions.The passage describes parallel computing as a way to distribute computation across more devices.
- Long Video Understanding: Token compression produces compact representations and high compression ratios, but detailed video understanding can lag behind image-focused MLLMs.The related-work discussion identifies preserving semantics during compression as an open improvement area.
- Evaluation: Long-video assessment spans object recognition, temporal reasoning, and memory retention, although recent benchmarks have focused mainly on question-answering tasks.The benchmark landscape includes egocentric videos, online content, movies, and television shows.
- Specific Vision Problems: Conventional MLLMs perform strongly on captioning and question answering but remain challenged by fine-grained tasks such as segmentation and temporal grounding.These tasks require precise predictions that standard MLLM outputs do not readily provide.
3 InternVideo2.5: Long and Rich Context Modeling
InternVideo2.5 models long and rich multimodal context through adaptive hierarchical token compression, specialized visual preference learning, and progressive training across short, long, and task-specific data.
- 3.1 Video Length Adaptive Token Representation For Long Multimodal Context: HiCo combines spatiotemporal-aware compression during visual encoding with adaptive multimodal context consolidation during language-model processing.The method compresses redundancy across events and within spatiotemporal signals.
- 3.1 Video Length Adaptive Token Representation For Long Multimodal Context: Adaptive temporal sampling uses 15 frames per second for short videos and 1 frame per second for minute- or hour-level videos.The sampling strategy targets motion granularity in shorter sequences and event-level understanding in longer ones.
- 3.1 Video Length Adaptive Token Representation For Long Multimodal Context: Spatiotemporal token merging maps M encoder tokens to N compressed tokens, with N < M, pooling semantically similar tokens rather than nearby tokens.The connector applies semantic similarity-based merging across spatiotemporal scales.
- 3.1 Video Length Adaptive Token Representation For Long Multimodal Context: Multimodal token dropout prunes uniformly in early language-model layers and uses attention-guided selection in deeper layers to retain task-relevant information.The mechanism reduces computational overhead while preserving structural integrity and relevant visual content.
- 3.2 Enhancing Visual Precision in Multimodal Context through Task Preference Optimization: Task preference optimization integrates dense visual annotations and specialized modules for temporal understanding and instance segmentation into the MLLM.The temporal module predicts boundaries and relevance scores, while the segmentation module produces pixel-level predictions.
- 3.3 Training Video Corpus for Multimodal Context Modeling: Three-stage progressive training jointly increases task complexity and video temporal length using visual-text aligned, long-video, and task-specific visual data.Direct training on long-form videos is intended to reduce the gap between training and deployment scenarios while mitigating degradation of general abilities.
4 Experiments
Experiments evaluate InternVideo2.5 on short- and long-video understanding, memory retention, qualitative video reasoning, specific vision tasks, and component compatibility. The results show stronger benchmark performance, substantially longer recall capacity, and expert-level capabilities on several densely annotated visual tasks.
- Experimental Setup: InternVideo2.5 addresses short and long video dialogue and typical vision tasks such as tracking after incorporating HiCo and TPO.The evaluation covers mainstream multimodal video understanding and classical vision tasks.
- Video Understanding: InternVideo2.5 achieves almost leading performance across popular short- and long-video question-answering benchmarks at around 7B LLM capacity.Relative to InternVL2.5, gains exceed 3 points on MVBench and Perception Test, and reach +12.4 on EgoSchema, +3.9 on MLVU, and +8.0 on LVBench.
- Memory Retention: InternVideo2.5 recalls frames from sequences up to 3,000 frames and processes over 10,000 frames without out-of-memory errors, whereas InternVL2.5-8B struggles within 500 frames and exceeds memory beyond 1,000 frames.The LRC variant maintains high recall up to 3,000 frames but degrades beyond that point, likely because training data is primarily short videos.
- Qualitative Evaluations: InternVideo2.5 provides detailed motion descriptions with precise times, target tracking or segmentation for reasoning, and understanding of long visual inputs.Qualitative examples include temporal references, tracking, moment retrieval, and abnormal-event detection in extended surveillance videos.
- Specific Vision Tasks: TPO enables moment retrieval, reference tracking, temporal grounding, and video referring segmentation with expert-model-level performance, outperforming other MLLMs on most tasks.The reported results support joint learning between multimodal question answering and classical vision tasks.
- Ablation Studies: Combining HiCo and TPO improves short- and long-video benchmark results and enables NIAH, with gains of 1.7 on MVBench, 3.5 on Perception Test, 1.0 on EgoSchema, 1.0 on LongVideoBench, and 1.3 on MLVU.The VideoMME increase is only 0.2, and TPO benefits vanish for inputs around or longer than 15 minutes.
5 Concluding Remarks
InternVideo2.5 improves video MLLM perception and understanding by modeling long and rich context through greater context length and fineness. The paper reports stronger benchmark performance, sixfold longer input sequences, and enhanced visual capabilities, while identifying computational cost and limited reasoning-focused LRC as remaining concerns.
- Concluding Remarks: InternVideo2.5 improves video MLLM perception and understanding through long and rich context modeling, targeting both context length and fineness.The approach uses direct preference optimization for dense visual annotations and adaptive hierarchical token compression for efficient spatiotemporal representation.
- Concluding Remarks: InternVideo2.5 achieves state-of-the-art performance across various video understanding benchmarks around the 7B model size and increases input video sequences sixfold over the applied MLLMs.The model also exhibits enhanced visual capabilities, including object tracking.
- Limitations: Processing extended contexts remains computationally expensive, and the implementation primarily focuses on visual context properties rather than reasoning-related areas.The paper identifies more efficient learning techniques and extending LRC to reasoning as future work.