Source-linked AI summary
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, Limin Wang
TL;DR
Video understanding requires models that transfer spatiotemporal representations across perception, semantics, dialogue, and long-context reasoning tasks. InternVideo2 addresses this with progressive training that unifies reconstruction, crossmodal contrastive learning, and next token prediction, alongside semantically segmented multimodal captions. It achieves state-of-the-art performance across more than 60 video and audio tasks, particularly in video dialogue and long-video understanding, while retaining limitations from fixed input representations and compressed tokens.
Problem
Embedding videos effectively into large multimodal models remains a central challenge for improving video understanding across perception and reasoning tasks.
Method
InternVideo2 uses progressive training that unifies masked video modeling, crossmodal contrastive learning, and next token prediction, with semantically segmented video-audio-speech captions.
Results
InternVideo2 achieves state-of-the-art performance on more than 60 video and audio tasks, with superior performance in video-related dialogue and long-video understanding.
Takeaways & Limitations
The results support InternVideo2 as a general video encoder for video understanding, including perception, reasoning, dialogue, and long-video analysis.
Takeaways & Limitations
Fixed input resolutions, sampling rates, and highly compressed tokens restrict rich video representation and fine-grained detail capture.
Abstract
from arXiv · showhide
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio tasks. Notably, our model outperforms others on various video-related dialogue and long video understanding benchmarks, highlighting its ability to reason and comprehend longer contexts. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2/.
1 Introduction
InternVideo2 is a video foundation model built with progressive training and enhanced multimodal data to support video perception, semantic alignment, and reasoning. It achieves state-of-the-art performance across more than 60 video and audio tasks, including video dialogue and long-video understanding.
- The three training stages learn spatiotemporal structure, align video with other modalities, and enhance open-ended dialogue capability.The stages use unmasked reconstruction, multimodal learning with audio and text, and next token prediction, respectively.
- The multimodal dataset contains 402M entries, including video-text pairs, video-audio-speech-text pairs, and image-text pairs.InternVid2 semantically segments videos and fuses audio, video, and speech captions into more comprehensive descriptions.
- InternVideo2 unifies masked reconstruction, crossmodal contrastive learning, and next token prediction in a progressive training scheme.The approach targets perceptive, semantic, and reasoning capabilities in video understanding.
- InternVideo2 achieves state-of-the-art performance on more than 60 video and audio tasks, with superior results in video dialogue and long-video understanding.The evaluation spans basic spatiotemporal perception and high-level reasoning tasks such as long-video and procedure-aware question answering.
- InternVideo2 is trained with an enhanced dataset that incorporates audio data and improved captioning to strengthen performance and generalization.The dataset emphasizes spatiotemporal consistency and labeling quality.
2 Related Work
Video foundation models commonly use video-text contrastive learning, masked video modeling, or next token prediction, with prior work also combining multiple objectives. InternVideo2’s framework organizes these learning schemes into consecutive stages with progressive initialization.
- Video Foundation Models: Typical video foundation models use video-text contrastive learning, masked video modeling, and next token prediction.
- InternVideo2 Framework: InternVideo2 arranges unmasked video token reconstruction, multimodal contrastive learning, and next token prediction as three consecutive training phases.The video encoder is trained from scratch in stage 1 and initialized from the preceding stage in stages 2 and 3.
- Video Foundation Models: Prior approaches have unified multiple pretraining objectives or combined masked modeling with video-text contrastive learning.Examples include All-in-one and UMT, which use unified or combined objectives for video and video-language tasks.
- Multimodal Large Language Models: Multimodal large language models extend language models to open-world tasks and have demonstrated zero- or few-shot performance across multimodal benchmarks.
3 Method
InternVideo2 uses progressive training to develop spatiotemporal perception, multimodal semantic alignment, and video-centric dialogue capabilities. Its method combines token reconstruction, crossmodal objectives, and LLM-based training.
- Overview: InternVideo2 is trained in three stages: spatiotemporal token reconstruction, video-audio-speech-language alignment, and joint training with an LLM.The stages progressively target perception, semantic alignment, and open-ended dialogue.
- Stage 1: Reconstructing Unmasked Video Tokens: Stage 1 reconstructs unmasked video tokens using InternVL-6B and VideoMAEv2-g as semantic and motion-aware expert teachers.The model masks 80% of tokens frame by frame and aligns only unmasked tokens using mean squared error.
- Stage 1: Reconstructing Unmasked Video Tokens: The reconstruction objective combines video-encoder outputs with representations from InternViT-6B and VideoMAEv2's ViT-g, weighted by α1 and α2.The video encoder aligns outputs from selected layers and final tokens of the expert models through learnable projections and an l2 norm.
- Stage 2: Aligning Video to Audio-Speech-Text: The multimodal encoders include a 12-layer BEATs-initialized audio transformer and BERT-Large-derived text and multimodal components.Audio uses 64-dimensional log Mel filterbank spectrograms from 10-second clips.
- Stage 2: Aligning Video to Audio-Speech-Text: Stage 2 aligns video, audio, image, speech, and text through crossmodal contrastive, matching, and masked language modeling losses.The combined objective is L = LCON + LMAC + LMLM, with matching based on paired or unpaired video-text inputs.
- Stage 3: Connecting to an LLM: Stage 3 connects InternVideo2 to an LLM through a QFormer-based video BLIP and adds high-definition post-training for fine-grained and long spatiotemporal understanding.The high-definition stage divides input videos into up to six segments.
4 Multimodal Video Data
InternVideo2's data pipeline combines diverse video resources with multimodal captions and temporally consistent clips. New datasets and VidCap annotation refine descriptions across visual, audio, and speech modalities.
- Dataset Construction: The pretraining data includes newly built K-Mash and InternVid2 alongside publicly available datasets.K-Mash is an unlabeled video set spanning first- and third-person views, short and long durations, and varied settings.
- Videos with Audio-Speech Modalities: InternVid2 contains 100M videos with video-audio-speech captions collected from multiple sources and segmented into clips.The dataset uses unimodal and crossmodal inputs for automatic annotation, with temporal segmentation emphasized during clip generation.
- Temporal Consistency: AutoShot segments videos by temporal semantic variation rather than pixel differences, producing semantically complete clips without inconsistent extra frames.This replaces the SceneDet filter from FFmpeg for temporal boundary detection.
- Video Multimodal Annotation: VidCap independently captions visual, audio, and speech streams, then uses an LLM to correct and fuse them into crossmodal captions.The system contains separate video, audio, and speech captioners plus an LLM for caption refinement and fusion.
- Instruction and High-Definition Data: An updated MVBench training setup reduces WebVid and CoCo caption data, adds S-MiT data for instruction diversity, and incorporates GPT-4 video annotations during high-definition training.The changes prioritize diversity and fine-grained training data rather than only increasing quantity.
5 Experiments
InternVideo2 is evaluated across recognition, retrieval, grounding, audio, and video-centric dialogue settings, including zero-shot, finetuning, and probing. It achieves state-of-the-art or superior results across many tasks, while experiments also expose scaling, data, and training-design trade-offs.
- Action recognition: 92.1%/91.9%/85.9% top-1 accuracy on K400/600/700 establishes new state-of-the-art end-to-end recognition results with 16 frames.The model also reports 95.9% on ActivityNet and 97.0% on HACS, while exceeding CoCa-g by 2.2% on Moments in Time.
- Probing and retrieval: InternVideo2 outperforms or matches competing methods in attentive probing and surpasses prior methods in video-text retrieval across zero-shot and finetuned settings.The main retrieval exception is video-to-text retrieval on MSR-VTT, where VideoPrism performs best.
- Action recognition: +3.2% on K400, +8.0% on SthSthV2, and +4.8% on UCF-101 versus DINOv2-g are reported for InternVideo2-1B under linear probing.Results trend upward with model scale, and multimodal pretraining in stage 2 further raises performance.
- Overall evaluation: InternVideo2 achieves state-of-the-art performance across more than 60 video and audio tasks, including video-related dialogue and long video understanding.The evaluation spans tasks from video recognition and retrieval to question answering and audio classification.
- Scaling and ablations: Scaling the video encoder from 1B to 6B improves zero-shot action recognition by 1.4% and zero-shot video retrieval by 1.9%, but finetuned action recognition rises only 0.4%.The ablation indicates that pretraining data scale must grow with model scale to avoid downstream saturation.
- Scaling and ablations: Fused video-audio-speech captions raise zero-shot MSR-VTT t2v R1 from 24.7 to 27.1, while AutoShot improves t2v retrieval by nearly 7 points over SceneDet.These findings support the contribution of audio-aware captioning and temporal segmentation to retrieval performance.
6 Conclusion and Discussion
InternVideo2 combines established learning objectives and data-processing strategies into a video foundation model that achieves state-of-the-art results across video and audio tasks. The authors also identify limits involving input representations, visual-world consistency, compute scalability, and caption-based bias analysis.
- InternVideo2 unifies masked video modeling, video-audio-text contrastive learning, and next token prediction in a progressive framework.
- InternVideo2 achieves state-of-the-art performance across various video and audio tasks, including video dialogue and long video understanding.
- Fixed input resolutions, sampling rates, and highly compressed tokens restrict rich video expression and fine-grained detail capture.
- Jointly learning all three objectives is computationally feasible, but scalability becomes an issue with limited resources.
- InternVideo2 cannot guarantee an implicit world model that ensures consistency in visual reasoning.
- Synthetic captions may not fully reflect corresponding videos, creating a gap between bias analysis and actual reality.
7 Broader Impact
InternVideo2 may inherit biases from its training data and teacher models, with reported caption keyword distributions differing across age, gender, and race categories. Such biases may have societal implications if they reinforce inequalities or prejudices.
- InternVideo2 may embed biases present in its training data, neural teachers, and language models.
- Biases in AI models can reinforce existing inequalities or prejudices and may produce unfair or discriminatory outputs.
B Video-centric Multimodal Data
InternVideo2 organizes training data according to its three progressive learning objectives, covering video-only reconstruction, multimodal alignment, and video instruction for interaction alignment.
- The training data includes video-only pretraining, Video-Audio-Speech-Text data for multimodal alignment, and video instruction data for human-computer interaction alignment.
B.1 Video-only Data
The video-only data collection combines established action-recognition datasets with additional YouTube videos to increase diversity. The expanded collection is used without labels for unsupervised learning.
- K-Mash sources videos from Kinetics-400, Something-Something, Moments in Time, ActivityNet, and HACS.
- K-Mash2M adds 844,000 YouTube videos to enhance dataset diversity.
- All K-Mash2M videos are used without labels, enabling unsupervised learning from diverse visual concepts.
B.2 Videos with Audio-Video-Speech Modalities
InternVid2 is constructed as a diverse audio-visual-speech dataset with semantically segmented clips and automatically generated, corrected, and fused multimodal captions. Filtering and sampling select caption-aligned video segments for InternVideo2 training.
- Collection: InternVid2 combines video, audio, speech, and textual descriptions, with data drawn from YouTube, anonymous sources, and a portion of Chinese videos.The collection is intended to improve source and cultural diversity.
- Trimming: Semantically varying clips are produced with AutoShot to avoid inconsistent context from pixel-difference-based scene cuts.AutoShot predicts temporal boundaries from semantic variations and aims to generate semantically complete cuts.
- Trimming: The dataset preserves clips longer than 2 seconds, randomly samples 30-second segments from longer clips, and removes still or extremely dynamic clips.These rules define the retained video-clip sampling range.
- Annotation: Visual, audio, and speech captions are automatically generated, corrected, and fused with an LLM into cross-modal captions for training.The approach uses dedicated visual, audio, and speech captioning components before fusion.
- Annotation: Each video receives five caption types: three unimodal captions—audio, video, and speech—and two multimodal captions—audio-video and audio-video-speech.The examples show that multimodal captions combine scene content, sound, and spoken information.
- Filtering & Sampling: Caption-video CLIP similarity is used to select the top 60 million InternVid2 video segments, while LAION-2B contributes samples in its top 158 million.The selection criterion prioritizes caption-aligned samples.
C.1.1 How InternVideo2 Works in Feature-based Tasks.
InternVideo2’s feature-based-task analysis examines which video-encoder layers are most useful for downstream temporal localization and evaluates zero-shot video QA performance. The strongest extracted features come from late encoder layers, while zero-shot QA generally improves over prior state-of-the-art results.
- Feature-based tasks: Feature-based evaluation follows the main-paper train and test protocols and focuses on temporal action localization.The study asks which InternVideo2s1 prediction features are suitable for feature-based tasks.
- Feature-based tasks: The best features appear between the video encoder’s last 5th and 7th layers, with effective representations concentrated near the final layers.This pattern is consistent with the reported similarity between feature-based temporal tasks and linear-probing classification.
- Video retrieval: Zero-shot video retrieval is reported with R@1, R@5, and R@10 across MSR-VTT, DiDeMo, LSMDC, ActivityNet, VATEX, and MSVD.The evaluation also records the input frame number used at evaluation.
- Video QA: InternVideo2 consistently improves zero-shot multi-choice QA accuracy over previous state-of-the-art methods except on LSMDC, where it is comparable with InternVideo.The evaluation covers MSR-VTT and VATEX in stage 2, while Table 25 also describes LSMDC.
- Movie understanding: MoVQA evaluates open-ended movie QA across multiple temporal scene lengths and six question types, including causal and hypothetical reasoning.InternVideo2-6B is evaluated on single-scene, multi-scene, and full-scene settings.