Source-linked AI summary
InternVideo: General Video Foundation Models via Generative and Discriminative Learning
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, Yu Qiao
TL;DR
Video foundation models are needed because image-focused foundation models and existing video models have limited coverage of dynamic, complex, and diverse video tasks. InternVideo combines masked video modeling with video-language contrastive learning through cross-representation coordination, achieving state-of-the-art results across 39 datasets and 10 tasks. The model is designed as a training-efficient general video representation, but it remains limited for long-term and high-order video understanding.
Problem
Existing vision foundation models focus mainly on image-level pretraining, while existing video models have limited transferability across diverse video applications.
Method
InternVideo combines masked video modeling and multimodal contrastive learning, then coordinates their complementary representations through cross-model attention and supervised training.
Results
InternVideo achieves state-of-the-art performance across 10 tasks and 39 video datasets spanning action understanding, video-language alignment, and open understanding.
Takeaways & Limitations
The results support InternVideo as a general and training-efficient video representation covering action, video-language, and open-world video applications.
Takeaways & Limitations
InternVideo processes videos using clips and can hardly address long-term video tasks or high-order tasks such as anticipating movie plots.
Abstract
from arXiv · showhide
The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .
1 Introduction
InternVideo addresses the limited generality and scalability of video foundation models by combining masked video modeling with multimodal contrastive learning. Its unified representations achieve strong performance across diverse video understanding tasks and datasets.
- Motivation: Video understanding remains less explored than image understanding because video processing is substantially more computationally expensive and many benchmarks can leverage image appearance features.The added temporal dimension can raise complexity by at least an order of magnitude when spatial resolutions and temporal sampling are comparable.
- Motivation: Existing video foundation models have narrow transferability, typically focusing on action understanding or video-language alignment rather than a broad application domain.The paper attributes this pattern to learning schemes and the lack of a comprehensive benchmark.
- Approach: InternVideo jointly develops masked video modeling and multimodal contrastive learning in modular designs to capture complementary spatiotemporal and semantic information.Masked modeling specializes in action understanding, while contrastive learning embeds semantics but can overlook concrete spatiotemporal modeling.
- Results: 86.9% top-1 accuracy is achieved on Kinetics-400 with the huge VideoMAE setup, compared with 81.01% for base and 85.35% for large.The scalability study links stronger performance to proper video diversity and increased training size.
- Approach: Cross-model attention combines separately trained masked-video and multimodal representations while keeping their backbones frozen during lightweight supervised interaction learning.This avoids directly joint-training objectives that may conflict and provides a computationally tractable fusion mechanism.
- Results: InternVideo outperforms state-of-the-art methods across 10 tasks and 39 datasets spanning action recognition, localization, video question answering, retrieval, and other applications.The benchmark covers action understanding, video-language alignment, and open-world video applications.
2 Related Work
Related work spans image foundation models, video foundation models, self-supervised pretraining, and multimodal pretraining. These lines of research motivate combining masked modeling, contrastive learning, and video-language modeling for broader video understanding.
- Image Foundation Models: Image foundation models use web-scale image-text data and contrastive learning to produce robust representations that support zero-shot transfer.The related work contrasts these image-focused models with the broader requirements of video understanding.
- Video Foundation Models: Prior video foundation models show promise for recognition or multimodal benchmarks but often target limited video-text data or specific task settings.Examples include masked language and video modeling, unified video-language pretraining, and masked language modeling.
- Self-supervised Pretraining: Self-supervised pretraining is commonly organized around contrastive learning and masked modeling as alternative pretext-task families.Contrastive methods construct augmented views and optimize positive-pair agreement against negative samples.
- Multimodal Pretraining: Video-language research has progressed from offline visual and textual features toward end-to-end large-scale video-text pretraining with downstream fine-tuning.The related work identifies this as a standard paradigm in the video-language area.
3 InternVideo
InternVideo combines masked video modeling and multimodal contrastive learning, then coordinates their representations through learnable cross-model interactions. It further uses supervised action classification and evaluates the resulting general video foundation model across diverse applications.
- InternVideo: InternVideo integrates masked modeling, multimodal learning, and supervised training to progressively improve video representations.Its self-supervised components include masked modeling and multimodal learning, while supervised action classification further enhances representation quality.
- Masked Video Modeling: VideoMAE reconstructs highly masked video inputs with an asymmetric ViT encoder-decoder for spatiotemporal representation learning.Temporal downsampled inputs are divided into non-overlapping 3D patches and projected into cube embeddings.
- Video-Language Contrastive Learning: The multimodal encoder performs video/image-text contrastive learning and video captioning using a CLIP-based architecture with UniFormerV2 temporal modeling.Video and text are separately encoded before cross-modal learning.
- Supervised Video Post-Pretraining: Action classification is used as a post-pretraining step because it transfers to diverse downstream video applications.The masked video and multimodal encoders are trained separately with supervised action classification on the Kinetics-710 benchmark.
- Cross-Representation Learning: InternVideo learns a unified representation by adding cross-model attention modules that align representations from masked modeling and video-language contrastive learning.The approach freezes both backbones except classification layers and multimodal query tokens, updating newly added components.
4 Experiments
The experiments evaluate InternVideo across action understanding, video-language alignment, and open understanding tasks. This setup is intended to measure downstream performance across a broad video-understanding benchmark.
- 4 Experiments: InternVideo is evaluated on three task groups: action understanding, video-language alignment, and open understanding.The experiments cover the downstream performance of InternVideo across these task categories.
4.1 Data for Pretraining
InternVideo uses large-scale, cross-domain video data for general video foundation model pretraining. The pretraining resources include public datasets and self-collected clips, with Kinetics-710 supporting supervised training and UnlabeledHybrid supporting masked pretraining.
- 4.1 Data for Pretraining: Six public datasets and self-collected video clips are used to create a diverse pretraining distribution.The data sources support general video foundation model pretraining across varied domains.
- 4.1 Data for Pretraining: Kinetics-710 contains 650K videos with 710 unique action labels and is used for supervised training.It combines unique training data from Kinetics-400, Kinetics-600, and Kinetics-700 while removing certain testing-overlapping training examples.
- 4.1 Data for Pretraining: UnlabeledHybrid combines Kinetics-710, Something-Something V2, AVA, WebVid2M, and self-collected videos for masked video pretraining.The construction includes sampled videos from WebVid2M and self-collected data, plus clips extracted from AVA training videos.
4.2 Implementations
The implementation uses large-scale video-text and image-text post-pretraining, long masked-video pretraining, and coordinated-model training. These stages rely on substantial batch sizes, multi-GPU hardware, and cosine-annealed optimization schedules.
- 4.2 Implementations: The multimodal model is post-pretrained on WebVid2M, WebVid10M, HowTo100M, and 100M LAION image-text pairs.Images and videos are alternated during training, with video-text batch size 14,336 and image-text batch size 86,016.
- 4.2 Implementations: VideoMAE-Huge is trained for 1200 epochs on UnlabeledHybrid using 64 80G-A100 GPUs.Training uses cosine annealing, 10% warmup, a learning rate of 2.5e −4, and MultiScaleCrop augmentation.
- 4.2 Implementations: Coordinated-model training freezes both backbones except classification layers and multimodal query tokens, updating newly added components.The setup uses tanh gating in added MHCA and FFN modules, zero-initialized dynamic weighted-sum parameters, and a 5-epoch cosine schedule.
4.3 Downstream Tasks
InternVideo is evaluated across action understanding, video-language alignment, and open-world video tasks, demonstrating strong performance across diverse benchmarks. Its video representations support both spatiotemporal reasoning and multimodal applications.
- Action Recognition: InternVideo improves action recognition across eight benchmarks and significantly surpasses previous state-of-the-art methods on nearly all evaluated datasets.The evaluation includes Kinetics, Something-in-Something, ActivityNet, HACS, and HMDB51, using top-1 accuracy.
- Temporal Action Localization: InternVideo outperforms previous methods on four temporal action localization datasets, with especially large gains on fine-grained benchmarks.The datasets are THUMOS-14, ActivityNet-v1.3, HACS Segment, and FineAction; evaluation uses mean Average Precision.
- Spatiotemporal Action Localization: InternVideo achieves state-of-the-art spatiotemporal action localization performance on AVA2.2 and AVA-Kinetics.Using ViT-Huge and a simple linear classification head, training with AVA-Kinetics also improves performance when testing on AVA alone.
- Video-Language Alignment: InternVideo significantly outperforms previous methods on video retrieval across MSR-VTT, MSVD, LSMDC, DiDeMo, ActivityNet, and VATEX.Results are reported with R@1 for both text-to-video and video-to-text retrieval.
- Video Question Answering: InternVideo outperforms previous state-of-the-art methods on video question answering across MSR-VTT, MSVD, and TGIF.The model uses cross-modality fusion through its multimodal video encoder and caption decoder.
- Video Open Understanding: InternVideo demonstrates broad open-world generalization, including 64.25% accuracy for zero-shot Kinetics-400 action recognition and consistent gains on open-set recognition.It also improves a visual-navigation baseline from 50.2% to 52.9% Success Rate and identifies unknown action classes without model calibration.
5 Concluding Remarks
InternVideo demonstrates the feasibility of versatile and training-efficient video foundation models across action, video-language, and open-understanding tasks. The study remains focused on clip-based video perception and does not address long-term or higher-order video understanding.
- Concluding Remarks: InternVideo achieves state-of-the-art performance on nearly 40 datasets spanning 10 tasks and three broad video-understanding areas.The model uses cross-model learning between VideoMAE and video-language contrastive modeling, together with supervised training.
- Concluding Remarks: InternVideo reaches generalized video representations with 64.5K GPU hours, compared with 245.76K TPU hours required by CoCa.The paper reports record-breaking results using simple task heads and suitable downstream adaptation tuning.
- Scope: The study focuses on current video perception tasks and processes videos using clips rather than introducing brand-new formulations or model designs.This scope supports practical video foundation modeling but does not cover all forms of video understanding.
- Limitations: Clip-based processing leaves long-term video tasks and higher-order tasks, such as anticipating a movie's plot from observed parts, unresolved.The paper identifies these capabilities as important directions for advancing video representation learning.
- Future Directions: Systematically coordinating models across modalities, pretraining tasks, and architectures remains an open challenge for improving video representations.Suggested technical routes include model distillation, unified pretraining objectives, and feature alignment.
- Future Directions: Future video foundation models are expected to extend from perceptual capabilities toward large-scale spatiotemporal analysis and cognitive understanding.The paper also points to combining foundation models with decision-making to form intelligent agents for new tasks.
6 Broader Impact
InternVideo is presented as a video foundation model spectrum supporting broad video understanding, with reported state-of-the-art performance across around 40 datasets. The authors also report reduced training power consumption and identify social-impact topics requiring further study.
- InternVideo delivers state-of-the-art performance on around 40 datasets spanning action discrimination, video-language alignment, and open understanding.
- Training InternVideo uses 23.19% of the power consumption reported for CoCa.
- The authors state that future impact studies should examine bias, risks, fairness, equality, and related social topics.