Source-linked AI summary
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Conghui He, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, Yu Qiao
TL;DR
Video-language learning lacks high-quality datasets that combine scale with strong video-text correspondence. This paper builds InternVid using multiscale LLM captioning and trains ViCLIP on it, obtaining strong zero-shot action recognition and competitive retrieval while supporting dialogue and video generation. The dataset’s coverage remains bounded by the diversity and representativeness of YouTube videos.
Problem
Video-language learning lacks a high-quality dataset that supports pretraining at scale while maintaining strong video-text correspondence.
Method
InternVid uses multiscale LLM-generated captions for large-scale video-text data, while ViCLIP applies ViT-L contrastive learning with video masking.
Results
ViCLIP achieves leading zero-shot action recognition, competitive video retrieval, and improves text-to-video generation when using InternVid subsets.
Takeaways & Limitations
InternVid provides resources for scalable video-language representation learning, video-centric dialogue, and video generation research.
Takeaways & Limitations
YouTube-based coverage excludes or scarcely collects some categories, limiting generalizability across all types of video data.
Abstract
from arXiv · showhide
This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.
1 Introduction
InternVid addresses the shortage of high-quality, scalable video-text data by combining large-scale collection with multiscale LLM-generated captions. Its dataset supports ViCLIP training and downstream video understanding, dialogue, and generation applications.
- Motivation: Web-scale video-language learning remains underexplored because existing large datasets often have weak semantic correspondence between videos and ASR-derived text.Improved video-text correlation has been shown to benefit video retrieval and video question answering.
- InternVid: InternVid contains over 7 million videos, 760,000 hours, and 234 million clips spanning 16 scenarios and around 6,000 motion descriptions.The dataset is designed to scale video-language modeling while maintaining high video-text correspondence.
- InternVid: Multiscale captioning uses a middle-frame description at coarse scale and frame-by-frame captions summarized by a language model at fine scale.This approach generates clip-level descriptions with minimal human intervention.
- ViCLIP: ViCLIP scales a ViT-L video-language transformer with contrastive learning and video masking, achieving notable improvement over previous Video CLIP variants, especially in zero-shot settings.The model is trained on InternVid and uses video masking to accelerate learning without compromising effectiveness.
- Applications: InternVid supports approximately 7 million interleaved video-text pieces for video-centric dialogue and provides 18 million filtered clips for video generation.Alongside WebVid-10M, InternVid significantly improves a stable-diffusion-based video generation model.
- Results: ViCLIP achieves zero-shot action-recognition scores of 75.7, 73.5, and 66.4 on K400, K600, and K700, respectively, while providing competitive video-retrieval performance.InternVid-Aesthetics also improves a simple text-to-video baseline from FVD: 705.3 -> 616.5.
2 Related Work
Prior video-language datasets and models established scalable multimodal pretraining, but often relied on ASR text or limited video-text data. Related work spans instructional datasets, higher-alignment web video collections, and masked or unified multimodal modeling approaches.
- Video-centric datasets: HowTo100M uses instructional YouTube videos with corresponding ASR subtitles to learn joint video-text representations.YT-Temporal and HD-VILA extend related efforts toward audio-visual-language learning and high-resolution video crossmodal learning.
- Video-centric datasets: WebVid was developed around the finding that video-text alignment matters more than quantity, using 10M videos with higher-quality alt-texts.Its limited scale and relatively low activity diversity constrain its use in current scaling studies.
- Multimodal pretraining: VIOLET, All-in-one, and LAVENDER combine masked language or video modeling with unified multimodal pretraining, but rely on limited video-text data.This limitation hampers performance in video-only tasks such as action recognition.
3 InternVid: A Video-Centric Multimodal Dataset
InternVid is constructed as a large-scale video-language dataset emphasizing temporal dynamics, semantic diversity, and strong video-text correspondence. It combines YouTube curation, multiscale captioning, clip-level features, and interleaved video-text formats.
- Dataset Design: The dataset construction targets substantial scale, temporal dynamics, rich semantics, and strong video-text correlations.Videos are gathered through category- and action/activity-based strategies, while descriptions are generated from frame-specific annotations.
- Data Curation: InternVid contains 7 million YouTube videos averaging 6.4 minutes across 16 topics, curated from popular content and action-based searches.The collection combines 2 million videos from popular channels with 5.1 million videos retrieved using action/activity verbs.
- Data Curation: 6,104 action queries are assembled from time-use data, public video datasets, visual-grounding corpora, and language-model extraction with manual checking.The resulting queries are used to search YouTube for action-focused videos.
- Data Curation: Videos are segmented by scene variance into clips averaging around 10 seconds, yielding 234 million clips after filtering still and extreme-dynamics segments.The resulting clip durations range from 2 seconds to more than 30 seconds.
- Captioning: Multiscale captioning describes common objects, actions, and scenes at fine scale while captioning only the central frame at coarse scale.The two strategies are designed to produce scalable, rich, and diverse video captions.
- Statistics and Features: InternVid spans 16 categories and multiple countries, with videos averaging 351.9 seconds; 85% of clips last 0–10 seconds and captions vary in length.Approximately half of captions contain 10–20 words, one-third contain fewer than 10 words, and 11% exceed 20 words.
- Statistics and Applications: The dataset provides aesthetic and clip-caption similarity scores, contains about ten times more verbs than WebVid10M, and supports 7.1M interleaved video-text pairs.InternVid-ICL organizes clips and captions into multiple interleaved formats, including sequential ordering and optional ASR text.
4 ViCLIP: Learning Video-Text Representation at Scale
ViCLIP combines CLIP-initialized video and text encoders with spatiotemporal attention, masking, and contrastive video-text alignment. Trained on InternVid, it achieves strong action-recognition, retrieval, dialogue, and generation results.
- Model and training: ViCLIP uses CLIP-initialized ViT video and text encoders, replacing native video attention with spatiotemporal attention and optimizing contrastive alignment.Random video patch masking reduces computational burden, while unmasked training helps bridge pretraining and downstream full-video use.
- Action recognition: ViCLIP trained on InternVid-10M-FLT sets new zero-shot action-recognition records on Kinetics-400, Kinetics-600, and Kinetics-700.The reported result exceeds ViCLIP trained on WebVid10M and other compared models, including EVA-CLIP-E.
- Action recognition: 87.9% in K400 and 73.6% in SthSthV2 are achieved by ViCLIP pretrained on 200M pairs in fine-tuned recognition, versus 86.8% and 71.2% with 10M.The comparison indicates that larger InternVid pretraining improves fine-tuned recognition, unlike the reported zero-shot subset discrepancy.
- Video retrieval: 3.7% is the average increase in fine-tuned text-to-video R@1 across five retrieval benchmarks when transitioning from InternVid-10M to WebVid10M.InternVid-10M-DIV and InternVid-10M-FLT show overall increases of 5.8% and 5.1% over WebVid10M, respectively.
- Text-to-video generation: InternVid-Aesthetics-18M substantially improves the WebVid10M text-to-video baseline in IS, FID, and CLIPSIM, with better visual quality and temporal coherence.The InternVid-based baseline also avoids the watermarks identified as a WebVid10M data bias.
- Video dialogue: 2.64 is VideoChat-ViCLIP’s average quantitative video-conversation score, up from 2.29 for vanilla VideoChat.Correctness rises from 2.23 to 2.86, contextual understanding from 2.53 to 3.08, and temporal understanding from 1.94 to 2.36.
5 Conclusion
InternVid supports multimodal video research by combining large-scale clip descriptions, metadata, and computed scores with a ViT-L video-text representation baseline. The paper reports benefits for crossmodal learning and text-to-video generation.
- Conclusion: InternVid contains over 230 million clips from 7 million high-resolution YouTube videos, with clip-level descriptions generated using existing models and a multiscale approach.The dataset includes annotations, metadata, and computed scores for multimodal research focused on videos.
- Conclusion: InternVid is designed to support both video understanding and generation through large-scale video-text data and caption-based resources.The conclusion specifically connects the dataset to crossmodal learning and text-to-video generation at scale.
A Data Availability Statement
The released data is publicly available and distributed through YouTube video IDs rather than raw video files. Usage is restricted to research, with collection and release practices described as aligned with YouTube policies.
- Data availability: The studies use publicly available data and do not use exclusive or private data sources.The statement frames the data source as publicly accessible.
- Data sharing: The release supplies YouTube video IDs needed to download content instead of providing original raw data.This policy follows the precedent of datasets such as Kinetics and HD-VILA.
- Usage rights: Commercial usage is not sanctioned because the released data is intended exclusively for research purposes.The usage-rights statement limits the stated purpose of the release.
- Policy compliance: The authors state that their collection and release practices comply with YouTube’s data privacy policies and do not violate user data or privacy rights.The statement presents compliance as a condition of the data practices.
- Licence: The stated data licence is CC BY 4.0.The licence is identified in the availability statement without additional conditions in the supplied passage.
B Limitations & Societal Impact
InternVid uses YouTube videos with content and privacy safeguards, but its coverage remains limited by platform and copyright constraints. The dataset may therefore not generalize across all video categories.
- Videos are downloaded from YouTube using Safe for Work queries and channels, with a binary NSFW filter excluding non-ethical content.
- Only YouTube IDs are shared, aligning the release with YouTube’s data protocols and privacy considerations.
- The dataset samples videos across 11 languages, including English, Chinese, Korean, and German.
- Coverage may exclude surveillance footage, sports competitions, movies, and documentaries because of copyright or platform restrictions.The authors caution that these omissions constrain generalizability across all video-data categories.
C More Statistics in InternVid
InternVid’s captions show broad semantic coverage and stronger action-oriented vocabulary than WebVid10M, while multilingual distributions support analysis and customization. These statistics characterize the dataset’s semantic and geographic richness.
- Actionness: 212,155 unique verbs occur in InternVid captions versus 109,485 in WebVid10M, indicating greater actionness in InternVid.The authors note that simple counting may make these totals approximate.
- Video Caption and Transcript Distribution: InternVid captions include objects, attributes, locations, scenes, and actions/events such as walking, eating, cutting, and holding.
- Multilingual distributions: Word distributions are provided for four languages and reflect trends across countries while enabling potential data customization through metadata.
D InternVid-ICL: Interleaved Video-Text for In-Context Video Learning
InternVid supports interleaved video-text formats for in-context learning by arranging clips, captions, and transcripts in temporal or concatenated sequences. These formats can vary the amount of video context.
- Format (a): Format (a) arranges clips and their descriptions sequentially according to temporal order within the same video.
- Context construction: Videos are randomly dropped with probability 0.3 to construct richer text context than sequential original video-text pair combinations.
- Interleaved formats: Format (b) displays each clip’s caption and ASR transcript, while format (a) is obtained by omitting ASR transcripts.
- Interleaved formats: Format (c) concatenates multiple videos that use interleaved video-text format (a).
E.1 ViCLIP
ViCLIP is evaluated for zero-shot and fine-tuned video understanding, while the paper also describes a text-to-video baseline and its evaluation protocol. The supplied material emphasizes action recognition, retrieval, and generation comparisons.
- Action Recognition: Zero-shot action recognition samples 8 frames per video and reports mean top-1 and top-5 accuracy on Kinetics-400, Kinetics-600, and Kinetics-700.
- Action Recognition: Full fine-tuning excludes or includes K710 in separate experiments, using either standard benchmark fine-tuning or a K710-first training procedure.
- Video Retrieval: Video retrieval fine-tuning combines video-text contrastive loss with video-text matching loss and samples 12 frames during training and testing.Zero-shot retrieval evaluation samples 8 frames.
- Text-to-Video Generation: The text-to-video baseline uses a U-Net transformer with interleaved spatiotemporal attention, visual-text cross-attention, feed-forward layers, and temporal attention.Its 2D convolutional kernels are extended from 3 × 3 to 1 × 3 × 3.
- Text-to-Video Evaluation: Text-to-video evaluation generates 2,020 UCF-101 videos and 2,990 MSRVTT videos, assessing synthesis quality with FID, FVD, and IS and semantic similarity with CLIPSIM.
- Text-to-Video Results: Adding InternVid-Aesthetics-18M to WebVid10M substantially improves the baseline’s IS, FID, and CLIPSIM and visual temporal quality versus other methods.
F More Results
The authors compare ViCLIP models trained with captions from their method and VideoChat, and evaluate InternVid-based text-to-video generation against other baselines. Their captions improve retrieval and action recognition, while InternVid-based generation performs strongly in visual quality and temporal coherence.
- Captioning baseline: ViCLIP-B trained with the authors’ captions outperforms VideoChat-captioned training on MSR-VTT retrieval and K400/600/700 action recognition.The training setup differs only in the source of the generated captions.
- Text-to-video generation: InternVid-Aes-18M and WebVid10M text-to-video baselines significantly outperform other methods in visual quality and temporal coherence.
- Text-to-video generation: InternVid-based text-to-video generation avoids the watermarks present in the WebVid10M data bias.