Source-linked AI summary
LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training
Andreas Hochlehnert, Marianna Nezhurina, Mehdi Cherti, Andrej Radonjic, Thaddäus Wiedemer, Christoph Schuhmann, Romain Beaumont, Wieland Brendel, Bernhard Schölkopf, A. Sophia Koepke, Jenia Jitsev, Matthias Bethge
TL;DR
Open video-language resources are comparatively small and difficult to process at web scale, despite videos offering aligned visual and audio signals. LAION-BVD addresses this gap with a large open dataset and scalable scene-level multimodal captioning pipeline, and its models show strong, consistently scaling performance across video, audio, and frame-based image-text evaluations.
Problem
Open video-language resources remain comparatively small, while collecting and processing web videos at scale requires substantial compute, memory, engineering effort, and platform access.
Method
LAION-BVD collects 1.3B CommonCrawl video URLs, downloads 80M videos, extracts scene-level clips and frames, and generates video, audio, and frame captions for multimodal pre-training.
Results
Models trained on LAION-BVD show strong performance across video-text, audio-text, and frame-based image-text evaluations, with consistent scaling trends as data and compute increase.
Takeaways & Limitations
The released dataset and metadata broaden access to large-scale open web video and support more reproducible joint video, audio, and image research.
Takeaways & Limitations
Automatically generated captions are intentionally short, limiting caption richness and potentially introducing systematic captioning errors or biases.
Abstract
from arXiv · showhide
We present LAION-BVD, a large-scale open video dataset for multimodal learning, which contains 1.3B platform-specific video URLs collected from CommonCrawl. From these, we download 80M videos with a total duration of 10 million hours. The dataset is designed for multimodal pre-training across the video, audio, and image modalities. Using content-aware scene detection, we extract clips for which we synthetically generate video and audio captions. Models trained on these data achieve competitive performance on standard video-text and audio-text benchmarks, with consistent improvements as training or model scale increases. Additionally, we explore video frames as an alternative source of image-text data by extracting scene-changing frames. These frames exhibit a visual distribution distinct from standard web image corpora, and models trained on this dataset achieve strong image-text retrieval performance. We release LAION-BVD to the research community. It significantly expands open access to multimodal videos at an unprecedented scale.
1 Introduction
LAION-BVD addresses the relative scarcity and practical difficulty of assembling open video-language resources by releasing a large-scale web video dataset with multimodal annotations. The dataset supports video-, audio-, and image-text pre-training and is evaluated across multiple model and data scales.
- Motivation: Open video-language resources remain comparatively small, while web videos provide visual content and aligned audio for multimodal learning.InternVid contains 7 million videos, and large-scale collection is constrained by processing demands and platform access restrictions.
- Dataset: LAION-BVD is built from 1.3B platform-specific video URLs collected from CommonCrawl and includes 80M videos totaling 10 million hours.The corpus is released as an open web video dataset.
- Dataset: A scalable annotation pipeline segments videos into scenes and generates clip-level video captions, audio captions, and frame-caption pairs.The pipeline supports multimodal pre-training across video, audio, and image modalities.
- Evaluation: ViCLIP, CLAP, and CLIP models trained with these data are used to validate video-language, audio-language, and image-text pre-training at different model and data scales.The evaluations cover standard video-text and audio-text benchmarks and frame-based image-text retrieval.
- Release: LAION-BVD is released with video URLs and subset captions, while research institutions can download raw data after accepting terms of use.This release is intended to broaden access to multimodal video research.
- Dataset: The released analyzed subsets include 55M clips and 300M scene-changing frames sampled from the larger corpus.The clips support video- and audio-language experiments, while frames support image-text analysis.
2 Related Work
Related work spans multimodal foundation models, automatic captioning, and open datasets for video-text, audio-text, and image-text learning. LAION-BVD is positioned within this landscape as a web-video resource that retains paired visual and audio streams and supplies frame-caption supervision.
- Multimodal foundation models: Multimodal foundation models commonly use paired data from at least two modalities, including image-text, video-text, and audio-text data.Embedding models learn shared cross-modal spaces, while multimodal models process images or videos and output text.
- Video-text datasets: Video-text datasets range from manually annotated domain-specific collections to larger general datasets using automatically derived supervision such as ASR, subtitles, and alt-text.Video captions typically cover clips from several seconds to a few minutes.
- LAION-BVD positioning: Table 1 compares LAION-BVD with public open-ended video-text datasets, while current ViCLIP and CLAP experiments use a recaptioned 55M-clip subset.The subset comes from roughly 2.4M videos, whereas the full released corpus provides the broader scale context.
- Audio-text datasets: Audio-text datasets support speech, music, and ambient-sound learning, with larger resources often using weak or automatically generated captions.Representative corpora include AudioCaps, Clotho, WavCaps, AudioSetCaps, and LAION-CLAP.
- LAION-BVD positioning: LAION-BVD differs from dedicated audio corpora by sourcing web videos while retaining the paired visual stream alongside audio.Its frame-caption pairs also provide supervision for image-text training.
- Image-text datasets: Open image-text datasets have scaled from 330k English pairs in MS-COCO to over 2B in LAION-5B’s English-language subset.Image-text pairs are comparatively easy to collect because web images often have captions or descriptive alt-text.
- LAION-BVD positioning: Table 2 compares representative public audio-text datasets and distinguishes the 55M recaptioned experimental subset from the fully released corpus.The captioned row covers clips used in audio-language experiments; the total row covers captioned and uncaptioned data.
- Automatic captioning: Automatic multimodal curation pipelines caption images, split videos into clips, and generate audio captions with pretrained models.This shared approach enables annotation across multiple modalities at scale.
3 Dataset
LAION-BVD is curated from CommonCrawl video URLs into a large multimodal corpus, then processed into captioned scene clips and scene-changing frames. The resulting dataset spans substantial scale, diverse sources and languages, and supports video-language evaluation with scaling gains.
- Data curation: 1.3B platform-specific video URLs were extracted from CommonCrawl, and distributed downloading produced 80M videos totaling 10M hours.The curation pipeline uses CommonCrawl metadata, platform-specific extractors, and distributed infrastructure.
- Clip extraction: 55M video and audio clips were extracted from 2.4M sampled videos after duration filtering, scene detection, and removal of effectively static segments.The shared preprocessing pipeline creates scene-level segments for both video-language and audio-language experiments.
- Frame extraction: 300M scene-changing keyframes were filtered from sampled videos to form the BVD-I-300M image-training subset.Black frames were removed, and ffmpeg scene detection used a threshold of 0.1.
- Captioning: Modality-specific captioners generated video captions, audio captions, and frame captions for the extracted clips and keyframes.Video clips were captioned with Qwen3-VL-2B-Instruct, audio segments with Audio Flamingo 3, and keyframes with DeepSeek-VL2-tiny.
- Dataset statistics: 94% of videos came from YouTube, with 4% from Vimeo and 2% from Dailymotion, while topics and languages remained diverse.Vlogs and music were the leading topics, and almost 60% of content was in English.
- Dataset validation: BVD-trained ViCLIP outperformed InternVid-trained ViCLIP and an image-only CLIP baseline, with performance improving as data and model scale increased.BVD-V-50M achieved the best overall average among the evaluated LAION-BVD subsets.
4 Experiments validating LAION-BVD
LAION-BVD is evaluated as a source of video-text, audio-text, and image-text training data. Across these settings, its minimally curated data supports competitive performance and scaling, while frame-caption pairs favor retrieval over classification.
- Evaluation setup: 55M synthetically annotated clips from 2.4M videos support video-text and audio-text evaluation, while 300M captioned frames support image-text analysis.ViCLIP and CLAP are used for video and audio; CLIP is used for static frames.
- Video-text validation: 3.3 and 4.0 percentage points are the improvements of BVD-V-10M and BVD-V-50M over InternVid-10M-FLT at 50M samples seen.Performance increases with dataset size, compute, model size, and compute under the reported evaluations.
- Video-text validation: 2.1 percentage points higher aggregate performance than InternVid-trained ViCLIP (FLT) is achieved by minimally curated LAION-BVD samples.The aggregate metric combines zero-shot classification and retrieval benchmarks.
- Audio-text validation: BVD-A-1.7M and BVD-A-10M match or exceed LAION-Audio across model scales for matched samples seen, although LAION-Audio + AudioSet performs better.Both BVD audio subsets also show consistent scaling with model size.
- Overall findings: LAION-BVD provides a viable audio-data source for multimodal pre-training across single- and mixed-dataset settings with favorable scaling behavior.The conclusion is supported by consistent CLAP performance in both training settings.
- Image-text validation: Frame-caption pairs provide strong, favorably scaling image-text retrieval supervision but weaker classification performance because caption style and class coverage differ.LAION-BVD shows weaker scaling for ImageNet-1k classification, while retrieval remains strong.
5 Limitations
The study’s limitations concern automatically generated captions, restricted model coverage, separate rather than joint modality evaluation, and residual dataset biases.
- Caption quality and style: Automatically generated short captions scale processing but constrain caption richness and may introduce systematic errors or biases.The captions are produced by small captioning models.
- Model scope: Evaluation focuses on contrastive video-text training with ViCLIP and does not cover generative video-language or diffusion-based approaches.The authors call for further evaluation on generative tasks.
- Audio-visual integration: Video-text, audio-text, and image-text settings are evaluated separately, leaving unified audio-visual representations and tight synchronization untested.The study does not train joint audio-visual models.
- Data biases: Minimal filtering leaves possible biases, stereotypes, and uneven representation across languages, regions, and topics that models may inherit.The source platforms apply moderation, but residual dataset bias remains possible.
6 Discussion and Conclusion
LAION-BVD provides a large-scale open resource for multimodal pre-training, spanning video, audio, and image data. Its evaluations show strong training signals across modalities and consistent scaling trends, while its release supports more reproducible research.
- 1.3B video platform links, 10 million video hours, 55M clip-level captions, and 300M frame captions define LAION-BVD’s multimodal scale.The dataset is designed for joint video, audio, and image-text pre-training.
- Video, audio, and individual frames each provide a strong training signal in the reported evaluations.ViCLIP outperforms InternVid-based models at matched model and data scale, CLAP remains competitive, and extracted frames support retrieval.
- Increasing data and compute produces consistent scaling trends across modalities, supporting LAION-BVD as a reliable multimodal pre-training source.
- Releasing the dataset, metadata, and documented curation pipeline broadens access and supports more reproducible multimodal foundation-model research.
A.1.1 Experimental setup
The ViCLIP experiments adapt CLIP to process video frames while retaining its text encoder, then train across selected data and optimization settings. The setup uses uniformly sampled frames and compares predefined hyperparameter configurations.
- ViCLIP combines a video encoder with a text encoder initialized from CLIP, modifying vision processing to handle frame sets and temporal information.
- Eight frames are uniformly sampled from each video for ViCLIP training.
- Training sweeps learning rates from 1e-6 to 4e-4 and compares global batch sizes of 8k and 32k.
- The reported ViCLIP hyperparameter settings are summarized in Table 11.
A.1.2 Additional results
Additional analyses examine scaling reliability, benchmark overlap, caption faithfulness, and the audio-training setup. Larger video datasets improve selected metrics, decontamination has limited impact, and most audited captions are faithful.
- At fixed compute, larger training data yields statistically clear improvements on HMDB51, MSR-VTT V2T, MSVD T2V, and the overall average.Several other metric differences remain within the uncertainty range.
- The overlap analysis counts shared unique YouTube video IDs between BVD-V-55M and K400, MSR-VTT, and MSVD.Decontaminated test sets remove examples whose source-video IDs occur in BVD-V-55M.
- Decontaminated MSR-VTT and MSVD test sets retain performance comparable to the original sets, indicating limited impact from observed overlap.
- 79.1% of 134 audited video captions were accurate, while 97.8% contained no major error.Minor errors affected 18.7% and major errors affected 2.2% of captions.
- Most caption errors involved incorrect actions, objects, or scene descriptions, while the audit found that captions were generally faithful and useful for supervision.
- CLAP training uses paired audio-caption data, shared audio-text embeddings, and symmetric contrastive loss across BVD-A-1.7M and BVD-A-10M.The setup studies scaling across dataset and model size.
- Audio evaluation uses UrbanSound8K classification plus AudioCaps and Clotho retrieval with text-to-audio and audio-to-text Recall@5.
- Table 17 reports computational training costs for different CLAP backbones at a fixed number of training samples.
A.2.2 Additional results
The additional audio-language results validate the CLAP training pipeline against the published LAION-CLAP baseline. The open pipeline consistently achieves higher performance, while BVD-A-1.7M remains competitive overall.
- The open CLAP implementation consistently outperforms the original LAION-CLAP model trained on the same dataset.
- At matched 158M model scale, the open pipeline reproduces and exceeds the published LAION-CLAP baseline at comparable or lower training compute.
- Replacing LAION-Audio with BVD-A-1.7M remains competitive overall and is particularly strong on UrbanSound8K and AudioCaps retrieval.
A.2.3 Human audit of audio captions
A human audit found that most synthetically generated audio captions were accurate, although substantial errors remained in a minority of cases.
- 106/134 (79.1%) audio captions were judged accurate in the human audit.
- 20/134 (14.9%) captions contained minor errors, while 8/134 (6.0%) contained major errors.
- 126/134 (94.0%) captions had no major error.
- The main hallucination types involved incorrect sound events, speech content, music descriptions, and speaker attributes.The audit recorded 10 incorrect sound events, 5 speech-content errors, 4 music-description errors, and 1 speaker-attribute error.
- Audio captioning remained more prone to substantial errors than video captioning.
A.2.4 Contamination analysis of Audio Flamingo 3 with downstream test sets
The analysis finds that Audio Flamingo 3 overlap with AudioCaps test audio does not explain BVD retrieval gains, while BVD caption statistics and frame-caption experiments characterize the dataset’s broader behavior.
- Contamination analysis: 356 of 975 AudioCaps test video IDs occurred in the AF3 AudioSkills subset, but with AudioSet-derived rather than AudioCaps reference captions.
- Contamination analysis: 607 of 975 AudioCaps test clips were available for evaluation after excluding clips absent from AF3 training.The evaluation compared these clean clips with a fixed random control set of the same size because the smaller gallery itself raises R@5 by approximately 8.3 percentage points.
- Contamination analysis: BVD-trained models were not more affected by removing AF3-seen examples, and BVD models showed a smaller pooled reduction than non-BVD models.
- Caption analysis: BVD captions have a broader length distribution than AudioCaps and Clotho captions, with longer captions dominating in the analyzed BVD subset.
- Frame-caption generation: Synthetic frame captions were generated by selecting a quality-throughput-balanced VLM, with DeepSeek-VL2-tiny achieving the highest CLIPScore and second-highest throughput.
- Frame-caption validation: Automatically generated captions reduced classification accuracy on ImageNet benchmarks in validation, motivating further analysis of caption length and semantic coverage.
B Why Strong Retrieval Does Not Translate to ImageNet-1k Accuracy
LAION-BVD frame captions support strong image-text retrieval but weaker ImageNet-1k zero-shot classification. The analysis associates this gap with caption style and class-name coverage rather than caption length or fluency alone.
- Retrieval performance: Up to 0.87 R@5 was achieved on Flickr30k image-to-text retrieval by models trained on LAION-BVD subsets.
- Caption statistics: LAION-BVD captions have a narrower distribution dominated by longer captions, whereas DataComp captions have a broader distribution with many short captions.
- ImageNet-1k class-name coverage: LAION-BVD captions contain approximately 21M ImageNet-1k class-name mentions, compared with 141M in DataComp captions.
- ImageNet-1k class-name coverage: DataComp captions align more strongly with ImageNet-1k semantic categories through greater density and diversity of class-associated terminology.
- Implications: Caption length or fluency alone is insufficient for improving downstream transfer, while class-centric semantic alignment may better support ImageNet-like zero-shot classification.