PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, Ying Tai

arXiv:2407.02371v3cs.CV

TL;DR

Text-to-video generation needs precise, high-quality text-video data and better use of textual information. The paper introduces OpenVid-1M, OpenVidHD-0.4M, and MVDiT, and reports superiority across text-to-video and video restoration evaluations, though intricate natural-scene dynamics remain difficult.

  • Problem

    Text-to-video generation lacks a precise, open-sourced, high-quality dataset, despite the importance of collecting accurate text-video pairs.

  • Method

    The paper curates OpenVid-1M with high-resolution clips and expressive captions, selects OpenVidHD-0.4M 1080p videos, and proposes MVDiT to jointly model visual structure and text semantics.

  • Results

    The authors report superiority of OpenVid-1M on text-to-video and video restoration tasks; their model achieves 73.46% VQAA and 68.58% VQAT.

  • Takeaways & Limitations

    OpenVid-1M provides a million-level, high-quality dataset with expressive captions for text-to-video research, while OpenVidHD-0.4M supports high-definition video generation.

  • Takeaways & Limitations

    The model still struggles with intricate dynamics and motions in natural scenes, sometimes producing unrealistic videos.

Abstract

from arXiv · show

Text-to-video (T2V) generation has recently garnered significant attention thanks to the large multi-modality model Sora. However, T2V generation still faces two important challenges: 1) Lacking a precise open sourced high-quality dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M, are either with low quality or too large for most research institutions. Therefore, it is challenging but crucial to collect a precise high-quality text-video pairs for T2V generation. 2) Ignoring to fully utilize textual information. Recent T2V methods have focused on vision transformers, using a simple cross attention module for video generation, which falls short of thoroughly extracting semantic information from text prompt. To address these issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive captions. This open-scenario dataset contains over 1 million text-video pairs, facilitating research on T2V generation. Furthermore, we curate 433K 1080p videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition video generation. Additionally, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of mining both structure information from visual tokens and semantic information from text tokens. Extensive experiments and ablation studies verify the superiority of OpenVid-1M over previous datasets and the effectiveness of our MVDiT.

1 INTRODUCTION

The paper addresses limited access to precise, high-quality text-video data and incomplete use of textual information in text-to-video generation. It introduces OpenVid-1M and MVDiT to address these dataset and modeling challenges.

  • Challenges: Existing datasets often trade off scale against quality, containing watermarks, blurry or flickering videos, or short and imprecise captions.UCF-101 uses category labels, while WebVid-10M and Panda-70M include quality and caption limitations.
  • Dataset: OpenVid-1M provides over 1 million high-quality, open-scenario video clips with detailed captions.The clips have resolutions of at least 512×512 and are designed to cover diverse scenes.
  • Dataset: OpenVid-1M combines million-scale coverage with expressive, precise captions for open-scenario generation.Its captions highlight nouns, verbs, and easily overlooked details.
  • Method: MVDiT uses parallel visual-text processing to mine structural information from visual tokens and semantic information from text tokens.Its architecture combines multimodal feature integration, self-attention, temporal attention, and cross-attention.
  • Evaluation: The paper validates OpenVid-1M with STDiT and MVDiT on text-to-video generation and demonstrates its superiority on video restoration.The contribution claims cover both dataset validation and the proposed multimodal model.

2 RELATED WORK

Prior text-to-video work uses both specific-scenario and open-scenario datasets, alongside UNet- and DiT-based generation methods. Existing datasets range from small specialized collections to very large web-scale corpora with documented quality limitations.

  • Text-to-video Datasets: Specific-scenario datasets contain limited text-video pairs tailored to particular contexts, such as action recognition or single-person performances.Examples include UCF-101 with 13,320 videos and Taichi-HD with 2,668 videos.
  • Data Processing: The data-processing pipeline separates its first three steps for parallel execution and processes later steps sequentially.This organization is presented as improving processing efficiency.
  • Text-to-video Datasets: Open-scenario datasets include WebVid-10M, Panda-70M, and InternVid, spanning 10.7 million to 234 million video clips.The datasets differ substantially in scale and construction approach.
  • Text-to-video Datasets: WebVid-10M contains low-quality watermarked videos, while Panda-70M includes many static, flickering, or low-quality samples.These limitations motivate more precise dataset curation.
  • Text-to-video Models: Text-to-video generation methods are organized into UNet-based and DiT-based approaches.The related work identifies both families as active directions for video synthesis.

3 CURATING OpenVid-1M

OpenVid-1M is built by filtering source datasets for aesthetics, temporal consistency, motion, and clarity, extracting single-scene clips, and adding expressive captions. A 1080p subset, OpenVidHD-0.4M, is also curated for high-definition generation.

  • Sources: OpenVid-1M is assembled from ChronoMagic, CelebvHQ, Open-Sora-plan, and Panda2, with filtering details focused on downloaded Panda-50M.The source datasets are combined into the final collection.
  • Filtering: Aesthetic, temporal-consistency, and motion filters select visually pleasing clips with suitable dynamics.The pipeline removes low-aesthetic, nearly static, flickering, and excessively rapid-motion videos.
  • Filtering: Clarity filtering retains the top 30% of clips by DOVER-Technical score after intersecting the aesthetic, temporal, and motion subsets.The resulting clips are intended to have clean textures and high clarity.
  • Clip Extraction: Cascaded Cut Detector splits multi-scene videos so that each extracted clip contains one scene.This step produces the Ours-0.6M subset from Panda-50M.
  • Captioning and HD Curation: Large multimodal-model recaptioning creates expressive descriptions, and 1080p videos are selected to form OpenVidHD-0.4M.LLaVA-v1.6-34b is used for recaptioning, while the HD subset targets high-definition generation.

4 DATA PROCESSING AND STATISTICAL COMPARISON

The paper compares OpenVid-1M with SVD-inspired processing and prior datasets using quality, motion, temporal consistency, caption length, and scale statistics. OpenVid-1M is reported to provide cleaner, more descriptive, and more balanced training data for open-scenario text-to-video generation.

  • Pipeline Comparison: OpenVid-1M adds DOVER-based clarity assessment to aesthetic filtering and differs from SVD in its motion-evaluation pipeline.The paper contrasts its processing choices with SVD’s use of Farneback optical flow and RAFT.
  • Statistical Comparison: OpenVid-1M subsets have higher aesthetics scores and a greater proportion of moderate-motion videos than Panda-50M.The comparison associates these distributions with superior visual quality and smoother, more consistent motion.
  • Statistical Comparison: OpenVid-1M shows more balanced temporal-consistency distributions and significantly longer captions than Panda-50M.Panda-50M is described as containing static or excessively dynamic videos, whereas OpenVid-1M provides richer semantic information.
  • Dataset Comparison: 1,019,957 clips averaging 7.2 seconds comprise OpenVid-1M, totaling 2,051 hours of open-scenario video.The dataset includes diverse categories such as portraits, scenic views, cityscapes, and metamorphic content.

5 METHOD

MVDiT uses a parallel visual-text architecture that repeatedly integrates visual and textual features through self-attention, temporal attention, and cross-attention. This design targets both structural and semantic information for text-to-video generation.

  • MVDiT Architecture: MVDiT emphasizes a parallel visual-text structure to mine structure from visual tokens and semantic information from text tokens.It diverges from prior DiT architectures that focus primarily on modeling visual content.
  • MVDiT Architecture: Each MVDiT layer extracts visual and linguistic features, applies multi-modal temporal attention, enables token interaction, and forwards both branches through a feedforward layer.The layer combines temporal attention, multi-modal self-attention, and multi-head cross-attention before the final feedforward stage.
  • Multi-Modal Self-Attention: Visual tokens and text tokens are concatenated into a multi-modal feature for self-attention, then split back into their respective branches.Text tokens are repeated across the temporal dimension, while self-attention promotes interaction between modalities within each frame.
  • Multi-Modal Temporal-Attention: Multi-modal temporal attention communicates along the temporal dimension using enhanced visual and text features.The design is intended to capture structural temporal consistency from visual information and semantic temporal consistency from textual information.
  • Multi-Head Cross-Attention: Cross-attention uses visual tokens as queries and text tokens as keys and values to directly embed semantic information into visual tokens.After repeated updates, the final visual feature predicts noise and covariance at time t.

6 EXPERIMENTS

Experiments evaluate OpenVid-1M and MVDiT through standardized T2V comparisons, dataset comparisons, qualitative analysis, restoration, and ablations. The results report strong video quality, temporal consistency, restoration performance, and benefits from higher resolution, multimodal architecture, and higher-quality data.

  • 6.1 EXPERIMENTAL SETTINGS: 700 prompts and public metrics provide a consistent basis for comparing generated videos across models.The evaluation uses the same prompt set for each model and measures visual quality, alignment, and temporal consistency.
  • 6.2 COMPARISON WITH STATE-OF-THE-ART MODELS: 73.46% VQAA and 68.58% VQAT place the model first in aesthetics and second in clarity, while 99.87% Clip_temp_score ranks second for temporal consistency.The reported results indicate robust performance across metrics while using less training data.
  • 6.2 COMPARISON WITH STATE-OF-THE-ART MODELS: OpenVid-1M-trained models achieve the best scores across nearly all metrics at 256 × 256 and show similar superiority at 1024 × 1024.OpenVidHD-0.4M also outperforms alternatives combined with ×4 super-resolution for high-definition generation.
  • 6.2 COMPARISON WITH STATE-OF-THE-ART MODELS: Qualitative comparisons show clearer, more aesthetic, and more detailed videos, stronger prompt understanding, and more realistic motion than competing models.Examples include depicting an android surrounded by Easter eggs and correctly rendering a car kicking up dust.
  • 6.2 COMPARISON WITH STATE-OF-THE-ART MODELS: OpenVidHD-0.4M-trained restoration models outperform Upscale-A-Video across fidelity and perception metrics and produce clearer textures and more accurate structures.The restoration model uses 130K synthesized training samples and does not require task-specific design optimizations.
  • 6.3 ABLATION STUDY: Higher resolution, MVDiT over STDiT, and more high-quality training data each improve reported metrics; removing MMTA prevents video generation.Additional ablations find that MHCA improves quality and alignment, while α improves quality and convergence.
  • 6.3 ABLATION STUDY: Long captions are strongly preferred over Panda’s short captions for richer element descriptions and temporal events, although motion modeling remains improvable.The comparison covers 1,117 validation samples evaluated by 10 volunteers across omission, hallucination, distortion, and temporal mismatch.

7 CONCLUSION

The paper presents OpenVid-1M and its OpenVidHD-0.4M subset alongside MVDiT for high-quality text-to-video generation. It reports efficacy against prior datasets while acknowledging continuing difficulty with intricate physical dynamics and natural-scene motion.

  • 7 CONCLUSION: OpenVid-1M contains over 1 million high-resolution clips with expressive descriptions, while OpenVidHD-0.4M supports high-definition generation.An automated pipeline prioritizes aesthetics, temporal consistency, fluid motion, clarity, and single-scene clips.
  • 7 CONCLUSION: MVDiT uses the dataset to generate visually compelling videos by making full use of multimodal information.The conclusion links the model and dataset to the reported efficacy over WebVid-10M and Panda-50M.
  • 7 CONCLUSION: The model still struggles with intricate dynamics and natural-scene motions, sometimes producing unrealistic videos.The authors suggest that more high-quality training data could help address this limitation.

A MORE IMPLEMENTATION DETAILS

The implementation combines quality filtering, caption enrichment, multimodal spatial attention, and temporal attention. These components assess and curate videos while enabling MVDiT to integrate visual, textual, and temporal information.

  • A MORE IMPLEMENTATION DETAILS: Aesthetics and clarity are assessed with the LAION Aesthetic Predictor and DOVER for efficient million-scale processing.The two predictors are selected for fast inference and alignment with human preferences.
  • A MORE IMPLEMENTATION DETAILS: Temporal consistency is measured by averaging CLIP cosine similarity between every pair of adjacent frames.The score represents coherence and consistency across video frames.
  • A MORE IMPLEMENTATION DETAILS: Motion amplitude is estimated from adjacent-frame UniMatch optical flow, using the squared average of predicted values.Higher flow scores indicate stronger motion effects.
  • A MORE IMPLEMENTATION DETAILS: A cascade of three cut detectors with different thresholds captures sudden changes, fade-ins, and fade-outs.The cascade addresses transitions missed by a single fixed-threshold detector.
  • A MORE IMPLEMENTATION DETAILS: Filtering thresholds and ratios are derived from repeated evaluator assessments of randomly sampled data subsets.Figure 8 visualizes videos with varying clarity, aesthetic, motion, and temporal-consistency scores.
  • A MORE IMPLEMENTATION DETAILS: MMSA repeats text tokens across time and concatenates them with frame tokens for spatial and within-frame self-attention.This adapts MMDiT-style multimodal self-attention to video inputs.
  • A MORE IMPLEMENTATION DETAILS: MMTA adds temporal attention so the multimodal architecture can capture information along the temporal dimension and generate videos.The module preserves the dual-branch structure used in MMSA.
  • A MORE IMPLEMENTATION DETAILS: MHCA directly communicates semantic information from text tokens to visual tokens.The module addresses the performance impact of insufficient semantic information in video generation.

B ABLATIONS ON DATA PROCESSING STEPS

Ablations isolate the effects of OpenVid-1M’s processing stages and examine computational considerations. The combined filtering pipeline achieves the strongest metric results across most evaluated measures.

  • B ABLATIONS ON DATA PROCESSING STEPS: Temporal screening improves Clip_temp_score and warping_error, indicating better temporal consistency.The result isolates the contribution of temporal filtering within the data-processing pipeline.
  • B ABLATIONS ON DATA PROCESSING STEPS: Aesthetic, temporal, and motion screening improves VQAA, VQAT, and Blip_bleu, suggesting gains in video quality and text understanding.These filtering stages jointly affect both visual and text-video evaluation measures.
  • B ABLATIONS ON DATA PROCESSING STEPS: Clarity screening significantly improves VQAA and VQAT, while combining all four processing steps yields the highest scores in most metrics.The data-processing ablation keeps training data, iterations, and resolution fixed across settings.
  • B ABLATIONS ON DATA PROCESSING STEPS: Attention computation has O(L^2) complexity in token length L, motivating caching because attention values remain temporally consistent across consecutive denoising steps.The observed redundancy is presented as a basis for reusing cached attention values.

D EXAMPLES OF OpenVid-1M DATASET

OpenVid-1M samples illustrate the dataset’s high visual quality and descriptive richness, while model showcases emphasize aesthetics, text alignment, and dynamic motion generation.

  • Dataset examples: OpenVid-1M samples at 512 × 512 and 1920 × 1080 demonstrate quality and descriptive richness in aesthetics, motion, temporal consistency, caption length, and clarity.These examples are attributed to the dataset’s data processing pipeline.
  • Model analysis: Figure 9 compares similarity across attention values and timesteps, alongside generation quality for accelerated and original models at 1024 resolution.
  • Dataset examples: Figure 10 presents examples of the OpenVid-1M dataset.
  • Model showcases: Model showcases highlight aesthetically pleasing painting-style content, accurate text alignment, and intricate dynamics such as motorcycle races and galloping.The examples include accurately depicting the phrase “crashed down.”

F VIDEO DURATIONS COMPARISON WITH OTHER DATASETS

OpenVid-1M is compared with other million-level text-to-video datasets in terms of video duration and visual generation quality. It contains 1,019,957 clips averaging 7.2 seconds, with 2,051 total hours, while the cited comparisons emphasize its high-quality, clean videos and stronger visual attributes.

  • Video duration comparison: 1,019,957 clips in OpenVid-1M average 7.2 seconds each and total 2,051 hours of video.
  • Dataset comparison: Compared with WebVid-10M and Panda-70M, OpenVid-1M is described as containing high-quality, clean videos.WebVid-10M is described as having low-quality videos with watermarks, while Panda-70M includes still, flickering, or blurry videos and short captions.
  • Video duration comparison: Figure 12 compares video durations between OpenVid-1M and other million-level text-to-video datasets.
  • Visual quality: Visual comparisons report clearer, more aesthetically pleasing, and more detailed videos from the authors’ model, attributed to high-resolution OpenVid-1M.
  • Prompt understanding: The authors’ model accurately depicts “android” and “surrounded by colorful Easter eggs,” demonstrating strong prompt understanding.
  • Motion quality: The authors’ model better captures “kicking up dust,” highlighting its reported superior motion quality.
Loading 2407.02371v3…