Source-linked AI summary
MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, Ying Shan
TL;DR
Existing public video datasets are poorly suited to Sora-like generation because they largely contain short, weak-motion videos with brief or inaccurate captions. The paper introduces MiraData with long, semantically consistent clips and structured GPT-4V captions, alongside MiraBench for broader evaluation, and reports stronger motion performance for models trained on MiraData. The authors also identify dataset biases, annotation errors, limited coverage, and metric inaccuracies in uncommon scenarios.
Problem
Existing public datasets contain short, low-motion videos and brief or inaccurate captions, limiting their suitability for Sora-like long-video generation.
Method
MiraData is curated from manually selected sources through segmentation, filtering, and captioning, while MiraBench evaluates generation with 17 metrics and 150 prompts.
Results
Experiments show that MiraDiT trained on MiraData improves motion strength while maintaining temporal and 3D consistency, with better visual quality and text-video alignment.
Takeaways & Limitations
MiraData complements existing datasets with long-duration, high-motion videos and detailed captions for evaluating and training long-video generation models.
Takeaways & Limitations
MiraData has inherent biases, potential annotation errors, insufficient coverage, and metrics that may be inaccurate for uncommon scenarios such as jitter or overexposure.
Abstract
from arXiv · showhide
Sora's high-motion intensity and long consistent videos have significantly impacted the field of video generation, attracting unprecedented attention. However, existing publicly available datasets are inadequate for generating Sora-like videos, as they mainly contain short videos with low motion intensity and brief captions. To address these issues, we propose MiraData, a high-quality video dataset that surpasses previous ones in video duration, caption detail, motion strength, and visual quality. We curate MiraData from diverse, manually selected sources and meticulously process the data to obtain semantically consistent clips. GPT-4V is employed to annotate structured captions, providing detailed descriptions from four different perspectives along with a summarized dense caption. To better assess temporal consistency and motion intensity in video generation, we introduce MiraBench, which enhances existing benchmarks by adding 3D consistency and tracking-based motion strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics covering temporal consistency, motion strength, 3D consistency, visual quality, text-video alignment, and distribution similarity. To demonstrate the utility and effectiveness of MiraData, we conduct experiments using our DiT-based video generation model, MiraDiT. The experimental results on MiraBench demonstrate the superiority of MiraData, especially in motion strength.
1 Introduction
MiraData addresses the inadequacy of existing video datasets for Sora-like generation by providing longer, higher-motion videos with detailed captions and a benchmark for broader evaluation.
- Motivation: Existing public datasets mainly contain short, low-quality or low-motion clips with brief captions, limiting their suitability for long-video generation.Typical clips last 5–18 seconds, while captions often contain only 12–30 words.
- MiraData: MiraData provides long videos averaging 72.1 seconds, high motion intensity, and structured captions averaging 318 words.The dataset is designed to improve duration, motion strength, and caption detail together.
- MiraData: Its curation pipeline collects diverse sources, segments semantically consistent clips, filters for visual quality and motion intensity, and enriches captions with GPT-4V.The pipeline includes downloading, segmentation, filtering, and annotation stages.
- MiraBench: MiraBench adds 3D-consistency and tracking-based motion metrics to existing evaluations, using 17 metrics and 150 prompts across six assessment dimensions.The dimensions include temporal consistency, motion strength, 3D consistency, visual quality, text-video alignment, and distribution similarity.
2 Related Work
Related work reveals a gap between existing video-generation resources and Sora-like goals: datasets and models generally provide short, weak-motion videos with limited captions, motivating MiraData.
- Video-Text Datasets: Open-source text-video datasets are constrained by short durations, limited caption lengths, and poor visual quality.These constraints reduce their suitability for general video generation.
- Video-Text Datasets: WebVid-10M is a commonly used open-source dataset, but its prominent video watermark is an additional data-quality concern.Many high-performing video-generation models instead rely heavily on internal datasets, restricting transparency and usability.
- Video Generation: Diffusion-based video generation improves visual quality and diversity but incurs substantial computational cost, creating a quality–duration trade-off.Practical resource constraints have encouraged short-duration generation.
- Video Generation: Previous methods primarily generate short clips, while coarse-to-fine approaches struggle with temporal continuity and strong motion magnitude.Transformer-based methods offer improved scalability and a better trade-off between computational complexity and performance.
- Video Generation: Existing methods commonly produce clips such as 2 seconds or 16 frames with weak motion strength, whereas Sora demonstrates longer videos with enhanced motion and 3D consistency.The paper identifies short duration, weak motion, and short or inaccurate captions as dataset limitations.
- MiraData: MiraData is introduced as a large-scale dataset with long-duration videos and structured detailed captions, built through collection, splitting and stitching, selection, and captioning.Its five-step construction process is presented as a resource for extended video sequences with enhanced motion and coherence.
3 MiraData Dataset
MiraData is a large-scale video dataset built from diverse sources, processed into semantically coherent clips, filtered for generation quality, and annotated with detailed structured captions.
- MiraData uses a five-step pipeline covering video collection, splitting and stitching, selection, and captioning.
- 3.1 Data Collection: The dataset combines manually selected YouTube channels with stock-video sources to provide diverse content, visual quality, long duration, and strong motion.Sources include YouTube, Videvo, Pixabay, and Pexels.
- 3.1 Data Collection: 156 manually selected YouTube channels yielded around 68K videos, which became around 34K videos and 173K clips after splitting and stitching.
- 3.2 Video Splitting and Stitching: The splitting and stitching process targets semantically coherent clips by detecting shots and recombining short segments across content-coherent transitions.
- 3.3 Data Selection: MiraData provides five quality-filtered versions based on color, aesthetic quality, motion strength, and NSFW content.
- 3.4 Video Captioning: Structured captions describe the main object, background, camera movements, and video style; dense and structured captions average 90 and 214 words, respectively.
4 MiraBench
MiraBench evaluates video generation with 150 prompts and 17 metrics spanning temporal behavior, motion, 3D consistency, visual quality, alignment, and distribution similarity.
- MiraBench forms 150 balanced prompts across human, animal, object, and landscape categories using precise video-text pairs.
- The benchmark defines 17 metrics across six perspectives: temporal consistency, motion strength, 3D consistency, visual quality, text-video alignment, and distribution consistency.
- Temporal Motion Strength: Tracking strength measures long-range point displacement with CoTracker, complementing optical-flow dynamic degree for motion evaluation.
- Temporal Motion Strength: In the illustrated comparison, tracking strength assigns 4.1 and 11.8 to the left and right motions, whereas dynamic degree assigns 1.2 and 0.7.
- Temporal Consistency: Temporal consistency uses adjacent-frame DINO and CLIP feature similarity plus temporal motion smoothness.
- The remaining metrics assess 3D reconstruction errors, aesthetic and imaging quality, five text-video alignment aspects, and FVD, FID, and KID distribution similarity.
5 Experiments
MiraDiT is a DiT-based long-video generator designed to use dense captions and support variable video conditions. Models trained on MiraData improve motion strength while preserving consistency and improving alignment-related quality.
- 5.1 Model Design of MiraDiT: MiraDiT uses a hybrid VAE and Flan-T5-XXL text encoder supporting up to 512 tokens for dense, structured captions.
- 5.1 Model Design of MiraDiT: Its spatial-temporal transformer separates spatial and temporal self-attention to reduce the computational load of long-video generation.
- 5.1 Model Design of MiraDiT: FPS-conditioned modulation enables motion-strength control during inference, while Frame n’ Pack supports training across temporal lengths and resolutions.
- 5.2 Comparison with Previous Video Generation Datasets: The experiments compare MiraDiT trained on WebVid-10M or MiraData at 384 × 240 resolution and 5s length using 14 MiraBench metrics.
- 5.2 Comparison with Previous Video Generation Datasets: MiraData training significantly improves motion strength while maintaining temporal and 3D consistency relative to WebVid-10M training.
- 5.3 Role of Caption Length and Granularity: Longer, more detailed captions increase dynamics, temporal consistency, generation control, and text-video alignment, but do not necessarily improve visual quality.
6 Conclusion and Discussion
MiraData complements existing video datasets with long-duration, detailed-caption, high-motion videos and shows advantages in evaluation with MiraDiT and MiraBench. The dataset and benchmark still have limitations, including bias, annotation errors, insufficient coverage, metric inaccuracies in uncommon scenarios, and potential societal harms.
- Conclusion: MiraData complements existing datasets with long-duration videos, detailed captions, and strong motion intensity, supporting high-motion, temporally consistent long-video generation.The dataset is curated from diverse sources and annotated with multiple high-performance models.
- Conclusion: MiraData shows advantages under the MiraBench evaluation framework when used with the MiraDiT model.
- Limitations: MiraData remains limited by inherent biases, potential annotation errors, insufficient coverage, and inaccurate MiraBench metrics in uncommon scenarios such as jitter or overexposure.
- Potential Negative Societal Impacts: Enhanced video generation capabilities may contribute to deepfakes, misinformation, privacy breaches, and harmful content generation.The authors propose ethical guidelines, privacy protections, and unbiased dataset curation as safeguards.