Source-linked AI summary
UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions
Zhucun Xue, Jiangning Zhang, Teng Hu, Haoyang He, Yinan Chen, Yuxuan Cai, Yabiao Wang, Chengjie Wang, Yong Liu, Xiangtai Li, Dacheng Tao
TL;DR
Existing public T2V datasets emphasize medium or low resolutions and simple captions, limiting support for UHD video generation. The paper introduces UltraVideo with a four-stage curation pipeline and extends Wan to UltraWan, which supports native high-resolution video generation and receives favorable VBench and human evaluations.
Problem
Existing T2V datasets mainly contain medium- and low-resolution videos with simple captions, falling short of UHD generation requirements for sharp details, rich textures, and precise semantic control.
Method
The paper curates UltraVideo through high-quality clip collection, statistical filtering, model-based purification, and comprehensive structured-caption generation, then extends Wan for native UHD generation.
Results
UltraVideo contains 42K short videos and 17K long videos, while UltraWan supports native UHD video generation and demonstrates superiority in VBench and human evaluations.
Takeaways & Limitations
UltraVideo provides a high-quality, open-source UHD-4K/8K dataset with structured captions for fine-tuning high-resolution video generation models.
Takeaways & Limitations
The authors note that proliferating fake videos and malicious uses of AI-generated content threaten information authenticity and social stability, motivating regulatory and technical responses.
Abstract
from arXiv · showhide
The quality of the video dataset (image quality, resolution, and fine-grained caption) greatly influences the performance of the video generation model. The growing demand for video applications sets higher requirements for high-quality video generation models. For example, the generation of movie-level Ultra-High Definition (UHD) videos and the creation of 4K short video content. However, the existing public datasets cannot support related research and applications. In this paper, we first propose a high-quality open-sourced UHD-4K (22.4\% of which are 8K) text-to-video dataset named UltraVideo, which contains a wide range of topics (more than 100 kinds), and each video has 9 structured captions with one summarized caption (average of 824 words). Specifically, we carefully design a highly automated curation process with four stages to obtain the final high-quality dataset: \textit{i)} collection of diverse and high-quality video clips. \textit{ii)} statistical data filtering. \textit{iii)} model-based data purification. \textit{iv)} generation of comprehensive, structured captions. In addition, we expand Wan to UltraWan-1K/-4K, which can natively generate high-quality 1K/4K videos with more consistent text controllability, demonstrating the effectiveness of our data curation.We believe that this work can make a significant contribution to future research on UHD video generation. UltraVideo dataset and UltraWan models are available at https://xzc-zju.github.io/projects/UltraVideo.
1 Introduction
UltraVideo addresses the lack of high-resolution, semantically detailed video data by combining UHD-focused collection with automated curation and structured captions. The paper also extends Wan for native UHD generation, evaluated with VBench and human assessments.
- Motivation: Existing T2V datasets emphasize medium or low resolution and simple captions, limiting support for UHD videos with detailed visual and semantic control.High-resolution generation also faces scalability problems, including artifacts, blurriness, and inconsistent content when models trained on low-resolution data are extrapolated.
- UltraVideo: UltraVideo provides an open-source UHD-4K/8K dataset with diverse videos and structured captions averaging 824 words per video.The dataset contains 42K short videos and 17K long videos, with 10 types of semantic tags.
- Data curation: Its four-stage curation pipeline covers diverse clip collection, statistical filtering, model-based purification, and comprehensive structured caption generation.Collection prioritizes videos at least 4K resolution, while filtering removes visual defects and model-based purification detects low-quality attributes and assesses aesthetic and motion consistency.
- Contribution: The work focuses on fine-tuning high-resolution video generation models with high-quality data and fine-grained structured captions.The dataset is designed to bridge the gap between academic resources and large corporate data for high-resolution generation applications.
- UltraWan: UltraWan extends Wan-T2V-1.3B with a high-resolution generation architecture and caption sampling strategy, supporting native UHD video generation.The authors report superiority through VBench evaluations and human assessments.
2 Curating UltraVideo Dataset
UltraVideo prioritizes high-quality UHD video data over scale through controlled sourcing, statistical and model-based filtering, and comprehensive captioning. The resulting dataset contains diverse resolutions, topics, durations, and structured descriptions while substantially reducing detected video-quality defects.
- Dataset focus: UltraVideo prioritizes high-quality UHD video data over the quantity emphasized by many existing datasets.The dataset targets high-resolution frames, native resolution and FPS, and comprehensive captions for model fine-tuning.
- Curation pipeline: The curation pipeline combines controlled source collection, scene splitting, duration and frame filtering, and three successive quality-screening stages.The process applies statistical checks for text, borders, exposure, and grayscale, followed by model-based aesthetic, motion, and caption-consistency filtering.
- Dataset diversity: UltraVideo covers seven major themes and 108 topics, supporting diverse distributions across scenes, subjects, actions, events, camera motion, genres, and emotions.The topic statistics were derived from captions with LLM assistance and manual confirmation.
- Curation pipeline: 42K short videos and 17K long videos remain after collection, statistical filtering, and model-based purification.Short videos span 3–10 seconds, while long videos are at least 10 seconds.
- Captions and analysis: Structured captions vary in length and type, with detailed descriptions intended to support fine-grained controllable video consistency.The dataset includes brief, detailed, summarized, and aggregated caption distributions.
- Quality comparison: Human evaluation found a 2.3% UltraVideo failure rate versus 41.5% for Koala-36M under the stated non-compliance criteria.Five people evaluated 1,000 randomly sampled videos from each dataset while ignoring resolution and image-quality differences.
3 UltraWan: Stand on the Shoulders of Giants
UltraWan adapts Wan-T2V-1.3B for native high-resolution generation through parameter-efficient fine-tuning and caption sampling. Its configuration balances resolution, frame count, and memory constraints while evaluations compare generation quality across model variants.
- Motivation: Directly extrapolating Wan-T2V-1.3B to 1K and 4K causes significant performance deterioration or ineffectiveness.UltraWan therefore introduces high-resolution-adaptable model parameters rather than relying on native extrapolation alone.
- Model configuration: UltraWan-1K preserves 81 output frames, whereas UltraWan-4K reduces output to 29 frames so one sample fits on a single GPU.The experiments use the smaller Wan-T2V-1.3B, batch size 1 per GPU, and no tensor parallelism.
- Caption training: Random caption sampling selects Brief, Detailed, or Summarized Description with probability 1/3, optionally supplemented by one of seven remaining categories.The strategy is designed to use comprehensive structured captions for fine-grained prompt control.
- Training data: Sub-clip sampling uniformly selects frames from the middle toward both sides while retaining the video’s native FPS.The procedure aligns sampled sub-clips with their captions without interval-based frame skipping.
- Parameter-efficient tuning: LoRA fine-tuning targets QKV and output-linear self-attention modules plus the first and third feedforward linear layers.The LoRA ranks are 64 for UltraWan-1K and 16 for UltraWan-4K.
4 Experiments
Experiments assess UltraWan under native high-resolution generation, using VBench comparisons, human preferences, and qualitative semantic-consistency evidence. The study is constrained by the computational cost of high-resolution experiments and limited metric reliability.
- Native 1K/4K experiments are conducted on the small-scale Wan-T2V-1.3B because high resolution substantially increases computational power and video-memory requirements.
- Scaling the official model directly to 1K causes a significant performance decline, while UltraWan-1K training improves 1K generation.
- LoRA-based UltraWan-1K is recommended over full-parameter training because it achieves stronger overall results under lower training costs.
- UltraWan-4K performs better on image-quality and temporal-stability indicators, although its lower frame rate worsens some other indicators.
- Human preference evaluation compares video aesthetics, temporal stability, and text consistency across outputs from the high-resolution generation models.
- Structured captions are associated with stronger semantic consistency in UltraWan generations.
5 Conclusion
The conclusion presents UltraVideo as an open UHD-4K/8K dataset with structured captions and a four-stage curation process, alongside UltraWan models for native high-resolution generation. It also notes both future applications and risks from maliciously generated fake videos.
- UltraVideo provides diverse UHD-4K/8K text-to-video data with comprehensive structured captions for each video.
- A four-stage automated curation process addresses resolution scalability and semantic granularity in the dataset.
- UltraWan-1K/-4K extends Wan to native high-resolution video generation with improved text controllability.
- Preserved native resolution, frame rate, and audio allow UltraVideo to be adapted to ultra-resolution tasks such as super-resolution, frame interpolation, codecs, editing, and music generation.
- The paper identifies fake-video proliferation and malicious use of generated content as risks to information authenticity and social stability.
Appendix
The appendix supplements the main paper with related work, caption statistics, qualitative VBench and UltraVideo results, and additional UltraWan findings.
- The supplementary material includes a related-work section.
- It reports additional statistical distributions for the ten structured caption types.
- It presents more qualitative results using VBench prompts.
- It provides additional qualitative results with UltraVideo prompts and experimental findings for UltraWan.
A.1 Text-to-Video Datasets
Text-to-video datasets have evolved from manually annotated, task-specific collections toward large-scale web video datasets and multimodal resources. Existing datasets still face limitations in quality, scale, or suitability for modern video-generation training.
- Early datasets such as UCF101, MSVD, and MSR-VTT target specific tasks, while manual annotation limits their scale and older videos often have poor quality.
- WebVid-10M expands dataset scale with 10.7 million general videos and alt-text, but its videos contain low-quality watermarks.
- Later efforts use automatic ASR annotation and construct video-centric multimodal datasets, including InternVid and Panda-70M.
- Video-generation research progressed from GANs to diffusion models, improving generated-video quality, diversity, and controllability.
A.3 Video Data Curation
The paper describes data curation as important for video-dataset quality and contrasts prevailing image-based tagging and manual-rule processes with the need to account for temporal characteristics.
- Prevailing curation relies on image models for tagging and manual rule-based filtering.Examples include CLIP for image-text consistency and LAION-Aesthetics for image aesthetics.
- Video metrics are often formed by averaging image-based measures over time.This approach does not account for temporal characteristics of videos.
- Figure A1 presents statistical distributions for the comprehensive structured captions.
C More Qualitative Results
This section presents additional qualitative results associated with the VBench prompts.
- Figure A2 shows additional qualitative results.
- The results are presented as visual examples rather than a quantitative evaluation.
- The section directs readers to inspect the qualitative outputs shown in Figure A2.
D Experimental Findings of UltraWan
The experiments compare Wan-T2V-1.3B with UltraWan-1K and UltraWan-4K across qualitative outputs and structured-caption distributions. UltraWan-1K shows competitive performance and stronger semantic alignment, while UltraWan-4K remains more artifact-prone under the reported training constraints.
- Comparative findings: UltraWan-1K and UltraWan-4K show superior semantic alignment to Wan-T2V-1.3B while generating native 1K video.The comparison uses identical prompts across the three models.
- Comparative findings: UltraWan-4K is more susceptible to subject artifacts than UltraWan-1K.The paper attributes this likely to the resolution gap complicating LoRA adaptation and possible undertraining limited to one epoch.
- Comparative findings: UltraWan-4K shows a preference for landscape and architectural scenes.
- Caption analysis: Figure A1 reports statistical distributions of the comprehensive structured captions.
- Qualitative results: Figures A2 and A3 show qualitative results with VBench and UltraVideo prompts, respectively.
- Qualitative results: Figures A4 and A5 provide additional UltraVideo-prompt results for visual inspection.