PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation

Wenhao Wang, Yi Yang

arXiv:2503.01739v2cs.CV

TL;DR

Existing video datasets are generally not curated around the topics text-to-video users want to create. VideoUFO builds a million-scale collection around real users’ focus, and experiments find that a VideoUFO-trained model performs especially well on models’ worst-performing topics. The authors also note that their current effectiveness validation uses a computationally demanding academic baseline whose generated-video quality remains limited.

  • Problem

    Recent video datasets are typically open-domain rather than centered on topics text-to-video users focus on, leaving user-topic coverage unaddressed.

    Source: §1 · p. 3

  • Method

    VideoUFO curates over 1.09 million clips across 1,291 user-focused topics and compares datasets and model performance on those topics.

    Source: §1 · p. 3, §7 · p. 9

  • Results

    Current 16 text-to-video models have poorly performing topics, while a VideoUFO-trained model achieves the highest similarities on worst-performing topics and maintains performance on best-performing ones.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that VideoUFO helps improve text-to-video model performance on user-focused topics.

    Source: §7 · p. 9

    Effectiveness validation currently uses an academic baseline requiring 32 A100 GPUs, and its generated-video quality remains limited.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Text-to-video generative models convert textual prompts into dynamic visual content, offering wide-ranging applications in film production, gaming, and education. However, their real-world performance often falls short of user expectations. One key reason is that these models have not been trained on videos related to some topics users want to create. In this paper, we propose VideoUFO, the first Video dataset specifically curated to align with Users' FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1) minimal (0.29%) overlap with existing video datasets, and (2) videos searched exclusively via YouTube's official API under the Creative Commons license. These two attributes provide future researchers with greater freedom to broaden their training sources. The VideoUFO comprises over 1.09 million video clips, each paired with both a brief and a detailed caption (description). Specifically, through clustering, we first identify 1,291 user-focused topics from the million-scale real text-to-video prompt dataset, VidProM. Then, we use these topics to retrieve videos from YouTube, split the retrieved videos into clips, and generate both brief and detailed captions for each clip. After verifying the clips with specified topics, we are left with about 1.09 million video clips. Our experiments reveal that (1) current 16 text-to-video models do not achieve consistent performance across all user-focused topics; and (2) a simple model trained on VideoUFO outperforms others on worst-performing topics. The dataset and code are publicly available at https://huggingface.co/datasets/WenhaoWang/VideoUFO and https://github.com/WangWenhao0716/BenchUFO under the CC BY 4.0 License.

1 Introduction

VideoUFO is a video dataset curated around real text-to-video users’ interests, addressing the mismatch between open-domain video collections and user-focused topics. It contains over 1.09 million clips across 1,291 topics, and the paper reports that it benefits model performance on difficult topics.

1 Introduction

  • Unlike open-domain datasets, VideoUFO concentrates on topics text-to-video users focus on, aiming to better cater to their needs without unnecessary dataset expansion.The paper reports only 0.29% overlap with existing datasets and says most recent datasets primarily reprocess HD-VILA-100M [11].Source: §1 · p. 3
  • VideoUFO retrieves videos through YouTube’s official API and selects Creative Commons-licensed videos, a data-compliance feature the paper says gives researchers greater flexibility.The paper notes that recent datasets do not explicitly address regulatory compliance in their collection processes.Source: §1 · p. 3
  • Current 16 text-to-video models have poorly performing topics, while a VideoUFO-trained model achieves the highest similarities on worst-performing topics and maintains performance on the best-performing ones.The benchmark selects 10 prompts per topic, generates videos, describes them with a multimodal language model, and compares descriptions with original prompts.Source: §1 · p. 3
  • VideoUFO is the first dataset curated around real text-to-video users’ focus, with over 1.09 million clips spanning 1,291 user-focused topics.Its topic-based approach analyzes prompts and retrieves related videos; the paper says the dataset can help align models with actual user needs.Source: §1 · pp. 2–3
  • The paper compares VideoUFO with recent datasets on fundamental attributes and topic coverage, and says its curation follows dataset-quality best practices.The introduction illustrates the topic-exposure motivation with a generated firefly that differs noticeably from its real-life counterpart.Source: Figure 2 · p. 2, §1 · p. 3

2 Related Works

Related work covers text-to-video models, video datasets, and preference alignment in diffusion models. The paper distinguishes VideoUFO by focusing on the topics real users want to create.

2 Related Works

  • Text-to-video models include commercial and open-source systems, and the paper notes that training data determines the upper limit of generated-video quality.It presents VideoUFO as a dataset with potential to help models better cater to real-world user preferences.Source: §2 · p. 3
  • Existing text-video datasets are not designed around text-to-video users’ preferences, leaving a gap between dataset topics and real-world applications.The paper contrasts those collections with VideoUFO’s user-focus perspective.Source: §2 · p. 3
  • A VideoUFO data point contains a clip, ID, topic, source-video timestamps, brief and detailed captions, and six VBench [28] quality scores.The figure caption describes the dataset record’s fields and quality-score additions.Source: Figure 3 · p. 4
  • Prior preference-alignment work applies preference optimization to diffusion models, including methods targeting image quality or video visual quality and semantic alignment.VideoUFO instead targets performance on real-world users’ focused topics.Source: §2 · p. 4

3 Curating VideoUFO

VideoUFO curation turns YouTube videos found for user-focused topics into verified clips with brief and detailed captions. The resulting collection also records clip quality and can be extended to new topics or modalities.

3 Curating VideoUFO

  • A dataset item includes a clip ID, topic, original-video time bounds, brief and detailed captions, and clip quality.Figure 4 visualizes the semantic distribution of user-focused topics using WIZMAP.Source: §3 · p. 4, Figure 4 · p. 4

Collecting videos

  • The collection starts from 586,490 YouTube videos searched through the official API under Creative Commons, with only 1,675 IDs (0.29%) found in the compared datasets.Search results were constrained to videos under four minutes and at least 720p.Source: §3 · pp. 4–5

Splitting videos and generating captions

  • Videos are segmented into semantics-consistent clips, each paired with brief and detailed captions generated using Panda-70M and Open-Sora-Plan pipelines.The captioning pipeline uses QWen2-VL-7B for detailed video annotations.Source: §3 · p. 5

Verifying clips

  • After topic verification, 1,091,712 clips remain from 3,181,873 segmented clips, with detailed captions used for verification to reduce cost.The verification substitutes detailed captions and GPT-4o mini for checking more than 3 million clips directly with GPT-4o.Source: §3 · p. 5

Video quality assessment

  • Each clip is scored on six VBench dimensions: subject and background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality.The scores provide automated quality assessments aligned with human perception.Source: §3 · p. 5

Extension

  • VideoUFO can be extended by searching more platforms, analyzing emerging prompt datasets, or adapting the topic pipeline to image-to-video prompts.The paper names scaling up, new focusing, and image-to-video as extension directions.Source: §3 · p. 5

4 Comparison with Other Video Datasets

The comparison finds that VideoUFO is sourced around real users’ interests and has a more flexible license than several recent datasets. Topic coverage remains incomplete across existing datasets, and the coverage analysis depends on their supplied captions.

4.1 Fundamental Attributes

  • VideoUFO is collected around real text-to-video users’ focus, while several recent datasets reuse HD-VILA-100M data and its commercial-use-restricting R-UDA license.VideoUFO instead searches YouTube for Creative Commons videos and is described as comparable to other datasets in several other attributes.Source: §4.1 · p. 6

4.2 Topics Coverage

  • Topic coverage is measured by semantic matching: a user-focused topic counts as covered when an existing-dataset topic has cosine similarity greater than 0.6.The threshold is empirical; embeddings are produced with SentenceTransformers.Source: §4.2 · p. 6
  • InternVid covers 821 user-focused topics (63.6%), while combining all compared datasets raises coverage to 79.5% but increases training computational burden.No existing dataset covers all user-focused topics.Source: §4.2 · pp. 6–7, Figure 5 · p. 6
  • Coverage counts need not be one-to-one: OpenVid-1M contains 671 topics yet covers 703 user-focused topics because similar VideoUFO topics can map to one topic.The paper says this mapping creates relative duplicate topics without compromising the dataset’s reflection of users’ interests.Source: §4.2 · p. 7
  • Topic analysis uses the existing datasets’ captions because downloading and recaptioning all videos is prohibitively expensive, and caption quality may affect extracted-topic counts.Some supplied captions may be uninformative.Source: §4.2 · p. 7

5 VideoUFO Benefits Video Generation

BenchUFO evaluates text-to-video models on user-focused topics using prompt-to-description similarity. The results show uneven topic performance, with VideoUFO training improving scores on the weakest topics while maintaining top-topic performance.

5.1 Benchmark

  • BenchUFO selects 791 concrete nouns, samples 10 VidProM prompts per topic, generates videos, describes them with QWen2-VL-7B, and averages prompt-description cosine similarity.It reports scores for the 10 or 50 worst- and best-performing topics.Source: §5.1 · p. 7
  • Concrete nouns reduce a potential mismatch between abstract-topic prompts and literal video descriptions that could make generation appear ineffective.The authors illustrate this with “freedom,” which may yield descriptions of a dove, star, or heart rather than the abstract concept.Source: §5.1 · p. 7
  • The authors justify QWen2-VL-7B as a video describer and argue that VBench covers fewer topics and omits user-focused examples such as forest, sunset, and beach.They cite VBench's 79 topics and assume the selected video-understanding model can comprehend user-focused topics.Source: §5.1 · p. 8

5.2 Observations

  • 0.233 to 0.314 separates top-10 from low-10 scores across current models, indicating inconsistent performance across user-focused topics.The authors suggest insufficient training on videos about topics such as giant squid, animal cell, Van Gogh, and ancient Egyptian may contribute to this pattern.Source: §5.2 · p. 8
  • +4.2% on low-10 scores: MVDiT trained on VideoUFO achieves the highest score while maintaining top-10 performance.The authors report that this training reduces the gap between worst- and best-performing topics.Source: §5.2 · p. 8
  • OpenVid training yields a low-10 score of 0.401 versus 0.400 for prior state of the art, while VidGen training performs even lower.These dataset comparisons are used to distinguish VideoUFO's contribution from the MVDiT architecture.Source: §5.2 · p. 8
  • The validation uses the academic video-generation method requiring 32 A100 GPUs, whose generated-video quality remains limited.The authors identify computational resources as the reason for following this baseline and suggest future exploration in large-scale foundational-model training.Source: §5.2 · p. 9

6 Ablation Studies on Curation of VideoUFO

The ablation studies examine two curation choices: the number of clusters used to identify user-focused topics and the method used to verify clips.

6 Ablation Studies on Curation of VideoUFO

  • The curation ablations investigate how cluster count affects topic analysis and how clip-verification methods affect dataset construction.Further details are referenced in Appendix Section C.Source: §6 · p. 9

7 Conclusion

VideoUFO is a million-scale text-to-video dataset focused on user needs, with minimal overlap with existing datasets and a CC BY license. The paper describes its curation, compares its attributes and topic coverage with other datasets, and introduces a benchmark on which VideoUFO helps improve generation performance.

7 Conclusion

  • VideoUFO is a newly collected million-scale dataset for text-to-video generation focused on user needs.The paper reports minimal overlap with existing datasets and release under the more permissive CC BY license.Source: §7 · p. 9
  • Its curation analyzes user-focused topics, searches for related videos, segments them into clips, and applies post-processing.Source: §7 · p. 9
  • The paper compares VideoUFO with existing datasets in fundamental attributes and topic coverage.Source: §7 · p. 9
  • The authors build a benchmark for evaluating text-to-video models on user-focused topics and report that VideoUFO helps improve performance in this regard.Source: §7 · p. 9

A The Prompt for Concluding Topics

The appendix prompts specify how topics are condensed and how videos are described. Topic outputs are restricted to short labels, while video descriptions are instructed to capture content and changes in chronological order.

A The Prompt for Concluding Topics

B The Prompt for Describing Videos

C Ablation Studies on Curation of VideoUFO

The ablations compare topic-clustering choices and clip-verification methods, while accompanying figures describe caption, clip-duration, and video-quality-score distributions.

The number of clusters

  • The authors chose 2,000 preset clusters after topic counts plateaued and began fluctuating around that level; DBSCAN and HDBSCAN did not yield reasonable results.The tested preset-cluster range was 100 to 3,000.Source: C Ablation Studies on Curation of… · p. 13

Ablation Studies on Curation of VideoUFO

  • Figure 10 summarizes distributions of brief and detailed caption lengths and clip durations; Figure 11 summarizes scores from six video-quality metrics.The score distributions can support filtering videos for specific training needs.Source: Figure 10 · p. 14, Figure 11 · p. 14

The method for clip verification

  • About 5,400× lower API cost with detailed captions retained about 80% prediction overlap against direct GPT-4o verification.Direct GPT-4o processing of 1,000 clips cost $162.23; the study selected detailed captions with GPT-4o-mini for verification.Source: C Ablation Studies on Curation of… · p. 14
  • GPT-4o-mini performed slightly better than GPT-4o on clip verification, while brief captions reduced prediction overlap by 14.9%.The paper attributes the lower overlap to brief captions not capturing all clip information.Source: C Ablation Studies on Curation of… · p. 14

D The Distributions of Caption Lengths and Clip Durations

This section points readers to Figure 10 for the distributions of caption lengths and clip durations.

D The Distributions of Caption Lengths and Clip Durations

E The Distributions of Video Quality Score

This section describes the video-quality dimensions whose score distributions are shown in Figure 11.

E The Distributions of Video Quality Score

NeurIPS Paper Checklist

The checklist affirms that the paper’s claims reflect its contributions and scope, and reports public data and code alongside its experimental and ethics disclosures.

NeurIPS Paper Checklist

  • The checklist marks the paper’s abstract and introduction claims as accurately reflecting its contributions and scope.Source: NeurIPS Paper Checklist · p. 15
  • The authors report that the paper provides the information needed to reproduce its main experiments and makes the dataset and code publicly available.Source: NeurIPS Paper Checklist · pp. 16–17
  • The checklist reports that error bars are omitted because they would be too computationally expensive, while the authors characterize the results as significant and stable.Source: NeurIPS Paper Checklist · p. 17
  • The authors state that they scrape only data with an explicit CC BY 4.0 license to avoid potential copyright infringement.Source: NeurIPS Paper Checklist · p. 19
  • The checklist says the paper does not involve crowdsourcing or research with human subjects and anticipates no positive or negative societal impacts from dataset release.Source: NeurIPS Paper Checklist · pp. 18, 20

Related papers

Loading 2503.01739v2…