VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

• VidProM comprises 1.67 million unique text-to-video prompts and 6.69 million videos generated by four state-of-the-art diffusion models.Source: Figure 1 · p. 1
• The dataset points to text-to-video prompt engineering and efficient video generation as research directions.Source: Figure 1 · p. 1
• The figure also identifies fake-video detection and video-copy detection for diffusion models as research directions.Source: Figure 1 · p. 1

• Text-to-video diffusion models rely heavily on prompts, yet no publicly available dataset focused on text-to-video prompts existed.Source: §1 · p. 2
• VidProM is a large-scale text-to-video prompt-gallery dataset containing 1.67 million unique prompts from real users and 6.69 million videos generated by four state-of-the-art diffusion models.The prompts are from official Pika Discord channels; each is assigned six NSFW probabilities, a UUID, and a timestamp.Source: §1 · p. 2
• Compared with DiffusionDB, VidProM prompts are generally more dynamic, complex, and longer, and the dataset is specialized for videos rather than images.Source: §1 · p. 2
• VidProM can support prompt-based evaluation, model distillation, prompt engineering, similar-prompt video reconstruction, and video detection research.Source: §1 · p. 2
• The paper presents VidProM as a rich database for future studies and argues that a specialized text-to-video prompt dataset is needed.Source: §1 · p. 2

2.1 Text-to-Video Diffusion Models

• Text-to-video diffusion models generate video content from textual prompts; VidProM uses four publicly accessible sources, including Pika, Text2Video-Zero [3], VideoCrafter2, and ModelScope [5].Source: §2.1 · p. 3

2.2 Existing Datasets

• Existing text-video datasets mainly pair captions with real videos, whereas VidProM pairs real-user prompts with videos generated by text-to-video diffusion models.Source: §2.2 · p. 3
• Figure 3 indicates that prompts with an NSFW probability above 0.2 are a very small fraction of the 1,672,243 unique prompts.Source: Figure 3 · p. 4
• Prior prompt datasets include text-to-text and text-to-image resources, while this paper introduces a prompt dataset specifically collected for text-to-video generation.Source: §2.2 · p. 4

• The dataset also records a UUID, timestamp, 3072-dimensional prompt embedding, and four generated videos for each data point.Source: §3 · p. 4
• Prompts and timestamps are extracted from official Pika Discord channels, while image-to-video prompts, prompts without associated videos, and duplicates are removed.Source: §3 · p. 4
• Each prompt receives six NSFW probabilities; fewer than 0.5% exceed 0.2, and users can choose thresholds for filtering potentially unsafe data.Source: §3 · pp. 4–5
• The curation process yields 1,672,243 unique prompts and 6.69 million videos, with generation requiring approximately 50,631 GPU hours.Source: §3 · pp. 4–5
• VidProS contains 1,038,805 semantically unique prompts, defined by pairwise cosine similarity below 0.8 using text-embedding-3-large embeddings.Source: §3 · p. 5

4.1 Basic Information

• VidProM has 40.6% more semantically unique prompts than DiffusionDB, and its prompts were collected over eight months rather than one.The paper also notes that VidProM uses text-embedding-3-large, which supports up to 8192 tokens versus CLIP’s 77.Source: §4.1 · pp. 5–6
• VidProM includes videos generated by four text-to-video models, with more than 14 million seconds of video produced using over 50,000 V100 GPU hours.The paper contrasts this with DiffusionDB’s images from Stable Diffusion and reports an average repetition rate per source of 1 for VidProM versus about 8.2 for DiffusionDB.Source: §4.1 · pp. 6–7

4.2 Semantics of prompts

• Video prompts more often describe time, dynamic actions, and duration, and are generally more complex and longer than image prompts.VidProM has nearly 60,000 prompts over 70 words versus about 15,000 in DiffusionDB, and over 25,000 over 100 words versus close to zero; Figure 4 compares prompt examples and long-prompt counts.Source: Figure 4 · p. 6, §4.2 · p. 7
• VidProM and DiffusionDB have significantly different prompt distributions: video users favor general human activities, while image users tend more toward art paintings.The comparison uses t-SNE on randomly selected 10,000-prompt samples and WizMap over all prompts; both groups show interest in cars, food, and cute animals.Source: §4.2 · p. 8

Video Generative Model Evaluation

• VidProM could support evaluating models on real-user prompts and large-scale data, making evaluations more practical and arguments more convincing.The paper contrasts this with current evaluation efforts using carefully designed, small-scale prompts.Source: §5 · p. 8

Text-to-Video Diffusion Model Development

• VidProM raises questions about whether caption-to-prompt domain gaps hinder model performance and whether models can be trained or distilled on prompt-generated-video pairs.The paper notes that some studies report irreversible defects from synthetic-data training, while proposed methods may prevent such collapse.Source: §5 · p. 8

Text-to-Video Prompt Engineering

• VidProM provides data for text-to-video prompt engineering, a research area the authors say had no related work in the community to their knowledge.They demonstrate automatic prompt completion by training a large language model on VidProM.Source: §5 · p. 8

Efficient Video Generation

• Searching VidProM for a prompt’s closest matches and reconstructing a video from existing videos is proposed as an alternative to generation from noise.The section gives generation times of 43 seconds for ModelScope and 51 seconds for VideoCrafter2 on a single V100 GPU.Source: §5 · p. 8

Fake Video Detection

• VidProM could support specialized fake-video detectors and provide prompts for generating videos from more diffusion models.The paper identifies generalization across generators and the time cost of examining video frames with image detectors as challenges.Source: §5 · p. 9

Video Copy Detection for Diffusion Models

• VidProM can be used to pre-filter potential video replications for manual annotation and possible use as training sources.The section distinguishes diffusion-model replication from existing copy-detection research focused on hand-crafted transformations such as flips and crops.Source: §5 · p. 9

Training details

• The authors treat prompt completion as general text generation and fine-tune Mistral-7B-v0.1 on VidProM using eight A100 GPUs for two hours.They report using DeepSpeed for fine-tuning.Source: §6 · p. 9

Demonstration

• The trained prompt-completion model is released for free download and testing under the CC BY-NC 4.0 License.The paper provides its Hugging Face model page.Source: §6 · p. 9
• Demonstrations show completions for inputs spanning cats, underwater worlds, landscapes, women, Spider-Man, and Chinese dragons.The examples include multiple generated prompts for each input.Source: §6 · pp. 9–10

7 Conclusion

• VidProM comprises 1.67 million unique text-to-video prompts and 6.69 million videos generated by four diffusion models, alongside related data.The conclusion also describes comparing VidProM prompt information and semantics with DiffusionDB and outlining research directions.Source: §7 · p. 10
• The current dataset’s generated videos are short and not of the highest quality, though the authors added 10,000-video examples from each of three newer models.The examples include 8-second videos at 720p for StreamingT2V and Open-Sora 1.2, and 6-second videos at 720p for CogVideoX-2B.Source: §7 · p. 10

A Semantic De-duplication Algorithm

• Semantic de-duplication retains prompts so that every pair has cosine similarity below θ = 0.8; the similarity-pair stage took about 0.604 hours for 1,672,243 prompts.The reported computation used eight A100 GPUs and 128 CPU cores with Faiss.Source: A Semantic De-duplication Algorithm · p. 15

B Domain Gap between Video Captions and Text-to-Video Prompts

• VidProM prompts and Panda-70M captions have significantly different distributions, including differences in topics and descriptive style.The paper suggests filtering training videos for topics of human interest or using style transfer to align captions and prompts; Panda-70M is identified as [18].Source: B Domain Gap between Video Captions… · pp. 15–16

C Benchmarking Existing Fake Image Detection on VidProM

• Traditional cues such as global texture statistics and frequency analysis are the most effective tested approaches, but all existing models struggle on fake-video detection.GAN-image detectors and methods designed for diffusion images or cross-model generalization also fail on videos.Source: C Benchmarking Existing Fake Image… · p. 16, Table 2 · p. 17, Table 3 · p. 17

D Video Copy Detection for Diffusion Models

• The video-copy-detection section investigates whether text-to-video diffusion outputs replicate training data or other existing videos.Such replication is the phenomenon examined in the section.Source: D Video Copy Detection for Diffusion… · p. 17

D.1 Experimental Setup

• The copy-detection setup samples one million videos from VideoCraft2 and ModelScope training sources and uses FCPL, a state-of-the-art, efficient system on DVSC2023.The authors note that this source selection may underestimate replication, but consider it sufficient to analyze the phenomenon.Source: §D.1 · p. 17

Qualitative observations

• Figure 7 illustrates generated videos replicating training or existing content, including Pika’s close replication of the copyrighted painting The Persistence of Memory.The authors note that such use may be acceptable for purposes such as education, news reporting, and parody, while profitable use without permission could potentially infringe copyright.Source: §D.2 · p. 17, Figure 7 · p. 18

Quantitative observations

• Only about 2% of Text2Video-Zero videos have a maximum normalized score above 0, and Pika has the lowest replication rate while Text2Video-Zero has the highest.A score above 0 generally indicates a high likelihood of replicated content.Source: §D.2 · p. 17, Figure 8 · p. 18
• The authors suggest that commercial models’ lower replication rates may partly reflect measures to prevent replication and mitigate legal risks.Source: §D.2 · pp. 17–18

D.3 Limitations and Future Directions

• Current copy-detection models can assign scores no higher than 0 to replicated videos because they do not generalize well beyond manually generated transformations.The training transformations include examples such as horizontal flips and random crops.Source: §D.3 · p. 18, Figure 9 · p. 19, Figure 10 · p. 19
• A proposed workflow uses existing copy-detection systems to filter candidate pairs, has human labelers verify them, then trains a new detection model on validated pairs.Source: §D.3 · p. 18

Composition

• 1,672,243 instances cover all available instances through February 2024, with each instance containing a prompt, metadata, an embedding, and four generated videos.The dataset is self-contained.Source: Composition · p. 20
• Some NSFW prompts and videos remain despite server rules; six NSFW probabilities let researchers choose a filtering threshold for their tasks.Sensitive information in prompts is possible but described as rare because of server rules and moderation.Source: Composition · pp. 20–21

Collection

• DiscordChatExporter collected Pika Discord messages from July 2023 to February 2024, and random prompt-video mappings were manually checked.Users were not notified of this specific collection, and no ethical review or impact analysis was conducted.Source: Collection · pp. 21–22

Preprocessing

• The dataset preserves the original prompts and generated videos without preprocessing; raw data is also saved.Source: Preprocessing · p. 22

Uses

• Proposed uses include model evaluation and development, prompt engineering, efficient video generation, fake-video detection, video copy detection, and multimodal learning.Use of the dataset should follow Pika Lab licensing policies.Source: Uses · p. 23

Distribution

• VidProM is publicly available online; Pika prompts and videos use CC BY-NC 4.0, while videos from the other three models retain their respective licenses.Source: Distribution · pp. 23–24

Maintenance

• The authors plan bimonthly updates, continue hosting older versions, and accept contributions; harmful videos can be reported and creators can request removal.Source: Maintenance · p. 24