CelebV-Text: A Large-Scale Facial Text-Video Dataset

1. Introduction

• A comprehensive construction pipeline combines data collection and processing, attribute annotation, and semi-automatic text generation.Grammar templates are flexibly combined and modified to balance the advantages of automatic and manual text methods.Source: §1 · p. 2
• The dataset contains 70,000 in-the-wild clips and 1,400,000 text descriptions, with 20 texts per clip.It covers six types of static and dynamic attributes; dynamic attributes have start and end timestamps, and manual texts describe labels that cannot be discretized.Source: §1 · p. 2
• CelebV-Text surpasses existing face video datasets [11] in resolution (over 2 times higher), sample count, and distribution diversity.Its texts also show higher diversity, complexity, and naturalness than those in text-video datasets [20] [73], and retrieval experiments validate text-video relevance [18].Source: §1 · p. 2
• The paper reports better relevance between generated face videos and texts than a state-of-the-art large-scale pretrained model [27].It also reports that text interpolation in [20] significantly improves temporal coherence and introduces a benchmark with representative models [20] across three text-video datasets.Source: §1 · p. 2
• The paper presents CelebV-Text as the first large-scale facial text-video dataset with high-quality videos and rich, highly relevant text descriptions.Its contributions also include statistical analyses of video and text quality, diversity, and relevance, plus self-evaluations and a benchmark intended to standardize evaluation.Source: §1 · p. 2

Text-to-Video Generation

• Text-to-video methods use encoded text inputs and include discrete-latent-code approaches [17] [60] and diffusion models [21] [25] [26] [55].Some discrete-code methods treat videos as independent images [20] [27] [64], whereas Phenaki [61] models temporal relations between frames.Source: §2 · p. 2

Multimodal Datasets

• Open-world multimodal datasets [3] [33] [36] [37] [42] [45] [53] [72] include manual annotations or internet-sourced text, while closed-world datasets include controlled or synthetic collections.Examples include CLEVR [29], a synthetic text-image dataset, and MUGEN [22], a video-audio-text dataset built using CoinRun [12].Source: §2 · p. 2

Multimodal Face Datasets

• Modified MUG and MM-Vox [20] describe static facial attributes, while CelebV-HQ [73] provides discrete facial labels and timestamps but no text descriptions.The passage notes that the static-only descriptions and limited annotations of Modified MUG and MM-Vox restrict text diversity for face-domain text-to-video research.Source: §2 · p. 3

3. Dataset Construction

• The construction pipeline has three stages: data collection and processing, data annotation, and semi-automatic text generation.The stages target a large, high-quality video collection and natural, diverse descriptions relevant to the videos.Source: §3 · p. 3, Figure 2 · p. 3

3.1. Data Collection & Processing

• Videos are collected from online resources using diverse search queries, then filtered for low resolution, short duration, and overlap with CelebV-HQ [73].Processing avoids resizing clips with face bounding boxes below the threshold and splits clips when backgrounds change, alongside checks for consistent motion and identity.Source: §3.1 · p. 3

3.2. Data Annotation

• The annotation scheme separates static attributes from dynamic attributes, which include actions, emotions, and light directions.Dynamic attributes receive dense start and end timestamps; automatic annotation is used for selected labels with accuracy of 85% or higher, while workers annotate actions, light directions, and detailed appearance descriptions.Source: §3.2 · p. 4

3.3. Semi-auto Text Generation

• Semi-automatic text generation combines manual descriptions of detailed appearance with template-generated text from annotated attributes.Templates are derived from common grammar structures identified through annotator-written descriptions and online corpora, then use synonym replacement to increase diversity.Source: §3.3 · pp. 4–5

4.1. Video Comparisons

• CelebV-Text contains 70,000 clips totaling around 279 hours, with 20 sentences per video, and has the best video quality among the compared datasets.Its image quality is comparable to CelebV-HQ and higher than MM-Vox; head-related actions make up around 60% of samples, and facial features around 45% of general-appearance labels.Source: §4.1 · p. 5

4.2. Text Comparisons

• CelebV-Text captions average 67.15 words, versus 28.39 for MM-Vox and 31.06 for CelebV-HQ, and show broader text styles and better 4-gram vocabulary naturalness and complexity.Figure 5 also reports better performance for CelebV-Text in both 4-gram and number-of-words distributions.Source: §4.2 · pp. 5–6, Figure 5 · p. 5

4.3. Text-Video Relevance

• With action descriptions, CelebV-Text achieves the best performance on most retrieval metrics; CelebV-HQ and CelebV-Text both outperform MM-Vox on appearance-only retrieval.Adding emotion descriptions produces similar results for CelebV-HQ and CelebV-Text, indicating comparable annotation accuracy for static appearance attributes.Source: Table 2 · p. 6, §4.3 · p. 6

5.1. High-relevance Text-to-Video Generation

• Static-attribute tests condition MMVID and CogVideo on descriptions of general appearance, face details, and light conditions.Figure 6 presents qualitative outputs for static attributes and dynamic attributes.Source: §5.1 · p. 6, Figure 6 · p. 7

5.2. Benchmark on Facial Text-to-Video Generation

• MMVID-interp outperforms MMVID on all reported metrics, while MMVID achieves better FVD, FID, and CLIPSIM than TFGAN.The benchmark also reports lower MMVID video quality when input text describes a dynamic state change.Source: Table 4 · p. 7, §5.2 · p. 8

5.1. High-relevance Text-to-Video Generation

• MMVID trained on CelebV-Text models dynamic attribute changes, and MMVID-interp stabilizes sampling when MMVID struggles to preserve attributes such as earrings.Both MMVID variants model described temporal changes, while CogVideo fails to reflect the described change in the qualitative comparison.Source: §5.1 · pp. 6–7
• CogVideo produces lower-quality, less text-relevant samples than MMVID trained solely on CelebV-Text despite its larger model and training data.The paper describes CogVideo as approximately 100 times larger than MMVID and trained on approximately 75 times more text-video data than CelebV-Text.Source: §5.1 · p. 7

5.2. Benchmark on Facial Text-to-Video Generation

• Across MM-Vox, CelebV-HQ, and CelebV-Text, the benchmark evaluates TFGAN and MMVID using static and dynamic descriptions.The experiments report FVD for temporal consistency, FID for frame quality, and CLIPSIM for text-video relevance.Source: §5.2 · pp. 7–8

6. Discussion

• CelebV-Text contains 70,000 clips, each paired with 20 sentences describing static and dynamic factors.The authors say they demonstrated the dataset's superiority and effectiveness through statistical analysis and experiments.Source: §6 · p. 8
• The authors plan to enlarge the dataset's scale and diversity and explore fine-grained face-video control, pretrained-model adaptation, and 3D-aware generation.These are stated as future directions rather than completed evaluations.Source: §6 · p. 8
• The dataset is intended for research; raw videos will not be released, and annotations, video links, and processing tools are subject to a legality check.The annotations exclude personal biometric information and cover generic attributes such as gender, hair color, and motion.Source: §6 · p. 8

A.2. Grouped Attribute Details

• The appendix groups 40 general appearances and 37 actions into named categories, including facial features, hairstyle, head movements, and daily actions.The listed action groups also include eyes, interaction, and feeling.Source: §A.2 · p. 11

A.3. More Distributions

• CelebV-Text videos average 14.34 seconds, compared with 6.68 seconds for CelebV-HQ; estimated age and ethnicity distributions are close to CelebV-HQ.Age and ethnicity are estimated using an off-the-shelf facial attribute analysis framework rather than manual labels.Source: §A.3 · p. 11, Figure A1 · p. 12, Figure A4 · p. 14
• The attribute inventory includes 40 general appearances, 5 detailed appearances, 6 light conditions, 37 actions, 8 emotions, and 6 light directions.The appendix shows distributions for general appearances, hair colors, actions, emotions, detailed appearances, color temperature, and brightness.Source: Table A1 · p. 12, Figure A2 · p. 13, Figure A3 · p. 13

A.4. Selected Algorithms

• Emotion labels are assigned per frame, smoothed over time with a sliding window, and reviewed by human annotators.The appendix also says light color temperature follows and perceived brightness simplifies light-intensity calculation.Source: §A.4 · p. 12

B. Template Designs

• Text templates are built by parsing annotator-written descriptions, modifying common parse trees, and using PCFGs and synonym replacement to diversify generated sentences.The template design includes attribute-dependent choices, such as gender-related attributes and personal nouns.Source: B. Template Designs · pp. 12, 14, Table A2 · p. 14

C. Results of n-grams

• CelebV-Text improves over MM-Vox in unique n-grams, supporting more diverse descriptions.The supplied passage does not state the numerical size of the difference.Source: C. Results of n-grams · p. 14, Table A3 · p. 14
• CelebV-Text also improves over CelebV-HQ in unique n-grams.The paper identifies the comparison as evidence of description diversity.Source: C. Results of n-grams · p. 14, Table A3 · p. 14
• Table A3 compares unique n-gram counts for MM-Vox, CelebV-HQ, and CelebV-Text.The passage reports an improvement for CelebV-Text over both comparison datasets, without specifying values in the supplied text.Source: Table A3 · p. 14, C. Results of n-grams · p. 14

• FVD, FID, and CLIPSIM assess temporal consistency, individual-frame quality, and generated-video/input-text relevance, respectively.Source: §D.1 · p. 14
• 2,048 randomly selected test videos serve as the real set, matched with 2,048 generated videos as the fake set.Source: §D.1 · p. 14
• 4 uniformly sampled frames per video provide 8,192 real and fake images for FID; CLIPSIM averages scores across frames.Source: §D.1 · p. 14

D.2. Performance Under Texts of Different Lengths

• Table A4 benchmarks text-to-video generation on different datasets, with lower values better for ↓ metrics and higher values better for ↑ metrics.Source: Table A4 · p. 15
• Lengthy inputs better match natural-language distributions and support attribute matching, but can reduce output fidelity on FVD and FID.MMVID trained on CelebV-Text with lengthy inputs produces satisfactory outputs for short test texts, while MMVID trained on MM-Vox with short texts can miss attributes in long inputs.Source: §D.2 · pp. 14–15

D.3. Unconditional Video Generation

• Unconditional video generation is evaluated with DIGAN, MoCoGAN-HD, and StyleGAN-V, with results shown in Figure A6.Source: §D.3 · p. 15

D.4. Static Face Video Generation

• MMVID outperforms TFGAN when trained on CelebV-Text descriptions of detailed appearances and light conditions.Source: §D.4 · p. 15
• For static attributes, MMVID trained on CelebV-Text produces samples that correspond better to text than larger-scale-trained CogVideo.Figure A7 shows detailed-appearance and light-condition samples; Figure A8 adds MMVID results for general appearances.Source: §D.4 · p. 15, Figure A7 · p. 18, Figure A8 · p. 19

D.5. Dynamic Face Video Generation

• Dynamic-attribute experiments report quantitative and qualitative results for MMVID, MMVID-interp, and CogVideo.Figures A9 and A10 show dynamic-emotion and dynamic-light-direction samples, respectively.Source: §D.5 · p. 15, Figure A9 · p. 20, Figure A10 · p. 21
• MMVID-interp splits a dynamic description into two sentences, repeats appearance in both encodings, and linearly interpolates them during sampling.The text-sequence-length-proportional α_i controls the interpolation; the authors describe the modification as simple and say it will be improved.Source: §D.5 · p. 15
• Figure A5 compares generation using short and lengthy input texts.Source: Figure A5 · p. 16