PaperlayerView exact revision on arXiv

Source-linked AI summary

Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation

Amber Yijia Zheng, Lu Liu, Raymond A. Yeh, Xi Yin

arXiv:2607.18789v1cs.CV

TL;DR

Training-data distribution and caption quality remain underexplored factors in text-to-video generation. The paper introduces Moving Alphabet for controlled experiments and finds that balanced, diverse data and high-quality captions are important, while post-training remedies only partially recover losses from poor pre-training data.

  • Problem

    How video distribution and caption quality affect text-to-video performance remains unclear, despite their central role in video-data pipelines.

  • Method

    The paper introduces Moving Alphabet, a controlled testbed that varies rendered-letter content, motion, duration, and caption precision or recall.

  • Results

    Balanced, diverse content and durations improve generalization, caption quality affects performance and training efficiency, and post-training remedies provide only partial recovery.

  • Takeaways & Limitations

    Text-to-video practitioners should prioritize balanced data curation and high-quality captions during pre-training rather than relying on post-training correction.

  • Takeaways & Limitations

    The findings are derived from a controlled testbed rather than real-world video data.

Abstract

from arXiv · show

Text-to-video generation has advanced significantly over the past five years through scaling of model size, data, and compute. Unlike model architecture, training data is often underexplored. Real-world data curation is complex and non-trivial, involving clip selection from raw videos and captioning to create video-text pairs for learning text-to-video mappings. We study how data distribution and caption quality impact text-to-video models. To enable controlled experiments, we introduce Moving Alphabet, a procedural testbed that renders letters with varying fonts, colors, sizes, and positions, moving in different directions and speeds against a black background. This design allows precise control over data distribution and caption quality by corrupting ground-truth metadata. Our experiments yield three findings: a) a diverse and balanced distribution of video content and duration is critical for generalization; b) caption quality significantly affects both model performance and training efficiency, suggesting that text-to-video models are bounded by video understanding capabilities; and c) classifier-free guidance and fine-tuning on high-quality data provide partial recovery from models trained on corrupted captions, but cannot fully compensate for poor pre-training data. We believe these insights can inform the development of large-scale text-to-video models, and we advocate for greater attention to the science of pre-training data.

1 Introduction

Moving Alphabet provides a controlled testbed for studying how video-data distribution and caption quality affect text-to-video generation. The findings emphasize balanced, diverse pre-training data and high-quality captions, while showing that post-hoc recovery methods cannot fully repair poor data.

  • Controlled testbed: Moving Alphabet renders controllable letter videos with varying fonts, colors, sizes, positions, speeds, and directions to isolate data-distribution and caption-quality effects.The testbed is designed to avoid the impracticality of directly controlling real-world video data.
  • Data distribution: Balanced mixtures of 1-letter, 2-letter, and 3-letter scenes match or exceed single-complexity specialists, with the same trend for mixing 2-seconds, 4-seconds, and 8-seconds videos.The result holds even when models are evaluated on a specialist’s own data distribution.
  • Caption quality: Caption corruption sharply reduces performance and training efficiency, with precision more damaging than recall and corrupted-caption models requiring up to 2–4× more compute to match ground-truth validation loss.Under a limited budget, caption correctness should be prioritized over completeness.
  • Post-hoc recovery: Classifier-free guidance improves prompt following somewhat, but clean-data fine-tuning recovers only up to 55% of lost FG PSNR under moderate caption corruption.These methods provide partial recovery rather than fully fixing poor pre-training data.
  • Implications: The study argues that training-data quality and composition deserve as much attention as architectural and model innovation in real-world text-to-video development.The authors acknowledge that the findings come from a controlled testbed while presenting them as guidance for real-world data curation.

2 Related Work

Prior work advances video generation, data curation, synthetic testbeds, and curriculum-based data mixing, but training recipes and the isolated effects of data axes remain insufficiently characterized for text-to-video models.

  • Video generation models: Diffusion-based video generation progressed from cascaded or spatiotemporal extensions of image diffusion to latent-diffusion models with transformer backbones.Examples include Make-A-Video, Imagen Video, and Latte.
  • Video generation models: Existing text-to-video systems report strong qualitative results, while their data composition, caption quality, and mixing strategies are typically described only at a high level.This work instead uses controlled experiments to isolate each training-data axis.
  • Training data curation for generative models: Generative-model research shows that data quality can rival scale, motivating recaptioning, large-scale filtering pipelines, and systematic data-selection benchmarks.Examples include DALL·E 3, LAION-5B, and DataComp.
  • Synthetic testbeds for understanding generative models: Controlled synthetic datasets support analysis of representation learning, disentanglement, compositional visual reasoning, and compositional text-to-image generation.Representative benchmarks include CLEVR, dSprites, 3D Shapes, and T2I-CompBench.
  • Curriculum and data mixing: Curriculum learning and data-mixing research study how example difficulty and source proportions affect training efficiency and final performance.DoReMi learns domain weights, The Pile combines diverse sources, and Data Mixing Laws predict performance from mixture proportions.

3 Moving Alphabet: A Controlled Video Generation Testbed

Moving Alphabet is a procedural text-to-video testbed that renders controllable letter appearance and motion, enabling precise manipulation of video distributions and caption quality. Its configurable complexity, duration, and metadata-derived captions support controlled experiments on training data.

  • Testbed construction: Moving Alphabet renders one to three moving letters on a black background, with attributes sampled from known distributions for measurable caption corruption.The procedural setup avoids ambiguous ground truth and confounding factors inherent in real-world videos.
  • Content complexity: Each letter varies across identity, font, color, size, rotation, position, speed, and direction, with motion reflected at canvas boundaries.Identity uses 52 classes, fonts use 8 typefaces, colors 10 options, sizes 3 levels, rotations 8 angles, and speed is sampled uniformly in [3, 10] pixels per frame.
  • Caption construction: Templated captions are generated from metadata and describe every letter’s appearance, normalized start and end positions, exact movement angle, speed, and boundary bounces.This provides captions with explicitly known ground-truth content and motion descriptions.
  • Evaluation: The evaluation uses metadata-aware pixel and attribute metrics because natural-video FVD features transfer poorly to the synthetic domain.Pixel metrics include foreground PSNR and scaled foreground MSE, while attribute classifiers measure generated colors, sizes, directions, and speeds against ground truth.

4 Investigation on Data Distribution

Balanced training distributions across scene complexity and video duration generalize better than single-level or heavily biased mixtures. Single-complexity and single-duration training cause out-of-distribution collapse, while equal mixing delivers robust performance across evaluation conditions.

  • Scene complexity: Single-complexity training collapses out of distribution: Pure 1L reaches 0.357 direction accuracy and 4.1 dB FG PSNR on 3L scenes, while Pure 3L reaches 0.485 direction accuracy and 8.2 dB FG PSNR on 1L scenes.The degradation is not graceful, and compositional skills do not emerge from single-object training alone.
  • Scene complexity: Mix Equal generalizes across complexity levels and reaches 12.9 dB FG PSNR on 3L evaluation versus 10.1 dB for Pure 3L.Heavy-1L underperforms Mix Equal on 1L evaluation despite containing more 1L data, with a 3.2 dB FG PSNR gap.
  • Video duration: Single-duration training memorizes duration: Pure 4s achieves 11.1 dB FG PSNR on 4-second videos but drops below 5.3 dB on other durations.The model fails to learn temporal representations that transfer across durations.
  • Video duration: Mix Equal achieves 12.2 dB average FG PSNR across lengths, with 12.0–12.4 dB across individual evaluation durations and gains over specialists at 4s and 8s.It uses 39.7B tokens, while Pure 2s uses 19.2B and Pure 8s uses 65.3B.
  • Video duration: Heavy-8s achieves 12.5 dB on 8-second videos but catastrophically falls to 6.6 dB on 2-second videos, showing the instability of heavy duration oversampling.Heavy-2s maintains FG PSNR above 11.6 dB at all lengths.

5 Investigation on Caption Quality

Caption precision and recall both impair video-generation quality, but precision more sharply reduces fidelity and attribute controllability. Caption corruption also raises compute requirements, while omitted attributes remain reliable only when other caption fields redundantly specify them.

  • Generation quality: Both caption precision and recall sharply reduce FG PSNR, with precision corruption costing 8.1 dB versus 7.6 dB for recall corruption.Reducing recall from β=1.0 to 0.3 lowers FG PSNR from 12.3 to 4.8, while reducing precision from α=1.0 to 0.3 lowers it from 12.3 to 4.3.
  • Generation quality: Precision is the primary driver of attribute controllability: color accuracy falls from 0.95 at α=1.0 to 0.08 at α=0.3 when β=1.0.Color accuracy reaches 0.21 at α=0.5 and near chance below that level, whereas precision advantage is positive across attributes.
  • Training efficiency: Low recall is the most expensive corruption axis, requiring 2.08 to over 3.91× baseline compute at β=0.3 versus 1.00–3.29× at β=1.0.At α=0.7, β=0.3, reaching the threshold takes 2.15× compute; severe corruption can prevent convergence within the 200-epoch budget.
  • Training efficiency: Precision also increases training cost: at fixed β=0.9, reducing α from 1.0 to 0.3 raises compute from 1.41× to 2.35×, peaking at 3.62× for α=0.5.Both axes increase training cost, but recall shows a more consistent gradient across precision levels, so higher-quality captions outperform simply training longer on noisy data.
  • Missing attributes: When an attribute is omitted, direction remains near-uniform with mean KL = 0.006, while size and color develop biases with mean KL = 0.029 and mean KL = 0.099.Direction is recoverable from retained start and end positions; size and color lack redundant cues and drift toward seed-dependent defaults.

6 Investigation on the Recovery from Poor Pre-training

Classifier-free guidance recovers moderate-corruption attribute accuracy but cannot restore pixel fidelity, while finetuning on clean captions partially repairs mild corruption and fails on severe damage. Across both strategies, omission is more recoverable than hallucination, and recovery remains incomplete for complex scenes.

  • Classifier-free guidance: With good captions, CFG has a sweet spot around CFG = 2–3, balancing pixel fidelity and attribute faithfulness.Ground-truth captions reach 12.6 dB FG PSNR at CFG = 2, declining to 11.5 dB at CFG = 10; direction accuracy reaches 0.97 at CFG = 5.
  • Classifier-free guidance: CFG improves moderate-corruption attribute accuracy but barely restores pixel quality, leaving a 7.3 dB FG PSNR gap to clean-caption training.At α=0.7, β=0.7, direction rises from 0.94 to 0.97 and size from 0.66 to 0.80 at CFG = 5+, while FG PSNR rises only from 5.0 to 6.3 dB.
  • Classifier-free guidance: Severe caption corruption is beyond CFG’s reach: FG PSNR varies by less than 0.3 dB, direction reaches only 0.49, and color remains near chance.At α=0.3, β=0.3, color accuracy is 0.07 at CFG = 1 and does not recover because CFG amplifies incorrect claims.
  • Cross-strategy comparison: Omission is more recoverable than hallucination: β=0.9 finetuning fully recovers FG PSNR, whereas low-β or low-α corruption leaves substantial damage.For β ≤0.7, recovery is under 20% or only 1–2 dB regardless of data size; hallucinated associations must be unlearned before clean data can help.
  • Finetuning recovery: Finetuning with clean captions partially recovers mild corruption but cannot rescue severe damage: at α=0.9, 5K samples yield 55% recovery, whereas α=0.3 remains unchanged.At α=0.9, 1L FG PSNR rises from 7.2 to 9.6 dB toward an 11.7 dB baseline; at α=0.3, it stays at 4.6 dB.
  • Finetuning recovery: Finetuning recovery declines with scene complexity and shows diminishing returns from additional data, with 10K outperforming 5K and 1K.At α=0.7, 1L reaches 43% recovery while 3L reaches 11% with 5K samples; severe corruption remains unrecoverable even with 10K samples.

7 Conclusion · Appendix

Moving Alphabet is introduced as a controlled testbed for studying how training data affects text-to-video generation. The experiments show that balanced data, diverse content and durations, and high-quality captions are important, while post-training remedies provide only partial recovery from poor pre-training data.

  • 7 Conclusion: Moving Alphabet provides a controlled testbed for studying how training data affects text-to-video generation.
  • 7 Conclusion: Balanced content complexity is important for generalization.
  • 7 Conclusion: Diverse video content supports generalization.
  • 7 Conclusion: Balanced and diverse clip durations are important for generalization.
  • 7 Conclusion: Caption quality is crucial for model performance and training efficiency.
  • 7 Conclusion: Classifier-free guidance and fine-tuning provide only partial recovery from poor pre-training data.
  • 7 Conclusion: The findings recommend prioritizing balanced data curation during pre-training.
  • 7 Conclusion: The findings recommend using high-quality captions at the pre-training stage.

A Attribute Specifications

The Moving Alphabet dataset defines its attributes through explicit value sets, with all attributes sampled independently and uniformly from those values.

  • Table 4 lists all attributes used in the Moving Alphabet dataset and their possible values.
  • All Moving Alphabet attributes are sampled independently and uniformly from their listed values.

B Training Details

Models are trained with AdamW using a warmed-up, constant learning rate and a fixed mixed-precision optimization setup, while 10% text dropout enables classifier-free guidance.

  • Optimization: Training uses AdamW with a learning rate of 5 × 10−4, linearly warmed up from 0 for 500 steps and then held constant.The setup also uses a per-GPU batch size of 24, bf16 mixed precision, and gradient clipping with maximum norm 1.0.
  • Classifier-Free Guidance: 10% text dropout is applied during training to enable classifier-free guidance at inference.This regularization supports classifier-free guidance during inference.

C Evaluation Metric Details

The evaluation isolates letters from the near-black background using foreground masks and, for multi-letter videos, ground-truth trajectory ROIs. It measures color, size, direction, and speed from detected pixel regions and centroids against ground truth.

  • Foreground extraction: Foreground masks use pixels whose brightest RGB channel exceeds 40/255; multi-letter videos additionally use trajectory-based ROIs at metadata-recorded positions.For 1L videos, the mask directly isolates the single letter.
  • Attribute metrics: Color accuracy assigns the median RGB of foreground pixels above intensity 200 to the nearest ground-truth color in Euclidean RGB space.A prediction is correct when the assigned color matches the ground-truth color.
  • Attribute metrics: Size accuracy bins the foreground bounding-box maximum, median over the first 5 frames, into small (<44 px), medium (44–68 px), or large (≥68 px).These bins correspond to ground-truth render sizes of 32, 56, and 80 px, respectively; correctness requires a matching bin.
  • Motion metrics: Direction accuracy converts first-to-last centroid displacement into atan2 angles and classifies them into right, down, left, or up quadrants.The prediction is correct when it matches the ground-truth cardinal direction.
  • Motion metrics: Speed correlation reports Pearson correlation between ground-truth speeds and median per-frame centroid displacements across evaluation videos.The median per-frame centroid displacement, measured in pixels, serves as predicted speed.

D Training Convergence

Training metrics plateau by 60K–70K steps, roughly 100 epochs with 150K training samples, so epoch 100 is used as the evaluation checkpoint throughout the experiments.

  • D Training Convergence: All three pixel-quality metrics plateau by training step 60K–70K in the Mix Equal condition across complexity levels and CFG values.The metrics are FG PSNR, BG PSNR, and FG MSE, evaluated at two CFG values across all three complexity levels.
  • D Training Convergence: Epoch 100 is selected as the evaluation checkpoint for all experiments.This checkpoint corresponds to roughly 100 epochs with 150K training samples.

E Caption Corruption: Additional Metrics · F Per-Attribute Breakdowns · G Marginal Bias: Single-Attribute Details

Additional caption-corruption analyses show that precision-favored allocations generally outperform recall-favored ones, especially for color and size, while higher-complexity attributes degrade more sharply under corruption. In single-attribute training, conditional caption dropping induces severe, seed-dependent size bias, unlike unconditional training, which preserves diversity and near-uniformity.

  • E Caption Corruption: Additional Metrics: Precision-favored corruption allocations win for most level pairs across all four attributes, with the largest margins for color and size.Direction and speed are noisier in the single-run comparison at CFG = 1.0.
  • F Per-Attribute Breakdowns: Per-level heatmaps show that higher-complexity models degrade more steeply as caption corruption increases.The breakdown covers complexity levels 1L, 2L, and 3L across the α × β grid at CFG = 1.0.
  • F Per-Attribute Breakdowns: The gap between caption-precision and caption-recall effects becomes more pronounced at higher complexity.The analysis reports this pattern across direction accuracy, size accuracy, speed correlation, and foreground PSNR.
  • G Marginal Bias: Single-Attribute Details: In the single-attribute setting, conditional caption dropping produces large, high-variance size biases, whereas unconditional training remains near-uniform.Figure 16 reports means over 4 seeds with error bars of ±1 standard deviation.
  • G Marginal Bias: Single-Attribute Details: 13.7× higher KL divergence occurs for conditional-drop models than unconditional models on size.Table 5 reports means ± standard deviations over 4 random seeds at CFG = 1.0.
  • G Marginal Bias: Single-Attribute Details: Each conditional-drop seed converges to a different dominant size, while unconditional models preserve roughly uniform size distributions.The differing seed outcomes indicate that the size default is arbitrary and depends on random initialization.
  • G Marginal Bias: Single-Attribute Details: The text encoder’s null representation for size acts as a strong default that varies with random initialization.This provides the mechanism for the seed-dependent collapse observed in conditional-drop models.
Loading 2607.18789v1…