A Recipe for Scaling up Text-to-Video Generation with Text-free Videos

1. Introduction

• Text-to-video progress trails text-to-image generation amid limited publicly available video-text data and high video-captioning costs.The introduction contrasts the costly captioning process with the easier collection of text-free clips.Source: §1 · p. 2
• TF-T2V separates spatial appearance generation into a content branch and temporal dynamics synthesis into a motion branch.The framework uses image-text data for spatial generation and video-only data to learn motion patterns without textual annotations; paired video-text data can also be co-optimized.Source: §1 · p. 2
• A temporal coherence loss explicitly constrains correlations between adjacent frames to improve generated-video continuity.The authors position this unified approach as avoiding complex cascading steps.Source: §1 · p. 2
• FID improves from 9.67 to 8.19 and FVD from 484 to 441 when randomly collected text-free videos double the training set.The authors report this as evidence of TF-T2V’s scaling potential.Source: §1 · p. 2
• TF-T2V is presented as a plug-and-play framework for text-to-video and compositional synthesis, with experiments supporting continuity, fidelity, controllability, and scalability.The introduction says the framework can integrate into existing generation frameworks.Source: §1 · p. 2

Text-to-image generation

• Text-to-image work moved from GAN-based methods toward diffusion models, including text-conditional generation and latent denoising.The section cites LAION-5B as a large-scale image-text dataset and describes Stable Diffusion as operating at the latent level.Source: §2 · p. 2

Compositional video synthesis

• TF-T2V’s pipeline pairs image-text data for spatial appearance generation with text-free videos for motion synthesis, jointly optimizing both branches.The figure also describes text-guided video generation at inference and integration with compositional synthesis.Source: Figure 2 · p. 3

Text-to-video generation

• Text-to-video research spans early recurrent or autoregressive methods and later diffusion models trained on large video-text datasets.Examples include cascaded pixel-level diffusion, two-step text-to-image-embedding approaches, and latent video diffusion models with temporal layers.Source: §2 · p. 3

Compositional video synthesis

• Compositional video synthesis adds structured signals such as depth or sketch for fine-grained control, while TF-T2V can be merged into existing controllable frameworks using text-free videos.VideoComposer is described as integrating textual, spatial, and temporal conditions.Source: §2 · p. 3

3.1. Preliminaries of video diffusion model

• Diffusion training progressively adds noise to clean data, then learns to denoise the resulting sample through an iterative reverse process.The denoising model uses v-prediction, and representative video diffusion models apply the objective per frame in latent feature space.Source: §3.1 · pp. 3–4

3.2. TF-T2V

• TF-T2V jointly optimizes a content branch for spatial appearance and a motion branch for temporal dynamics, using image-text data and text-free videos in one model.Image-text data guides text-conditioned appearance generation, while image-conditioned video synthesis trains temporal modules on text-free clips.Source: §3.2 · p. 4, §3.2.2 · p. 4

3.2.1 Spatial appearance generation

• The content branch uses text and image embeddings to guide appearance generation and randomly drops either condition so each can control content separately.CLIP encoders provide the embeddings.Source: §3.2.1 · p. 4

3.2.2 Motion dynamic synthesis

• The motion branch learns image-conditioned video synthesis from text-free clips, using the original video's center-frame image embedding to recover the video.The temporal modules learn motion dynamics through this image-to-video task.Source: §3.2.2 · p. 4
• When paired video-text data is available, TF-T2V also trains on text-to-video and image-to-video generation to enhance textual control.Its motion branch can be co-optimized with paired video-text data.Source: §3.2.2 · p. 4
• The temporal coherence loss compares predicted and ground-truth adjacent-frame differences to help alleviate flicker and support seamless transitions.The method adds frame differences as an additional supervisory signal because per-frame losses do not consider temporal correlations.Source: §3.2.2 · pp. 4–5

3.2.3 Training and inference

• The total loss combines the base loss with the coherence loss, and the framework can generate text-guided videos or integrate structural conditions for compositional synthesis.The balance coefficient λ is set empirically to 0.1.Source: §3.2.3 · p. 5

4.1. Experimental setup

• The setup uses ModelScopeT2V and VideoComposer baselines, samples 16 frames at 4 FPS, and treats WebVid10M’s roughly 10.7M pairs as text-free; Internal10M adds about 10M unlabeled videos.LAION-5B supplies image-text pairs; evaluation uses FID, FVD, and CLIPSIM for text-to-video and depth error, sketch error, and EPE for controllability.Source: §4.1 · p. 5

4.2. Evaluation on text-to-video generation

• TF-T2V trained on WebVid10M and Internal10M performs better than its WebVid10M-only counterpart, with reported improvements in continuity and visual quality.The text-to-video evaluation also reports that human assessment on 100 randomly synthesized videos supports the qualitative observations.Source: Table 2 · p. 5, §4.2 · p. 6, Figure 3 · p. 6

4.3. Evaluation on compositional video synthesis

• In comparisons with existing methods on compositional video generation, TF-T2V exhibits high-fidelity and consistent generation; evaluations address depth, sketch, and motion control, and human preferences on 100 randomly sampled videos further support its superiority.Source: §4.3 · p. 6, Table 3 · p. 6, Table 4 · p. 6

4.4. Ablation study

• Temporal coherence loss helps alleviate temporal discontinuity, including color shifts, in the qualitative ablation.The ablation evaluates frame consistency using average CLIP similarity between consecutive frames in 1,000 videos.Source: §4.4 · p. 7, Figure 6 · p. 8

4.5. Evaluation on semi-supervised setting

• TF-T2V-Semi reaches the best text-to-video performance, outperforming ModelScopeT2V trained on labeled WebVid10M and producing better prompt alignment and temporal smoothness.The comparison uses labeled WebVid10M together with text-free Internal10M; the semi-supervised result is reported in Table 8.Source: §4.5 · p. 8, Table 8 · p. 8

• TF-T2V is presented as a versatile video-generation framework that exploits text-free videos and explores scaling.Source: §5 · pp. 8–9
• The framework decomposes video generation into spatial appearance generation and motion-dynamics synthesis.Source: §5 · p. 9
• A temporal coherence loss constrains learning correlations between adjacent frames; experiments demonstrate effectiveness and potential in fidelity, controllability, and scalability.Source: §5 · p. 9

• In the compositional motion-to-video comparison, TF-T2V produces more realistic and appealing results than VideoComposer.The videos use textual descriptions and motion vectors as conditions.Source: Figure 8 · p. 12
• Table 9 presents an ablation study of different training manners.Source: Table 9 · p. 12
• The supplementary material adds experimental details and results and discusses limitations and future work.Source: Supplementary Material · p. 12

• The compositional motion-to-video experiment uses a Stable Diffusion 2.1 VAE with a downsample factor of 8 to encode latent features.Source: §6 · p. 12
• TF-T2V's network structure is basically consistent with ModelScopeT2V and VideoComposer to facilitate fair comparison.Source: §6 · p. 12
• Human evaluation randomly samples 100 generated videos, with ratings from 0% to 100% averaged for each evaluation content.Source: §6 · p. 12

• The results in Table 9 demonstrate the rationality of jointly optimizing TF-T2V's spatial and temporal blocks.The comparison is between joint training and sequential two-stage spatial and temporal modeling.Source: §7 · pp. 12–13
• Under semi-supervised settings, FVD (↓) gradually decreases as the number of text-free videos increases.Source: §7 · p. 13, Figure 9 · p. 13

8. More experimental results

• 896 × 512 videos show that TF-T2V can be applied to high-definition text-to-video synthesis.The passage also notes 448 × 256 video generation.Source: §8 · p. 13
• 1280 × 640 and 1280 × 768 videos demonstrate TF-T2V’s application potential for high-definition compositional synthesis.Source: §8 · p. 13
• To demonstrate its application potential in high-definition compositional video synthesis, TF-T2V additionally synthesizes videos at 1280 × 640 and 1280 × 768, alongside a 896 × 512 text-to-video model.The results are displayed in Figures 11 and 12.Source: §8 · p. 13

9. Limitations and future work

• Only a doubling of the training set was explored, leaving 10× or 100× scaling untested because of computational constraints.Source: §9 · p. 13
• 16 frames were sampled per clip, so long-video generation with text-free videos remains unexplored.Source: §9 · p. 13
• Text-free TF-T2V may struggle with temporal prompts such as “from right to left” or “rotation”; semi-supervised training helped alleviate this problem.Source: §9 · p. 13
• The examples include depth-to-video outputs at 448 × 256, 1280 × 640, and 1280 × 768 resolutions.Source: Figure 11 · p. 15
• The sketch-to-video examples likewise include 448 × 256, 1280 × 640, and 1280 × 768 resolutions.Source: Figure 12 · p. 16