Source-linked AI summary
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, Humphrey Shi
TL;DR
Text-to-video methods often require costly training and large video datasets. This paper adapts pretrained text-to-image diffusion models without training or optimization, adding latent motion and cross-frame attention; it reports high-quality, temporally consistent videos across multiple applications.
Problem
Existing text-to-video approaches rely on computationally heavy training and large-scale video datasets, motivating a zero-shot, training-free setting.
Method
The method modifies a pretrained text-to-image diffusion model by adding motion dynamics to latent codes and cross-frame attention to the first frame.
Results
The method produces high-quality, time-consistent videos while preserving foreground context, appearance, and identity across frames.
Takeaways & Limitations
The approach extends beyond text-to-video to conditional and specialized generation and instruction-guided video editing without additional training.
Takeaways & Limitations
Without the proposed modifications, generated frames share prompt semantics but lack object-appearance and motion coherence.
Abstract
from arXiv · showhide
Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets. In this paper, we introduce a new task of zero-shot text-to-video generation and propose a low-cost approach (without any training or optimization) by leveraging the power of existing text-to-image synthesis methods (e.g., Stable Diffusion), making them suitable for the video domain. Our key modifications include (i) enriching the latent codes of the generated frames with motion dynamics to keep the global scene and the background time consistent; and (ii) reprogramming frame-level self-attention using a new cross-frame attention of each frame on the first frame, to preserve the context, appearance, and identity of the foreground object. Experiments show that this leads to low overhead, yet high-quality and remarkably consistent video generation. Moreover, our approach is not limited to text-to-video synthesis but is also applicable to other tasks such as conditional and content-specialized video generation, and Video Instruct-Pix2Pix, i.e., instruction-guided video editing. As experiments show, our method performs comparably or sometimes better than recent approaches, despite not being trained on additional video data. Our code will be open sourced at: https://github.com/Picsart-AI-Research/Text2Video-Zero .
1. Introduction
The paper introduces zero-shot, training-free text-to-video synthesis by adapting pretrained text-to-image diffusion models. Two lightweight modifications enforce temporal consistency, while the approach also supports several video generation and editing applications.
- Zero-shot text-to-video synthesis generates videos from textual prompts without optimization or fine-tuning.
- The method adapts pretrained text-to-image diffusion models to video generation without additional training.
- Motion dynamics are encoded in latent codes to maintain global scene and background consistency across frames.
- Cross-frame attention is used to preserve foreground context, appearance, and identity throughout the sequence.
- The method is demonstrated for conditional and specialized video generation as well as instruction-guided video editing.
2. Related Work
Related work spans autoregressive, transformer, and diffusion approaches for text-to-video synthesis, generally leveraging substantial video-related training. The paper distinguishes its training-free approach from methods requiring optimization or reference videos, while noting temporal consistency challenges when adapting image diffusion models.
- Earlier text-to-image methods progressed from template and feature-matching techniques to GAN-, transformer-, and diffusion-based systems.
- Applying image diffusion models to video is difficult because probabilistic generation does not straightforwardly ensure temporal consistency.
- Existing text-to-video methods use autoregressive transformers, diffusion processes, or hybrid architectures to generate videos from text.
- Several approaches require video or image-video training, including tuning, joint training, unsupervised video-data use, or multi-stage video diffusion.
- Unlike prior approaches, Text2Video-Zero is training-free and avoids massive computing requirements, whereas Tune-A-Video still requires optimization and a reference video.
3. Method
The method adapts Stable Diffusion for zero-shot text-to-video by generating sequences of latent codes with motion dynamics and replacing self-attention with cross-frame attention. These modifications target temporal consistency in the global scene, background, and foreground appearance and identity.
- Stable Diffusion: Stable Diffusion operates in latent space and uses DDIM sampling to decode latent codes into generated images.The model uses an autoencoder latent representation and a UNet-like denoising architecture with convolutional and attention blocks.
- Problem formulation: Zero-shot text-to-video generates temporally consistent video frames without training or fine-tuning on video data.The method defines a function from a text description and frame count to video frames, using pretrained image synthesis models.
- Naive baseline: Independent Gaussian sampling produces frames that share prompt semantics but lack object-appearance and motion coherence.This motivates constructing a temporally related sequence of latent codes rather than sampling every frame independently.
- 3.3.1 Motion Dynamics in Latent Codes: Motion dynamics warp latent codes using frame-dependent global translation vectors before DDPM forward steps create the sequence used for video diffusion.The translation vector is δ_k = λ·(k−1)δ, where λ controls the amount of global motion; DDPM adds freedom to object motion.
- 3.3.2 Reprogramming Cross-Frame Attention: Cross-frame attention replaces each self-attention layer so every frame attends to keys and values from the first frame.This carries object and background appearance, structure, and identity from the first frame to subsequent frames.
- 3.3.3 Background smoothing (Optional): Background smoothing optionally combines background-masked latent codes from the first frame and frame k to further improve background temporal consistency.The paper reports this is especially helpful for text-prompt video generation with one or no initial image and no further guidance.
3.4. Conditional and Specialized Text-to-Video
The method extends ControlNet and specialized DreamBooth models to conditional and specialized video generation while addressing temporal inconsistencies through motion-aware latents and cross-frame attention. It also supports applying SD-based instruction-guided image editing to video without training or fine-tuning.
- Conditional generation: Directly applying ControlNet to video causes temporal inconsistencies and severe changes in object appearance, identity, and background.
- Conditional generation: ControlNet conditions video generation on inputs such as edges, scribbles, pose, depth maps, and segmentation maps.The video procedure applies the pretrained ControlNet branch per frame while modifying the main diffusion process.
- Conditional generation: The proposed modifications combine motion-enriched latent codes with cross-frame attention to improve temporal consistency in ControlNet-guided videos.
- Specialized generation: Specialized DreamBooth models produce specialized time-consistent video generations when their weights are adopted for conditional generation.
- Video editing: The approach enables SD-based text-guided image editing algorithms, including Instruct-Pix2Pix, to operate on video without training or fine-tuning.
4. Experiments
Experiments evaluate Text2Video-Zero across text-to-video, conditional generation, specialized generation, and instruction-guided editing, including comparisons with CogVideo and Tune-A-Video. The method produces temporally consistent videos while preserving scene, background, object identity, and appearance.
- Experimental setup: The framework generates 8-frame, 512 × 512 videos from Stable Diffusion 1.5 and supports conditional, specialized, and instruction-guided video generation.It uses ControlNet, DreamBooth, and Instruct Pix2Pix codebases for the respective applications.
- Qualitative results: The method maintains global scene and background consistency while preserving the foreground object's context, appearance, and identity throughout sequences.These properties are reported across the evaluated applications.
- Comparison with baselines: 31.19 CLIP score is obtained by the method versus 29.63 for CogVideo on text-to-video generation.The method slightly outperforms CogVideo despite CogVideo's 9.4 billion parameters and large-scale video training.
- Comparison with baselines: Compared with CogVideo, the method shows better text-video alignment in examples involving a man riding a bicycle and a man running in snow.Both methods preserve object identity and background with good temporal consistency in the reported qualitative comparison.
- Video editing: Video Instruct-Pix2Pix produces temporally consistent edits, while per-frame Instruct-Pix2Pix lacks temporal consistency and Tune-A-Video is less aligned to instructions.The method also addresses local edits while preserving details of the input sequence and background.
- Ablation study: Combining motion-coherent latent codes with cross-frame attention achieves the best temporal coherence in the ablation study.Cross-frame attention improves preservation of object identities and appearances across generations.
5. Conclusion
The paper introduces zero-shot text-to-video synthesis using a method for time-consistent generation without optimization or fine-tuning. It demonstrates effectiveness across multiple video synthesis and editing applications.
- The paper addresses zero-shot text-to-video synthesis and proposes a method for time-consistent video generation.
- The approach requires no optimization or fine-tuning, making text-to-video generation and its applications affordable.
- The method is demonstrated for conditional and specialized video generation and for Video Instruct-Pix2Pix instruction-guided video editing.
Appendix
The appendix extends the evaluation with additional qualitative results, ablations, conditional generation experiments, and comparisons for instruction-guided video editing.
- Additional qualitative results investigate text-to-video quality and compare the method with CogVideo.
- Qualitative ablations analyze the relevance of the proposed procedures.
- Conditional text-to-video experiments include pose-guided and edge-guided generation with extensive qualitative results and ablations.
- The appendix provides additional instruction-guided video-editing results and comparisons with Tune-A-Video.
6. Additional Experiments for Text-to-Video Unconditional Generation
Additional unconditional text-to-video experiments examine qualitative generation and the contributions of motion latents, cross-frame attention, background smoothing, and motion timing. The combined components yield consistent scenes, plausible motion, and improved preservation of object identity and background.
- 6.1. Qualitative results: The method produces temporally consistent style and scene representations and plausible motions in unconditional text-to-video generation.Qualitative examples include skiing, waving, and dancing.
- 6.1. Qualitative results: The method performs comparably or sometimes better than CogVideo despite using an optimization-free approach without large-scale video training.Several examples show better text alignment and higher-quality motion than CogVideo.
- 6.2. Ablation Studies: Applying motion to latent codes at Δt = 0 mainly produces global shifts without individual object motion.The experiments find it crucial to apply motion at T′ < T and empirically set Δt = 60 for good object motions.
- 6.2. Ablation Studies: Background smoothing makes the background more consistent and better preserved, including a red sign in the reported example.
- 6.2. Ablation Studies: Without motion information and cross-frame attention, scenes differ across frames and the main object's identity is not preserved.Motion improves pose alignment, while cross-frame attention substantially improves identity and background preservation; combining both gives the best results.
7. Text-to-Video with Edge Guidance
With edge guidance, the method combines cross-frame attention and motion information in latents to improve identity preservation and temporal consistency.
- Cross-frame attention preserves the person’s identity better in edge-guided video generation.
- Motion information in latents further improves temporal consistency under edge guidance.
- The method presents additional edge-guided video-generation results alongside an ablation of its two components.
8. Text-to-Video with Pose Guidance
The method supports pose-guided video generation, with experiments examining the effects of cross-frame attention and motion information in latents.
- The method generates videos guided by pose information.
- The pose-guided experiments evaluate cross-frame attention and motion information in latents.
- Additional pose-guided results are presented alongside the component-effect study.
9. Video Instruct-Pix2Pix
The approach extends to instruction-guided video editing, producing high-fidelity outputs that follow editing instructions and outperform the compared baselines in reported examples.
- The method outperforms naive frame-by-frame Instruct-Pix2Pix and Tune-A-Video in the reported editing comparisons.
- Tune-A-Video is described as limited in localized editing and as struggling to transfer style and color information.