Source-linked AI summary
Video Diffusion Models
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, David J. Fleet
TL;DR
High-fidelity, temporally coherent video generation remains an important generative-modeling challenge. The paper extends standard diffusion models to video, combines image and video modeling, and introduces reconstruction-guided conditional sampling; it reports state-of-the-art sample-quality scores on video prediction and unconditional generation and promising text-conditioned results.
Problem
The paper addresses the challenge of generating high-fidelity, temporally coherent video and validating diffusion models in the video modality.
Method
The paper extends standard Gaussian diffusion models with video-suitable architectures, joint image-video training, and reconstruction-guided conditional sampling for video extension.
Results
The models achieve state-of-the-art sample quality scores on video prediction and unconditional video generation, with promising first results for text-conditioned video generation.
Takeaways & Limitations
Diffusion models can provide effective generative models for unconditional, text-conditioned, and video-prediction tasks, while reconstruction guidance supports longer spatial and temporal video generation.
Takeaways & Limitations
The models were not released because of potential harmful uses, and their outputs reflect biases in their training datasets.
Abstract
from arXiv · showhide
Generating temporally coherent high fidelity video is an important milestone in generative modeling research. We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results. Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find to reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods. We present the first results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation. Supplementary material is available at https://video-diffusion.github.io/
1 Introduction
This work applies diffusion models to video generation in both unconditional and conditional settings, extending recent image-generation results to a new data modality. It reports high-quality video generation using a largely standard Gaussian diffusion formulation and initial results on video generation tasks.
- The paper presents diffusion-model results for video generation in both unconditional and conditional settings.
- The work evaluates diffusion models on video generation as a new data modality beyond image and audio generation.
- High-quality videos can be generated with essentially the standard Gaussian diffusion formulation and straightforward architectural changes for video data.The changes accommodate video within the memory constraints of deep learning accelerators.
2 Background
The background presents diffusion models as continuous-time latent-variable generative models that learn denoising-based reverse processes. It describes training objectives, ancestral and predictor-corrector sampling, and classifier-free guidance for conditional generation.
- A continuous-time diffusion model defines Gaussian latents z_t through a forward process from data x, with noise increasing until q(z_1) approximately follows N(0, I).
- Training: Training learns to denoise noisy latents into estimates of the data using weighted mean squared error, commonly parameterized by predicting noise.The paper samples times t over [0, 1] and also uses v-prediction for certain models.
- Sampling: The ancestral sampler starts from Gaussian noise and reverses the diffusion process through discretized stochastic steps.Its stochasticity is controlled by γ, with s and t following a uniformly spaced sequence from 1 to 0.
- Sampling: The predictor-corrector sampler alternates an ancestral step with a Langevin correction intended to better match the forward-process marginals.The Langevin step uses a fixed step size δ of 0.1 in the described implementation.
- Conditional generation: Classifier-free guidance adjusts conditional and unconditional model predictions to emphasize conditioning, typically trading lower diversity for higher sample quality.
3 Video diffusion models
The paper adapts diffusion models to video with a factorized 3D U-Net, jointly training on images and videos. It also introduces reconstruction-guided conditional sampling for extending videos spatially and temporally while addressing coherence problems in prior replacement sampling.
- Video diffusion models: The model jointly represents fixed-length video blocks at fixed spatial resolution using a 3D U-Net adapted from image diffusion architectures.The architecture factorizes processing over space and time and modifies 2D convolutions for video frames.
- Video diffusion models: Factorized space-time attention permits masking temporal interactions, enabling joint training on video and image generation.The paper reports that joint training is important for sample quality.
- Video diffusion models: Autoregressive conditional sampling extends fixed-size samples to longer videos, higher frame rates, and higher spatial resolutions.The same framework can append future frames, interpolate missing frames, or upsample low-resolution videos.
- Video diffusion models: The replacement method can produce individually good segments that are incoherent with the conditioning segment because it omits the required conditional dependence.The needed quantity is the conditional expectation given both the noisy latent and the conditioning video, rather than the expectation given only the latent.
- Video diffusion models: Reconstruction guidance approximates the missing conditional term using the model’s reconstruction of the conditioning data and an adjustable weighting factor.Larger weighting factors tend to improve sample quality, especially with predictor-corrector sampling and Langevin diffusion.
4 Experiments
The experiments evaluate video diffusion across unconditional generation, video prediction, and text-conditioned generation. They show benefits from joint image-video training, classifier-free guidance, and reconstruction guidance for extending videos.
- 4.3.3 Autoregressive video extension for longer sequences: A two-model cascade extends 16x64x64 frameskip-4 samples to 64x128x128 frameskip-1 videos through simultaneous super-resolution and autoregressive extension.Both models are conditioned on the text prompt.
- 4.3.1 Joint training on video and image modeling: Adding 4 or 8 independent image frames per video improves video and image sample-quality metrics in text-conditioned generation.The added images reduce gradient variance while introducing some bias for the video-modeling objective.
- 4.3.2 Effect of classifier-free guidance: Higher classifier-free guidance weights improve Inception Score-like metrics, while FID-like metrics improve before degrading as guidance increases.Guidance also increases individual-frame fidelity and strengthens the conditioning signal.
- 4.3.3 Autoregressive video extension for longer sequences: Reconstruction guidance improves perceptual quality over replacement guidance when generating longer videos.Figure 4 reports that replacement produces temporally incoherent blocks, whereas reconstruction guidance maintains temporal coherence.
5 Related work
The paper situates video diffusion among prior autoregressive, VAE, GAN, and normalizing-flow approaches, contrasting its jointly modeled video-diffusion architecture with concurrent framewise diffusion work.
- Related work: Prior video-generation work commonly uses autoregressive models, VAEs, GANs, and normalizing flows.
- Related work: A concurrent diffusion approach predicts individual frames with an image diffusion model inside an RNN temporal autoregressive model, unlike this paper’s jointly modeled video approach.
6 Conclusion
The paper extends diffusion modeling to video through space-time U-Nets and factorized attention, covering unconditional, text-conditioned, and video prediction settings. It also introduces reconstruction-guided sampling for video extension while highlighting unresolved societal risks and the need for further bias evaluation.
- Contributions: Diffusion models for video use straightforward 2D-to-3D U-Net extensions with factorized space-time attention.The approach applies the standard diffusion formulation to video data in space and time.
- Contributions: The framework supports unconditional, text-conditioned, and video prediction models.
- Contributions: Reconstruction-guided conditional sampling outperforms replacement or imputation methods and enables temporal interpolation, extrapolation, and spatial super-resolution.
- Limitations and societal implications: The authors withhold model release because generative video systems could facilitate fake content, harassment, misinformation, and other unethical uses.
- Limitations and societal implications: The models may reflect training-data biases, motivating future audits and benchmark evaluations for social and cultural bias in video generation.
A Details and hyperparameters
This appendix section points to additional samples and documents the hyperparameters, training details, and compute resources used for each model.
- Details and hyperparameters: Figure 5 provides more samples accompanying Figure 2.
- Details and hyperparameters: The appendix lists model hyperparameters, training details, and compute resources.
A.1 UCF101
The UCF101 appendix entry specifies the training configuration for one model, including optimizer, architecture, schedule, hardware, and training duration.
- UCF101: The model uses 256 base channels, channel multipliers 1, 2, 4, 8, and two blocks per resolution.
- UCF101: Training uses Adam with learning rate 0.0003, batch size 128, cosine noise schedule, and 60,000 training steps.
- UCF101: The configuration uses 128 TPU-v4 chips, attention resolutions 8, 16, 32, dropout 0.1, and EMA 0.9999.
A.2 BAIR Robot Pushing
The BAIR Robot Pushing appendix entry specifies a separate training configuration with its own architecture, optimization settings, compute allocation, and training duration.
- BAIR Robot Pushing: The model uses 128 base channels, channel multipliers 1, 2, 3, 4, and three blocks per resolution.
- BAIR Robot Pushing: Training uses Adam with learning rate 0.0002, batch size 128, cosine noise schedule, and 660,000 training steps.
- BAIR Robot Pushing: The configuration uses 128 TPU-v4 chips, attention resolutions 8, 16, 32, dropout 0.1, and EMA 0.999.
A.3 Kinetics
The Kinetics configuration uses a 256-channel diffusion model trained with Adam, cosine noise scheduling, and 256 TPU-v4 chips.
- The model uses 256 base channels with channel multipliers 1, 2, 4, and 8.
- Training uses Adam with β1 = 0.9, β2 = 0.99, and a learning rate of 0.0002.
- The configuration trains with batch size 256 for 220,000 steps on 256 TPU-v4 chips.
- The diffusion model uses cosine noise scheduling, attention at resolutions 8, 16, and 32, and dropout 0.1.
- Joint training includes 8 independent images per video, with a conditioning embedding dimension of 1024.
A.4 Text-to-video
The text-to-video configurations vary model width, optimization duration, conditioning design, and the number of independent images jointly trained per video.
- The configurations use 128 or 256 base channels with channel multipliers 1, 2, 4, and 8; one configuration adds multiplier 16.
- Training runs for 200,000, 700,000, or 800,000 steps on 64 or 128 TPU-v4 chips.
- All configurations use Adam, cosine noise scheduling, two blocks per resolution, and attention resolutions 8, 16, and 32.
- The conditioning embedding uses dimension 1024, while the MLP depth is 4 layers in all three configurations.
- Joint training uses 0, 4, or 8 independent images per video in the first configuration and 8 or 7 in the other configurations.