Source-linked AI summary
Imagen Video: High Definition Video Generation with Diffusion Models
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, Tim Salimans
TL;DR
Text-to-video generation requires models that produce high-definition, temporally consistent, and controllable videos. Imagen Video addresses this with a cascaded video-diffusion system that combines base generation with spatial and temporal super-resolution, achieving high-fidelity videos and capabilities including text animation, artistic styles, and 3D object understanding.
Problem
Prior video-generation work focused on restricted datasets or moderate-resolution settings, motivating text-to-video generation with high frame fidelity, temporal consistency, and language understanding.
Method
Imagen Video uses a frozen T5 text encoder, a base video diffusion model, and interleaved spatial and temporal super-resolution models in a seven-model cascade.
Results
Imagen Video generates high-fidelity videos with good temporal consistency while retaining image-system capabilities such as accurately spelling text.
Takeaways & Limitations
The system demonstrates controllable video generation spanning diverse videos, text animations, artistic styles, and 3D object understanding.
Takeaways & Limitations
The model and frozen T5-XXL encoder were trained on problematic data, and social biases and stereotypes remain challenging to detect and filter.
Abstract
from arXiv · showhide
We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models. Given a text prompt, Imagen Video generates high definition videos using a base video generation model and a sequence of interleaved spatial and temporal video super-resolution models. We describe how we scale up the system as a high definition text-to-video model including design decisions such as the choice of fully-convolutional temporal and spatial super-resolution models at certain resolutions, and the choice of the v-parameterization of diffusion models. In addition, we confirm and transfer findings from previous work on diffusion-based image generation to the video generation setting. Finally, we apply progressive distillation to our video models with classifier-free guidance for fast, high quality sampling. We find Imagen Video not only capable of generating videos of high fidelity, but also having a high degree of controllability and world knowledge, including the ability to generate diverse videos and text animations in various artistic styles and with 3D object understanding. See https://imagen.research.google/video/ for samples.
1 INTRODUCTION
Imagen Video extends diffusion-based generative modeling to text-to-video generation, targeting high-definition, temporally consistent, language-aligned videos. Its contributions include a cascaded design, transferred image-generation techniques, video-specific diffusion findings, and qualitative controllability.
- 1 INTRODUCTION: Imagen Video generates high-definition videos with high frame fidelity, strong temporal consistency, and deep language understanding.The system scales to 128 frames at 1280×768 resolution and 24 frames per second.
- 1 INTRODUCTION: The system uses a frozen T5 text encoder, a base video diffusion model, and interleaved spatial and temporal super-resolution diffusion models.
- 1 INTRODUCTION: The authors demonstrate cascaded diffusion models for high-definition video generation and transfer frozen text conditioning and classifier-free guidance from text-to-image systems.
- 1 INTRODUCTION: The paper identifies v-prediction for sample quality and progressive distillation of guided models as effective findings for text-conditioned video diffusion.
- 1 INTRODUCTION: Imagen Video shows qualitative controllability through 3D object understanding, text animations, and videos in diverse artistic styles.
2 IMAGEN VIDEO
Imagen Video is implemented as a continuous-time diffusion cascade that progressively generates and refines video across spatial and temporal dimensions. The models denoise noisy latents through learned reverse dynamics while conditioning super-resolution stages on text and lower-resolution video.
- 2 IMAGEN VIDEO: Imagen Video comprises seven video diffusion sub-models for text-conditional generation, spatial super-resolution, and temporal super-resolution.The cascade includes one base model, three spatial super-resolution models, and three temporal super-resolution models.
- 2.1 DIFFUSION MODELS: The diffusion process uses continuous-time latents with a forward Gaussian process and a learned reverse-time model that generates samples from noise.The reverse process runs from t = 1 to t = 0 under a continuous cosine noise schedule.
- 2.1 DIFFUSION MODELS: Training minimizes a noise-prediction objective, while the implementation uses v-parameterization rather than directly predicting noise or clean data.
- 2.1 DIFFUSION MODELS: For spatial and temporal super-resolution, conditioning includes the text, the previous-stage low-resolution video, and conditioning-augmentation strength.The text embedding is provided to all super-resolution models.
- 2.1 DIFFUSION MODELS: Sampling can use an ancestral sampler with stochasticity controlled by γ or a deterministic DDIM sampler for fast progressive distillation.
2.2 CASCADED DIFFUSION MODELS AND TEXT CONDITIONING
Imagen Video uses a cascade that progressively increases spatial and temporal resolution, while conditioning every model on text embeddings and supporting broader video super-resolution use.
- Cascaded diffusion models first generate low-resolution video, then sequentially increase spatial and temporal resolution through super-resolution models.
- The pipeline contains one frozen text encoder, one base model, three spatial super-resolution models, and three temporal super-resolution models.
- The super-resolution models can be trained independently in parallel and applied to real videos or outputs from other generative models.
- Text embeddings are injected into all models in the cascade, not only the base model.
- The base model uses spatial and temporal attention, while super-resolution models use temporal convolutions and omit spatial attention at the highest resolution for memory efficiency.
- Frozen T5-XXL contextual embeddings are critical for text-video alignment and support evidence of deeper language understanding.
2.3 VIDEO DIFFUSION ARCHITECTURES
Imagen Video extends the U-Net to video with separable spatial and temporal processing, using attention in the base model and cheaper convolutions for high-resolution upsampling.
- The Video U-Net applies spatial operations independently across frames with shared parameters and temporal operations across frames.
- The base model uses temporal attention for long-term dependencies, whereas SSR and TSR models use temporal convolutions for local temporal consistency during upsampling.
- Temporal convolutions reduce memory and computation costs, which is crucial for super-resolution at high frame rates and spatial resolutions.
- Spatial attention improves fidelity in the base and first two SSR models, but higher-resolution models switch to fully convolutional architectures to reduce memory and compute.
- The highest-resolution SSR model trains on random lower-resolution crops and generalizes to full-resolution sampling.
2.4 v-PREDICTION
Imagen Video uses v-prediction across its models to improve numerical stability for progressive distillation and avoid artifacts at higher resolutions.
- v-prediction parameterization is used for all models and provides numerical stability throughout the diffusion process for progressive distillation.
- At higher resolutions, v-parameterization avoids color-shifting artifacts and, in video, avoids temporal artifacts.
2.5 CONDITIONING AUGMENTATION
Imagen Video adds noise conditioning augmentation to every super-resolution model, using randomized training noise and a small fixed sampling noise level to reduce cascade mismatches and artifacts.
- Noise conditioning augmentation is applied to all temporal and spatial super-resolution models.
- The method supports parallel cascade training by reducing sensitivity to domain gaps between stages.
- Training adds Gaussian noise with a random signal-to-noise ratio, while sampling uses a fixed ratio such as 3 or 5.
- Small sampling-time augmentation helps remove artifacts from the previous stage while preserving most of the conditioning video's structure.
2.6 VIDEO-IMAGE JOINT TRAINING
Imagen Video jointly trains on images and videos, while classifier-free guidance strengthens text adherence and specialized guidance schedules mitigate saturation artifacts.
- 2.6 VIDEO-IMAGE JOINT TRAINING: Joint training packs images as single-frame videos while masking temporal computations, enabling use of larger image-text datasets.This strategy disables temporal convolutions and cross-frame attention for independent images.
- 2.6 VIDEO-IMAGE JOINT TRAINING: Joint image-video training enables video dynamics in different artistic styles.
- 2.6.1 CLASSIFIER FREE GUIDANCE: Classifier-free guidance is critical for generating high-fidelity samples that respect text prompts.The adjusted prediction combines conditional and unconditional models using guidance strength w.
- 2.6.1 CLASSIFIER FREE GUIDANCE: For w > 0, guidance emphasizes conditioning, typically trading lower diversity for higher quality.
- 2.6.2 LARGE GUIDANCE WEIGHTS: Oscillating guidance alternates high and low weights to preserve text alignment while limiting saturation artifacts.It uses an initial high-weight phase followed by alternating weights, but was not beneficial beyond 80×48 spatial resolution.
2.7 PROGRESSIVE DISTILLATION WITH GUIDANCE AND STOCHASTIC SAMPLERS
The paper uses two-stage progressive distillation with classifier-free guidance and stochastic sampling to reduce the number of video diffusion sampling steps.
- 2.7 PROGRESSIVE DISTILLATION WITH GUIDANCE AND STOCHASTIC SAMPLERS: Progressive distillation repeatedly halves the required sampling steps while retaining much perceptual quality.
- 2.7 PROGRESSIVE DISTILLATION WITH GUIDANCE AND STOCHASTIC SAMPLERS: The authors report that guided progressive distillation works well for video generation.
- 2.7 PROGRESSIVE DISTILLATION WITH GUIDANCE AND STOCHASTIC SAMPLERS: The first distillation stage learns one model matching the guided combination of jointly trained conditional and unconditional models.
- 2.7 PROGRESSIVE DISTILLATION WITH GUIDANCE AND STOCHASTIC SAMPLERS: The second stage applies progressive distillation to that guided model to produce models requiring fewer sampling steps.
- 2.7 PROGRESSIVE DISTILLATION WITH GUIDANCE AND STOCHASTIC SAMPLERS: A stochastic N-step sampler combines a deterministic DDIM update using twice the original step size with a noise-perturbed backward step.
3 EXPERIMENTS
Experiments show Imagen Video generates high-fidelity, stylistically diverse, text-animated videos with 3D structure understanding, while scaling and v-prediction improve quality and distillation accelerates sampling.
- 3.1 UNIQUE VIDEO GENERATION CAPABILITIES: Imagen Video generates high-fidelity videos with artistic styles, approximate 3D structural consistency, and varied text animation styles.The paper presents these capabilities as evidence of controllable generation and useful video priors.
- 3.2 SCALING: Video modeling performance is not saturated at the tested model sizes, suggesting further scaling benefits for video generation.
- 3.2 SCALING: Scaling the base video U-Net from 500M to 1.6B to 5.6B parameters improves both FVD and CLIP scores.Both metrics are computed on 4096 video samples.
- 3.3 COMPARING PREDICTION PARAMETERIZATIONS: At high resolution, v-prediction produces better generations and faster sample-quality convergence than ϵ-prediction.ϵ-prediction also exhibits color shifts and inconsistencies across frames.
- 3.4 PERCEPTUAL QUALITY AND DISTILLATION: Distillation makes the cascade about 18× faster while producing videos of similar perceptual quality.The distilled models are also about 36× more efficient in FLOPs because guidance is folded into one model.
4 LIMITATIONS AND SOCIETAL IMPACT
Imagen Video offers creative benefits but raises unresolved safety and ethical concerns. The authors describe filtering efforts while acknowledging that problematic training data and social biases remain challenging.
- Video generative models may amplify and augment human creativity but may also be misused to generate fake, hateful, explicit, or harmful content.
- The authors apply input prompt filtering and output video content filtering in internal trials.
- Imagen Video and its frozen T5-XXL text encoder were trained on problematic data.
- Although internal testing suggests much explicit and violent content can be filtered, social biases and stereotypes remain challenging to detect and filter.
- The authors withheld the model and source code until these safety concerns are mitigated.
5 CONCLUSION
Imagen Video extends text-to-image diffusion methods into text-conditioned video generation through a cascade of video diffusion models. The resulting system produces high-fidelity, temporally consistent videos while retaining text rendering, and progressive distillation accelerates sampling.
- Imagen Video is a text-conditional video generation system based on a cascade of video diffusion models.
- The system extends Imagen into the time domain and trains jointly on video and images to generate high-fidelity videos with good temporal consistency.
- The model retains image-system capabilities such as accurately spelling text in generated videos.
- Progressive distillation speeds up video diffusion models at sampling time.