Source-linked AI summary
Photorealistic Video Generation with Diffusion Models
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, José Lezama
TL;DR
Video generation must handle the computational cost of transformer attention and the tension between image-data reuse and video compression. W.A.L.T. combines unified image-video latent compression with windowed attention, achieving state-of-the-art image and video benchmark results and high-resolution text-to-video generation.
Problem
High-dimensional video makes full transformer attention prohibitively expensive, while spatial and spatiotemporal latent compression impose different trade-offs for complexity and image-data reuse.
Method
W.A.L.T. jointly encodes images and videos into a unified latent space and uses alternating window-restricted spatial and spatiotemporal attention in a transformer diffusion model.
Results
W.A.L.T. reports state-of-the-art results on UCF-101, Kinetics-600, and ImageNet without classifier free guidance, plus state-of-the-art zero-shot FVD on UCF-101 for text-to-video generation.
Takeaways & Limitations
The framework supports joint image-video learning and photorealistic, temporally consistent text-to-video synthesis at 512 × 896 resolution and 8 frames per second.
Takeaways & Limitations
Some default settings are not optimal, and further tuning may improve performance.
Abstract
from arXiv · showhide
We present W.A.L.T, a transformer-based approach for photorealistic video generation via diffusion modeling. Our approach has two key design decisions. First, we use a causal encoder to jointly compress images and videos within a unified latent space, enabling training and generation across modalities. Second, for memory and training efficiency, we use a window attention architecture tailored for joint spatial and spatiotemporal generative modeling. Taken together these design decisions enable us to achieve state-of-the-art performance on established video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without using classifier free guidance. Finally, we also train a cascade of three models for the task of text-to-video generation consisting of a base latent video diffusion model, and two video super-resolution diffusion models to generate videos of $512 \times 896$ resolution at $8$ frames per second.
1 Stanford University 2 Google Research 3 Georgia Institute of Technology
W.A.L.T. generates high-resolution, temporally consistent photorealistic videos from text prompts. The shown samples use 512 × 896 resolution for 3.6 seconds at 8 frames per second.
- W.A.L.T. generates photorealistic videos from text prompts.
- The generated videos are high-resolution and temporally consistent.
- The samples have 512 × 896 resolution, 3.6 seconds duration, and 8 frames per second.
1. Introduction
Video generation remains difficult for transformer architectures because full attention is prohibitively expensive on high-dimensional video, while latent-space choices trade off image-data reuse against complexity. W.A.L.T. addresses these constraints with unified image-video latents and windowed attention, achieving strong benchmark and text-to-video results.
- Full transformer attention scales quadratically with sequence length, making high-dimensional video processing prohibitively expensive.
- Spatial compression enables reuse of pretrained image models but increases network complexity, whereas spatiotemporal compression limits access to large paired image-text datasets.
- W.A.L.T. uses an autoencoder to map images and videos into a unified latent space for joint training and lower-cost high-resolution video generation.
- Window-restricted spatial and spatiotemporal attention alternates within transformer blocks to reduce computation while modeling image and video structure.
- W.A.L.T. reports state-of-the-art results on UCF-101, Kinetics-600, and ImageNet without classifier free guidance.
- A three-model cascade generates photorealistic text-to-video outputs at 512 × 896 resolution and 8 frames per second, with state-of-the-art zero-shot FVD on UCF-101.
2. Related Work
Prior work uses diffusion and transformer-based approaches for image and video generation, but video transformers face long-sequence sampling and efficiency challenges. W.A.L.T. contributes a transformer backbone for jointly training image and video latent diffusion models.
- Video diffusion models operate in pixel or latent space, with latent approaches providing important efficiency advantages for video modeling.
- Joint training on image and video data has been used to improve text-conditioned video generation and leverage larger image datasets.
- Transformer generative models include autoregressive and diffusion systems for images and videos, but autoregressive video sampling is typically impractical because sequences are very long.
- W.A.L.T. provides an empirical demonstration of a transformer backbone for jointly training image and video latent diffusion models.
3. Background
Diffusion models learn generation by reversing a gradual noising process, while latent diffusion reduces computation by operating on compressed continuous representations produced by an autoencoder.
- Gaussian diffusion gradually adds noise to real data and trains a reverse process that denoises samples from a noise distribution.
- The noise schedule γ(t) decreases monotonically from 1 to 0 as diffusion time t ranges from 0 to 1.
- The denoiser fθ uses conditioning such as class labels or text prompts and may predict noise, the denoised input, or velocity v; the experiments use v-prediction.
- Latent diffusion encodes high-resolution videos into lower-dimensional continuous latent representations, then decodes those latents into reconstructions.
- Adversarial and perceptual losses can improve reconstruction quality in the decoder.
4. W.A.L.T
W.A.L.T. uses a unified causal latent representation for images and videos, then generates with alternating spatial and spatiotemporal window attention. Conditioning mechanisms and a three-model cascade support multiple generation tasks while reducing computational demands.
- Learning Visual Tokens: A causal encoder independently tokenizes the first frame, allowing static images to share a unified spatiotemporally compressed latent space with videos.The resulting latent tensors can represent either a video or a stack of independent images, and are real-valued and quantization-free.
- Learning to Generate Images and Videos: Patchification converts each latent frame into non-overlapping patches, while learnable embeddings encode spatial and temporal positions.Images receive the temporal embedding associated with the first latent frame.
- Learning to Generate Images and Videos: Window attention alternates spatial windows over individual frames with spatiotemporal windows over latent-frame neighborhoods to model spatial and temporal relationships efficiently.For images, an identity attention mask passes image-frame value embeddings through spatiotemporal layers unchanged.
- High-Resolution Generation: W.A.L.T. achieves computational efficiency, supports joint image-video training, and avoids flickering artifacts associated with independently encoded video frames.For high-resolution video, the base model generates 128 × 128 videos that are upsampled twice by two super-resolution stages.
- Conditional Generation: The model supports conditioning on timesteps and external information through cross-attention, adaptive normalization, and self-conditioning mechanisms.Text-conditioned generation uses cross-attention, while AdaLN parameters are regressed from condition and timestep embeddings.
- Conditional Generation: AdaLN-LoRA reduces a ViT-g model’s MLP parameters from 475M to 12M when r = 2.The reduction follows from using a low-rank parameterization with r much smaller than dmodel.
5. Experiments
W.A.L.T. is evaluated across image, video, frame-prediction, and text-to-video tasks, with ablations examining its architectural and training choices. It achieves strong benchmark results and produces high-resolution, temporally consistent text-conditioned videos.
- Visual Generation: W.A.L.T significantly outperforms prior works on UCF-101 video generation and Kinetics-600 video prediction using 50 DDIM inference steps.It achieves state-of-the-art performance with fewer model parameters than prior video diffusion models.
- Visual Generation: W.A.L.T outperforms prior ImageNet image-generation methods without specialized schedules, convolutional inductive bias, improved diffusion losses, or classifier-free guidance.VDM++ has a slightly better FID but uses significantly more parameters, namely 2B.
- Ablation Studies: Reducing patch size improves performance, while full self-attention is unnecessary for good performance in the ablation studies.The default ablations evaluate spatial and spatiotemporal window configurations on UCF-101 using FVD and inception scores.
- Ablation Studies: Increasing self-conditioning rate from 0.0 to 0.9 improves FVD by 44%, while increasing AdaLN-LoRA bottleneck dimension improves performance and model size.The AdaLN-LoRA bottleneck provides a trade-off between parameter count and generation performance.
- Ablation Studies: Enforcing zero terminal SNR improves video-generation performance, and latent dimension c = 8 provides a strong operating point across evaluated datasets and tasks.Increasing latent dimension improves reconstruction quality, but both lower and higher values can produce poor generation FVD scores.
6. Conclusion
W.A.L.T. is a simple, scalable, efficient transformer framework for latent video diffusion that unifies image and video generation. The authors report state-of-the-art image and video generation and high-resolution text-to-video synthesis.
- W.A.L.T. is a transformer-based framework for latent video diffusion.The framework is described as simple, scalable, and efficient.
- Windowed attention forms the transformer backbone for image and video generation.
- A three-model cascade synthesizes high-resolution, temporally consistent photorealistic videos from natural-language descriptions.
A. Implementation Details
The implementation uses established architectures and task-specific training procedures. These include aspect-ratio finetuning, autoregressive continuation for long videos, and adapted evaluation prompts for UCF-101.
- Training configuration: The first stage follows Yu et al.'s architecture and hyperparameters, while the second stage uses specified spatial and spatiotemporal windows.The second-stage settings include 1 × 16 × 16 spatial and 5 × 8 × 8 spatiotemporal windows.
- Aspect-ratio finetuning: Aspect-ratio finetuning adapts a square-trained base stage to generate videos with a 9 : 16 aspect ratio.The procedure interpolates position embeddings and scales window sizes.
- Long video generation: For long video generation, the model generates 17 initial frames and uses encoded recent frames as context for autoregressive continuation.The last 5 frames are encoded into 2 latent frames to help maintain motion continuity.
- UCF-101 evaluation: UCF-101 text-to-video evaluation follows prior work while adapting prompts to better describe the dataset classes.
B.1. Image Generation
W.A.L.T. is evaluated for class-conditional ImageNet image generation alongside qualitative image and video samples. Compared with prior image-generation methods, it achieves better Inception Score and competitive FID scores.
- ImageNet results: W.A.L.T. performs better than prior works on Inception Score and achieves competitive FID scores.The comparison concerns class-conditional image generation at 256 × 256 resolution.
- Evaluation setup: ImageNet class-conditional generation is evaluated at 256 × 256 using the ADM protocol and classifier-free guidance.
- Qualitative results: Figure 5 presents qualitative samples for ImageNet class-conditional generation.
B.3. Image-to-Video
The image-to-video setup jointly trains frame prediction and conditions on an initial latent frame. Qualitative Kinetics-600 frame-prediction samples are shown in Figure 6.
- Image-to-video method: Joint frame-prediction training conditions on 1 latent frame to predict subsequent frames.This leverages the high-quality first frame from the image generator as context.
- Implementation: The section includes training and evaluation hyperparameters for the image-to-video task.
- Qualitative results: Figure 6 shows Kinetics-600 frame-prediction samples with ground truth above and generation below.Unobserved ground-truth frames are shaded.