Photorealistic Video Generation with Diffusion Models

• W.A.L.T. generates photorealistic videos from text prompts.Source: Figure 1 · p. 1
• The generated videos are high-resolution and temporally consistent.Source: Figure 1 · p. 1
• The samples have 512 × 896 resolution, 3.6 seconds duration, and 8 frames per second.Source: Figure 1 · p. 1

• Full transformer attention scales quadratically with sequence length, making high-dimensional video processing prohibitively expensive.Source: §1 · p. 2
• Spatial compression enables reuse of pretrained image models but increases network complexity, whereas spatiotemporal compression limits access to large paired image-text datasets.Source: §1 · p. 2
• W.A.L.T. uses an autoencoder to map images and videos into a unified latent space for joint training and lower-cost high-resolution video generation.Source: §1 · p. 2
• Window-restricted spatial and spatiotemporal attention alternates within transformer blocks to reduce computation while modeling image and video structure.Source: §1 · p. 2
• W.A.L.T. reports state-of-the-art results on UCF-101, Kinetics-600, and ImageNet without classifier free guidance.Source: §1 · p. 2
• A three-model cascade generates photorealistic text-to-video outputs at 512 × 896 resolution and 8 frames per second, with state-of-the-art zero-shot FVD on UCF-101.Source: §1 · p. 2

• Video diffusion models operate in pixel or latent space, with latent approaches providing important efficiency advantages for video modeling.Source: §2 · p. 2
• Joint training on image and video data has been used to improve text-conditioned video generation and leverage larger image datasets.Source: §2 · p. 2
• Transformer generative models include autoregressive and diffusion systems for images and videos, but autoregressive video sampling is typically impractical because sequences are very long.Source: §2 · p. 2
• W.A.L.T. provides an empirical demonstration of a transformer backbone for jointly training image and video latent diffusion models.Source: §2 · p. 3

• Gaussian diffusion gradually adds noise to real data and trains a reverse process that denoises samples from a noise distribution.Source: §3 · p. 3
• The noise schedule γ(t) decreases monotonically from 1 to 0 as diffusion time t ranges from 0 to 1.Source: §3 · p. 3
• The denoiser fθ uses conditioning such as class labels or text prompts and may predict noise, the denoised input, or velocity v; the experiments use v-prediction.Source: §3 · p. 3
• Latent diffusion encodes high-resolution videos into lower-dimensional continuous latent representations, then decodes those latents into reconstructions.Source: §3 · p. 3
• Adversarial and perceptual losses can improve reconstruction quality in the decoder.Source: §3 · p. 3

Learning Visual Tokens

• A causal encoder independently tokenizes the first frame, allowing static images to share a unified spatiotemporally compressed latent space with videos.The resulting latent tensors can represent either a video or a stack of independent images, and are real-valued and quantization-free.Source: §4.1 · pp. 3–4

Learning to Generate Images and Videos

• Patchification converts each latent frame into non-overlapping patches, while learnable embeddings encode spatial and temporal positions.Images receive the temporal embedding associated with the first latent frame.Source: §4.2 · p. 4
• Window attention alternates spatial windows over individual frames with spatiotemporal windows over latent-frame neighborhoods to model spatial and temporal relationships efficiently.For images, an identity attention mask passes image-frame value embeddings through spatiotemporal layers unchanged.Source: §4.2 · p. 4

High-Resolution Generation

• W.A.L.T. achieves computational efficiency, supports joint image-video training, and avoids flickering artifacts associated with independently encoded video frames.For high-resolution video, the base model generates 128 × 128 videos that are upsampled twice by two super-resolution stages.Source: §4.2 · p. 4, §4.5 · p. 5

Conditional Generation

• The model supports conditioning on timesteps and external information through cross-attention, adaptive normalization, and self-conditioning mechanisms.Text-conditioned generation uses cross-attention, while AdaLN parameters are regressed from condition and timestep embeddings.Source: §4.3 · p. 4
• AdaLN-LoRA reduces a ViT-g model’s MLP parameters from 475M to 12M when r = 2.The reduction follows from using a low-rank parameterization with r much smaller than dmodel.Source: §4.3 · p. 4

Visual Generation

• W.A.L.T significantly outperforms prior works on UCF-101 video generation and Kinetics-600 video prediction using 50 DDIM inference steps.It achieves state-of-the-art performance with fewer model parameters than prior video diffusion models.Source: §5.1 · p. 5
• W.A.L.T outperforms prior ImageNet image-generation methods without specialized schedules, convolutional inductive bias, improved diffusion losses, or classifier-free guidance.VDM++ has a slightly better FID but uses significantly more parameters, namely 2B.Source: §5.1 · p. 5

Ablation Studies

• Reducing patch size improves performance, while full self-attention is unnecessary for good performance in the ablation studies.The default ablations evaluate spatial and spatiotemporal window configurations on UCF-101 using FVD and inception scores.Source: §5.2 · p. 6, Table 3 · p. 6
• Increasing self-conditioning rate from 0.0 to 0.9 improves FVD by 44%, while increasing AdaLN-LoRA bottleneck dimension improves performance and model size.The AdaLN-LoRA bottleneck provides a trade-off between parameter count and generation performance.Source: §5.2 · p. 6
• Enforcing zero terminal SNR improves video-generation performance, and latent dimension c = 8 provides a strong operating point across evaluated datasets and tasks.Increasing latent dimension improves reconstruction quality, but both lower and higher values can produce poor generation FVD scores.Source: §5.2 · p. 6

• W.A.L.T. is a transformer-based framework for latent video diffusion.The framework is described as simple, scalable, and efficient.Source: §6 · p. 8
• Windowed attention forms the transformer backbone for image and video generation.Source: §6 · p. 8
• A three-model cascade synthesizes high-resolution, temporally consistent photorealistic videos from natural-language descriptions.Source: §6 · p. 8

Training configuration

• The first stage follows Yu et al.'s architecture and hyperparameters, while the second stage uses specified spatial and spatiotemporal windows.The second-stage settings include 1 × 16 × 16 spatial and 5 × 8 × 8 spatiotemporal windows.Source: A. Implementation Details · p. 12

Aspect-ratio finetuning

• Aspect-ratio finetuning adapts a square-trained base stage to generate videos with a 9 : 16 aspect ratio.The procedure interpolates position embeddings and scales window sizes.Source: A. Implementation Details · p. 12

Long video generation

• For long video generation, the model generates 17 initial frames and uses encoded recent frames as context for autoregressive continuation.The last 5 frames are encoded into 2 latent frames to help maintain motion continuity.Source: A. Implementation Details · p. 12

UCF-101 evaluation

• UCF-101 text-to-video evaluation follows prior work while adapting prompts to better describe the dataset classes.Source: A. Implementation Details · p. 12

ImageNet results

• W.A.L.T. performs better than prior works on Inception Score and achieves competitive FID scores.The comparison concerns class-conditional image generation at 256 × 256 resolution.Source: §B.1 · p. 12

Evaluation setup

• ImageNet class-conditional generation is evaluated at 256 × 256 using the ADM protocol and classifier-free guidance.Source: Table 7 · p. 12

Qualitative results

• Figure 5 presents qualitative samples for ImageNet class-conditional generation.Source: Figure 5 · p. 12

Image-to-video method

• Joint frame-prediction training conditions on 1 latent frame to predict subsequent frames.This leverages the high-quality first frame from the image generator as context.Source: §B.3 · p. 12

Implementation

• The section includes training and evaluation hyperparameters for the image-to-video task.Source: Table 8 · p. 13

Qualitative results

• Figure 6 shows Kinetics-600 frame-prediction samples with ground truth above and generation below.Unobserved ground-truth frames are shaded.Source: Figure 6 · p. 13