Source-linked AI summary
Latte: Latent Diffusion Transformer for Video Generation
Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, Yu Qiao
TL;DR
Latte addresses the challenge of generating high-quality videos with complex spatio-temporal information by modeling video distributions in latent space with a Transformer. It introduces efficient spatial-temporal variants and identifies design practices through ablations, achieving state-of-the-art results on four video-generation benchmarks and competitive text-to-video performance.
Problem
High-quality video generation is difficult because videos combine complex spatio-temporal information with high-resolution frames.
Method
Latte encodes videos into latent features, extracts spatio-temporal tokens, applies Transformer blocks, and uses four variants that disentangle spatial and temporal dimensions.
Results
Latte achieves state-of-the-art results across four standard video-generation benchmarks and competitive results on text-to-video generation.
Takeaways & Limitations
The study identifies best practices for Transformer-based latent video diffusion, including model variant 1, uniform frame patch embedding, S-AdaLN, absolute position embedding, and image-video joint training.
Abstract
from arXiv · showhide
We propose Latte, a novel Latent Diffusion Transformer for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, i.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to the text-to-video generation (T2V) task, where Latte achieves results that are competitive with recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation.
1 Introduction
Latte applies a latent-space video Transformer to the challenging problem of modeling high-dimensional spatio-temporal video data. It introduces efficient spatial-temporal decompositions, evaluates key design choices, and achieves strong benchmark and text-to-video results.
- High-quality video generation remains challenging because videos contain complex spatio-temporal information within high-resolution frames.
- Latte encodes videos into latent features, extracts tokens, and applies Transformer blocks to model their distribution.
- Four Transformer variants disentangle spatial and temporal dimensions to handle the large number of extracted video tokens.
- The study systematically examines patch embedding, model variants, timestep-class injection, temporal positional embedding, and learning strategies.
- Latte outperforms state-of-the-art methods across four standard video-generation benchmarks and achieves competitive text-to-video results.
2 Related Work
Prior video-generation research includes GAN-based extensions of image generators and broader Transformer architectures. These approaches address video synthesis through different modeling paradigms, each with documented challenges or successes.
- GAN-based video-generation methods extend powerful image generators but often encounter mode collapse.
- Transformers have achieved notable success across image restoration, vision, face recognition, and natural-language-processing tasks.
3 Methodology
Latte extends latent diffusion models to video by using a Transformer backbone over spatio-temporal latent tokens. It introduces four efficient variants and evaluates design choices for embedding, information injection, and model training.
- Latent diffusion video modeling: Latte compresses video frames into latent space, extracts tokens, and applies a Transformer to model latent spatial and temporal information.The diffusion process operates on video latents, while the Transformer predicts denoising outputs.
- Model variants: Four Latte variants decompose spatial and temporal processing through separate blocks, fusion strategies, or factorized attention.Variants include spatial-temporal blocks with interleaved or late fusion, sequential spatial-then-temporal attention, and attention split across heads.
- Model variants: The token pipeline reshapes video-latent tokens for spatial processing across each temporal index and temporal processing across frames.The token sequence receives spatio-temporal positional embeddings before these reshaping operations.
- Architectural design choices: The architecture and experiments examine efficient spatio-temporal modeling and best practices for Transformer-based latent diffusion video generation.The four variants are presented as alternatives for capturing video information efficiently, and empirical analysis targets crucial integration choices.
- Architectural design choices: Compression frame patch embedding extends ViT patch embedding into three dimensions by extracting temporal tubes, inherently incorporating spatio-temporal information.Uniform frame patch embedding instead embeds each frame individually using non-overlapping image patches.
- Architectural design choices: Timestep or class information is integrated either as tokens or through adaptive normalization, including scalable adaptive layer normalization with residual-connection scaling.S-AdaLN computes γ_c, β_c, and α_c from conditioning information c.
4 Experiments
Latte’s experiments identify design choices for Transformer-based video diffusion and evaluate the resulting model across video-generation settings. The study analyzes embedding, attention variants, conditioning, positional encoding, initialization, joint training, scale, and benchmark performance.
- Ablation study: Uniform frame patch embedding outperforms compression frame patch embedding, which may lose spatio-temporal signal.The authors report that this finding differs from video-understanding results with ViViT.
- Ablation study: S-AdaLN significantly outperforms injecting timestep-class information into all tokens.The authors attribute this to adaptive encoding in each Transformer block, which appears to improve information transmission and convergence.
- Ablation study: Absolute temporal positional embedding performs slightly better than the alternative and is selected because Latte does not require variable-duration or multi-resolution adaptability.
- Ablation study: ImageNet pre-training accelerates early training, but random initialization achieves better late-stage performance on FaceForensics.The authors relate this behavior to the distribution gap between ImageNet and FaceForensics, especially for small datasets.
- Ablation study: Image-video joint training significantly improves FID and FVD, while different video sampling intervals converge to similar performance.The selected sampling interval is 3 to maintain reasonable continuity in generated videos.
- Model variants and scale: Variant 1 performs best, and increasing Latte’s model size generally improves performance.The variant analysis links excessive consecutive spatial attention to disrupted temporal coherence, while parallel fusion performs poorly because spatial and temporal features are mismatched.
5 Conclusion
Latte is presented as a simple, general video diffusion method using a video Transformer backbone. It achieves state-of-the-art results across four video-generation benchmarks, while its text-to-video results are comparable to current approaches and may inform future Transformer-based diffusion research.
- Latte uses a video Transformer backbone in a simple and general video diffusion method.
- The paper determines best practices for clip patch embedding, model variants, timestep-class injection, temporal positional embedding, and learning strategies.
- Latte achieves state-of-the-art results across four standard video-generation benchmarks.
- The method produces text-to-video results comparable to current T2V approaches.
- Latte provides insights for integrating Transformer-based backbones into diffusion models for video generation and other modalities.
A.1 The sampled video frames
The appendix provides sampled video frames from different methods for visual comparison across four datasets.
- Sampled video frames from different methods are provided in the appendix.
- The visual samples are presented as video frames rather than full textual benchmark results.
- The samples support comparisons among methods through the videos shown in Figure 9.
A.2 The structure of S-AdaLN
The appendix shows the structure of S-AdaLN in Figure 10.
- Figure 10 presents the structure of S-AdaLN.
- The S-AdaLN visualization is included in the appendix.
- The appendix provides a structural view of the S-AdaLN component rather than a benchmark comparison.
A.3 Discussion about the difference from concurrent works
Latte is related to concurrent Transformer-based video-generation works, including methods using an architecture akin to its Variant 3. Its distinction is a systematic analysis of Transformer backbones and associated design practices for video generation.
- Discussion about the difference from concurrent works: VDT, GenTron, and W.A.L.T. use architectures akin to Latte’s Variant 3.
- Discussion about the difference from concurrent works: Concurrent works explore similar Transformer-based ideas for video generation and related video tasks.
- Discussion about the difference from concurrent works: Latte’s primary difference is its systematic analysis of different Transformer backbones for video generation.
- Discussion about the difference from concurrent works: The paper also analyzes relative best practices associated with Transformer backbones for video generation.
- Discussion about the difference from concurrent works: Figure 9 provides sampled videos from different methods on UCF101, Taichi-HD, FaceForensics, and SkyTimelapse.
- Discussion about the difference from concurrent works: Figure 10 depicts S-AdaLN and the vanilla Transformer block used in the paper’s architecture figures.