Source-linked AI summary
Video Probabilistic Diffusion Models in Projected Latent Space
Sihyun Yu, Kihyuk Sohn, Subin Kim, Jinwoo Shin
TL;DR
High-resolution, temporally coherent video synthesis is difficult, and existing video diffusion models are costly in computation and memory. PVDM learns video distributions in a factorized low-dimensional latent space using an autoencoder and specialized diffusion model, achieving strong benchmark results while supporting efficient long-video generation. Its scope remains limited by a gap between generated and real videos and by the lack of large-scale text-to-video experiments.
Problem
High-resolution and temporally coherent video generation is challenging because videos are high-dimensional and exhibit complex spatiotemporal dynamics, while video diffusion models are computation- and memory-inefficient.
Method
PVDM uses an autoencoder that factorizes videos into three 2D latent vectors and a diffusion model specialized for this image-like latent space.
Results
PVDM consistently outperforms prior video-generation methods and achieves 639.7 FVD on UCF-101 128-frame videos versus 1773.4 for the prior state-of-the-art.
Takeaways & Limitations
PVDM supports efficient high-resolution and long-video synthesis under limited computational resources, including arbitrary-length generation through joint unconditional and frame-conditional modeling.
Takeaways & Limitations
A gap remains between real and generated videos, and the authors did not conduct large-scale video experiments for challenging text-to-video generation because of limited resources.
Abstract
from arXiv · showhide
Despite the remarkable progress in deep generative models, synthesizing high-resolution and temporally coherent videos still remains a challenge due to their high-dimensionality and complex temporal dynamics along with large spatial variations. Recent works on diffusion models have shown their potential to solve this challenge, yet they suffer from severe computation- and memory-inefficiency that limit the scalability. To handle this issue, we propose a novel generative model for videos, coined projected latent video diffusion models (PVDM), a probabilistic diffusion model which learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources. Specifically, PVDM is composed of two components: (a) an autoencoder that projects a given video as 2D-shaped latent vectors that factorize the complex cubic structure of video pixels and (b) a diffusion model architecture specialized for our new factorized latent space and the training/sampling procedure to synthesize videos of arbitrary length with a single model. Experiments on popular video generation datasets demonstrate the superiority of PVDM compared with previous video synthesis methods; e.g., PVDM obtains the FVD score of 639.7 on the UCF-101 long video (128 frames) generation benchmark, which improves 1773.4 of the prior state-of-the-art.
1. Introduction
Video synthesis remains difficult because videos combine high-dimensional pixels with complex spatiotemporal dynamics, while video diffusion models are computationally and memory inefficient. PVDM addresses these challenges by learning video distributions in a factorized 2D latent space and achieves strong quality and efficiency results.
- High-resolution video generation remains challenging because videos contain high-dimensional data and complex spatiotemporal dynamics.
- Latent diffusion models improve image-generation efficiency, but latent diffusion for video remained overlooked.
- PVDM introduces a two-stage video latent diffusion framework combining an autoencoder with a diffusion model operating in projected latent space.
- The autoencoder factorizes videos into three 2D image-like latent vectors, representing common temporal content and motion across spatial directions.
- PVDM uses a 2D convolutional diffusion architecture instead of computation-heavy 3D convolutions and jointly trains unconditional and frame-conditional generation for arbitrary-length videos.
- 74.40 IS on UCF101 16-frame 256×256 videos and 639.7 FVD on UCF-101 128-frame videos, improving the prior state-of-the-art from 1773.4.On a single NVIDIA 3090Ti 24GB GPU, PVDM supports batch size 7 for 16-frame 256×256 videos, whereas a competing video diffusion model nearly fills memory at 128×128 with batch size 1.
2. Related work
Prior video-generation approaches include GAN extensions and diffusion models, while latent-space approaches reduce the cost of distribution modeling. The related work motivates PVDM as a video diffusion model built around low-dimensional latent representations.
- Video-generation research has included extensions of image GANs, but these methods often suffer from mode collapse and scaling difficulty on complex datasets.
- Diffusion models learn data distributions through iterative denoising and have shown strong sample quality and mode coverage in image synthesis.
- Latent diffusion amortizes data in a low-dimensional latent space, reducing computation while retaining strong generation quality.
3. Projected latent video diffusion model
PVDM learns video distributions in a factorized low-dimensional latent space, replacing conventional 3D video representations with three 2D latent vectors and a diffusion model. A single model jointly supports unconditional and conditional generation, enabling arbitrary-length videos by sequentially generating and concatenating clips.
- Diffusion model: PVDM applies a 2D convolutional diffusion architecture to the image-like latent space, avoiding computation-heavy 3D convolutional architectures.The latent-space diffusion model is trained as a denoising autoencoder rather than operating repeatedly in the high-dimensional input space.
- Autoencoder: The autoencoder maps videos into latent codes and reconstructs them using pixel-level reconstruction and perceptual-similarity objectives.It uses an encoder f_φ and decoder g_ψ, with discrete latent codes motivated by perceptual image compression.
- Autoencoder: PVDM represents each video with three 2D latent vectors instead of a conventional 3D latent tensor.The vectors are z_s ∈ R^(C×H′×W′), z_h ∈ R^(C×S×W′), and z_w ∈ R^(C×S×H′).
- Autoencoder: The shared-content vector z_s captures time-invariant content, while z_h and z_w encode motion along two spatial axes.This factorization exploits temporal coherence and represents video motion with two spatial grids.
- Long video generation: A single diffusion model jointly learns unconditional p(x) and conditional p(x₂|x₁) generation by introducing a null frame for unconditional training.This avoids maintaining separate unconditional and conditional models.
- Long video generation: Long videos are generated by sampling an initial clip, repeatedly sampling future clips conditioned on the previous clip, and concatenating the results.The procedure supports arbitrary length L > 1.
4. Experiments
Experiments evaluate PVDM on UCF101 and SkyTimelapse using standard video-generation metrics, qualitative comparisons, reconstruction quality, and efficiency measurements. PVDM produces realistic, temporally coherent videos while improving generation quality and computational efficiency over prior methods.
- Experiment setup: PVDM is evaluated on UCF101 and SkyTimelapse using 16- or 128-frame clips at 256×256 resolution.Training uses the train split, while evaluation reports IS and FVD under fixed protocols.
- Experiment setup: Evaluation reports IS for 16-frame unconditional generation and FVD16/FVD128 for 16- and 128-frame clips, respectively.IS is higher-is-better, whereas FVD is lower-is-better; 10,000 clips are generated for IS and 2,048 real/fake clips for FVD.
- Qualitative results: PVDM produces realistic 256×256 videos on UCF-101 and SkyTimelapse, including small-object motion and large frame-to-frame transitions.The method also generates plausible videos on the complex UCF-101 dataset, where other baselines often fail.
- Quantitative results: PVDM consistently outperforms prior video-generation methods across diverse quantitative metrics and achieves state-of-the-art results for 128-frame videos.The reported comparisons include UCF-101 and SkyTimelapse, with particular improvement on the complicated, multi-class UCF-101 dataset.
- Reconstruction quality: The autoencoder produces accurate reconstructions similar to ground-truth videos, with sufficiently small R-FVD indicating preserved perceptual similarity after compression.Reconstruction quality is assessed using PSNR and R-FVD; Figure 5 visualizes frames with stride 4.
- Efficiency analysis: PVDM achieves ≈17.6× better computation efficiency than VDM, requiring ≈7.88 seconds to synthesize a 256×256, 16-frame video versus VDM’s >2 minutes.The comparison uses the same sampler setup and measures training batch size, synthesis time, and memory on a single NVIDIA 3090Ti 24GB GPU.
5. Conclusion
PVDM learns video distributions in an image-like 2D latent space, aiming to scale video synthesis efficiently under limited computational resources.
- PVDM parameterizes videos in an image-like 2D latent space for diffusion-based distribution learning.
- The authors position PVDM as a foundation for scaling video synthesis research.
A. Detailed description of training objective
The autoencoder training objective combines reconstruction, perceptual, adversarial, and vector-quantization terms, with tanh activation used to control the latent representation.
- The autoencoder objective combines pixel reconstruction, perceptual similarity, adversarial, and vector-quantization terms.
- A tanh activation after the projection network supports latent-diffusion training without degrading autoencoder training.
- The adversarial term uses hinge loss and feature similarity between real samples and reconstructions.
- The training sets λ1 = 1 and λ3 = 0.25, while λ2 changes from 0 to 0.25 after the other terms converge.
B.1. Datasets
The evaluation uses UCF-101 and SkyTimelapse, with video-generation comparisons spanning GAN, autoregressive, and diffusion-based methods.
- Datasets: UCF-101 contains 13,320 human-action videos at 320×240 resolution, with 9,357 training videos used after resizing to 256 × 256.
- Datasets: SkyTimelapse contains 5,000 sky time-lapse videos, center-cropped and resized to 256 × 256 for training and evaluation.
- Evaluation: Inception score evaluation on UCF-101 uses a C3D network and 10,000 generated samples.
- Evaluation: FVD evaluation follows the fixed protocol proposed by StyleGAN-V, sampling videos before selecting fixed-length clips.
- Compared methods: Additional baselines use motion-content decomposition, progressive generation, low-dimensional discriminator kernels, Transformers, or separate spatial and temporal discriminators.
- Compared methods: The compared methods include video GANs, autoregressive latent models, and diffusion models with 3D convolutional architectures.
D. More details on training setup
Training uses fixed optimization settings for the autoencoder and diffusion model, alongside Transformer-based 3D-to-2D projections and configurable PVDM variants.
- Training setup: Autoencoder training uses batch size 24 and learning rate 1e-4 until FVD and PSNR converge.
- Training setup: The 3D-to-2D projections use a 4-layer Transformer with 4 heads, hidden dimension 384, and MLP dimension 512.
- Training setup: The latent codebook dimension is set to 4.
- Model configurations: Table 6 reports model configurations for the PVDM-S and PVDM-L variants.
E. Qualitative comparison of generated results
PVDM produces overall high-quality first-frame results compared with recent video synthesis baselines, particularly on the complex UCF-101 dataset.
- PVDM shows overall high-quality synthesis results compared with other recent video synthesis methods, especially on UCF-101.The qualitative comparison concerns generated first frames, with baseline results drawn from the StyleGAN-V website.
F. More analysis in our autoencoder architecture
The autoencoder analysis examines how projected 2D-shaped latents capture perceptual detail and temporal coherence, while contrasting PVDM with concurrent latent video diffusion methods.
- Ablation study on our autoencoder: The projected autoencoder components improve perceptual-detail capture, achieving better R-FVD than alternative variants at equal model sizes.The ablation is conducted on UCF-101 using equalized model sizes.
- Ablation study on our autoencoder: The 2D autoencoder degrades substantially at dim(z) = 8,192 because it lacks temporal layers for capturing temporal coherence.
- Comparison with concurrent work: MagicVideo encodes videos frame-wise, increasing latent dimension with length and limiting scalability, unlike PVDM.
- Comparison with concurrent work: LVDM uses 3D CNNs and cubic-shaped latent tensors, whereas PVDM uses projected 2D-shaped latents for diffusion modeling.
H. Limitations and future work
PVDM remains separated from real videos, and its performance on large-scale text-to-video generation has not been experimentally evaluated because of limited resources.
- A gap remains between real and generated videos, motivating better diffusion architectures and more efficient latent structures.
- Large-scale video datasets and challenging text-to-video generation remain experimentally untested because of limited resources.The authors leave this evaluation and application direction for future work.