PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Video Diffusion Models for High-Fidelity Long Video Generation

Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, Qifeng Chen

arXiv:2211.13221v2cs.CVcs.AI

TL;DR

Photorealistic video synthesis remains difficult because existing methods struggle with fidelity, length, or computational cost. The paper introduces latent-space video diffusion with hierarchical generation, conditional latent perturbation, and unconditional guidance, achieving strong short- and long-video results while generating videos beyond one thousand frames.

  • Problem

    Photorealistic video synthesis remains challenging because existing methods provide unsatisfactory visual quality or video length, while diffusion-based video synthesis requires substantial computational resources.

  • Method

    LVDM compresses videos into a low-dimensional 3D latent space, then uses hierarchical latent diffusion, conditional latent perturbation, and unconditional guidance for long-video generation.

  • Results

    The model achieves state-of-the-art results on three benchmarks in short- and long-video settings and generates videos with more than one thousand frames.

  • Takeaways & Limitations

    The framework provides an efficient video-diffusion baseline that supports long-video generation and extends to open-domain text-to-video generation.

Abstract

from arXiv · show

AI-generated content has attracted lots of attention recently, but photo-realistic video synthesis is still challenging. Although many attempts using GANs and autoregressive models have been made in this area, the visual quality and length of generated videos are far from satisfactory. Diffusion models have shown remarkable results recently but require significant computational resources. To address this, we introduce lightweight video diffusion models by leveraging a low-dimensional 3D latent space, significantly outperforming previous pixel-space video diffusion models under a limited computational budget. In addition, we propose hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. To further overcome the performance degradation issue for long video generation, we propose conditional latent perturbation and unconditional guidance that effectively mitigate the accumulated errors during the extension of video length. Extensive experiments on small domain datasets of different categories suggest that our framework generates more realistic and longer videos than previous strong baselines. We additionally provide an extension to large-scale text-to-video generation to demonstrate the superiority of our work. Our code and models will be made publicly available.

1. Introduction

Video synthesis is computationally difficult because videos are high-dimensional and their real-world distributions are statistically complex. LVDM addresses these challenges with latent-space diffusion, hierarchical long-video generation, and techniques that mitigate degradation during extension.

  • Video synthesis is challenging and computationally expensive because videos are high-dimensional and their real-world distributions are statistically complex.
  • LVDM introduces an efficient diffusion-based video-generation approach by first compressing videos into tight, low-dimensional latents.
  • A hierarchical framework operates in video latent space to generate videos longer than the temporal training length.
  • The model achieves state-of-the-art results on three benchmarks for both short and long video generation settings.
  • The framework also provides results for open-domain text-to-video generation, indicating effectiveness and generalization of the models.
  • Conditional latent perturbation and unconditional guidance mitigate performance degradation during long-video generation.

2. Related Work

Prior video-generation methods used GANs, autoregressive models, VAEs, normalizing flows, and pixel-space diffusion, but faced limitations in scalability, fidelity, or computational cost. LVDM compresses videos into compact latent representations and extends them with hierarchical diffusion and guidance techniques.

  • GAN-based video methods face mode collapse and training instability, making them difficult to scale to complex and diverse video distributions.
  • Diffusion models offer stable training and scalability, yet prior video diffusion approaches perform denoising in pixel space and require substantial computational resources.
  • LVDM extends latent image diffusion to video using a 3D auto-encoder for video compression.
  • The framework adds hierarchical latent diffusion, conditional perturbation, and unconditional guidance to support longer videos and reduce performance degradation.

3. Method

LVDM performs video diffusion in a compressed 3D latent space, supporting unified unconditional and conditional generation. Hierarchical generation, conditional latent perturbation, and unconditional guidance address long-video extension and accumulated quality degradation.

  • Video Autoencoder: A lightweight 3D video autoencoder compresses videos into lower-dimensional latent representations for diffusion and reconstruction.The encoder uses 3D convolutions and spatial and temporal downsampling; the decoder reconstructs the video latent.
  • Base LVDM: Diffusion is performed on compressed video latents, with a spatial-temporal factorized 3D U-Net estimating noise during denoising.The backbone uses space-only 3D convolutions and temporal attention in selected layers.
  • Hierarchical LVDM for Long Video Generation: A unified conditional latent diffusion model predicts future latent codes from preceding ones, enabling autoregressive video extension beyond the training length.Binary masks identify conditional frames and frames to predict, while random mask patterns jointly train unconditional generation and conditional prediction.
  • Hierarchical LVDM for Long Video Generation: Hierarchical generation first models sparse-frame storylines and then interpolates missing frames to reduce the number of autoregressive prediction steps.The interpolation model masks middle frames between sparse frames during training.
  • Hierarchical LVDM for Long Video Generation: Conditional latent perturbation adds controlled noise to conditioning latents during training and sampling to mitigate errors inherited from previous generation steps.A maximum timestep threshold limits the perturbation to a minor noise level so conditional information remains preserved.
  • Hierarchical LVDM for Long Video Generation: Unconditional guidance combines unconditional and conditional scores to guide sampling and alleviate quality degradation during autoregressive long-video generation.The guidance strength is represented by w, and the unconditional score is unaffected by accumulated autoregressive error.

4. Experiments

Experiments evaluate LVDM for unconditional short and long video generation across multiple datasets, including efficiency against pixel-space diffusion models and quality against strong baselines. LVDM achieves strong short-video results and slower long-video quality degradation than TATS.

  • Experimental Setup: LVDM is evaluated on UCF-101, Sky Time-lapse, and Taichi for unconditional video generation using FVD and KVD metrics.Short-video evaluation uses 16-frame clips; models are trained at 256^2 resolution.
  • Efficiency Comparison: LVDM achieves better FVD than pixel-space VDM and MCVD under similar training time, parameter counts, and hardware.The comparison uses approximately 4.5 days of training on 8 V100s.
  • Short Video Generation: FVD on Sky-Timelapse decreases from 116.5 to 95.18 at 256^2 resolution, while LVDM reaches new state-of-the-art results on UCF-101 and Taichi at the same resolution.At 128^2 resolution, LVDM also surpasses prior methods on Sky-Timelapse and is comparable to the best results on UCF-101 and Taichi.
  • Short Video Generation: LVDM produces higher-fidelity and more diverse samples than DIGAN and TATS in qualitative short-video comparisons.DIGAN shows coordinate-like artifacts, while TATS produces flatter and less diverse content.
  • Long Video Generation: For 1024-frame videos, LVDM outperforms TATS in both autoregressive and hierarchical settings across two datasets.The comparison reports lower FVD scores and slower quality degradation over time for LVDM.

5. Extension for Text-to-Video Generation

The paper extends LVDM to controllable open-domain text-to-video generation by scaling the model and leveraging pretrained text-to-image spatial parameters. An ablation study examines conditional latent perturbation and unconditional guidance.

  • Text-to-Video Extension: LVDM is scaled to billion-level parameters and trained on a 2 million-video subset of WebVid for text-to-video generation.The extension targets more controllable, open-domain generation.
  • Ablation Study: The ablation experiments study conditional latent perturbation and unconditional guidance.These techniques are proposed to mitigate performance degradation during long-video generation.
  • Text-to-Video Extension: Spatial convolutions and spatial attention are initialized from pretrained Stable Diffusion weights, while temporal modules are learned for video generation.This reuses pretrained spatial content-generation capacity while adapting the model to motion dynamics.

6. Conclusion

The conclusion presents LVDM as an efficient latent-space diffusion framework for short and long video generation, including videos exceeding one thousand frames. It reports state-of-the-art results and identifies future work in architecture design and sampling efficiency.

  • Conclusion: LVDM reduces video data dimensionality and speeds up diffusion-model training and sampling through an efficient latent-space framework.The approach is intended as a strong baseline for future video-generation research.
  • Conclusion: Hierarchical latent-space generation produces videos with more than one thousand frames.The conclusion links hierarchical generation to long-video synthesis beyond the training length.
  • Conclusion: LVDM achieves new state-of-the-art results across short- and long-video settings and extends to open-domain text-to-video generation.The conclusion also reports effectiveness of unconditional guidance and conditional latent perturbation for reducing accumulated autoregressive error.
  • Conclusion: Future work includes better architecture designs and further acceleration of video diffusion training and sampling.
Loading 2211.13221v2…