VideoLCM: Video Latent Consistency Model

Xiang Wang; Shiwei Zhang; Han Zhang; Yu Liu; Yingya Zhang; Changxin Gao; Nong Sang

Source-linked AI summary

VideoLCM: Video Latent Consistency Model

Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, Nong Sang

arXiv:2312.09109v1Dec 14, 2023cs.CVcs.AI

TL;DR

Video diffusion can require many sampling steps, and consistency models remain relatively unexplored for video generation. VideoLCM applies consistency distillation to latent video diffusion models; it reports high-fidelity synthesis in four steps versus approximately 50 DDIM steps, with potential for real-time synthesis.

Problem
Video diffusion methods can require substantial sampling, while consistency models’ use in video synthesis remains largely unexplored.
Source: §1 · p. 2
Method
VideoLCM builds on latent video diffusion models and uses consistency distillation to train a video latent consistency model.
Source: §1 · p. 2, §5 · p. 8
Results
Four steps yield high-fidelity video synthesis with VideoLCM, compared with approximately 50 DDIM steps for prior methods.
Source: §5 · p. 8
Takeaways & Limitations
The authors present VideoLCM as a simple, effective baseline for subsequent research, with potential for fast video synthesis.
Source: §1 · p. 2, §5 · p. 8
VideoLCM requires a strong teacher and finetuning, and despite reducing inference to 4–6 steps, it does not achieve real-time video generation.
Source: §4.5 · pp. 7–8

Abstract

from arXiv · show

Consistency models have demonstrated powerful capability in efficient image generation and allowed synthesis within a few sampling steps, alleviating the high computational cost in diffusion models. However, the consistency model in the more challenging and resource-consuming video generation is still less explored. In this report, we present the VideoLCM framework to fill this gap, which leverages the concept of consistency models from image generation to efficiently synthesize videos with minimal steps while maintaining high quality. VideoLCM builds upon existing latent video diffusion models and incorporates consistency distillation techniques for training the latent consistency model. Experimental results reveal the effectiveness of our VideoLCM in terms of computational efficiency, fidelity and temporal consistency. Notably, VideoLCM achieves high-fidelity and smooth video synthesis with only four sampling steps, showcasing the potential for real-time synthesis. We hope that VideoLCM can serve as a simple yet effective baseline for subsequent research. The source code and models will be publicly available.

1. Introduction

Video diffusion models can require many sampling steps, while consistency models have been little explored for video. VideoLCM applies consistency distillation to video generation and reports high-fidelity synthesis with fewer steps.

Video diffusion methods can require substantial sampling, such as 50-step DDIM, and consistency models remain largely unexplored for video synthesis.Source: §1 · p. 2
High-fidelity video synthesis takes 4–6 steps with VideoLCM, compared with approximately 50 DDIM steps for prior methods.Compositional depth-to-video synthesis can use as few as 1 step.Source: §1 · p. 2
VideoLCM builds on latent video diffusion models and uses consistency distillation to train a video latent consistency model.Source: §1 · p. 2
The authors present VideoLCM as a simplified, effective baseline for future fast-video-synthesis research.Source: §1 · p. 2

2. Related Work

Related work covers latent-space image generation, consistency models, and video-generation approaches. VideoLCM’s pipeline applies consistency distillation to video diffusion models.

Text-to-image work includes two-stage text-embedding generation, latent-space diffusion, and conditional inputs for spatial control.Examples include DALLE-2, Stable Diffusion, Control-Net, and Composer.Source: §2 · p. 2
Consistency models map points along a probability-flow ODE trajectory to its starting image, supporting one-step generation and iterative refinement.Latent Consistency Models conduct this optimization in latent space to reduce memory use and improve inference efficiency.Source: §2 · p. 2
Figure 2 depicts noising a source video, teacher and student predictions, and a consistency loss between student outputs in latent space.The teacher estimate is fed into the EMA student; conditional guidance is omitted from the figure for clarity.Source: Figure 2 · p. 3
Video-generation research moved from GANs to diffusion models, while approaches use temporal modules, super-resolution, frame interpolation, and controllable synthesis.The cited video methods still rely on extensive iterative denoising for satisfactory results.Source: §2 · pp. 2–3

3. Method

The method extends latent consistency models to video using consistency distillation. It presents the consistency-model preliminaries, VideoLCM’s training design, and its inference settings.

3.1. Preliminaries

Self-consistency requires the model to give the same output for inputs at different points along the denoising trajectory.Here, ϵ is a time step, T the overall denoising step, and x_t the noised input.Source: §3.1 · p. 3
Consistency distillation trains against an estimate produced with an ODE solver and uses an EMA of the model parameters for the target model.The solver is denoted Φ, and the EMA parameters are θ*; the estimate is ˆx_tn.Source: §3.1 · p. 3

3.2. VideoLCM

VideoLCM performs consistency optimization in latent space and distills pretrained video diffusion models, using DDIM to estimate the earlier state.The teacher can be a text-to-video or compositional video-synthesis model; conditions may be text prompts or combined signals.Source: §3.2 · pp. 3–4
Classifier-free guidance is used during distillation with a fixed guidance scale, keeping the consistency model’s structure aligned with the teacher.Guidance is applied only to the teacher during training and is not required at consistency-model inference.Source: §3.2 · p. 4
Text-to-video synthesis uses 4–6 LCM steps, while compositional synthesis can use 2–4 steps and sometimes 1 step.Source: §3.2 · p. 4

4. Experiments

Experiments evaluate VideoLCM on text-to-video and compositional synthesis through latency measurements, step-count ablations, and qualitative examples. Results show that low-step quality depends on conditioning, while training relies on teacher models and finetuning.

4.1. Experimental setup

Training uses WebVid10M and LAION-5B, with pretrained TF-T2V teacher parameters fixed during consistency distillation.The reported setup uses approximately 4k training iterations on four NVIDIA A100 GPUs.Source: §4.1 · p. 4

4.2. Time efficiency

88 seconds are saved at 16x448x256 and 50 seconds at 16 × 256 × 256 versus the 50-DDIM-step baseline [47].The comparison measures inference time for text-to-video synthesis and includes encoding and decoding overhead.Source: §4.2 · p. 4

4.3. Ablation study on inference steps

At 4–6 steps, text-to-video quality is comparable to DDIM’s 50 steps; 2–4 steps produce relatively stable compositional synthesis.For depth-to-video, structured control can support good visual quality even at one step.Source: §4.3 · p. 6

4.4. More qualitative results

Four-step visualizations show quality and temporal continuity in text-to-video, plus stable, input-controllable depth, sketch, and inpainting synthesis.The figures show text-to-video examples in Fig. 5 and compositional examples in Figs. 6–8.Source: §4.4 · pp. 6–7, Figure 5 · p. 6, Figure 6 · p. 7

4.5. Limitations

The method depends on a strong teacher and finetuning; distillation may yield unsatisfactory results when teacher-training data are unavailable or from different domains.The authors also state that reducing inference to 4–6 steps does not achieve real-time video generation.Source: §4.5 · pp. 7–8

5. Conclusion

VideoLCM extends latent consistency models to video generation by applying consistency distillation to existing latent video diffusion models. The conclusion reports high-fidelity synthesis in four sampling steps, compared with approximately 50 DDIM steps for prior methods.

VideoLCM extends latent consistency models to video generation using existing latent video diffusion models and consistency distillation.Source: §5 · p. 8
High-fidelity video synthesis takes four sampling steps, compared with approximately 50 DDIM steps for prior methods.Source: §5 · p. 8
The authors propose VideoLCM as a simple, effective baseline for subsequent research.Source: §5 · p. 8