PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, Hang Zhao

arXiv:2310.04378v1cs.CVcs.LG

TL;DR

Latent diffusion models generate high-resolution images well but require slow iterative sampling. LCMs distill guided diffusion into latent-space consistency models that predict PF-ODE solutions directly, enabling few-step inference, while LCF adapts them to customized datasets. The method uses only 32 A100 GPU hours for 2- or 4-step inference and achieves state-of-the-art performance on LAION-5B-Aesthetics.

  • Problem

    Iterative reverse sampling makes high-quality latent diffusion image generation computationally intensive and slow.

  • Method

    LCMs apply one-stage guided distillation in Stable Diffusion’s latent space by solving an augmented PF-ODE, and LCF adapts pretrained LCMs to customized datasets.

  • Results

    LCMs achieve state-of-the-art performance on LAION-5B-Aesthetics with 2- or 4-step inference using 32 A100 GPU hours of training.

  • Takeaways & Limitations

    LCMs provide high-fidelity few-step or one-step sampling for pretrained latent diffusion models, including Stable Diffusion.

  • Takeaways & Limitations

    Future work must extend LCMs to text-guided image editing, inpainting, and super-resolution.

Abstract

from arXiv · show

Latent Diffusion models (LDMs) have achieved remarkable results in synthesizing high-resolution images. However, the iterative sampling process is computationally intensive and leads to slow generation. Inspired by Consistency Models (song et al.), we propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs, including Stable Diffusion (rombach et al). Viewing the guided reverse diffusion process as solving an augmented probability flow ODE (PF-ODE), LCMs are designed to directly predict the solution of such ODE in latent space, mitigating the need for numerous iterations and allowing rapid, high-fidelity sampling. Efficiently distilled from pre-trained classifier-free guided diffusion models, a high-quality 768 x 768 2~4-step LCM takes only 32 A100 GPU hours for training. Furthermore, we introduce Latent Consistency Fine-tuning (LCF), a novel method that is tailored for fine-tuning LCMs on customized image datasets. Evaluation on the LAION-5B-Aesthetics dataset demonstrates that LCMs achieve state-of-the-art text-to-image generation performance with few-step inference. Project Page: https://latent-consistency-models.github.io/

1 INTRODUCTION

Latent Consistency Models address the slow iterative sampling of latent diffusion models by enabling high-fidelity image generation in only a few steps. The paper introduces efficient guided distillation and fine-tuning methods for pretrained Stable Diffusion models.

  • LCMs use consistency models in image latent space to accelerate high-resolution text-to-image generation.
  • Latent Consistency Models enable fast few-step or one-step high-fidelity sampling from pretrained latent diffusion models such as Stable Diffusion.
  • The method distills Stable Diffusion through a one-stage guided consistency procedure for 2∼4-step or one-step sampling.
  • 32 A100 GPU hours of training support 2- and 4-step inference while achieving state-of-the-art performance on LAION-5B-Aesthetics.
  • Latent Consistency Fine-tuning adapts pretrained LCMs to customized datasets while preserving fast inference.

2 RELATED WORK

Prior work established diffusion models as high-quality but slow generators, while latent diffusion reduced computation for high-resolution synthesis. Consistency models offered faster sampling, but their high-resolution text-to-image potential remained unexplored.

  • Diffusion models generate samples through iterative reverse denoising, which creates a bottleneck in generation speed.
  • Existing acceleration methods include training-free solvers and predictor-corrector techniques, plus training-based discretization, truncation, neural operators, and distillation.
  • Latent diffusion models perform diffusion in a compressed data latent space, improving computational efficiency for high-resolution text-to-image synthesis.
  • Consistency models learn mappings from PF-ODE trajectory points to their origin, enabling one-step generation while preserving generation quality.

3 PRELIMINARIES

The preliminaries describe diffusion sampling through stochastic and probability-flow dynamics, classifier-free guidance, and consistency mappings. These concepts provide the basis for distilling fast latent consistency models.

  • Diffusion models progressively add Gaussian noise and generate samples by reversing the denoising process.
  • The forward diffusion process can be represented as a continuous-time stochastic differential equation with a noise schedule and Brownian motion.
  • The Probability Flow ODE describes the evolution of diffusion marginals and provides a deterministic formulation for sampling.
  • Diffusion models train a noise predictor to approximate the negative score function, which yields an empirical PF-ODE for sampling.
  • Classifier-Free Guidance combines conditional and unconditional noise predictions using a guidance scale to improve generated sample quality.
  • Consistency models learn a function mapping any PF-ODE trajectory point to its origin and enforce self-consistency with a parameterized network and EMA target model.

4 LATENT CONSISTENCY MODELS

Latent Consistency Models extend consistency modeling to Stable Diffusion’s latent space, directly predicting augmented PF-ODE solutions for fast, guided few-step image generation. The method combines latent consistency distillation, one-stage classifier-free-guided distillation, SKIPPING-STEP training, and fine-tuning for customized datasets.

  • 4 LATENT CONSISTENCY MODELS: LCMs apply consistency models in the image latent space of Stable Diffusion to target high-resolution text-to-image generation in 2–4 or even 1 step.The approach addresses the unexplored use of consistency models for higher-resolution text-to-image tasks.
  • 4.1 CONSISTENCY DISTILLATION IN THE LATENT SPACE: The latent consistency function maps a noisy image latent, text condition, and time directly to the PF-ODE solution at t = 0.The function is parameterized using the teacher diffusion model’s noise prediction model.
  • 4.3 ACCELERATING DISTILLATION WITH SKIPPING TIME STEPS: The one-stage guided distillation procedure uses DDIM, DPM-Solver, or DPM-Solver++ during training, while these solvers are not used during inference.The method adapts to Stable Diffusion’s discrete 1,000-step schedule and uses solver-based skipping during distillation.
  • 4.2 ONE-STAGE GUIDED DISTILLATION BY SOLVING AUGMENTED PF-ODE: LCMs integrate classifier-free guidance by learning an augmented consistency function conditioned on the guidance scale and solving the corresponding augmented PF-ODE.The guided noise model combines conditional and unconditional predictions, while the augmented consistency model adds guidance-scale conditioning.
  • 4.3 ACCELERATING DISTILLATION WITH SKIPPING TIME STEPS: SKIPPING-STEP training accelerates convergence, and LCM surpasses baselines in the 1–4 step region at both 512×512 and 768×768 resolutions.The reported comparisons use DDIM-Solver with skipping step k = 20 on the LAION-Aesthetic datasets.
  • 4.4 LATENT CONSISTENCY FINE-TUNING FOR CUSTOMIZED DATASET: Latent Consistency Fine-tuning adapts a pretrained LCM to customized datasets while retaining few-step inference without requiring a teacher diffusion model trained on those datasets.The method is presented as an alternative to traditional diffusion-model fine-tuning.

5 EXPERIMENT

The experiments evaluate LCMs for text-to-image generation, solver and skipping-step choices, guidance scales, and customized-dataset adaptation. Across these settings, LCMs support efficient few-step generation while retaining strong image quality and enabling customized styles.

  • Text-to-Image Generation: LCMs notably outperform baseline methods at 512×512 and 768×768 resolutions, especially in the 1∼4-step regime.The baselines include DDIM, DPM, DPM++, and Guided-Distill.
  • Text-to-Image Generation: LCM requires only one forward pass per sampling step, saving time and memory relative to CFG-based DDIM, DPM, and DPM++ sampling.LCM also uses one-stage rather than Guided-Distill’s two-stage guided distillation.
  • ODE Solvers & Skipping-Step Schedule: Skipping-step schedules with k = 5, 10, or 20 accelerate convergence, while very small k values converge slowly and very large values can degrade DDIM results.The main experiments choose k = 20 because it provides competitive performance across the three solvers.
  • The Effect of Guidance Scale ω: Larger guidance scales ω improve CLIP-based sample quality but can worsen FID, while performance gaps between 2, 4, and 8 steps remain negligible.One-step inference shows a noticeable gap, leaving room for improvement.
  • Downstream Consistency Fine-Tuning Results: Latent Consistency Fine-tuning adapts pretrained LCMs to Pokemon and Simpsons datasets, producing customized styles with few-step inference.Each dataset contains hundreds of customized text-image pairs, with 90% used for fine-tuning and 10% for testing.

6 CONCLUSION

The paper presents LCMs and one-stage guided distillation for few-step or one-step inference on pretrained latent diffusion models. It also introduces LCF for few-step generation on customized image datasets, with experiments demonstrating LCM performance and efficiency.

  • LCMs apply consistency models in latent space to enable few-step or even one-step inference on pretrained latent diffusion models.
  • The one-stage guided distillation method supports efficient inference on pretrained latent diffusion models, while LCF adapts LCMs to customized image datasets.
  • Experiments on the LAION-5B-Aesthetics dataset demonstrate the superior performance and efficiency of LCMs.
  • Future work includes extending LCMs to text-guided image editing, inpainting, and super-resolution.

A.1 DIFFUSION MODELS

Diffusion models formulate generation as reverse-time stochastic dynamics, while the equivalent PF-ODE preserves marginals and supports deterministic sampling. Consistency distillation then trains latent models to predict trajectory origins directly, with iterative refinement for high-quality few-step generation.

  • Reverse-time SDE sampling starts from an approximately Gaussian terminal distribution but typically requires many discretization steps and is slow.
  • The PF-ODE has the same marginal distribution as the diffusion SDE and enables ODE-based sampling.
  • Consistency distillation maintains an EMA target model and minimizes a distance between consistency predictions obtained at paired time steps.
  • Latent consistency sampling directly predicts the origin z0 of the augmented PF-ODE trajectory instead of predicting zt−1 from zt.
  • Alternating noise injection and denoising refinement produces high-quality images in 2–4 steps, with the second iteration substantially improving generation quality in the reported experiments.
  • Latent consistency fine-tuning applies the same Gaussian noise at two separated time steps and enforces self-consistency without requiring the pretrained teacher diffusion model.

D DIFFERENT WAYS TO PARAMETERIZE THE CONSISTENCY FUNCTION

The consistency function can be parameterized according to the teacher diffusion model’s prediction target. The paper uses epsilon prediction at 512×512 and v-prediction at 768×768, with several PF-ODE solvers for distillation.

  • For epsilon prediction, the consistency function is parameterized according to the teacher diffusion model’s epsilon-prediction output.
  • For x-prediction, fθ combines cskip(t)z with cout(t)xθ(zt, c, t), where xθ is the teacher’s data-prediction model.
  • For v-prediction, fθ combines cskip(t)z with a data estimate formed from αtzt − σtvθ(zt, c, t).
  • The 512×512 LCM uses epsilon parameterization with Stable-Diffusion-V2.1-Base, whereas the 768×768 LCM uses v-parameterization with Stable-Diffusion-V2.1.
  • DDIM, DPM-Solver, and DPM-Solver++ are used as PF-ODE solvers, with the reported DPM-Solver and DPM-Solver++ cases restricted to order 2.

F TRAINING DETAILS OF LATENT CONSISTENCY DISTILLATION

Training uses resolution-matched Stable Diffusion teachers and distinct LAION aesthetics datasets for 512×512 and 768×768 LCM distillation.

  • The 512×512 setting uses LAION-Aesthetics-6+, containing 12M text-image pairs with predicted aesthetics scores above 6.
  • The 768×768 setting uses LAION-Aesthetic-6.5+, containing 650K text-image pairs with predicted aesthetics scores above 6.5.
  • The 512×512 LCM is trained from Stable-Diffusion-V2.1-Base, originally trained at 512×512 with epsilon prediction.
  • The 768×768 LCM is trained from Stable-Diffusion-V2.1, originally trained at 768×768 with v-prediction.

G REPRODUCTION DETAILS OF GUIDED-DISTILL

Guided-Distill is reproduced as a comparison baseline using a two-stage distillation procedure, with reduced batch sizes to fit available computational resources. Additional 768×768 four-step LCM results are shown after 4,000 training iterations.

  • Guided-Distill is reproduced as a guided-distillation baseline, although its implementation is not open-sourced.
  • The reproduction reduces Guided-Distill’s 512-resolution batch size from 512 to 72 because of limited computational resources.
  • Guided Distill uses two distillation stages, beginning with a student fitted to outputs of a pretrained guided diffusion model across classifier-free guidance scales.
  • 4,000 training iterations are used for the additional Dreamer-V7 SD distillation results with LCM four-step inference at 768×768 resolution.

H MORE FEW-STEP INFERENCE RESULTS

LCM synthesizes 768×768 images with only 2- or 4-step inference and can be derived from pre-trained Stable Diffusion in 4,000 training steps.

  • LCM synthesizes high-resolution 768×768 images using only 2 or 4 inference steps.
  • 4,000 training steps, equivalent to around 32 A100 GPU hours, suffice to derive LCM from any pre-trained Stable Diffusion model.
  • Figure 8 presents additional 768×768 image results generated with LCM using 2-step inference after distilling Dreamer-V7 SD.
Loading 2310.04378v1…