PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Diffusion Models via Early Stop of the Diffusion Process

Zhaoyang Lyu, Xudong XU, Ceyuan Yang, Dahua Lin, Bo Dai

arXiv:2205.12524v2cs.CV

TL;DR

DDPMs can require hundreds or thousands of denoising steps, making inference slow. ES-DDPM stops the forward diffusion early and uses samples from a pre-trained generator to begin reverse denoising from a non-Gaussian distribution. Across the reported experiments, the combined model improves sample quality over vanilla DDPM and the adopted generator while reducing denoising steps.

  • Problem

    High-quality DDPM sampling often requires hundreds or thousands of iterative denoising steps, while existing acceleration approaches can compromise quality or offer limited practical speedup.

    Source: §1 · pp. 1–2

  • Method

    ES-DDPM truncates forward diffusion and uses a pre-trained GAN or VAE to provide samples for the non-Gaussian starting distribution of reverse denoising.

    Source: §1 · p. 2

  • Results

    The combined model improves sample quality over vanilla DDPM and the adopted pre-trained generator, reducing denoising from 1000 to 100 steps in the best experimental case.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ES-DDPM can be combined with other acceleration methods and, with GANs, supports both global semantic and local pixel-level control.

    Source: §6 · p. 9, §5.4 · p. 9

    On multi-category datasets, poor GAN mode coverage can require many denoising steps before sample quality surpasses full-length DDPM; more study is also needed to prevent learned biases and misuse.

    Source: §6 · p. 9

Abstract

from arXiv · show

Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive performance on various generation tasks. By modeling the reverse process of gradually diffusing the data distribution into a Gaussian distribution, generating a sample in DDPMs can be regarded as iteratively denoising a randomly sampled Gaussian noise. However, in practice DDPMs often need hundreds even thousands of denoising steps to obtain a high-quality sample from the Gaussian noise, leading to extremely low inference efficiency. In this work, we propose a principled acceleration strategy, referred to as Early-Stopped DDPM (ES-DDPM), for DDPMs. The key idea is to stop the diffusion process early where only the few initial diffusing steps are considered and the reverse denoising process starts from a non-Gaussian distribution. By further adopting a powerful pre-trained generative model, such as GAN and VAE, in ES-DDPM, sampling from the target non-Gaussian distribution can be efficiently achieved by diffusing samples obtained from the pre-trained generative model. In this way, the number of required denoising steps is significantly reduced. In the meantime, the sample quality of ES-DDPM also improves substantially, outperforming both the vanilla DDPM and the adopted pre-trained generative model. On extensive experiments across CIFAR-10, CelebA, ImageNet, LSUN-Bedroom and LSUN-Cat, ES-DDPM obtains promising acceleration effect and performance improvement over representative baseline methods. Moreover, ES-DDPM also demonstrates several attractive properties, including being orthogonal to existing acceleration methods, as well as simultaneously enabling both global semantic and local pixel-level control in image generation.

1 Introduction

DDPMs generate samples by iteratively denoising Gaussian noise, but their many denoising steps make inference slow. ES-DDPM stops diffusion early and uses a pre-trained generative model to start denoising from a non-Gaussian distribution.

1 Introduction

  • Vanilla DDPMs often require thousands of iterative denoising steps for a high-quality sample, each involving a neural-network evaluation.Prior jumping-step methods reduce step count but break the Gaussian assumption and can lower sample quality; conditioning-based approaches also have limited acceleration in practice.Source: §1 · pp. 1–2
  • ES-DDPM stops diffusion after the initial steps and starts reverse denoising from a non-Gaussian distribution, reducing the number of required denoising steps.It uses a pre-trained generator such as a GAN or VAE to obtain samples that are diffused to that distribution, with minor computational overhead.Source: §1 · p. 2
  • When combined with a pre-trained generator, ES-DDPM improves sample quality over both vanilla DDPM and that generator, and reduces denoising from 1000 to 100 steps in the best experimental case.The approach is also described as accelerating both training and inference, and as compatible with other DDPM acceleration methods.Source: §1 · p. 2
  • With StyleGAN2, ES-DDPM supports both global semantic control and local pixel-level control of generated images.The paper attributes this combination to the complementary control properties of DDPMs and GANs.Source: §1 · p. 2

2 Background on DDPMs

DDPMs define a forward process that gradually adds noise to data and a learned reverse process that generates images by denoising. The iterative reverse sampling is slow because it requires many neural-network evaluations.

2 Background on DDPMs

  • The forward diffusion process gradually adds noise to a clean image x0, with T chosen so that xT is close to Gaussian noise.Typical choices of T are 1000–4000 steps.Source: §2 · p. 3
  • In the reverse process, a neural network parameterizes the mean of each transition, and sampling proceeds from xT to the generated image x0.The procedure samples xT from Gaussian noise, then repeatedly samples xt−1 from pθ(xt−1|xt).Source: §2 · p. 3
  • Sampling is slow because the reverse process iteratively evaluates the neural network hundreds or thousands of times.Each evaluation is used to sample from a transition distribution pθ(xt−1|xt).Source: §2 · p. 3

3 Methodology

ES-DDPM truncates diffusion at T′ and begins reverse denoising from a partially diffused image rather than Gaussian noise. A separately trained VAE or other generator can supply samples for this starting distribution.

3 Methodology

  • ES-DDPM cuts the diffusion chain at T′<T, then denoises the partially diffused image xT′ to generate x0 in fewer reverse steps.The sampling pipeline draws a latent code, decodes it to an image, diffuses that image to xT′, and applies the partial reverse process.Source: §3 · pp. 3–4
  • The encoder maps clean images x0 to a low-dimensional latent code z while modeling a partial diffusion process that need not make xT′ Gaussian.The encoder distribution qψ(z|x0) is Gaussian with neural-network-parameterized mean and standard deviation.Source: §3 · p. 3
  • Training minimizes the ordinary DDPM loss over steps 1 through T′ together with a VAE loss, allowing the ES-DDPM and VAE to be trained separately.The paper states that training ES-DDPM is faster than training a full-length DDPM.Source: §3 · p. 4
  • The VAE decoder is trained to match the partially diffused image distribution, while its latent-space term follows the standard VAE objective.The decoder also reconstructs the original clean image from the encoder's latent code.Source: §3 · p. 4
  • At sampling time, only the decoder is needed, so ES-DDPM can also use a GAN generator or a conditional VAE or GAN.The encoder is not used to generate samples, and the paper reports that combining ES-DDPM with a GAN is more effective than combining it with a VAE.Source: §3 · p. 4

4 Related Works

Prior DDPM acceleration methods alter reverse-step schedules or add conditioning; ES-DDPM instead truncates the forward diffusion process and starts denoising from a partially diffused generator sample. The paper presents it as compatible with existing accelerators and pre-trained generators.

4 Related Works

  • Several works by Ho et al. (2020), Kong & Ping (2021), and Nichol & Dhariwal (2021) accelerate DDPM sampling with reverse-step jumps; Watson et al. (2021) optimize schedules and sample-quality scores.Salimans & Ho (2022) progressively distill trained DDPMs for fast sampling; Xiao et al. (2021) propose denoising diffusion GANs that balance sample quality and mode coverage.Source: §4 · p. 4
  • ES-DDPM is described as orthogonal to other DDPM acceleration methods, which can be coupled to accelerate its remaining T′ denoising steps.The paper reports that combining the methods further improves acceleration.Source: §4 · p. 5
  • Unlike VAE-conditioned approaches that retain a complete diffusion process, ES-DDPM starts denoising from a partially diffused image produced by a VAE or GAN.The related approaches also require training a DDPM specific to the VAE model, whereas ES-DDPM can combine a pre-trained DDPM with other generative models.Source: §4 · p. 5
  • Compared with the concurrent approach by Zheng et al. (2022), ES-DDPM diffuses samples from a pre-trained generator instead of directly learning the xT′ distribution.The paper says this avoids extra learning complexity, allows one generator to serve any T′, and enables reuse of pre-trained models without retraining.Source: §4 · p. 5

5 Experiments

Across five image datasets, the experiments examine generation quality, coupled acceleration, training speed, and controllable generation. ES-DDPM combines early-stopped denoising with GAN or VAE outputs, with reported gains in sample quality, speed, and control.

5.1 Image Generation

  • CelebA needs T′=100 or 200 to beat its 1,000-step DDPM, whereas CIFAR-10 needs 700–800 steps; stronger GANs improve quality in fewer steps.On CIFAR-10, the combined model can even outperform the original DDPM; increasing T′ generally improves quality.Source: §5.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
  • On ImageNet-64, BigGAN-deep needs T′ around 900 for lower FID than the full-length DDPM, while LSUN single-category models beat full-length DDPM FID in 100 steps.For complex multi-category data, the combined model trades denoising steps against sample diversity; on LSUN-Bedroom and LSUN-Cat, it improves GAN recall with fewer steps.Source: Table 3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

5.2 Coupled With Other DDPM Acceleration Methods

  • DDIM and Timestep-Respacing further accelerate ES-DDPM, which has better sample quality than other acceleration methods at the same denoising steps in most cases.The method applies to the final T′ denoising steps of a full-length DDPM.Source: §5.2 · p. 7, Table 6 · p. 8, Table 7 · p. 8

5.3 Accelerate Training

  • 4.5 × 10^4 iterations (~4 hours) reached FID 3.09 on CelebA-64, versus 9 × 10^5 iterations (~80 hours) for FID 3.26; StyleGAN2-inclusive total was 29 hours.On CelebA-128, the full-length DDPM did not reach a comparable FID even after very long training.Source: Figure 2 · p. 8, §5.3 · p. 8

5.4 Controllable Image Generation

  • The combination supports GAN/VAE semantic control plus pixel-level edits: StyleGAN sets bedroom lighting, and ES-DDPM refines local edits when T′ is not too large.The denoising process also mitigates artifacts in the edited image.Source: Figure 3 · p. 9, §5.4 · p. 9

6 Conclusion

ES-DDPM combines early stopping in a DDPM with another generative model, such as a VAE or GAN. The paper reports gains in sample quality, sampling speed, and training speed, while noting a multi-category limitation tied to GAN mode coverage.

6 Conclusion

  • Combining an early-stopped DDPM with a VAE or GAN can outperform the original DDPM in sample quality, sampling speed, and training speed.The combined model can also be further accelerated with other DDPM acceleration methods.Source: §6 · p. 9
  • Combining DDPMs with GANs enables control of both high-level semantics and local image details.Source: §6 · p. 9
  • On multi-category datasets, ES-DDPM still needs many denoising steps to outperform full-length DDPM in sample quality, which the authors attribute to GANs’ poor mode coverage.The authors suggest trying generative models with better mode coverage for complex multi-category datasets.Source: §6 · p. 9

A.1 ELBO Proof

The appendix derives an evidence lower bound using variational inference and sets a loss involving VAE and DDPM terms. The proof invokes the concavity of the logarithm.

A.1 ELBO Proof

  • The proof applies variational inference to express the data log-likelihood using an expectation over q(x1:T′, z|x0).Source: §A.1 · p. 12
  • The derivation cites the concavity of log in its proof step.Source: §A.1 · p. 12
  • The stated loss combines terms involving the latent prior and encoder with terms involving the decoder and partial reverse diffusion process.Source: §A.1 · pp. 12–13

A.2 Conditional Generation

For conditional generation, the method combines a conditional DDPM with a VAE-like model trained separately. Its reverse diffusion chain covers only steps 1 through T′, and sampling proceeds from a condition-dependent latent code to an output image.

A.2 Conditional Generation

  • The condition c can encode information about the clean image, such as its class, a low-resolution version, or a grayscale version.Source: §A.2 · p. 14
  • The conditional encoding factorizes into q(x1:T′|x0) and qψ(z|c), with qψ(z|c) as a Gaussian whose mean and standard deviation are neural-network parameterized.Source: §A.2 · p. 14
  • The conditional DDPM loss trains only the first several steps, from 1 to T′, while the VAE-like loss trains a conditional generative model; the two models can be trained separately and combined.Source: §A.2 · p. 15
  • Sampling draws z from qψ(z|c), decodes it to an image, samples xT′, then runs the conditional DDPM in reverse to output x0.The decoder may be a conditional GAN or another conditional generation model.Source: §A.2 · p. 15

A.3 Computational Cost

The computational-cost section reports training and generation runtimes for CelebA, ImageNet-64, LSUN-Bedroom, and LSUN-Cat. Runtimes vary by dataset, model, step count, and hardware.

A.3 Computational Cost

  • On CelebA-64, StyleGAN2 training took 10^5 iterations (around 25 hours), versus 9 × 10^5 iterations (around 80 hours) for full-length DDPM.On CelebA-128, StyleGAN2 took 2.4 × 10^5 iterations (around 183 hours), versus 4 × 10^6 iterations (around 448 hours) for full-length DDPM.Source: §A.3 · p. 15
  • Generating 50,000 CelebA-128 images with the 1000-step DDPM took around one day on 8 NVIDIA RTX 2080Ti GPUs.Source: §A.3 · p. 16
  • Generating 50,000 ImageNet-64 images with the 250-step DDPM took about 8 hours on 4 NVIDIA A100 GPUs.Source: §A.3 · p. 16
  • Generating 50,000 LSUN-Bedroom or LSUN-Cat images with the 1000-step DDPM took about one day on 24 NVIDIA A100 GPUs.Source: §A.3 · p. 16

A.4 DDPM Acceleration Experiement Results on LSUN Datasets

The LSUN tables compare DDPM acceleration methods by FID, while additional experiments show ES-DDPM preserving high-level controls and refining local edits.

A.4 DDPM Acceleration Experiement Results on LSUN Datasets

A.5 Additional Controllable Generation Experiments

  • StyleGAN2 age and gender controls are preserved on CelebA-128; at the same denoising-step count, ES-DDPM achieves better sample quality than VAE-combination methods.The compared methods are those of Preechakul et al. (2021) and Pandey et al. (2022).Source: §A.5 · p. 16, Figure 4 · p. 16
  • A pre-trained DDPM can be paired with any generative model capable of high-level semantic control, without training a DDPM specific to that model.The authors contrast this with the cited VAE-combination methods, which require altering DDPM training and training a model specific to the paired generator.Source: §A.5 · pp. 16–17
  • At T′ = 300, ES-DDPM refines stroke-based edits to StyleGAN2-generated LSUN-Cat images and removes artifacts.The authors also report that the edited images become photorealistic.Source: §A.5 · p. 17, Figure 5 · p. 17
Loading 2205.12524v2…