Source-linked AI summary
On Fast Sampling of Diffusion Probabilistic Models
Zhifeng Kong, Wei Ping
TL;DR
Fast sampling methods face a trade-off between sampling speed and sample quality, while prior approaches differ in whether they can reuse pretrained DDPMs and whether continuous noise conditioning matches DDPM quality. FastDPM unifies fast sampling without retraining by using continuous diffusion steps and shorter approximate processes, and finds that method performance depends on domain and conditional information.
Problem
Fast sampling must reduce diffusion-process length while preserving sample quality, but prior methods either cannot reuse pretrained DDPMs or have uncertain quality on challenging image and audio synthesis tasks.
Method
FastDPM generalizes discrete diffusion steps to continuous steps, maps them bijectively to continuous noise levels, and constructs approximate forward and reverse processes of length S much smaller than T without retraining.
Results
FastDPM produces improved sample quality at small S, with DDIM-rev outperforming DDPM-rev for image generation while DDPM-rev outperforms DDIM-rev for audio synthesis.
Takeaways & Limitations
The preferred FastDPM method and reverse-process stochasticity should be selected according to the data domain and the amount of conditional information.
Takeaways & Limitations
FID is not an appropriate metric for conditional generation.
Abstract
from arXiv · showhide
In this work, we propose FastDPM, a unified framework for fast sampling in diffusion probabilistic models. FastDPM generalizes previous methods and gives rise to new algorithms with improved sample quality. We systematically investigate the fast sampling methods under this framework across different domains, on different datasets, and with different amount of conditional information provided for generation. We find the performance of a particular method depends on data domains (e.g., image or audio), the trade-off between sampling speed and sample quality, and the amount of conditional information. We further provide insights and recipes on the choice of methods for practitioners.
1 Introduction
FastDPM is a unified, no-retraining framework that shortens diffusion-model sampling while generalizing prior approaches. Its evaluations show that the best reverse-process design depends on domain and conditional information.
- Motivation: Prior acceleration methods either retrain models for continuous noise levels or approximate pretrained DDPM reverse processes with shorter, discrete-step processes.The two families differ in whether pretrained diffusion models can be reused.
- FastDPM: FastDPM generalizes discrete diffusion steps to continuous steps and uses a bijective mapping to construct approximate diffusion and reverse processes of length S ≪ T.The framework is designed without retraining.
- FastDPM: FastDPM provides approximate-process choices based on selecting original steps or variances, and reverse-process choices using stochastic DDPM-rev or deterministic DDIM-rev.These choices control reverse-process stochasticity.
- Findings: FastDPM produces new algorithms with improved sample quality at small S and is evaluated across image and audio generation tasks.The evaluation also varies the amount of conditional information.
- Findings: Deterministic DDIM-rev performs better for image generation, whereas stochastic DDPM-rev performs better for audio synthesis.The preferred stochasticity also changes with the amount of conditional information.
2 Related Work
Fast-sampling research reduces the reverse process from T to S ≪ T steps. FastDPM approximates pretrained DDPMs while combining continuous noise-level flexibility with mappings that generalize earlier methods.
- Acceleration methods: Two method families accelerate synthesis by reducing the reverse-process length from T to S ≪ T steps.They differ in whether acceleration is addressed during training or by approximating the original reverse process.
- Acceleration methods: One family retrains networks on continuous noise levels, then selects S ≪ T levels for a short reverse process.This approach assumes the network can predict added noise at arbitrary noise levels.
- FastDPM: FastDPM constructs a bijection between continuous diffusion steps and continuous noise levels to approximate the original DDPM model.This retains the flexibility of continuous noise levels while targeting the original model.
- FastDPM: FastDPM generalizes DiffWave through Gamma-function noise-level computation and generalizes DDIM through noise levels corresponding exactly to integer steps.These constructions extend prior methods across continuous and discrete domains.
3 Diffusion Models
DDPM uses forward and reverse Markov chains: the forward chain adds Gaussian noise, while the learned reverse chain removes it to generate data. Training simplifies a variational objective into noise prediction.
- Model structure: DDPM consists of a T-step diffusion process that adds Gaussian noise and a T-step reverse process that removes it.The diffusion process moves data toward a latent Gaussian distribution.
- Diffusion process: The forward transition uses a variance schedule β_1, ···, β_T, whose values determine the Gaussian noise added at each step.The schedule is specified through the transition distributions q(x_t|x_t−1).
- Model structure: The reverse sampler starts from x_T drawn from a standard Gaussian, iteratively samples x_t−1, and outputs x_0.The reverse process proceeds from t = T down to 1.
- Training: DDPM training is based on the variational evidence lower bound, which can be largely simplified under Ho et al.’s parameterization.The simplified unweighted objective is associated with higher generation quality.
- Diffusion process: The model can directly sample x_t given x_0 by expressing it as a scaled data point plus scaled Gaussian noise.This direct form underlies the interpretation of diffusion as progressively adding noise.
4 FastDPM: A Unified Framework for Fast Sampling in Diffusion Models
FastDPM approximates pretrained DDPMs with much shorter diffusion and reverse processes by extending diffusion steps continuously and mapping them bijectively to noise levels. Its framework supports multiple schedules and reverse samplers, including DDPM-rev and DDIM-rev.
- Overview: FastDPM approximates pretrained DDPM sampling with diffusion and reverse processes of length S ≪ T, avoiding retraining.The method targets the large number of network evaluations required when T is large.
- Continuous diffusion steps and noise levels: FastDPM extends discrete diffusion steps to continuous values and uses a bijection between diffusion steps t and noise levels r.The mapping is r = R(t) and t = T(r), where T is the inverse of R.
- Continuous diffusion steps and noise levels: The noise-level mapping defines R(t) = √ᾱ_t, interpreting each noisy state as a mixture of data and white noise.R(t) is extended from integer to real-valued diffusion steps using the variance schedule and a Gamma-function-based continuation.
- Approximate diffusion process: FastDPM constructs approximate diffusion schedules either by selecting S variances or by selecting S original diffusion steps.Step-based noise levels are a special case of variance-based noise levels.
- Approximate reverse process: The approximate reverse process supports stochastic DDPM-rev and implicit DDIM-rev sampling, with stochasticity controlled by κ.DDPM-rev samples from the stochastic reverse process, while DDIM-rev uses the DDIM procedure.
- Relation to prior methods: DDPM-rev is a special case of DDIM-rev when κ = 1, and existing DDIM and DiffWave methods correspond to specific FastDPM schedule–sampler choices.DDIM uses step-selected noise levels with DDIM-rev; DiffWave uses variance-selected noise levels with DDPM-rev.
5 Experiments
FastDPM is evaluated through extensive image and audio experiments comparing approximate diffusion and reverse processes at different sampling lengths and conditioning levels. Results show that the better configuration depends on domain, reverse-process stochasticity, and the amount of conditional information.
- Experimental setup: Experiments compare VAR and STEP approximate diffusion processes and DDPM-rev and DDIM-rev approximate reverse processes across image and audio tasks.FastDPM length S and noise-level schedules are varied, using FID, IS, accuracy, and MOS as appropriate.
- Experimental setup: FastDPM is tested on CIFAR-10, CelebA, LSUN-bedroom, SC09, and LJSpeech using pretrained checkpoints from DDPM, DDIM, and DiffWave.The image checkpoints use T = 1000, while the audio checkpoints use T = 200.
- Observations and Insights: The experiments provide method-selection insights and recipes for FastDPM across domains, datasets, and conditioning settings.The paper also notes that FID is not an appropriate metric for conditional generation.
- Observations and Insights: VAR marginally outperforms STEP for small S across the evaluated image and audio experiments.The relative advantage changes with dataset and S; for example, VAR leads on CIFAR-10 at S = 10 and LJSpeech at S = 5.
- Observations and Insights: DDIM-rev significantly outperforms DDPM-rev in image generation, whereas DDPM-rev significantly outperforms DDIM-rev in audio synthesis.In images, reducing κ from 1.0 to 0.0 consistently improves sample quality; in audio, increasing κ from 0.0 to 1.0 consistently improves it.
- Observations and Insights: The amount of conditional information affects which reverse process performs best in audio generation.DDPM-rev is best unconditionally, DDIM-rev with κ = 0.5 slightly leads with class conditioning, and the gap narrows for mel-spectrogram conditioning.
6 Conclusion
FastDPM is a unified, no-retraining framework for fast diffusion-model sampling that generalizes prior methods and offers greater flexibility. Its quality still degrades relative to original DDPM when S is small.
- FastDPM provides a unified framework for fast sampling in diffusion models without retraining.
- FastDPM generalizes prior methods and provides more flexibility across image and audio generation tasks.
- When S is small, FastDPM still produces lower quality than the original DDPM.The authors plan to study higher-quality algorithms for extremely small S.
A.1 Derivation of q(xt|x0)
The diffusion process yields a Gaussian marginal q(x_t|x_0), obtained first from its stepwise definition and then by recursion.
- The diffusion process is defined through successive noisy transitions, with each ϵ_t an independent standard Gaussian.
- Recursion expresses x_t directly in terms of x_0 and accumulated independent Gaussian noise.
- The resulting q(x_t|x_0) remains Gaussian, with mean √¯α_t x_0 and covariance (1 − ¯α_t)I.
B.1 Unconditional Generation on CIFAR-10
The CIFAR-10 figures compare FastDPM generated samples across different lengths S and approximate diffusion processes, using DDIM-rev or VAR as specified.
- Figure 1 compares CIFAR-10 samples across different S and approximate diffusion processes using DDIM-rev with κ = 0.0.
- Figure 2 compares CIFAR-10 samples across different S and approximate reverse processes with VAR as the approximate diffusion process.
B.2 Unconditional Generation on CelebA
The figures compare FastDPM samples across CelebA and LSUN-bedroom settings, varying S and approximate diffusion processes, while audio spectrograms compare reverse-process choices.
- Figure 3 compares CelebA samples across different S using DDIM-rev with κ = 0.0.
- Figures 4–7 compare LSUN-bedroom samples across approximate diffusion processes at S values of 100, 50, 20, and 10.
- At S = 50, Figure 8 compares DDIM-rev with κ = 0.0 and DDPM-rev, with DDPM-rev producing the clearest utterances.
- At S = 50, Figure 9 compares DDIM-rev with κ = 0.0, DDIM-rev with κ = 0.5, and DDPM-rev, with κ = 0.5 clearest.
- Figure 10 compares ground-truth and generated LJ001-0001 mel-spectrograms under DDIM-rev and DDPM-rev, with both generating high-quality speech.