Source-linked AI summary
Semi-Implicit Denoising Diffusion Models (SIDDMs)
Yanwu Xu, Mingming Gong, Shaoan Xie, Wei Wei, Matthias Grundmann, Kayhan Batmanghelich, Tingbo Hou
TL;DR
Fast, diverse, high-quality generation remains challenging: DDPM sampling can require many iterative steps, while DDGANs have difficulty scaling to large datasets. SIDDM matches noisy-data marginals implicitly and forward conditional distributions explicitly; the authors report performance comparable to diffusion models and better than models using few sampling steps.
Problem
DDPMs can require many iterative steps for high-quality sampling, and DDGANs face scalability limitations on large datasets.
Source: §1 · pp. 1–2Method
SIDDM matches noisy-data marginal distributions with an implicit GAN objective and forward conditional distributions with an explicit L2 reconstruction loss.
Source: §1 · p. 2, §3.2 · p. 4Results
SIDDM achieves generative performance comparable to DDPM while surpassing models with fewer sampling steps; experiments also report better quality and diversity than GAN-based methods and superior quantitative results to DDGAN.
Source: §5.1 · p. 8, §7 · p. 10Takeaways & Limitations
The authors report that leaving the reverse step unparameterized permits larger inference steps while retaining the exact diffusion process.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Despite the proliferation of generative models, achieving fast sampling during inference without compromising sample diversity and quality remains challenging. Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. The Denoising Diffusion Generative Adversarial Networks (DDGAN) attempted to circumvent this limitation by integrating a GAN model for larger jumps in the diffusion process. However, DDGAN encountered scalability limitations when applied to large datasets. To address these limitations, we introduce a novel approach that tackles the problem by matching implicit and explicit factors. More specifically, our approach involves utilizing an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. This combination allows us to effectively match the joint denoising distributions. Unlike DDPM but similar to DDGAN, we do not enforce a parametric distribution for the reverse step, enabling us to take large steps during inference. Similar to the DDPM but unlike DDGAN, we take advantage of the exact form of the diffusion process. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.
1 Introduction
The introduction frames generative modeling as a trilemma of sample quality, mode coverage, and speed, then presents SIDDM as a combination of implicit and explicit objectives aimed at addressing it.
1 Introduction
- Diffusion models can require thousands of steps for high-quality sampling, while fewer steps violate the small-noise Gaussian reverse-step assumption and bias samples.The introduction identifies quality, mode coverage, and speed as the generative-model trilemma.Source: §1 · pp. 1–2
- DDGANs enable faster sampling, but the authors identify limited scalability on large datasets and hypothesize that high-dimensional adjacent-state discrimination is difficult.The proposed explanation is that the discriminator processes concatenated adjacent variables.Source: §1 · p. 2
- SIDDM combines an implicit GAN objective for noisy-data marginals with an explicit L2 loss for the conditional forward diffusion, and adds discriminator regularization.The authors state that the regularizer stabilizes the discriminator without additional computational burden.Source: §1 · p. 2
- The authors present SIDDM as a solution targeting fast sampling, high generation quality, large-scale scalability, and improved model performance through regularization.Source: §1 · p. 2
2 Background
The background explains why Gaussian reverse-step modeling becomes unsuitable for large diffusion jumps and describes DDGAN's adversarial alternative, whose high-dimensional discriminator may limit scalability.
2 Background
- The forward process gradually corrupts sampled data with Gaussian noise, while the reverse process is parameterized as Gaussian when adjacent-step noise is sufficiently small.The reverse model's mean is predicted by a denoising function, and its variance depends on the noise schedule.Source: §2 · pp. 2–3
- With only a few sampling steps, the Gaussian reverse-step assumption fails, so L2 reconstruction cannot model the resulting KL divergence.The background notes that the usual objective indirectly maximizes the data likelihood's ELBO.Source: §2 · p. 3
- DDGANs use conditional adversarial learning to match denoising distributions and permit large noise additions between adjacent steps.Source: §2 · p. 3
- The authors hypothesize that adversarial learning on concatenated adjacent states is statistically inefficient for high-dimensional data, limiting DDGAN scalability.Source: §2 · p. 3
3 Semi-Implicit Denoising Diffusion Models
SIDDM reformulates denoising as matching marginal and conditional distributions, then develops a training objective and discriminator regularizer intended to improve training and performance.
3.1 Revisiting Denoising Distribution and the Improved Decomposition
- SIDDM factors the joint denoising distribution into implicit marginals and forward-diffusion conditionals, matching the former adversarially and the latter via KL to approximately match the joint.The conditional forward diffusion is named auxiliary forward diffusion (AFD); Figure 1 contrasts this decomposition with DDPM and DDGAN.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
3.2 Semi-Implicit Objective
- The training objective combines adversarial marginal matching with a conditional KL term, whose cross-entropy is tractable via mean square error and whose negative entropy is estimated adversarially.A conditional estimator participates in a min-max game to minimize the denoiser's negative conditional entropy.Source: §3.2 · pp. 4–5
- Compared with DDGAN's purely adversarial objective, SIDDM uses a less complex conditional objective for stable denoiser updates, shares DDPM network structure, and adds no overhead.The authors report steady performance improvement from the decomposition.Source: §3.2 · p. 5
3.3 Regularizer of Discriminator
- The discriminator regularizer reconstructs the ground-truth data from the discriminator's denoising output and applies only to sampled data q(xt).The authors state that it adds no model-capacity restriction, overhead, or dataset-specific hyperparameter grid search, and can be plugged into SIDDM and DDGAN.Source: §3.3 · p. 5
4 Related Works
Related work traces diffusion models and efforts to accelerate sampling, then distinguishes SIDDM's denoising-distribution decomposition from prior GAN approaches.
4 Related Works
- Discrete-time diffusion models introduced parameterized reverse processes [10] [1], while score-based work developed continuous-time models [16] and adversarial score matching targets a different goal [19].ADM, UDM, and EDM introduce classifier guidance [2], unbounded score matching [20], and a sampling strategy [21].Source: §4 · p. 6
- Diffusion sampling acceleration has included distillation [23], learned noise schedules [24], non-Markovian DDIM and FastDPM [25] [26], faster SDE solvers [27], and few-step DDGAN sampling [11].Diffusion models typically require thousands of steps for their best quality.Source: §4 · p. 6
- SIDDM proposes a different denoising-distribution decomposition from DDGAN [11] and, unlike AC-GAN and TAC-GAN, applies its decomposition to both unconditional and conditional generation.The authors report better results than DDGAN.Source: §4 · p. 6
5 Experiments
SIDDM recovers the 5×5 Mixture of Gaussians distribution at multiple denoising-step counts where DDGAN fails. On real data, it outperforms DDGAN, while ImageNet results are comparable to diffusion-based models at the same capacity.
5 Experiments
- Experiments evaluate the method on simulated Mixture of Gaussians and Cifar10-32, CelebA-HQ-256 [34], and ImageNet1000-64[35]; generator and discriminator use identical Unet structures, differing only in input and output channels.The Mixture of Gaussians setup samples 25 groups with different means and the same variance.Source: §5 · p. 6, §5.1 · p. 6
5.1 MOG Synthetic Data
- On a 5 × 5 Mixture of Gaussians sampled as 25 groups with distinct means and shared variance, the proposed method recovers the distribution at 2 and 16 denoising steps, whereas DDGANs fail.Removing the auxiliary forward term prevents recovery of the original distribution.Source: §5.1 · pp. 6–7, Figure 3 · p. 8
- SIDDM outperforms DDGAN quantitatively and GAN-based methods in quality and diversity, while remaining slightly below diffusion-based models with faster sampling.The reported sampling uses four denoising steps for CIFAR10 and two for CelebA-HQ.Source: §5.1 · p. 8, Table 2 · p. 8, Table 3 · p. 8
5.2 Generation on Real Data
- On ImageNet, SIDDM achieves results comparable to diffusion-based models at the same model capacity, while DDGAN fails to retain high-fidelity samples without additional inductive bias.The ImageNet experiments use four denoising steps.Source: §5.2 · pp. 8–9, Table 4 · p. 8
6 Ablations
Ablations indicate that both adversarial and auxiliary forward diffusion terms contribute to recovering the image distribution; the discriminator regularizer further enhances performance.
6 Ablations
- Removing either the adversarial term or AFD prevents recovery of the original image distribution, while FID is insensitive to weights when both terms are present.Figure 4 compares the full model with adversarial-only and AFD-only cases; Table 5 reports FID across term weights and settings.Source: §6 · p. 9, Figure 4 · p. 9, Table 5 · p. 9
- The proposed auxiliary tasks for the discriminator further enhance performance under the full formulation.The study also evaluates the model without the discriminator regularizer.Source: §6 · p. 9, Table 5 · p. 9
7 Conclusion
SIDDM combines implicit and explicit components to address slow DDPM sampling and DDGAN scalability limits. It allows larger reverse steps and reports performance comparable to DDPM, while surpassing models with fewer sampling steps.
7 Conclusion
- SIDDM combines implicit matching of reverse-process marginals with explicit modeling between variable pairs, using KL divergence and a min-max game for negative entropy.The method also uses L2 reconstruction loss for the KL divergence cross-entropy term.Source: §7 · p. 9
- SIDDM imposes no parametric reverse-step distribution, like DDGAN, and uses the exact diffusion process, like DDPM.These choices distinguish its reverse-step formulation from DDPM and its use of the diffusion process from DDGAN.Source: §7 · pp. 9–10
- SIDDM has generative performance comparable to DDPM and surpasses models with fewer sampling steps.The conclusion describes DDPM as slow because of iterative steps and DDGAN as limited on large-scale datasets.Source: §7 · pp. 9–10
S I. Derivation of Training Objective
The derivation starts from the forward posterior and uses the Gaussian form of the forward diffusion. It parameterizes the denoised data and rewrites the distribution-matching objective into its training terms.
S I. Derivation of Training Objective
- The forward posterior is formulated from Bayes’ rule and rewritten as a Gaussian using the forward-diffusion components.The derivation gives the posterior conditional on x_t and x_0.Source: S I. Derivation of Training Objective · p. 13
- The denoised x′_0 is parameterized by replacing x_0 with its prediction in the forward-posterior expression.This yields the parameterized x′_{t−1}.Source: S I. Derivation of Training Objective · p. 13
- The distribution-matching objective is rewritten into GAN matching and cross-entropy terms using the forward-diffusion transition and a regression-model distribution.The passage notes that the main-paper formulation swaps β_t and 1 − β_t; the authors identify this as a typo to correct.Source: S I. Derivation of Training Objective · pp. 13–14
S II. Derivation of Theorem 1
The derivation bounds joint total-variation distance using marginal and conditional distribution differences. It then relates total variation to Jensen–Shannon divergence to rewrite the bound.
S II. Derivation of Theorem 1
- The joint-distribution bound follows by applying the triangle inequality and separately bounding marginal and conditional differences.The marginal bound uses Hölder’s inequality, and the conditional bound uses an upper-bound constant.Source: S II. Derivation of Theorem 1 · p. 14
- The joint total-variation distance is bounded by a weighted sum of marginal and conditional total-variation distances.The weights are constants c1 and c2 in the displayed bound.Source: S II. Derivation of Theorem 1 · p. 14
- The derivation invokes a stated relation between total variation and Jensen–Shannon divergence to rewrite the bound.The supplied passage gives 1/2 d_TV(P,Q)^2 ≤ JSD(P,Q) ≤ 2 d_TV(P,Q).Source: S II. Derivation of Theorem 1 · p. 14
S III. Societal impact
The supplementary material describes possible applications, implementation choices, and generated samples. It notes a training-resource trade-off, details shared network components, and shows sampled images from three datasets.
S III. Societal impact
- SIDDMs could accelerate inference for text-to-image generation and editing, including real-time image generation or instant editing.The societal-impact passage says training takes more time and resources because of additional adversarial objectives.Source: S III. Societal impact · p. 14
S IV. More Implementation Details
- The implementation shares most discriminator layers with the regression model and regularizer, adding only separate linear heads and no obvious extra overhead over baseline DDGANs.The discriminator is a UNet with a symmetric generator structure; models are trained until their best FID score.Source: S IV. More Implementation Details · pp. 14–15, Table 6 · p. 15
S V. More Generated Results
- The supplementary figures show 768 CIFAR10 samples, 192 CelebHQ 256 samples, and 768 ImageNet 64 samples from the model.The captions identify these as randomly sampled generated images used to produce the paper results.Source: Figure 5 · p. 16, Figure 6 · p. 17, Figure 7 · p. 18