PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion-GAN: Training GANs with Diffusion

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, Mingyuan Zhou

arXiv:2206.02262v4cs.LGstat.ML

TL;DR

GAN training is often unstable, and instance noise has lacked effective demonstrations on high-dimensional image data. Diffusion-GAN uses adaptive forward diffusion with timestep-dependent discrimination and differentiable generator updates. It improves stability, fidelity, and diversity over strong GAN baselines on benchmark datasets, while adding comparable computational costs.

  • Problem

    Instance noise is a promising GAN-stabilization technique, but suitable noise distributions are difficult to choose and successful high-dimensional image results had not been demonstrated.

  • Method

    Diffusion-GAN applies the same adaptive, variable-length forward diffusion process to real and generated data, then trains a timestep-dependent discriminator while backpropagating through diffusion.

  • Results

    Diffusion-GAN outperforms corresponding state-of-the-art GAN baselines in fidelity and diversity according to FID and Recall, while preventing discriminator overfitting and capturing all 25 Gaussian modes in a toy example.

  • Takeaways & Limitations

    Diffusion-GAN provides model- and domain-agnostic differentiable augmentation for stable, data-efficient GAN training without requiring a costly reverse diffusion chain.

  • Takeaways & Limitations

    The paper’s prior-work claim is limited to the authors’ knowledge regarding empirical success of instance noise on high-dimensional image data.

Abstract

from arXiv · show

Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice. In this paper, we propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate Gaussian-mixture distributed instance noise. Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator. Both the observed and generated data are diffused by the same adaptive diffusion process. At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.

1 INTRODUCTION

GANs can generate realistic images but often train unstably, while instance noise has been difficult to use effectively on high-dimensional images. Diffusion-GAN addresses this with differentiable, adaptive diffusion-based noise and reports stronger stability and generation performance.

  • GANs suffer from non-convergence, training instability, and mode collapse despite their success in synthesizing high-resolution images.
  • Instance noise can widen distribution support and reduce discriminator overfitting, but selecting a suitable noise distribution is challenging.
  • The discriminator compares diffused real and fake images at every diffusion step using timestep-dependent discrimination.
  • No prior work had empirically demonstrated successful instance-noise training for GANs on high-dimensional image data.
  • Diffusion-GAN generates Gaussian-mixture instance noise through a differentiable diffusion process whose length depends on the data and generator.
  • Diffusion-GAN provides model- and domain-agnostic differentiable augmentation and improves stability and generation performance across strong GAN baselines.

2 PRELIMINARIES: GANS AND DIFFUSION-BASED GENERATIVE MODELS

GANs learn a target data distribution through adversarial interaction between a generator and discriminator. Diffusion-based generative models instead define a forward chain that progressively adds noise according to a variance schedule and supports closed-form intermediate sampling.

  • A GAN generator maps random noise to samples resembling the data, while its discriminator classifies real and generated samples.
  • The generator seeks to fool the discriminator, whereas the discriminator seeks to distinguish generated samples from real data.
  • Although GAN objectives are often modified for stability and performance, adversarial learning between generator and discriminator remains central.
  • Diffusion-based generative models represent data with latent variables and a forward diffusion chain of the same dimensionality as the data.
  • The forward chain gradually adds Gaussian noise over T steps using a predefined variance schedule β_t and variance σ^2.
  • An arbitrary noisy state x_t can be sampled directly from a Gaussian conditional distribution given the original data x_0.
  • A variational lower bound is used to optimize the reverse diffusion chain.

3 DIFFUSION-GAN: METHOD AND THEORETICAL ANALYSIS

Diffusion-GAN injects differentiable, Gaussian-mixture instance noise into both real and generated data through a diffusion process, while adapting its intensity and conditioning the discriminator on timestep. Theoretical analysis shows that this smooths optimization and, under stated conditions, matching perturbed distributions is equivalent to matching the original distributions.

  • Diffusion-based instance noise: Diffusion-GAN applies a differentiable diffusion process to real and generated samples, with the number of steps depending on the data and generator.The process gradually adds Gaussian noise and supports generator training by backpropagation through the perturbation chain.
  • Diffusion-based instance noise: The noisy-sample distribution is a mixture over diffusion steps, using the same Gaussian transition process for real and generated data.Each component has a timestep-specific noise level, and sampling first selects a timestep according to mixture weights.
  • Adversarial training: The adversarial objective trains a timestep-dependent discriminator to distinguish perturbed real and generated samples while the generator seeks to fool it at every timestep.The objective is a diffusion-augmented version of the original GAN game and is differentiable with respect to generator parameters.
  • Theoretical analysis: For every noise level, diffusion makes the relevant f-divergence smooth, continuous, and differentiable, avoiding singular or discontinuous GAN objectives.The analysis applies to any convex f-divergence, including JS divergence.
  • Theoretical analysis: A mixture across diffusion steps supplies informative gradients even when individual noise levels produce nearly flat divergence regions.In the toy example, increasing t smooths JS-divergence curves, but some levels remain nearly constant; mixing all steps addresses this issue.
  • Theoretical analysis: Under the theorem’s one-to-one reparameterization and known-noise assumptions, equality of the noisy real and generated distributions holds if and only if equality of the original distributions holds.Thus the noise injection can improve training without changing the target distribution being matched.

4 EXPERIMENTS

Experiments evaluate Diffusion-GAN across image benchmarks, domain-agnostic feature spaces, and a 25-Gaussians toy task. Results indicate improved diversity, fidelity, discriminator behavior, and compatibility with multiple GAN backbones, with comparable computational costs.

  • Experimental setup: Experiments cover image datasets from 32 × 32 to 1024 × 1024, including CIFAR-10, STL-10, LSUN, AFHQ, and FFHQ.The evaluation also includes low- and high-dimensional feature-vector settings and a 25-Gaussians toy task.
  • Image generation results: Diffusion StyleGAN2 produces photo-realistic, diverse images across resolutions from 32 × 32 to 1024 × 1024.The comparison evaluates StyleGAN2, StyleGAN2 + DiffAug, and StyleGAN2 + ADA using FID and Recall.
  • Image generation results: Diffusion StyleGAN2 achieves higher Recall than all GAN baselines on all 6 benchmark datasets and a clear FID advantage on 5 of 6.Recall measures generation diversity, while lower FID indicates better fidelity.
  • Data augmentation comparison: Diffusion StyleGAN2 consistently outperforms StyleGAN2 across datasets, while ADA and DiffAug can impair performance on sufficiently large datasets.The paper attributes this possible impairment to augmentation leakage overshadowing augmentation benefits.
  • Adaptive diffusion analysis: Adaptive diffusion adjusts T according to discriminator overfitting, while the diffusion-based mixture keeps discriminator outputs well behaved and useful for generator learning.T increases during Diffusion StyleGAN2 training but first rises and then falls for Diffusion ProjectedGAN.
  • Resources: Diffusion-GAN has memory and time costs generally comparable to its GAN baseline, with negative added memory and training-time costs when replacing ADA, and zero added inference-time cost.The paper reports these costs for diffusion-based augmentation relative to ADA.
  • Toy-domain analysis: On the 25-Gaussians task, Diffusion-GAN captures all 25 modes, unlike vanilla GANs, whose discriminator quickly overfits and whose generator severely collapses modes.The paper connects the improvement to non-leaking augmentation and adaptively adjusted diffusion-based noise injection.
  • Domain-agnostic augmentation: Applied to ProjectedGAN feature vectors, Diffusion ProjectedGAN improves both FID and Recall and reaches state-of-the-art FID on STL-10, LSUN-Bedroom, and LSUN-Church.This supports the method’s reported domain-agnostic behavior in high-dimensional feature space.

5 CONCLUSION

Diffusion-GAN uses a variable-length forward diffusion chain to provide Gaussian-mixture instance noise and differentiable augmentation without a reverse diffusion chain. The paper reports theoretical and empirical evidence of stable, non-leaking training and strong image-generation performance.

  • Conclusion: Diffusion-GAN generates instance noise with a variable-length forward diffusion chain and Gaussian mixture distribution.The generator maps noise to a sample in one step, avoiding a reverse diffusion chain during training and generation.
  • Conclusion: Theoretical analysis and experiments indicate that Diffusion-GAN prevents discriminator overfitting and avoids augmentation leakage.These properties are presented as advantages of its diffusion-based transformation.
  • Conclusion: Experiments report high-fidelity and diverse high-resolution images, outperforming corresponding state-of-the-art GAN baselines on FID and Recall.The comparison covers standard benchmark datasets.
  • Conclusion: Diffusion-GAN can train and generate as quickly as a vanilla GAN with the same generator size, unlike reverse-chain diffusion models.The paper gives a 50k-image sampling comparison between DDPM and Diffusion-GAN.
  • Conclusion: The method provides model- and domain-agnostic differentiable augmentation that can be applied beyond image data, including latent features.The paper contrasts this with augmentation methods requiring domain-specific knowledge.

B PROOF

The proof analyzes noisy distributions produced by differentiable transformations and establishes continuity and differentiability properties under stated assumptions. It also discusses several reparameterization forms that satisfy the theorem’s conditions.

  • Proof: Under mild conditions, the transformed real and generated densities are continuous functions over the observation variable.The argument first establishes continuity for the real density and states that the generated-density proof is analogous.
  • Proof: The divergence between transformed real and generated distributions is differentiable when the transformed densities and generator function are differentiable.The proof invokes the chain rule and differentiability of the relevant density terms.
  • Proof: The proof represents real and generated transformed samples using differentiable functions of data and auxiliary noise variables.The construction writes transformed variables as y = f(x) + g(ϵ) and yg = f(xg) + g(ϵg).
  • Proof: Moment-generating-function uniqueness is used to relate equality of transformed distributions to equality of the underlying generator distributions.The proof assumes the relevant mappings and auxiliary-noise constructions satisfy the stated identifiability conditions.
  • Proof: The theorem accommodates tractable inverse-CDF transformations, location-scale distributions, and implicit neural-network distributions.Examples include Cauchy, Logistic, Gaussian, Laplace, Student’s t, Uniform, and Triangular distributions.

C DERIVATIONS

The derivations show why the JS divergence can fail to provide useful generator gradients for disjoint-support distributions and how diffusion smooths this objective. Increasing diffusion produces smoother but potentially flatter learning signals.

  • Divergence definitions: The JS divergence is defined through KL divergences to the mixture distribution Pm = (Pr + Pg)/2.The derivation also introduces the broader f-divergence framework.
  • Toy example: For a toy example with disjoint supports, the JS divergence is not continuous and can provide no usable gradient for generator training.The derivation gives a divergence that is 0 when θ = 0 and log 2 when θ ≠ 0.
  • Toy example: The same discontinuity issue can occur when low-dimensional manifolds intersect only on a measure-zero set.The text presents Wasserstein-1 distance as a common remedy for non-overlapping supports.
  • Diffusion-based noise injection: Diffusion gives the real and generated toy distributions full support in R2, with overlap depending on the diffusion timestep.As t increases, the density weight on the original data decreases toward zero.
  • Diffusion-based noise injection: Increasing diffusion changes the JS-divergence curves from discontinuous to smoother and flatter, providing more useful but eventually smaller gradients.At intense diffusion such as t = 800, the flatter curve implies slower learning.
  • Diffusion-based noise injection: Diffusion makes the optimal discriminator well defined across the space and adjusts its behavior from assertive at smaller t to neutral at larger t.The derivation links timestep-dependent diffusion to a scale balancing discriminator behavior and gradient smoothness.

E DATASET DESCRIPTIONS

The experiments use image datasets spanning 32×32 to 512×512 resolutions and apply the Diffusion-GAN algorithm to the experimental models.

  • Dataset descriptions: CIFAR-10 contains 50k 32×32 training images across 10 categories, while STL-10 contains 100k unlabeled images resized to 64×64.The datasets provide low- and moderate-resolution image benchmarks.
  • Dataset descriptions: The LSUN experiments use 200k Bedroom images and 125k Church images, resized to 256×256 resolution.The passage specifies separate sample counts for the two LSUN subsets.
  • Experimental procedure: The paper provides Diffusion-GAN in Algorithm 1.The algorithm is part of the experimental implementation materials.

G HYPERPARAMETERS

Diffusion-GAN adds diffusion-specific hyperparameters while preserving the original GAN backbones’ learning hyperparameters and architectures. Its diffusion schedule, timestep range, and sampling strategy are configured to control noise injection across datasets.

  • Configuration: Diffusion-GAN introduces four hyperparameters: noise standard deviation σ, Tmax, T increasing threshold dtarget, and timestep sampling distribution pπ.The original GAN backbone learning hyperparameters remain unchanged.
  • Recommended settings: For images rescaled to [-1, 1], σ = 0.05 is fixed, while Tmax is typically 500 or 1000 and larger values are recommended for diverse datasets.A general starting configuration is (σ = 0.05, Tmax = 500, dtarget = 0.6, pπ = ‘uniform’).
  • Threshold and sampling: StyleGAN2-based models use dtarget = 0.6 across datasets except FFHQ, where dtarget = 0.8 performs slightly better in FID.The reported FFHQ setting is an exception to the general StyleGAN2 configuration.
  • Training procedure: Algorithm 1 alternates discriminator and generator updates using diffused real or generated samples, sampled timesteps, and the shared objective in Equation (3).The timestep list combines zeros with 32 samples from pπ, producing 64 timestep entries.
  • Reported configurations: The appendix reports separate diffusion and model configuration tables for the StyleGAN2- and ProjectedGAN-based experiments.StyleGAN2 settings are borrowed from Karras et al., while ProjectedGAN uses its recommended default configuration.
  • Diffusion config: The forward diffusion schedule uses linearly increasing βt, with T adaptively ranging from Tmin = 5 to Tmax = 1000 for pixel-level StyleGAN2 injection.For StyleGAN2, β0 = 0.0001, βT = 0.02, and σ = 0.05.

H IMPLEMENTATION DETAILS

The implementation samples diffusion timesteps and perturbed images during optimization, then integrates this process differently into StyleGAN2, ProjectedGAN, and InsGen. Adaptive diffusion adjusts and clips the total number of forward steps.

  • Shared pipeline: At each update, the method samples a timestep for each data point, draws a diffused sample from the analytic Gaussian distribution, and optimizes using y and t instead of x.The adaptive T is clipped to [Tmin, Tmax].
  • Implementation pipeline: The implementation includes an additional diffusion sampling pipeline whose configurations are specified in Appendix G.The pipeline is used during both discriminator and generator optimization.
  • Diffusion StyleGAN2: Diffusion StyleGAN2 injects timestep t into the discriminator’s mapping network and trains both networks with diffused samples y and t.The original class-label input is replaced by the discrete timestep.
  • Diffusion ProjectedGAN: Diffusion ProjectedGAN treats diffusion as data augmentation, feeding the discriminator diffused images y while ignoring t in Dϕ(y).This minimizes architectural modifications to ProjectedGAN.
  • Diffusion InsGen: Diffusion InsGen preserves its contrastive-learning component and injects timestep t into the discriminator similarly to Diffusion StyleGAN2.Training uses diffused samples y together with t.

I ABLATION ON THE MIXING PROCEDURE AND T ADAPTIVENESS

Ablations examine timestep adaptiveness and the priority-versus-uniform mixing procedure, while additional experiments test diffusion noise injection across GAN architectures and report CIFAR-10 inception scores.

  • Mixing procedure: Uniform mixing can outperform priority mixing on some datasets, indicating that the timestep mixing procedure remains a target for future optimization.Priority mixing is based on intuition, and the comparison is reported in Table 7.
  • T adaptiveness: Adaptive diffusion produces faster-converging training curves and lower final FIDs than a non-adaptive strategy.This comparison is reported in Figure 7.
  • Architecture transfer: Diffusion-DCGAN and Diffusion-SNGAN clearly outperform their corresponding baseline GANs on CIFAR-10 FID.The comparison is summarized in Table 8.
  • CIFAR-10 evaluation: Diffusion StyleGAN2 is evaluated using CIFAR-10 Inception Score alongside state-of-the-art GAN and diffusion-model baselines.Table 9 also reports sampling time using number of function evaluations (NFE).

L MORE GENERATED IMAGES

The paper provides additional randomly generated images from Diffusion ProjectedGAN, Diffusion InsGen, and Diffusion StyleGAN2 across LSUN, AFHQ, and FFHQ datasets.

  • Dataset coverage: The additional image examples cover LSUN-Bedroom, LSUN-Church, AFHQ, and FFHQ datasets.The examples are distributed across Figures 8, 9, and 10.
  • Diffusion ProjectedGAN: Additional Diffusion ProjectedGAN images are shown for LSUN-Bedroom with FID 1.43 and Recall 0.58, and LSUN-Church with FID 1.85 and Recall 0.65.These results are presented in Figure 8.
  • Diffusion InsGen: Additional Diffusion InsGen images are shown for AFHQ-Cat with FID 2.40, AFHQ-Dog with FID 4.83, and AFHQ-Wild with FID 1.51.These results are presented in Figure 9.
  • Diffusion StyleGAN2: Additional Diffusion StyleGAN2 images are shown for FFHQ with FID 3.71 and Recall 0.43.These results are presented in Figure 10.
Loading 2206.02262v4…