PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models Beat GANs on Image Synthesis

Prafulla Dhariwal, Alex Nichol

arXiv:2105.05233v4cs.LGcs.AIcs.CVstat.ML

TL;DR

GANs offered stronger visual quality, while likelihood-based models generally provided better coverage but remained behind on sample quality. This paper improves diffusion architectures and adds classifier guidance, showing diffusion models can surpass state-of-the-art GANs while retaining broader distribution coverage.

  • Problem

    Likelihood-based generative models offered diversity and scalability but still lagged GANs in visual sample quality and sampling speed.

  • Method

    The paper improves diffusion-model architecture and uses classifier guidance to trade diversity for fidelity through classifier gradients.

  • Results

    Diffusion models achieve the best FID on each evaluated task, outperform higher-resolution GANs while maintaining higher recall, and can sample in 25 diffusion steps.

  • Takeaways & Limitations

    Diffusion models can match or exceed GAN image quality across unconditional and class-conditional synthesis while preserving better distribution coverage.

  • Takeaways & Limitations

    Diffusion models remain slower than GANs at sampling time because they require multiple denoising steps and forward passes.

Abstract

from arXiv · show

We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128$\times$128, 4.59 on ImageNet 256$\times$256, and 7.72 on ImageNet 512$\times$512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256$\times$256 and 3.85 on ImageNet 512$\times$512. We release our code at https://github.com/openai/guided-diffusion

1 Introduction

The introduction frames diffusion models as scalable, distribution-covering alternatives to GANs, whose sample-quality advantages come with diversity and training challenges. It proposes improving diffusion architectures and adding a controllable diversity–fidelity tradeoff to close the visual-quality gap.

  • Motivation: GANs lead image-generation benchmarks by sample-quality metrics, but may capture less diversity and often collapse without carefully chosen training settings.The cited metrics include FID, Inception Score, and Precision.
  • Motivation: Likelihood-based models generally capture more diversity and scale or train more easily than GANs, but produce lower-quality samples and usually sample more slowly.VAEs are identified as an exception to the slower-sampling comparison.
  • Diffusion models: Diffusion models generate images by gradually removing noise while offering distribution coverage, a stationary training objective, and easy scalability.Their training objective is expressed as a reweighted variational lower-bound, and they already achieve state-of-the-art results on CIFAR-10.
  • Contributions: The authors hypothesize that diffusion models trail GANs because GAN architectures are more refined and GANs trade diversity for fidelity, motivating architectural improvements and guided sampling.The stated aim is to bring both benefits to diffusion models: stronger architecture and a controllable diversity–fidelity tradeoff.
  • Paper organization: The paper introduces architecture improvements in Section 3 and classifier-gradient guidance during diffusion sampling in Section 4.The introduction states that the architecture changes substantially improve FID and that classifier gradients guide sampling.

2 Background

Diffusion models generate samples by reversing a gradual noising process, with a noise predictor trained to estimate the noise in corrupted data. The section also summarizes improved reverse-process parameterization, alternative sampling, and metrics for evaluating sample quality.

  • Diffusion models: Diffusion sampling starts from noise xT and repeatedly produces less-noisy states until reaching a final sample x0.Each timestep corresponds to a noise level, and xt mixes the signal x0 with noise ϵ.
  • Diffusion models: The model ϵθ(xt, t) predicts the noise component of xt and is trained with the mean-squared error ||ϵθ(xt, t) − ϵ||2.Training samples randomly draw x0, timestep t, and noise ϵ to construct xt.
  • Diffusion models: Reverse transitions pθ(xt−1|xt) are modeled as diagonal Gaussians whose mean is computed from ϵθ(xt, t), while the variance can be fixed or learned.Nichol and Dhariwal’s learned variance and hybrid objective enable sampling with fewer steps without much drop in sample quality.
  • Recent improvements: DDIM provides an alternative non-Markovian sampling process, and setting reverse noise to 0 makes the mapping from latents to images deterministic.The authors adopt this sampling approach when using fewer than 50 sampling steps.
  • Evaluation metrics: FID is the default metric because it captures both diversity and fidelity, while Precision or IS measures fidelity and Recall measures diversity or distribution coverage.The section notes that these metrics are useful but imperfect proxies for human judgment.

3 Architecture Improvements

The architecture study evaluates changes to depth, attention, residual blocks, and embedding injection for diffusion models. The final architecture combines multi-resolution attention, BigGAN residual blocks, variable width, and adaptive group normalization, while excluding increased depth from further experiments because it slows training.

  • Architecture ablations: The ablations vary depth versus width, attention-head count, attention resolutions, and BigGAN residual blocks for upsampling and downsampling.The compared attention resolutions are 32×32, 16×16, and 8×8 rather than only 16×16.
  • Architecture ablations: Increased depth improves performance but increases training time and takes longer to reach the same performance as a wider model, so it is not used further.Aside from rescaling residual connections, the other architectural modifications improve performance and compound positively.
  • Attention configurations: More attention heads or fewer channels per head improve FID in the tested attention configurations.These experiments use 128 base channels, 2 residual blocks per resolution, multi-resolution attention, and BigGAN up/downsampling.
  • Embedding injection: Adaptive group normalization improves FID when injecting timestep and class embeddings into residual blocks.AdaGN applies a learned scale and bias after group normalization using a projection of the timestep and class embeddings.
  • Final architecture: The final default uses variable width, 2 residual blocks per resolution, multiple heads with 64 channels per head, attention at 32, 16, and 8 resolutions, BigGAN residual blocks, and AdaGN.This architecture is used for the rest of the paper.

4 Classifier Guidance

Classifier guidance conditions diffusion sampling with gradients from a class classifier, shifting or modifying predictions toward the desired label. Increasing gradient scale sharpens conditioning, improving fidelity while reducing diversity, and guidance substantially improves sample quality.

  • Stochastic sampling: Classifier guidance shifts each stochastic reverse-transition mean by Σg, where g is the classifier gradient, with an optional scale factor s.The resulting Gaussian transition preserves the unconditional covariance while moving its mean toward classifier-preferred samples.
  • DDIM sampling: For DDIM, classifier guidance instead modifies the noise prediction using a score-based conditioning trick and then applies the regular DDIM sampling procedure.The stochastic derivation does not apply to deterministic DDIM sampling.
  • Practical implementation: Classifier guidance uses ImageNet classifiers trained on the diffusion model’s noising distribution, with a UNet downsampling trunk and attention pooling at the 8x8 layer.Random crops are added to reduce overfitting before incorporating the classifier into diffusion sampling.
  • Gradient scaling: For s > 1, guidance samples from a sharper distribution proportional to p(y|x)^s, focusing on classifier modes for higher fidelity but less diversity.The scale factor amplifies larger classifier probabilities and therefore strengthens class conditioning.
  • Empirical effects: Classifier guidance improves precision at the cost of recall, while scaling beyond 1.0 smoothly trades diversity for higher precision and Inception Score.FID and sFID achieve their best values at an intermediate scale because they depend on both diversity and fidelity; guidance also outperforms BigGAN-deep in the reported FID–Inception Score trade-off.

5 Results

The improved diffusion architecture achieves state-of-the-art image generation across LSUN and ImageNet, while classifier guidance substantially improves higher-resolution results and preserves broader distribution coverage than GANs. Guidance and upsampling improve sample quality along complementary axes, with their combination producing the best FIDs.

  • Evaluation setup: Experiments evaluate unconditional diffusion models on LSUN bedroom, horse, and cat, and classifier-guided conditional models on ImageNet at 128×128, 256×256, and 512×512.The ImageNet models use 250 sampling steps, or 25 with DDIM; LSUN models use 1000 steps.
  • Sample quality: The improved architecture achieves the best FID on every task and the best sFID on all but one, reaching state-of-the-art quality on LSUN and ImageNet 64×64 without guidance.At higher ImageNet resolutions, classifier guidance enables substantial gains over the best GANs while maintaining higher distribution coverage.
  • Guidance and upsampling: Upsampling improves precision while retaining high recall, whereas classifier guidance provides a tradeoff between diversity and much higher precision.These methods therefore improve different dimensions of sample quality.
  • Guidance and upsampling: The best FIDs result from applying guidance at lower resolution before upsampling, indicating that guidance and upsampling complement one another.When combined, only the lower-resolution model is guided.

6 Related Work

Prior work connected score-based models and diffusion models, improving sample quality, audio generation, sampling speed, and ImageNet results. Related methods also explored diversity–fidelity control, likelihood-based generation, and classifier-controlled generative models.

  • Score-based and diffusion models: Score-based generative models model data distributions through gradients and Langevin dynamics, while diffusion models leveraged this connection to achieve excellent sample quality.Ho et al. connected score-based modeling with diffusion models after their introduction by Song and Ermon.
  • Subsequent diffusion-model advances: Subsequent work applied diffusion models to audio, improved samples with GAN-like setups, used stochastic differential equations, and accelerated sampling.These directions were pursued by Kong et al., Chen et al., Jolicoeur-Martineau et al., Song et al., and Nichol and Dhariwal.
  • Diversity–fidelity trade-offs: Previous diffusion-model work lacked a mechanism for trading diversity for fidelity, unlike GAN truncation and classifier rejection sampling approaches.GAN truncation decreases diversity while increasing fidelity as truncation increases.
  • Likelihood-based models: VQ-VAE and VQ-VAE-2 reduce training costs on large images with quantized latent codes, but their samples fall short of GANs without expensive rejection sampling and special metrics.These autoregressive likelihood-based models produce diverse, high-quality images but can remain blurry relative to GANs.
  • Classifier-controlled generation: Other research controlled generative models with pre-trained classifiers, including CLIP-guided GAN latent optimization and classifier-conditioned diffusion generation.Song et al. used a classifier to generate class-conditional CIFAR-10 images with a diffusion model.

7 Limitations and Future Work

Diffusion models remain slower than GANs because sampling requires multiple denoising steps, while classifier guidance is limited to labeled datasets. Future work includes faster sampling, unlabeled-data guidance, broader conditioning, and pretraining on large unlabeled datasets.

  • Limitations: Diffusion models sample more slowly than GANs because they require multiple denoising steps and forward passes.Distilling DDIM sampling into one step produces samples that are not yet competitive with GANs but outperform previous single-step likelihood models.
  • Limitations: Classifier guidance currently applies only to labeled datasets and lacks an effective diversity–fidelity tradeoff strategy for unlabeled data.Proposed extensions include clustering samples for synthetic labels or training discriminative models to distinguish true-distribution samples from sampling-distribution samples.
  • Future Work: Classifier gradients could condition pretrained generative models in diverse ways, including image generation from text captions using noisy CLIP.This direction is analogous to recent methods that guide GANs with text prompts.
  • Future Work: Large unlabeled datasets could pretrain powerful diffusion models that classifiers later improve using desirable properties.

8 Conclusion … B Detailed Formulation of DDPM

The paper concludes that improved diffusion architectures and classifier guidance achieve superior sample quality to state-of-the-art GANs, while its appendices analyze compute efficiency and DDPM formulation. The formulation describes Gaussian noising, learned reverse steps, variational training, and a simplified noise-prediction objective that produces better samples in practice.

  • 8 Conclusion: Diffusion models achieve better sample quality than state-of-the-art GANs through an improved architecture for unconditional generation and classifier guidance for class-conditional generation.Classifier-gradient scale controls the tradeoff between diversity and fidelity.
  • A Computational Requirements: The authors compare compute requirements because greater compute typically improves machine-learning results, targeting performance better than StyleGAN2 and BigGAN-deep at similar or lower budgets.
  • A.1 Throughput: Throughput is benchmarked using theoretical FLOPs assuming 100% NVIDIA Tesla V100 utilization and measured wall-clock time, including inter-machine communication when required.Each machine has 8 V100s.
  • A.1 Throughput: A naive PyTorch 1.7 implementation uses only 20-30% of hardware, while larger per-GPU batches and fused GroupNorm-Swish and Adam CUDA operations improve utilization.For ImageNet 128×128, the per-GPU batch size increases from 4 to 32 while fitting in GPU memory.
  • A.2 Early stopping: The ImageNet 128×128 model beats BigGAN-deep’s FID (6.02) after 500K iterations, while the 256×256 model does so after 750K iterations.These correspond to one eighth and roughly a third of training, respectively.
  • B Detailed Formulation of DDPM: DDPM defines a Markovian Gaussian noising process that gradually transforms data x0 into x1 through xT according to a variance schedule βt, with 1 − ᾱt giving arbitrary-timestep noise variance.The forward conditional q(xt|x0) is Gaussian, and q(xt−1|xt, x0) is Gaussian by Bayes theorem.
  • B Detailed Formulation of DDPM: Sampling starts from nearly isotropic Gaussian q(xT) and uses a neural network to approximate reverse transitions, trained with a variational lower bound whose practical alternative predicts noise ϵ.Ho et al. found the noise-prediction objective produces better samples in practice; reverse variance can be fixed to βtI or β̃tI.

C Nearest Neighbors for Samples · D Effect of Varying the Classifier Scale · E LSUN Diversity Comparison

The paper examines whether classifier guidance sacrifices sample uniqueness, how classifier scale changes generations, and how diffusion samples compare visually with StyleGAN2 and training data on LSUN. Nearest-neighbor results indicate that guided samples remain unique rather than being memorized.

  • C Nearest Neighbors for Samples: FID 4.59 is obtained with classifier scale 1 and 250 diffusion sampling steps on ImageNet 256×256.These settings correspond to the top samples in Figure 7.
  • C Nearest Neighbors for Samples: FID 5.44 is obtained with classifier scale 2.5 and 25 DDIM steps on ImageNet 256×256.These settings correspond to the bottom samples in Figure 7.
  • C Nearest Neighbors for Samples: Classifier guidance reduces generation diversity to achieve the models’ best FID, motivating a test for possible training-image recall.The concern is especially relevant as classifier scale increases.
  • C Nearest Neighbors for Samples: Nearest neighbors in InceptionV3 feature space show that classifier-guided ImageNet samples are unique and not stored in the training set.The analysis examined nearest neighbors for a handful of generated samples.
  • D Effect of Varying the Classifier Scale: Increasing classifier scale from 0.0 to 5.5 drastically changes some images while leaving others relatively unaffected.Each row uses a fixed noise seed, isolating the effect of classifier scale.
  • E LSUN Diversity Comparison: Figure 9 compares StyleGAN2 samples with truncation 1.0 against diffusion-model samples and training-set samples.For bedrooms, the comparison uses StyleGAN rather than StyleGAN2.

F Interpolating Between Dataset Images Using DDIM … J Using Fewer Sampling Steps on LSUN

The paper explores DDIM latent interpolation, reduced-temperature sampling, conditional diffusion, implementation choices, and sampling schedules for using fewer steps on LSUN. These experiments clarify reconstruction behavior, conditioning mechanisms, temperature effects, training defaults, and schedule-dependent sample quality.

  • F Interpolating Between Dataset Images Using DDIM: DDIM provides a deterministic implicit latent space that can encode real images and support interpolations between their latents.The process reverses an ODE to obtain latents for given real images.
  • F Interpolating Between Dataset Images Using DDIM: 250 reverse steps yield reasonable reconstructions, while reversing the first 249 steps avoids noise artifacts and enables cosine interpolation.Latents, class embeddings, and classifier log probabilities are interpolated using cos(θ)x0 + sin(θ)x1 as θ spans 0 to π.
  • F Interpolating Between Dataset Images Using DDIM: Without guidance, DDIM achieves almost perfect reconstructions, whereas classifier scale 2.5 produces only approximately similar reconstructions.The comparison uses a class-conditional 256×256 model and real dataset endpoints.
  • G Reduced Temperature Sampling: Temperature scaling on ImageNet 128×128 produced no substantial improvement in FID, Precision, or Recall, and low temperatures reduced both precision and recall.Noise scaling and epsilon scaling behaved similarly in the experiment.
  • H Conditional Diffusion Process: Conditional sampling uses pθ(xt|xt+1) multiplied by a classifier approximation pφ(y|xt) trained on noised images.The classifier approximates the noisy label distribution needed to condition the reverse diffusion process.
  • I Hyperparameters: Classifier-scale sweeps covered [0.5, 1, 2] for ImageNet 128×128 and ImageNet 256×256, and [1, 2, 3, 3.5, 4, 4.5, 5] for ImageNet 512×512.DDIM used separate scale sweeps for each resolution.
  • I Hyperparameters: The experiments generally used Adam or AdamW, 16-bit training with loss scaling, 32-bit weights and optimizer state, EMA rate 0.9999, and PyTorch.Architecture ablations used batch size 256 and 250 sampling steps, while 25-step DDIM used Song et al.'s uniform stride on ImageNet.
  • J Using Fewer Sampling Steps on LSUN: An improved sampling-time noise schedule largely closed the gap between 250-step and 1000-step LSUN sampling, with the sweep schedule outperforming uniform 250 steps on all three datasets.The sweep was not exhaustive because it required significant sampling compute, so superior schedules may exist.

K Samples from ImageNet 512×512

The best ImageNet 512×512 model produces samples with FID 3.85 across diverse and difficult classes. Guided samples using 250 steps and classifier scale 4.0 have FID 7.72.

  • Best 512×512 model: FID 3.85: The best 512×512 model generates samples spanning goldfish, arctic foxes, butterflies, elephants, flamingos, and tennis balls.The figure also includes cheeseburgers, fountains, balloons, tabby cats, lorikeets, and agarics.
  • Difficult classes: FID 3.85: The best 512×512 model generates difficult-class samples including bassoons, cabs, barbershops, tandem bicycles, ballplayers, and espresso makers.
  • Guided 512×512 model: FID 7.72: The guided 512×512 model uses 250 steps with classifier scale 4.0 to generate samples across the same ImageNet classes.Shown classes include goldfish, arctic foxes, monarch butterflies, elephants, flamingos, and tennis balls, as well as cheeseburgers, fountains, balloons, tabby cats, lorikeets, and agarics.
  • Random samples: FID 3.85 and FID 7.72: Random samples are shown from the best and guided ImageNet 512×512 models, respectively.

L Samples from ImageNet 256×256 · M Samples from LSUN

The ImageNet 256×256 samples demonstrate strong guided and best-model results across named classes, while LSUN samples report FID scores for bedroom, horse, and cat models. The figures specify sampling steps and classifier scales for the guided ImageNet models.

  • L Samples from ImageNet 256×256: FID 3.94 is reported for the best 256×256 model across samples from twelve ImageNet classes.The classes include goldfish, arctic fox, monarch butterfly, African elephant, flamingo, tennis ball, cheeseburger, fountain, balloon, tabby cat, lorikeet, and agaric.
  • L Samples from ImageNet 256×256: FID 4.59 is reported for the guided 256×256 model using 250 steps and classifier scale 1.0.The samples cover the same twelve listed ImageNet classes.
  • L Samples from ImageNet 256×256: FID 5.44 is reported for the guided 256×256 model using 25 DDIM steps and classifier scale 2.5.The samples cover the same twelve listed ImageNet classes.
  • L Samples from ImageNet 256×256: Random samples are shown from the best 256×256 model, which has FID 3.94.This figure presents random samples without additional sampling-step or classifier-scale specifications.
  • L Samples from ImageNet 256×256: Random samples are shown from the guided 256×256 model using 250 steps with classifier scale 1.0 and FID 4.59.The figure presents random samples from the guided model.
  • M Samples from LSUN: FID 1.90 is reported for the LSUN bedroom model using 1000 sampling steps.The figure shows random samples from this model.
  • M Samples from LSUN: FID 2.57 is reported for the LSUN horse model using 1000 sampling steps.The figure shows random samples from this model.
  • M Samples from LSUN: FID 5.57 is reported for the LSUN cat model using 1000 sampling steps.The figure shows random samples from this model.
Loading 2105.05233v4…