PaperlayerView exact revision on arXiv

Source-linked AI summary

Cascaded Diffusion Models for High Fidelity Image Generation

Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, Tim Salimans

arXiv:2106.15282v3cs.CVcs.AIcs.LG

TL;DR

Earlier diffusion-model work demonstrating high-quality samples focused mainly on modest datasets or data with strong conditioning, leaving performance on large, high-fidelity data less established. This paper uses cascaded diffusion models with conditioning augmentation for class-conditional ImageNet generation and reports that they outperform BigGAN-deep and VQ-VAE-2 on FID and classification accuracy.

  • Problem

    Prior diffusion work demonstrating high-quality samples focused mainly on modest datasets or strongly conditioned data, leaving large, high-fidelity generation less explored.

    Source: §1 · p. 2

  • Method

    The paper cascades separately trained diffusion models from low to higher resolutions and augments the conditioning inputs to its super-resolution models.

    Source: §1 · pp. 2–3

  • Results

    On class-conditional ImageNet, cascaded models with conditioning augmentation outperform BigGAN-deep and VQ-VAE-2 on FID and classification accuracy without classifier assistance.

    Source: §1 · p. 3, §6 · p. 23

  • Takeaways & Limitations

    Conditioning augmentation combats compounding error from train-test mismatch in super-resolution models.

    Source: §6 · p. 23

    Avoiding classifier guidance requires thousands of diffusion timesteps in low-resolution models, and ADM with classifier guidance outperforms these models on FID and Inception scores.

    Source: §5 · p. 23

Abstract

from arXiv · show

We show that cascaded diffusion models are capable of generating high fidelity images on the class-conditional ImageNet generation benchmark, without any assistance from auxiliary image classifiers to boost sample quality. A cascaded diffusion model comprises a pipeline of multiple diffusion models that generate images of increasing resolution, beginning with a standard diffusion model at the lowest resolution, followed by one or more super-resolution diffusion models that successively upsample the image and add higher resolution details. We find that the sample quality of a cascading pipeline relies crucially on conditioning augmentation, our proposed method of data augmentation of the lower resolution conditioning inputs to the super-resolution models. Our experiments show that conditioning augmentation prevents compounding error during sampling in a cascaded model, helping us to train cascading pipelines achieving FID scores of 1.48 at 64x64, 3.52 at 128x128 and 4.88 at 256x256 resolutions, outperforming BigGAN-deep, and classification accuracy scores of 63.02% (top-1) and 84.06% (top-5) at 256x256, outperforming VQ-VAE-2.

1. Introduction

This paper improves class-conditional ImageNet generation with cascaded diffusion models and introduces conditioning augmentation to improve super-resolution sample quality. The approach achieves better benchmark comparisons than BigGAN-deep and VQ-VAE-2 without an auxiliary classifier.

1. Introduction

  • The work targets high sample quality on large, high-fidelity datasets without strong conditioning signals, using straightforward techniques and no extra image classifier to boost quality metrics.Earlier diffusion work demonstrating high-quality samples had focused mostly on modest-size datasets or data with strong conditioning.Source: §1 · p. 2
  • The cascade uses separately trained diffusion models: a low-resolution base model generates samples, then super-resolution models sequentially upsample them to higher resolutions.Although other generative model types could be used in cascades, this work restricts its pipeline to diffusion models.Source: §1 · p. 2
  • Conditioning augmentation—strongly augmenting super-resolution models’ conditioning inputs—is critical to high sample fidelity and alleviates compounding error from train-test mismatch.Gaussian augmentation is key for low-resolution upsampling, Gaussian blurring for high-resolution upsampling, and amortized training enables post-training searches for optimal augmentation levels.Source: §1 · p. 3
  • Cascaded Diffusion Models outperform BigGAN-deep and VQ-VAE-2 on FID and classification accuracy, with the latter advantage reported by a large margin, without an auxiliary classifier.The introduction also reports better FID than BigGAN-Deep at any truncation value and better classification accuracy than VQ-VAE-2.Source: §1 · p. 3

2. Background

The background presents diffusion as a learned reverse process, then extends it to conditional generation and describes how U-Nets receive conditioning inputs.

2.1 Diffusion Models

  • A diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020) corrupts x0 over T steps toward Gaussian noise and learns a reverse process by optimizing an ELBO.Gaussian forward-process posteriors make the ELBO’s KL terms closed-form, enabling stochastic-gradient training on randomly selected objective terms.Source: §2.1 · pp. 3–4

2.2 Conditional Diffusion Models

  • Conditional diffusion learns pθ(x0|c) by feeding c into the reverse-process network; c may be a class label or low-resolution image, while the forward process remains unchanged.Source: §2.2 · pp. 4–5

2.3 Architectures

3. Conditioning Augmentation in Cascaded Diffusion Models

A cascaded diffusion model generates a low-resolution image first, then uses one or more super-resolution diffusion models to produce higher-resolution images; the pipeline can also be class-conditioned. The section’s central method is conditioning augmentation: augmenting low-resolution inputs during super-resolution training to improve sample quality.

3. Conditioning Augmentation in Cascaded Diffusion Models

  • A cascade factors high-resolution generation into a low-resolution diffusion model and a conditional super-resolution model, allowing independent training and resolution-specific architecture choices.Most modeling capacity can be dedicated to low resolutions, which the authors report are empirically most important for sample quality and are more computationally efficient to train and sample.Source: §3 · p. 5

3.1 Blurring Augmentation

  • For 128×128 and 256×256 upsampling, Gaussian blurring with a 3×3 filter and randomly sampled σ is effective; it is applied to 50% of training examples, not at inference.More generally, the authors found Gaussian noise most effective at low resolutions and blur at high resolutions; they sometimes amortized over augmentation strengths and selected one by post-training search.Source: §3 · p. 6, §3.1 · p. 7

3.2 Truncated Conditioning Augmentation

  • Truncated conditioning augmentation stops the low-resolution reverse process at s > 0 and trains the super-resolution model on z_s corrupted by the low-resolution forward process.A single super-resolution model can be trained over random truncation times s by taking s as an additional timestep embedding input; low-resolution model training remains unchanged.Source: §3.2 · pp. 7–8

3.3 Non-truncated Conditioning Augmentation

  • Non-truncated augmentation uses the full low-resolution reverse process, then corrupts its output for the super-resolution model; its sample-quality metrics are similar to truncation.The authors report this similarity empirically in Section 4.3.Source: §3.3 · pp. 8–9

4. Experiments

Experiments evaluate class-conditional ImageNet cascades and show that conditioning augmentation is central to their quality, including at higher resolutions. The study also tests whether its benefits extend to unconditional LSUN generation.

4.1 Main Cascading Pipeline Results

  • CDM outperforms VQ-VAE-2 and BigGAN-deep on CAS at 128×128 and 256×256, and its samples exceed real-data classifier accuracy on 96 classes, versus 6 for BigGAN-deep and 31 for VQ-VAE-2.Figure 7 compares classwise scores for real and generated data at 256×256.Source: §4.1 · pp. 12–13, Figure 7 · p. 13, Table 1 · p. 14

4.2 Baseline Model Improvements

  • For the non-cascaded 64×64 baseline, dropout slowed convergence but improved best FID and Inception scores over longer training, while adding forward-process noise to timestep 2000 and restarting reverse sampling further improved training-set FID and Inception score.The baseline reimplemented and improved a 4000-timestep model from Improved DDPM.Source: §4.2 · pp. 13, 16, Table 2 · p. 16

4.1 Main Cascading Pipeline Results

  • CDM outperforms BigGAN-deep on FID at 64×64, 128×128, and 256×256, though GANs achieve higher Inception scores with optimized truncation; CDM also beats diffusion models without classifier guidance.The main results compare the cascaded model with image-generation baselines across the reported resolutions.Source: §4.1 · p. 12, Table 1 · p. 14

4.3 Conditioning Augmentation Experiments up to 64×64

  • At 64×64, sufficient conditioning augmentation reverses a cascade’s FID degradation from 2.35 to 6.02, reaching 2.13 and outperforming the non-cascaded baseline.This small-scale experiment used a 16×16 base model and a 16×16→64×64 super-resolution model; without augmentation, the cascade performed worse than the baseline.Source: §4.3 · p. 16, Table 2 · p. 16
  • Conditioning augmentation works by reducing train-test mismatch and compounding error when super-resolution models condition on generated inputs; moderate augmentation helps, but too much degrades quality.With ground-truth conditioning, quality instead degrades monotonically as truncation time increases; truncated and non-truncated augmentation are approximately equally effective, with non-truncated augmentation recommended for practical benefits.Source: §4.3 · p. 18

4.4 Experiments at 128×128 and 256×256

  • For 256×256 upsampling, Gaussian blur improves sample scores over no blur, with σ ∼U(0.4, 0.6) best; class conditioning and increasing batch size from 256 to 1024 further improve performance.Blur was applied to half of training examples and omitted at inference; random flips yield marginal improvements.Source: §4.4 · pp. 18, 21, Table 4 · p. 20
  • Using 100 inference steps for high-resolution super-resolution models, the 64×64→256×256 FID deteriorates only marginally at 4 steps, with no concrete improvement from increasing steps from 100 to 1000.The authors tune the noise schedule for each inference step count without retraining.Source: Figure 10 · p. 20, §4.4 · p. 21

4.5 Experiments on LSUN

  • On LSUN Bedroom and Church, nonzero conditioning augmentation improves sample quality in unconditional 64×64→128×128 cascades, with quality improving and then degrading as truncation time increases.Each validation set contains only 300 examples, which the authors note explains the relatively large FID scores.Source: §4.5 · p. 21

5. Related Work

The work distinguishes itself through cascaded diffusion and conditioning augmentation, improving sample quality without classifier guidance. It contrasts this approach with ADM’s classifier-guided results and explains the associated timestep tradeoff.

5. Related Work

  • A single diffusion process with intermediate downsampling would require reverse upsampling transitions not guaranteed to be conditional Gaussian; dedicated conditional diffusion models make those steps more flexible.The paper contrasts its construction with transitions that are guaranteed conditional Gaussian under sufficiently slow diffusion.Source: §5 · p. 21
  • The work is distinguished by extensive study of conditioning augmentation and deeper cascades, and its augmentation resembles scheduled sampling (Bengio et al., 2015).It builds on DDPM (Ho et al., 2020) with modifications from Improved DDPM (Nichol and Dhariwal, 2021); prior diffusion cascades include SR3 (Saharia et al., 2021) and ADM (Dhariwal and Nichol, 2021).Source: §5 · p. 22
  • The work improves sample quality through cascading alone and avoids an extra image classifier because FID and Inception Score use ImageNet classifier activations (Dhariwal and Nichol, 2021).The authors argue classifier guidance risks gaming those metrics.Source: §5 · pp. 22–23
  • Thousands of low-resolution diffusion steps are needed without classifier guidance, versus ADM’s hundreds; guided ADM outperforms their models on FID and Inception scores, while unguided ADM is outperformed (Dhariwal and Nichol, 2021).The authors expect classifier guidance and cascading to complement each other and improve their results when combined.Source: §5 · p. 23

6. Conclusion

Cascaded diffusion models outperform state-of-the-art ImageNet class-conditional generators when paired with conditioning augmentation. The conclusion attributes this improvement to reduced compounding error and frames the work as a conceptual study with potential beneficial applications and possible misuse.

6. Conclusion

  • Cascaded diffusion models with conditioning augmentation outperform BigGAN-deep and VQ-VAE-2 on ImageNet class-conditional generation by FID and classification accuracy.The conclusion says conditioning augmentation introduces data augmentation into super-resolution models.Source: §6 · p. 23
  • Conditioning augmentation improves sample quality by combating compounding error from train-test mismatch in super-resolution models, with practical training and testing methods amortized over varying augmentation levels.Source: §6 · p. 23
  • The work is presented as a conceptual study of diffusion models’ image-synthesis capabilities in their original form with minimal extra techniques, with potential benefits for data compression and risk of malicious use.The authors hope it inspires future advances in diffusion-model capabilities.Source: §6 · p. 23

Appendix A. Samples

The appendix presents generated samples from ImageNet classes grouped by relative classification accuracy, alongside LSUN bedroom and church samples.

Appendix A. Samples

  • Figure 11 shows samples from six ImageNet classes with the best relative classification accuracy score, including Tiger Cat, Gong, Coffee Mug, Squirrel Monkey, Miniature Schnauzer, and Corn.Each row represents one class.Source: Figure 11 · p. 24
  • Figure 12 shows samples from six ImageNet classes with the worst relative classification accuracy score, including Letter Opener, Plate, Overskirt, Tobacco Shop, Black-and-tan Coonhound, and Bathtub.Each row represents one class.Source: Figure 12 · p. 25
  • Figure 13 presents LSUN 128x128 samples, with bedroom examples in the first six rows and church examples in the last six rows.Source: Figure 13 · p. 26

Appendix B. Hyperparameters · B.1 ImageNet

Appendix B.1 describes diffusion processes, U-Net architectures, and training hyperparameters for the ImageNet cascading-pipeline models. It identifies the cosine noise schedule and hybrid variance-learning loss with Improved DDPM (Nichol and Dhariwal, 2021).

B.1 ImageNet

  • The ImageNet model descriptions cover diffusion processes, U-Net architecture, and training hyperparameters, including a cosine noise schedule and hybrid reverse-variance loss.The passage also notes that some models use ¯αt for post-training sampler tuning (Chen et al., 2021; Saharia et al., 2021).Source: §B.1 · p. 27
  • Listed attention resolutions are either 8 and 16, or 16 alone.The passages do not identify which models use each setting.Source: §B.1 · pp. 27–29
  • The cited hardware entries report TPU-v3 configurations at both 256 cores and 128 cores, establishing that the listed research setups include these two distinct core counts.Source: §B.1 · pp. 27–29
  • Continuous noise conditioning is listed for two model entries.The passages do not identify the models using it.Source: §B.1 · pp. 28–29

B.2 LSUN

The LSUN Bedroom and Church pipelines used the same hyperparameters. Their configuration included three residual blocks per resolution, 64-dimensional attention heads, 64 TPU-v3 cores, and continuous noise conditioning.

B.2 LSUN

  • The passage states that the LSUN Bedroom and Church pipelines used the same hyperparameters for both datasets, but does not list the individual settings in the cited text.Source: §B.2 · p. 30
  • Each pipeline used 3 residual blocks per resolution and attention heads with dimension 64.The same architecture settings are listed for both datasets.Source: §B.2 · p. 30
  • Training used 64 TPU-v3 cores and continuous noise conditioning.Both settings are included in the LSUN pipeline hyperparameters.Source: §B.2 · p. 30

Related papers

Loading 2106.15282v3…