Source-linked AI summary
Cascaded Diffusion Models for High Fidelity Image Generation
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, Tim Salimans
TL;DR
The paper addresses high-fidelity image generation on class-conditional ImageNet without auxiliary classifier guidance. It uses cascaded diffusion models with conditioning augmentation, achieving superior benchmark results while relying on thousands of low-resolution diffusion timesteps and leaving classifier guidance unused.
Problem
Prior diffusion-model demonstrations of high-quality samples focused mainly on modest datasets or strongly conditioned data, motivating evaluation on large high-fidelity class-conditional ImageNet.
Method
The method cascades separately trained diffusion models across increasing resolutions and augments the low-resolution conditioning inputs of super-resolution models.
Results
Cascaded diffusion models outperform BigGAN-deep and VQ-VAE-2 on ImageNet using FID and classification accuracy, without auxiliary classifiers.
Takeaways & Limitations
Conditioning augmentation improves cascading pipelines by combating compounding error from train-test mismatch in super-resolution models.
Takeaways & Limitations
Without classifier guidance, the models use thousands of low-resolution diffusion timesteps and underperform ADM with classifier guidance on FID and Inception scores.
Abstract
from arXiv · showhide
We show that cascaded diffusion models are capable of generating high fidelity images on the class-conditional ImageNet generation benchmark, without any assistance from auxiliary image classifiers to boost sample quality. A cascaded diffusion model comprises a pipeline of multiple diffusion models that generate images of increasing resolution, beginning with a standard diffusion model at the lowest resolution, followed by one or more super-resolution diffusion models that successively upsample the image and add higher resolution details. We find that the sample quality of a cascading pipeline relies crucially on conditioning augmentation, our proposed method of data augmentation of the lower resolution conditioning inputs to the super-resolution models. Our experiments show that conditioning augmentation prevents compounding error during sampling in a cascaded model, helping us to train cascading pipelines achieving FID scores of 1.48 at 64x64, 3.52 at 128x128 and 4.88 at 256x256 resolutions, outperforming BigGAN-deep, and classification accuracy scores of 63.02% (top-1) and 84.06% (top-5) at 256x256, outperforming VQ-VAE-2.
1. Introduction
The paper improves diffusion-model image quality on class-conditional ImageNet using cascaded multi-resolution generation and conditioning augmentation for super-resolution stages.
- 1. Introduction: The paper studies cascaded diffusion pipelines for class-conditional ImageNet generation, extending prior diffusion work beyond modest datasets or strongly conditioned data.The stated goal is to improve sample quality on large, high-fidelity data.
- 1. Introduction: Cascading uses separately trained models at increasing resolutions, with a low-resolution base model followed by super-resolution models.Sampling proceeds sequentially from the base model through the super-resolution stages.
- 1. Introduction: Conditioning augmentation applies strong data augmentation to each super-resolution model’s conditioning input and is crucial for high-resolution sample fidelity.The authors introduce this technique to address cascading-pipeline errors.
- 1. Introduction: Conditioning augmentation alleviates compounding error caused by train-test mismatch during sampling in cascading pipelines.The paper relates this mismatch to exposure bias in sequence modeling.
- 1. Introduction: CDM yields high-fidelity samples superior to BigGAN-deep and VQ-VAE-2 on FID and classification accuracy without combining the generative models with classifiers.The classification-accuracy advantage over VQ-VAE-2 is described as large.
2. Background
Diffusion models gradually corrupt data and learn a reverse process that reconstructs it, with training based on an evidence lower bound or modified losses. Conditional diffusion incorporates signals such as class labels or low-resolution images into the reverse model, typically using U-Net architectures.
- Diffusion models: A diffusion model progressively destroys data over T timesteps until the terminal state is approximately standard normal.The reverse process starts from this standard normal prior and learns to match the forward process.
- Diffusion models: The reverse process is trained by optimizing an evidence lower bound whose Gaussian posterior terms can be evaluated in closed form.This permits stochastic gradient updates using randomly selected terms of the objective.
- Training objectives: Modified objectives can improve sample quality at the cost of log likelihood, with the specific loss depending on whether Σθ is learned or fixed.The non-learned case uses a simplified denoising-style loss, while the learned case uses a hybrid objective.
- Conditional generation: Conditional diffusion models learn pθ(x0|c) by providing a conditioning signal c to the reverse-process neural network.Examples include class labels for class-conditional generation and low-resolution images for super-resolution.
- Architectures: U-Nets map corrupted images to reverse-process parameters, while scalar conditions are embedded in intermediate layers and image conditions are concatenated channelwise.These architectural choices preserve the spatial dimensions needed for image diffusion.
3. Conditioning Augmentation in Cascaded Diffusion Models
Cascaded diffusion models generate high-resolution images through sequential low-resolution and super-resolution diffusion models. Conditioning augmentation modifies the low-resolution inputs to super-resolution models, with Gaussian noise and blur serving as effective variants for different resolutions.
- Cascaded Diffusion Pipelines: A cascading pipeline models high-resolution data with a low-resolution diffusion model followed by super-resolution diffusion models.The models can extend across more than two resolutions and can be conditioned on class information or other signals.
- Cascaded Diffusion Pipelines: Cascading allocates modeling capacity to low resolutions, which are empirically important for sample quality and computationally efficient to train and sample.Individual models can also be trained independently, allowing architecture choices to be tuned at each resolution.
- Conditioning Augmentation: Conditioning augmentation trains each super-resolution model on an augmented version of its low-resolution input.The most effective low-resolution augmentation adds Gaussian noise, while high-resolution upsampling benefits from random Gaussian blur.
- Conditioning Augmentation: Gaussian blurring uses a 3 × 3 filter with randomly sampled sigma during training, applied to 50% of examples, but not during inference.The method was most effective for upsampling to 128 × 128 and 256 × 256 images; inference-time blurring was not helpful in initial experiments.
- Truncated Conditioning Augmentation: Truncated conditioning augmentation stops the low-resolution reverse process at timestep s and conditions super-resolution on the resulting noisy sample.It requires modifications to super-resolution training and architecture but does not change the initial low-resolution model, and was most useful below 128 × 128 resolution.
- Truncated Conditioning Augmentation: Training with truncated augmentation separately optimizes the low- and high-resolution models, with the super-resolution model receiving conditioning corrupted through the low-resolution forward process.The truncation timestep s determines how far the low-resolution process is trained and how strongly the conditioning signal is corrupted.
- Non-truncated Conditioning Augmentation: Non-truncated conditioning augmentation samples the low-resolution output fully and then corrupts it before super-resolution, while achieving sample-quality metrics similar to truncated augmentation.Amortizing a super-resolution model over uniformly sampled s enables post-training search over augmentation strength without retraining.
4. Experiments
Experiments show that cascaded diffusion improves ImageNet sample quality across resolutions when paired with conditioning augmentation. The results also identify a trade-off: augmentation must be moderate, and avoiding classifier guidance requires many low-resolution diffusion timesteps.
- 4.1 Main Cascading Pipeline Results: CDM outperforms BigGAN-deep in FID across the evaluated 64×64, 128×128, and 256×256 ImageNet resolutions.GANs achieve better Inception scores when their truncation parameter is optimized for that metric.
- 4.1 Main Cascading Pipeline Results: CDM outperforms VQ-VAE-2 and BigGAN-deep at 128×128 and 256×256 on Classification Accuracy Score by a significant margin.The comparison suggests better potential performance on downstream tasks.
- 4.3 Conditioning Augmentation Experiments up to 64×64: Without conditioning augmentation, cascading degrades FID from 2.35 to 6.02, whereas sufficient augmentation improves it to 2.13.The augmented cascading pipeline therefore outperforms the non-cascaded 64×64 baseline in this experiment.
- 4.3 Conditioning Augmentation Experiments up to 64×64: Moderate conditioning augmentation improves sample quality, but excessive augmentation makes the super-resolution model behave like an unconditioned model.With generated conditioning, quality changes non-monotonically as truncation time increases; with ground-truth conditioning, it degrades monotonically.
- 4.3 Conditioning Augmentation Experiments up to 64×64: Conditioning augmentation combats compounding error caused by train-test mismatch between generated low-resolution inputs and ground-truth training inputs.The paper recommends non-truncated augmentation for practical benefits because truncated and non-truncated variants are approximately equally effective.
- 4.4 High-Resolution Super-Resolution: Gaussian blur augmentation improves 256×256 samples over no blur, with σ ∼U(0.4, 0.6) producing the best reported results.Additional gains come from class conditioning, large-batch training, and random-flip augmentation.
5. Related Work
The paper situates cascaded diffusion among diffusion-model developments and contrasts it with classifier-guided diffusion. Its approach avoids auxiliary classifiers, but this choice requires thousands of low-resolution diffusion timesteps.
- 5. Related Work: Cascaded diffusion modifies multi-resolution generation by using separately trained models that progressively upsample samples.The related formulation considers downsampling in the forward process and upsampling in reverse transitions, but does not guarantee conditional Gaussian reverse transitions.
- 5. Related Work: Concurrent ADM work uses classifier guidance to improve ImageNet samples by modifying the sampler with gradients from an auxiliary image classifier.The paper instead focuses on improving sample quality through cascading without introducing that extra model element.
- 5. Related Work: The authors avoid classifier guidance partly because FID and Inception Score are computed from ImageNet classifier activations, creating a risk of metric gaming.They characterize cascading and classifier guidance as complementary techniques that could be applied together.
- 5. Related Work: Avoiding classifier guidance requires thousands of diffusion timesteps in low-resolution models, whereas ADM uses hundreds.ADM with classifier guidance outperforms these models on FID and Inception scores, while these models outperform ADM without classifier guidance.
6. Conclusion
The conclusion presents cascaded diffusion with conditioning augmentation as a way to outperform state-of-the-art generative models on class-conditional ImageNet. It attributes the improvement to reducing compounding error from train-test mismatch in super-resolution models.
- 6. Conclusion: Cascaded diffusion models outperform BigGAN-deep and VQ-VAE-2 on class-conditional ImageNet generation using FID and classification accuracy score.The reported approach uses conditioning augmentation in the super-resolution models.
- 6. Conclusion: Conditioning augmentation improves sample quality by combating compounding error from train-test mismatch in cascading super-resolution models.The paper also proposes amortized training and testing over varying augmentation strengths.
- 6. Conclusion: The work is presented as a conceptual study of diffusion models in their original form with minimal extra techniques.The authors hope it will inspire future advances in diffusion-model capabilities.
- 6. Conclusion: Image generation may have malicious uses, although the authors identify potential benefits for downstream applications such as data compression.This is stated as a possible negative impact and potential benefit rather than an evaluated result.
Appendix A. Samples
The appendix provides ImageNet samples grouped by relative classification accuracy and LSUN samples from bedroom and church subsets. The supplied captions identify the classes represented in the best- and worst-scoring ImageNet groups.
- Appendix A. Samples: Figure 11 shows samples from the classes with the best relative classification accuracy score, with each row representing one ImageNet class.The listed classes include Tiger Cat, Gong, Coffee Mug, Squirrel Monkey, Miniature Schnauzer, and Corn.
- Appendix A. Samples: Figure 12 shows samples from classes with the worst relative classification accuracy score, including Letter Opener, Plate, Overskirt, Tobacco Shop, Black-and-tan Coonhound, and Bathtub.Each row represents a specific ImageNet class.
- Appendix A. Samples: Figure 13 presents 128×128 LSUN samples from the bedroom subset in its first six rows and the church subset in its last rows.
B.1 ImageNet
The ImageNet cascading pipelines use separately specified diffusion architectures and training configurations across the base and super-resolution models.
- Architecture: The ImageNet pipelines use U-Net architecture hyperparameters including base channel counts and channel multipliers per resolution.The paper describes each pipeline model through its diffusion process, neural network architecture, and training hyperparameters.
- Compute: 256 TPU-v3 cores are used for the higher-compute ImageNet configurations.
- Compute: 128 TPU-v3 cores are used for the listed ImageNet configurations.
- Conditioning: Continuous noise conditioning is included in some listed ImageNet configurations.
- Architecture: Attention is applied at resolutions 8, 16, or both, depending on the ImageNet configuration.
B.2 LSUN
The LSUN Bedroom and Church pipelines share the same listed hyperparameters, including residual-block, attention-head, hardware, and noise-conditioning settings.
- Shared configuration: The LSUN Bedroom and Church pipelines use the same hyperparameters.
- Architecture: Each listed LSUN configuration uses 3 residual blocks per resolution.
- Architecture: The listed LSUN configurations use attention heads with dimension 64.
- Compute: The LSUN pipelines use 64 TPU-v3 cores in the listed configurations.
- Conditioning: Continuous noise conditioning is used in the listed LSUN configurations.