PaperlayerView exact revision on arXiv

Source-linked AI summary

Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise

Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, Tom Goldstein

arXiv:2208.09392v1cs.CVcs.LG

TL;DR

Diffusion models are commonly built around Gaussian noise, leaving unclear whether randomness is necessary for their generative behavior. This paper replaces noise with arbitrary deterministic image transforms and finds that the resulting framework can restore images degraded by blur, inpainting, and downsampling.

  • Problem

    It remains unclear whether Gaussian noise or any randomness is necessary for diffusion models to work in practice.

  • Method

    The paper trains restoration networks for arbitrary image degradations and alternates restoration with degradation updates to generate images.

  • Results

    The framework removes random noise entirely and restores images afflicted by deterministic degradations including blur, inpainting, and downsampling.

  • Takeaways & Limitations

    These results support a more diverse landscape of diffusion models beyond the Gaussian noise paradigm.

  • Takeaways & Limitations

    When the restoration operator is not a perfect inverse, Algorithm 1 incurs errors because x0 is not a fixed point of its update rule.

Abstract

from arXiv · show

Standard diffusion models involve an image transform -- adding Gaussian noise -- and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. The success of these fully deterministic models calls into question the community's understanding of diffusion models, which relies on noise in either gradient Langevin dynamics or variational inference, and paves the way for generalized diffusion models that invert arbitrary processes. Our code is available at https://github.com/arpitbansal297/Cold-Diffusion-Models

1 Introduction

The section questions whether Gaussian noise or randomness is necessary for diffusion models, proposing generalized models that invert arbitrary image transformations such as blurring and downsampling. It argues that successful noise-free diffusion challenges prevailing theoretical accounts and enables new generative-model designs.

  • 1 Introduction: Diffusion models traditionally train denoising networks to remove Gaussian noise and generate images from pure Gaussian noise [Ramesh et al., 2022].Their theoretical foundations also emphasize Gaussian noise through Langevin-dynamics interpretations [Sohl-Dickstein et al., 2015, Song and Ermon, 2019].
  • 1 Introduction: This work generalizes diffusion models beyond Gaussian noise by training restoration networks to invert arbitrary image transformations, including blurring and downsampling.The proposed framework examines models built around image deformations rather than restricting diffusion to noise-based processes.
  • 1 Introduction: The existence of cold diffusions without Gaussian noise or randomness during training or testing challenges current theory and opens possibilities for generative models with different properties.These models motivate reconsidering the limits of theoretical explanations centered on noise.

2 Background

The background places diffusion models within broader generative modeling and inverse-problem research, while motivating the paper’s central investigation of whether Gaussian noise is necessary. Prior interpretations rely on Gaussian-noise properties, whereas earlier deterministic experiments were limited in scope.

  • Generative-model background: Generative models span language and images and extend to image restoration, while diffusion models have recently become competitive with established GAN-based image synthesis.The cited literature includes natural-language and image generation, image restoration, GANs, and diffusion models.
  • Diffusion-model background: Diffusion-model interpretations based on Langevin dynamics, variational inference, and score matching rely on properties or roles attributed to Gaussian noise.Noise is thought to expand low-dimensional training distributions to full-measure ambient support and provide data augmentation.
  • Inverse problems: Diffusion models have been applied to deblurring, denoising, super-resolution, and compressive sensing, building on earlier iterative neural inverse-problem methods.Related work also studied learned noise schedules and links between Gaussian-noise removal and noisy-density gradients in deterministic linear inverse problems.
  • Contribution: The paper argues definitively that noise is not necessary in diffusion models and examines removing it across multiple inverse problems.Earlier work included deterministic image-generation experiments and selected inverse problems, but this paper systematically investigates the consequences of removing noise.

3 Generalized Diffusion

Generalized diffusion replaces Gaussian-noise degradation with arbitrary continuous transformations and learns a restoration operator to approximately invert them. An improved sampler is especially stable for smooth degradations, achieving exact reconstruction for a class of linear operators even with imperfect restoration.

  • Generalized degradation: Generalized diffusion uses degradations such as blurring, masking, and downsampling, with severity varying continuously and D(x0, 0) = x0.Standard diffusion is recovered when D adds Gaussian noise with variance proportional to t.
  • Restoration operator: A neural restoration operator Rθ is trained to approximately invert D by minimizing an ℓ1 reconstruction loss over random images.The restoration network is parameterized by θ, and the experiments use the ℓ1 norm.
  • Stability analysis: For linear degradations, Algorithm 2 produces exact iterates xs = D(x0, s) for any restoration operator, while Algorithm 1 accumulates errors when R is imperfect.This robustness follows from cancellation in the update and holds for the model D(x, s) ≈ x + s · e.
  • Empirical comparison: Algorithm 2 succeeds on smooth deblurring where Algorithm 1 produces compounding artifacts and fails to generate a new image.Figure 2 compares both samplers on CelebA and shows high-quality noise-free sampling with Algorithm 2.

4 Generalized Diffusions with Various Transformations

This section evaluates generalized diffusion with blur, masking, downsampling, and synthetic snow degradations across deblurring, inpainting, super-resolution, and desnowification. Sampling produces sharper or context-consistent reconstructions and succeeds even for exotic transforms lacking blur-like structure.

  • Experimental scope: The experiments apply improved Algorithm 2 to deblurring, inpainting, super-resolution, and synthetic snow removal using distinct deterministic degradations.The section focuses on conditional generation; unconditional generation is deferred to Section 5.
  • Deblurring: For blur, sampling sequentially adds removed frequencies because each update is a difference-of-Gaussians band-pass filter.The forward process progressively blurs images through convolution with Gaussian kernels, while the model predicts clean images for inversion.
  • Inpainting: Inpainting reconstructions recover features qualitatively consistent with the unmasked image context.Effectiveness is assessed with distributional similarity metrics before and after reconstruction in Table 2.
  • Super-resolution: Super-resolution reconstructions produce features qualitatively consistent with low-resolution inputs after repeated twofold downsampling.Table 3 compares distributional similarity metrics between degraded or reconstructed images and test samples.
  • Synthetic snow removal: Synthetic snow removal demonstrates generalized diffusion on an exotic transform lacking blur’s scale-space and compositional properties [Hendrycks and Dietterich, 2019].CIFAR-10 examples with lighter snow are reconstructed near-perfectly, while CelebA examples with heavy snow show visually distinctive restoration.

5 Cold Generation

Cold diffusion extends generation beyond Gaussian-noise degradation by using deterministic sampling, modeling simple terminal distributions for blur, and reversing other transformations. Blur-based generation produces high-fidelity but low-diversity images, while preliminary inpainting, super-resolution, and animorphosis experiments demonstrate broader applicability.

  • Deterministic generation using Gaussian noise: Noise-based generation applies Algorithm 2 with a fixed noise pattern or with noise vectors deterministically estimated at each reconstruction step.The fixed-noise and estimated-noise methods are evaluated quantitatively on CelebA and AFHQ in Table 5.
  • Unconditional generation using deblurring: For blur, fully degraded images collapse to channel-wise means represented by a 3-vector, which can be sampled from a one-component GMM before applying Algorithm 2.The generation pipeline progressively blurs images with a 27x27 Gaussian kernel over 300 steps, then expands the sampled 3D mean into a 128 × 128 image.
  • Unconditional generation using deblurring: Blur-based cold diffusion generates high-fidelity but low-diversity images because channel-wise mean sampling creates perfect pixel correlation, while symmetry breaking further improves FID scores.The blur schedule uses Gaussian kernels over 300 steps, and Table 5 compares hot diffusion with cold diffusion using blur transformation.
  • Other transformations: Cold generation extends to inpainting, super-resolution, and animorphosis by choosing degradation routines with predictable final distributions.The reported CelebA FID scores are 90.14 for inpainting, 92.91 for super-resolution, and 48.51 for animorphosis.
  • Other transformations: Animorphosis reverses an interpolation that progressively adds a random AFHQ animal image to a CelebA face, and in principle can connect any two initial data distributions.Generation samples a random animal image and uses Algorithm 2 to reverse the transformation.

6 Conclusion

The work removes Gaussian noise from diffusion models, replacing it with arbitrary transforms to restore images degraded by deterministic processes such as blur, inpainting, and downsampling.

  • 6 Conclusion: Diffusion models can remove Gaussian noise entirely and replace it with arbitrary transforms in both their framework and sampling procedures.This generalizes diffusion models beyond Gaussian-noise forward and reverse processes.
  • 6 Conclusion: The generalized framework restores images affected by deterministic degradations, including blur, inpainting, and downsampling.
  • 6 Conclusion: The approach points toward a more diverse landscape of diffusion models based on processes beyond random Gaussian noise.

A Appendix · A.1 Deblurring

The appendix describes the deblurring experiment setup and presents qualitative reconstructions and iterative sampling examples for MNIST, CIFAR-10, and CelebA. It also reports additional random-image samples to illustrate qualitative performance.

  • A Appendix: Deblurring models were trained for 700,000 gradient steps using Adam [Kingma and Ba, 2014] with a 2 × 10−5 learning rate.Training used batch size 32 with gradient accumulation every 2 steps.
  • A Appendix: The final deblurring model is an exponential moving average with decay 0.995, updated after every 10 gradient steps.
  • A.1 Deblurring: For MNIST, the deblurring process recursively applies blur 40 times using a discrete Gaussian kernel.
  • A.1 Deblurring: Figures 19 and 20 illustrate the iterative sampling process of deblurring models on ten example images from each dataset.
  • A.1 Deblurring: Figure 9 provides nine additional images for each of MNIST, CIFAR-10, and CelebA to show qualitative deblurring results.
  • A.1 Deblurring: Figure 21 presents 400 random images as further qualitative evidence for the deblurring models.
  • A.1 Deblurring: Figure 9 compares degraded inputs, direct reconstructions, Algorithm 2 sampled reconstructions, and original images across MNIST, CIFAR-10, and CelebA.

A.2 Inpainting

The inpainting models use a fixed training configuration with randomized Gaussian masks and discretized masked inputs to prevent floating-point information leakage. Examples across MNIST, CIFAR-10, and CelebA illustrate iterative sampling and additional outputs.

  • A.2 Inpainting: Inpainting models were trained for 60,000 gradient steps with Adam [Kingma and Ba, 2014], learning rate 2×10−5, batch size 64, two-step gradient accumulation, and EMA decay 0.995.The EMA model was updated every 10 gradient steps.
  • A.2 Inpainting: Masked images were rounded to eight significant digits before inference to prevent potential information leakage from floating-point Gaussian-mask computation.
  • A.2 Inpainting: Figures show iterative sampling for one image per dataset and nine additional inpainting examples on MNIST, CIFAR-10, and CelebA.

A.3 Super-Resolution

The super-resolution models are trained with a 700,000-iteration Adam setup and dataset-dependent resolution schedules. Sampling progressively increases resolution, with examples shown for MNIST, CIFAR-10, and CelebA.

  • A.3 Super-Resolution: The final model uses an exponential moving average with decay 0.995, updated every 10 gradient steps, after 700,000 Adam-training iterations.Training uses learning rate 2 × 10−5, batch size 32, and gradient accumulation every 2 steps.
  • A.3 Super-Resolution: The schedule uses 3 time-steps for MNIST and CIFAR10 and 6 for CelebA, reducing images to 4 × 4 or 2 × 2, respectively.CIFAR10 additionally uses random cropping and horizontal flipping for regularization.
  • A.3 Super-Resolution: Figures 12 and 13 show progressive resolution increases and nine additional super-resolution examples for each of MNIST, CIFAR-10, and CelebA.Figure 12 illustrates the sampling process, while Figure 13 provides additional dataset examples.

A.4 Colorization … A.7 Generation using blur transformation: Further Details

The supplementary sections extend cold diffusion to colorization and snowification, relate noise-based sampling to DDIM, and show that Algorithm 2 substantially improves blur-based unconditional generation. These experiments also demonstrate visually successful recolorization, snowification controls, and symmetry-preserving generation.

  • A.4 Colorization: Colorization iteratively desaturates RGB images with scheduled three-channel 1 × 1 filters until the final image is fully grayscale.The degradation is D(x, t) = z(αt) ∗x, with xT = z(1) ∗x0 producing a gray image; the schedule removes less color information early and more later.
  • A.4 Colorization: Recolorization produces semantically correct regional color separation and diverse object colors, with only minor visual differences from direct reconstruction.Figure 14 compares degraded grayscale inputs, direct reconstructions, sampled reconstructions, and original images across CIFAR-10 and CelebA.
  • A.5 Image Snow: Snowification constructs clipped snow patterns from Gaussian seed matrices, spline zooming, and randomly oriented motion blur, then adds them to the image.The process is abstracted as h(x0, SA, c0, c1), with c0 controlling snow severity and c1 controlling apparent windiness across T steps.
  • A.6 Generation using noise : Further Details: Algorithm 2’s sampling update is mathematically equivalent to the deterministic sampling method described in DDIM [Song et al., 2021a].The derivation estimates noise from xt and the restored clean image x̂0, then rewrites the degradation at consecutive steps.
  • A.7 Generation using blur transformation: Further Details: Figure 16 shows promising 128×128 CelebA and AFHQ samples generated with Method 2 while preserving perfect within-channel symmetry.The result concerns generation without breaking symmetry within each channel.
  • A.7 Generation using blur transformation: Further Details: FID is 49.45 for CelebA and 54.68 for AFHQ with Algorithm 2, versus 257.69 and 214.24 for direct reconstruction in broken-symmetry generation.The passage identifies Algorithm 2 as markedly superior, and Figure 17 provides a qualitative comparison of the two approaches.
  • A.7 Generation using blur transformation: Further Details: Algorithm 2 is necessary for high-quality unconditional generation because direct reconstruction performs substantially worse in the broken-symmetry case.Figure 18 further compares Algorithm 1 and Algorithm 2 on 128 × 128 CelebA, where Algorithm 1 fails completely to generate a new image.
Loading 2208.09392v1…