PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Diffusion: Score Matching for General Corruptions

Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, Peyman Milanfar

arXiv:2209.05442v2cs.CVcs.AIcs.LG

TL;DR

General corruption processes lack a principled design framework, and prior alternatives to additive noise usually underperformed. The paper proposes Soft Score Matching and associated corruption-selection and sampling methods; it reports state-of-the-art CelebA-64 FID 1.85 and faster models than vanilla Gaussian denoising diffusion.

  • Problem

    A common framework for designing general corruption processes is missing, while results using alternatives to additive noise are usually inferior to additive-noise diffusion.

    Source: §1 · p. 1

  • Method

    Soft Score Matching predicts a clean image whose transformed version matches the diffused observation, alongside principled intermediate-distribution selection and Momentum Sampler.

    Source: §1 · p. 2

  • Results

    FID 1.85 on CelebA is state of the art for linear diffusion models, which are significantly faster than models trained with vanilla Gaussian denoising diffusion.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Changing the corruption process can improve sample quality and provide computational benefits within the demonstrated framework.

    Source: §6 · p. 10

    With blur paired with small noise, the objective may learn only locally around blurry images and may not reduce blurriness.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

We define a broader family of corruption processes that generalizes previously known diffusion models. To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft Score Matching incorporates the degradation process in the network. Our new loss trains the model to predict a clean image, \textit{that after corruption}, matches the diffused observation. We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes. We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler. We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking. We achieve state-of-the-art FID score $1.85$ on CelebA-64, outperforming all previous linear diffusion models. We also show significant computational benefits compared to vanilla denoising diffusion.

1 INTRODUCTION

Diffusion models generate samples by reversing corruption, but general corruption processes lack a principled design framework and have often underperformed additive-noise diffusion. The paper introduces Soft Score Matching and related design and sampling methods, reporting a CelebA-64 FID of 1.85 and faster generation than vanilla Gaussian denoising diffusion.

1 INTRODUCTION

  • Diffusion models invert corruption processes, and score-based models and DDPMs are unified under a shared framework (Song et al., 2021b).Diffusion applications include image, audio, and video generation and inverse problems.Source: §1 · p. 1
  • Prior work identified noise scheduling, network parameterization, and sampling as design stages; this paper adds corruption choice, for which a common design framework was missing.Recent alternative-corruption results were usually inferior to additive-noise diffusion.Source: §1 · p. 1
  • FID 1.85 on CelebA-64 is state of the art for linear diffusion models, and the models are significantly faster than vanilla Gaussian denoising diffusion.The reported blur-and-noise model achieves the FID result; the figure also illustrates blur/noise and masking/noise reversals.Source: Figure 1 · p. 2, §1 · p. 2
  • The objective incorporates the filtering process and predicts a clean image whose corruption matches the diffused observation.This formulation is described for linear corruption processes.Source: §1 · p. 2
  • Soft Score Matching provably learns scores for a wide family of regular corruption processes and supports learning with limited randomness.Its guarantee requires a regularity condition: any image must be transformable into any other image with nonzero likelihood.Source: §1 · pp. 1–2
  • The framework selects intermediate distributions by minimizing Wasserstein distance along the path and introduces Momentum Sampler, which combines corruptions from different diffusion levels.The paper reports training models on CelebA and CIFAR-10.Source: §1 · p. 2

2 BACKGROUND

Diffusion models generate samples by reversing a corruption process indexed by time, from clean images to fully corrupted ones. This section describes score-based and DDPM formulations, their SDE connection, and reverse-time sampling.

2 BACKGROUND

  • Diffusion models invert a corruption process indexed by time, with t = 0 denoting clean images and t = 1 fully corrupted images; the two general classes are score-based models and DDPMs.The diffusion process may be discrete or continuous.Source: §2 · p. 2
  • Score-based models typically add Gaussian noise of increasing magnitude, and the denoising score-matching objective trains time-conditioned scores used to denoise samples from the final distribution.The score approximates the gradient of the log likelihood; the final variance is chosen large so the final distribution is approximately Gaussian.Source: §2 · pp. 2–3
  • DDPMs rescale images and add noise through a Markov chain, typically ending at a unit Gaussian, and use a denoising score-matching objective derived from an ELBO.The passage specifies β1 = 1 for the final distribution.Source: §2 · p. 3
  • Score-based models and DDPMs can be expressed as SDE solutions (Song et al., 2021b), and their forward SDEs are reversible to a reverse SDE.In practice, the reverse process approximates the score with a neural network and generates samples by solving the reverse SDE.Source: §2 · p. 3
  • The SDE variants differ in their terminal variance: VE uses large noise, VP ends at unit variance, and subVP has bounded variance and experimentally yields better likelihoods.These variants are attributed to Song et al. (2021b).Source: §2 · p. 3

3 METHOD

The method extends diffusion to linear corruptions with a filtered-space training objective, a sampler that combines deblurring and denoising, and a principled way to choose corruption levels. The objective learns marginal scores under stated regularity conditions.

3.1 TRAINING OBJECTIVE

  • The corruption family combines a deterministic linear transform of the clean image with noise of nondecreasing variance, including VE, VP, and subVP SDEs as special cases.Time runs from natural images at t = 0 toward progressively harder recovery at larger t.Source: §3.1 · p. 3
  • Under fully supported, differentiable conditional distributions, minimizing the conditional-score DSM objective is equivalent to minimizing the marginal score-matching objective.The paper notes that additive noise is not the only way to ensure every corrupted observation has nonzero probability.Source: §3.1 · p. 4
  • Soft Score Matching uses the known corruption in its parameterization and compares the predicted residual in filtered space, addressing local learning around blurry images under small noise.With no filtering matrix, the objective recovers DSM; residual prediction also gives a smoother optimization landscape than direct clean-image prediction.Source: §3.1 · p. 4
  • FID 1.85 for Soft Score Matching versus 5.91 for the clean-image prediction baseline under the same setting.The comparison is reported for CelebA.Source: §3.1 · p. 5

3.2 SAMPLING

  • Momentum Sampler predicts a clean image, re-corrupts it at t and t − Δt, and uses the difference between those filtered predictions for deblurring alongside denoising.The authors report that the naive sampler produces images lacking diversity; the Momentum Sampler update generalizes its update with a momentum term.Source: §3.2 · pp. 5–6

3.3 SCHEDULING

  • The scheduling method selects intermediate corruption distributions by minimizing path cost for a fixed network capacity, then interpolates the chosen path for continuous time.Estimating all pairwise distances becomes more costly as the candidate set grows; the paper gives T = 256 as a practical example.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate Soft Diffusion on CelebA-64 and CIFAR-10, reporting its strongest benchmark results on CelebA, lower sampling cost, and ablations of the objective, schedule, and sampler. They also test masking, which demonstrates broader applicability but samples slowly.

4 EXPERIMENTS

  • Soft Diffusion requires fewer NFEs than NCSN++ (VE SDE) to achieve the same or better quality on CelebA and CIFAR-10.Figure 3 plots FID against NFEs for CelebA-64.Source: §4 · p. 8, Figure 3 · p. 8
  • FID 1.85 on CelebA-64 is state of the art; CIFAR-10 reaches FID 3.86 with Probability Flow Momentum Sampler and 3.91 with Momentum Sampler.Most experiments use Gaussian blur paired with low-magnitude noise, with dataset-specific schedules.Source: §4 · pp. 7–8, Table 1 · p. 8
  • The Momentum Sampler improves sample variety and detail, lowering FID from 27.82 to 1.85 compared with the repetitive, less detailed Naive Sampler.Source: §4 · p. 8, Figure 4 · p. 8
  • FID rises from 1.85 to 5.91 when replacing Soft Score Matching with Bansal et al.'s loss under the same pipeline.Source: §4 · p. 8
  • FID is 8.35 for the MSE-matched scheduling baseline versus 1.85 for the found schedule; lowering maximum noise degrades results, and very small noise fails.The Momentum Sampler also works when noise and blur change simultaneously.Source: §4 · p. 9
  • Masking predictions focus on observed noisy regions and grow as images become cleaner, but FID is 30.92 at 1024 steps and 12.37 at 4096.The conditional-mean predictions for blur and masking are shown at different diffusion times.Source: §4 · p. 9, Figure 5 · p. 9

5 RELATED WORK

The related-work discussion positions general corruption as a distinct design choice and contrasts the paper's scheduling and sampling approaches with prior methods. It also reports comparisons with concurrent degradation-based diffusion work.

5 RELATED WORK

  • The paper treats corruption choice as an additional diffusion design step beyond noise scheduling, network parametrization, and sampling.Source: §5 · p. 9
  • Unlike Bao et al.'s Gaussian-diffusion schedule minimizing KL, this work estimates Wasserstein distances from data to schedule general corruption processes.Source: §5 · p. 9
  • Under the same setting, Soft Score Matching reports FID 1.85 versus 5.91 for concurrent losses; the paper also reports CelebA FID 1.85 and CIFAR-10 FID 3.86.The discussion notes Hoogeboom & Salimans (2022) report CIFAR-10 FID 3.17 using blur with a VP SDE.Source: §5 · p. 9

6 CONCLUSIONS AND FUTURE WORK

The conclusion presents a framework for training and sampling diffusion models that reverse general corruption processes, reporting sample-quality and computational benefits from changing the corruption process. It identifies corruption optimization, mixed corruptions, and the role of noise as future research directions.

6 CONCLUSIONS AND FUTURE WORK

  • The framework trains and samples diffusion models that reverse general corruption processes.Source: §6 · p. 10
  • Changing the corruption process is reported to yield sample-quality improvements and computational benefits.Source: §6 · p. 10
  • Future directions include optimizing corruption for inverse problems, mixing corruptions, and clarifying noise's theoretical and practical roles.Source: §6 · p. 10

A.1 PROOFS

The appendix states a relationship between two objectives under regularity assumptions, then describes a probability-flow ODE sampler and its discretization. The sampler uses neural estimates, while the text notes that more sophisticated ODE solvers are possible.

A.1 PROOFS

  • Under supported, differentiable conditional distributions, Theorem 3.1 states that J1(θ) equals J2(θ) up to a θ-independent constant.The proof follows calculations of Vincent (2011).Source: §3.1 · p. 4, §A.1 · p. 13

B PROBABILITY FLOW ODE

  • The probability-flow ODE has the same marginal distributions as the backward SDE, and its sampler estimates Ctα and the score with a neural network.The resulting neural ODE update is identified as the Probability Flow ODE Momentum Sampler.Source: B PROBABILITY FLOW ODE · pp. 14–15
  • The sampler uses a first-order ODE discretization, though the paper notes that more sophisticated solvers such as Dormand–Prince are possible.Source: B PROBABILITY FLOW ODE · pp. 14–15

C SCHEDULINGS

The blur schedule is selected without supervision by minimizing Wasserstein distances between consecutive distributions, and the resulting schedule differs by dataset.

C SCHEDULINGS

  • CelebA-64 and CIFAR-10 receive slightly different blur schedules, reflecting that the diffusion depends on the data being modeled.The Gaussian blur kernel support is 161 × 161 for CelebA-64 and 65 × 65 for CIFAR-10.Source: C SCHEDULINGS · p. 15, Figure 6 · p. 15
  • The blur parameters are selected without supervision by minimizing the sum of Wasserstein distances between consecutive distributions.The optimization starts from 256 blur levels, tunes ϵ for a shortest path of 32 distributions, then uses linear interpolation for the continuous case.Source: Figure 6 · p. 15, C SCHEDULINGS · p. 15

D TRAINING DETAILS

Training uses the Song et al. architecture with Adam and a loss weighted by the noise level; blur settings and evaluation are dataset-specific.

D TRAINING DETAILS

  • Training uses Adam at learning rate 2e−4, batch size 128, gradient clipping above norm 1, and 5000 warmup steps.The models train for 1–2M iterations, based on observed FID performance.Source: D TRAINING DETAILS · p. 15
  • Gaussian blur uses a fixed kernel size with varying variance, zero-padding, and a final level chosen to yield noise over almost a single color.Kernel half sizes are 80 for CelebA-64 and 32 for CIFAR-10; standard deviations range from 0.01 to 23 and 0.01 to 18, respectively.Source: D TRAINING DETAILS · p. 15
  • The architecture follows Song et al. (2021b) unchanged, and the loss is scaled by w(t) = 1/σ²_t.The weighting follows Song & Ermon (2019; 2020) and Song et al. (2021b).Source: D TRAINING DETAILS · p. 16
  • Blur models run at 6 iterations per second on CelebA and 11 on CIFAR-10, with no overhead beyond efficient projection to measurement space.FID evaluation uses 50,000 samples, retaining the best checkpoint among those saved every 10,000 steps.Source: D TRAINING DETAILS · p. 16

E LIMITATIONS AND THINGS THAT DID NOT WORK

The method has operator, theoretical, and scope constraints, while some masking settings sample slowly and experiments with less stochasticity failed.

E LIMITATIONS AND THINGS THAT DID NOT WORK

  • The method requires a known diffusion operator, offers no guidance on which operators are useful, and currently concerns linear diffusions.The authors describe extension to nonlinear diffusions as relatively straightforward, and say it remains open whether another corruption will outperform blur.Source: E LIMITATIONS AND THINGS THAT DID NOT… · p. 16
  • The theoretical account only intuitively explains the reparametrization, and the method requires conditional log-likelihood to be defined everywhere.Some distributions do not satisfy this condition but may still be learned to some extent with heuristic methods such as Bansal et al. (2022).Source: E LIMITATIONS AND THINGS THAT DID NOT… · p. 16
  • Soft Score Matching can sample more slowly than clean-image prediction; masking needs many steps to transition smoothly between levels.With masking, the model is penalized only in the observed region and has no incentive to expand that region.Source: E LIMITATIONS AND THINGS THAT DID NOT… · p. 16
  • Experiments with less stochasticity did not work, and the appropriate amount of randomness at each diffusion step remains unresolved.The authors leave the cause of failure and the required per-step randomness for future investigation.Source: E LIMITATIONS AND THINGS THAT DID NOT… · p. 16

F.1.1 ABLATION STUDIES FOR NOISE

Noise is added initially and then held fixed in the main setup; lower fixed noise degraded samples, while changing noise alongside blur yielded a competitive FID.

F.1.1 ABLATION STUDIES FOR NOISE

  • The main experiments add noise initially, then hold it fixed while deterministic corruption such as blur or masking proceeds.The ablation assesses the magnitude of this fixed noise on CelebA-64, with maximum values σmax = 0.1, 0.05, and 0.025.Source: §F.1.1 · p. 16
  • Figure 7 shows that both lower-noise ablation models failed to produce realistic samples, with quality worsening as noise decreased.The samples were significantly worse than those from the state-of-the-art model; the authors caution that this study is not conclusive.Source: Figure 7 · p. 17, §F.1.1 · p. 17
  • The authors note that the shared weighting w(t) = 1/σ²_t might cause instability at low noise, so different weights could change the outcome.The instability possibility is cited to Nichol & Dhariwal (2021).Source: §F.1.1 · p. 17
  • 4.31 FID was achieved when noise and blur changed simultaneously, using geometric noise scheduling from 0.1 to 0.01.The blur parameters matched those of the state-of-the-art model; jointly selecting blur and noise schedules was outside this ablation's scope.Source: §F.1.1 · p. 17

F.1.2 ABLATION STUDY FOR BLUR

The blur-scheduling ablation compares a VE-guided baseline with the paper’s schedule, then presents nearest-neighbor comparisons and additional uncurated samples.

F.1.2 ABLATION STUDY FOR BLUR

  • The VE-guided baseline selects blur parameters so clean-to-corrupted-image MSE decays at the same rate as in Gaussian denoising diffusion.The passage introduces the comparison as a natural baseline for selecting diffusion levels.Source: §F.1.2 · p. 17
  • FID rose from 1.85 to 8.35 with the VE-guided schedule, and its best score took 2000 steps versus 280 with the paper’s schedule.The architecture and hyperparameters were held constant; only blur scheduling was ablated.Source: §F.1.2 · p. 17

F.4 UNCURATED SAMPLES

  • The blur model offers significant performance benefits at low NFEs on CIFAR-10.Additional uncurated samples are shown for blur models trained on CelebA and CIFAR-10, and a masking model trained on CelebA.Source: Figure 8 · p. 18, §F.4 · p. 18, Figure 10 · p. 19

F.3 NEAREST SAMPLES IN TRAINING DATA

  • Generated images and their nearest training-set neighbors are presented as evidence that the model produces new samples rather than simply memorizing the dataset.The nearest neighbor is selected by L2 pixel distance.Source: §F.3 · p. 17, Figure 9 · p. 18
Loading 2209.05442v2…