PaperlayerView exact revision on arXiv

Source-linked AI summary

Elucidating the Design Space of Diffusion-Based Generative Models

Tero Karras, Miika Aittala, Timo Aila, Samuli Laine

arXiv:2206.00364v2cs.CVcs.AIcs.LGcs.NEstat.ML

TL;DR

Diffusion-model formulations can make sampling and training choices appear tightly coupled. This paper separates their design choices and improves sampling, network preconditioning, and training. The changes improve sampling efficiency and achieve state-of-the-art image quality on CIFAR-10 and ImageNet-64, including ImageNet-64 FID 1.36.

  • Problem

    Diffusion-model formulations can appear tightly coupled, leaving unclear which sampling and training components can be modified independently.

    Source: §1 · p. 1

  • Method

    A common framework separates diffusion-model design choices, guiding improvements to sampling, network preconditioning, and training.

    Source: §1 · pp. 1–2, §6 · p. 10

  • Results

    Across CIFAR-10 and ImageNet-64, the improvements yield state-of-the-art image quality and more efficient sampling, including ImageNet-64 FID 1.36.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The improved sampler can serve as a drop-in replacement for several widely used diffusion models [37].

    Source: §1 · p. 1

    Best stochastic-sampling results require model-specific heuristic choices, which the authors tuned case by case using grid search.

    Source: §4 · p. 8

Abstract

from arXiv · show

We argue that the theory and practice of diffusion-based generative models are currently unnecessarily convoluted and seek to remedy the situation by presenting a design space that clearly separates the concrete design choices. This lets us identify several changes to both the sampling and training processes, as well as preconditioning of the score networks. Together, our improvements yield new state-of-the-art FID of 1.79 for CIFAR-10 in a class-conditional setting and 1.97 in an unconditional setting, with much faster sampling (35 network evaluations per image) than prior designs. To further demonstrate their modular nature, we show that our design changes dramatically improve both the efficiency and quality obtainable with pre-trained score networks from previous work, including improving the FID of a previously trained ImageNet-64 model from 2.07 to near-SOTA 1.55, and after re-training with our proposed improvements to a new SOTA of 1.36.

1 Introduction

The paper makes diffusion models’ design choices explicit through a practical analysis of training and sampling components. Its sampling and training improvements together achieve record FIDs on CIFAR-10 and 64×64 ImageNet.

1 Introduction

  • A practical design-space analysis clarifies relationships and available choices among training and sampling components that theory-heavy formulations can obscure.The scope is neural modeling of scores of Gaussian-corrupted data distributions across noise levels, within denoising score matching. Explicitly tabulating key ingredients is intended to support targeted exploration of individual components.Source: §1 · pp. 1–2
  • Improved time discretization and higher-order Runge–Kutta sampling reduce synthesis steps, providing a drop-in replacement sampler for widely used diffusion models [37].The work identifies the best-performing time discretization, evaluates sampler schedules, and analyzes the usefulness of sampling stochasticity.Source: §1 · p. 1
  • The first principled analysis of network input, output, and loss preconditioning derives improved training practices while retaining DDPM [16] and NCSN architectures.The authors also propose an improved training noise-level distribution and find non-leaking augmentation [25], typically used with GANs, beneficial for diffusion models.Source: §1 · p. 2
  • 1.79 FID on CIFAR-10 and 1.36 FID on 64×64 ImageNet are reported as record results from the combined improvements.The implementation and pre-trained models are publicly available.Source: §1 · p. 2

2 Expressing diffusion models in a common framework

The framework expresses diffusion sampling through a probability flow ODE and separates denoising, noise scheduling, signal scaling, and numerical integration into independently selectable components.

2 Expressing diffusion models in a common framework

  • Diffusion sampling starts from approximately Gaussian noise and progressively denoises through decreasing noise levels until the endpoint follows the data distribution.The starting noise standard deviation must greatly exceed the data standard deviation, and intermediate samples should follow the corresponding Gaussian-noised data distributions.Source: §2 · p. 2

ODE formulation

  • The probability flow ODE makes sampling deterministic after initialization, with a noise schedule controlling evolution along a score field.Backward steps move samples toward the data distribution. The noise schedule has practical implications and should not be chosen solely for theoretical convenience.Source: §2 · p. 2

Denoising score matching

  • An expected-L2-optimal denoiser supplies the score, motivating a noise-conditioned neural network trained for denoising.The score does not depend on the density’s generally intractable normalization constant. The neural denoiser may include pre- and post-processing, such as input scaling.Source: §2 · pp. 2–3

Time-dependent signal scaling

  • Time-dependent signal scaling changes sampling trajectories, while undoing scaling before score evaluation keeps the underlying noisy-data density independent of the scale schedule.The additional schedule s(t) defines the scaled variable x = s(t) x̂ and generalizes the ODE.Source: §2 · p. 3

Solution by discretization

  • Numerical integration requires choosing both a solver and discrete sampling times, with a second-order solver offering a better computational tradeoff than Euler.Many prior works rely on Euler’s method; the second-order comparison is developed in Section 3.Source: §2 · p. 3

Putting it together

  • Three earlier methods are reproduced as deterministic variants within a framework that exposes independently selectable design components.Within reasonable choices, changing one component need not require changes elsewhere to preserve convergence to the data distribution, although practical performance depends on the combination.Source: §2 · pp. 3–4, Table 1 · p. 3

3 Improvements to deterministic sampling

Combining Heun’s method, noise-dependent step sizes, and a simplified noise schedule substantially reduces deterministic sampling cost across three pretrained model families. These consistent improvements support the hypothesis that sampling choices can be separated from training choices.

3 Improvements to deterministic sampling

  • Sampling is treated as independent of network architecture and training, tested by replacing samplers for three pretrained VP, VE, and iDDPM model families.Tests use Song et al.’s unconditional CIFAR-10 models and Dhariwal and Nichol’s [9] class-conditional ImageNet-64 model. The denoiser is treated as a black box.Source: §3 · p. 4
  • 50,000 generated images underpin FID [15] evaluation, with unified-framework reimplementations achieving similar but consistently better results than original samplers.FID compares generated images against all available real images; NFE counts denoiser evaluations per image and directly reflects sampling speed. Improvements reflect corrected implementation oversights and more careful handling of DDIM’s discrete noise levels.Source: §3 · p. 4

Discretization and higher-order integrators

  • O(h^3) local error makes Heun’s second-order method more accurate than Euler’s O(h^2), costing one additional denoiser evaluation per step.Its correction accounts for changing derivatives, but the sampler reverts to Euler when stepping to zero noise to avoid division by zero. Heun achieves the same FID as Euler with considerably lower NFE across all evaluated cases.Source: §3 · pp. 4–5
  • ρ = 7 defines the adopted noise-level schedule, shortening steps at lower noise levels without requiring per-sample adaptation.Although ρ = 3 nearly equalizes truncation error across steps, values from 5 to 10 produce much better images. This suggests that errors near the minimum noise level matter disproportionately.Source: §3 · p. 5

Trajectory curvature and noise schedule

  • Choosing σ(t) = t and s(t) = 1, as in DDIM, considerably improves VP and VE while leaving ImageNet-64 results unchanged.The trajectory tangent always points toward the denoiser output, motivating largely linear trajectories with substantial curvature confined to a narrow noise range. DDIM already uses these choices.Source: §3 · p. 5

Discussion

  • 7.3×, 300×, and 3.2× reductions in NFE needed for high-quality results are achieved for VP, VE, and DDIM, respectively.The combined improvements enable 26.3 high-quality CIFAR-10 images per second on one NVIDIA V100. Adaptive RK45 with the proposed schedule costs more than its benefits justify.Source: §3 · p. 6

4 Stochastic sampling

A tailored stochastic sampler combines second-order ODE integration with noise injection, improving an ImageNet-64 model [9] from FID 2.07 to 1.55 through sampler changes alone. Its practical effectiveness depends on model-specific heuristic tuning to limit detail loss and color oversaturation.

Background

  • Stochasticity corrects earlier sampling errors through Langevin diffusion, but discretization introduces additional errors, making the optimal amount an empirical question.β(t) controls how quickly existing noise is replaced with fresh noise. Deterministic ODE sampling permits image-to-latent inversion but tends to yield worse output quality.Source: §4 · p. 6

Our stochastic sampler

  • The sampler alternates explicit noise injection with a second-order backward ODE step evaluated from the post-injection state.Unlike Euler–Maruyama, it evaluates the denoiser after noise injection, a distinction that appears important for low NFE with large steps. Its sub-steps preserve the correct distribution up to ODE truncation error; it is not a general-purpose SDE solver.Source: §4 · pp. 6–7

Practical considerations

  • Excessive noise addition and removal causes gradual detail loss, while very low and high noise levels exhibit drift toward oversaturated colors.The authors suspect practical denoisers produce slightly nonconservative vector fields that violate Langevin assumptions. Analytical denoisers did not show this degradation in their experiments.Source: §4 · p. 7
  • The sampler restricts stochasticity to a noise-level interval, caps injected noise, and inflates its standard deviation slightly to mitigate degradation.Within [Stmin, Stmax], γ_i = Schurn/N, capped so new noise never exceeds existing noise. Setting Snoise slightly above 1 partially counteracts detail loss, suggesting the denoiser removes too much noise.Source: §4 · p. 7

Evaluation

  • 1.55 FID on ImageNet-64, down from 2.07 for the original model [9], is achieved through sampler improvements alone.The stochastic sampler outperforms previous samplers [24] [37], especially at low step counts.Source: §4 · p. 7, Figure 4 · p. 8
  • Best results require model-specific grid searches over Schurn, Stmin, Stmax, and Snoise, limiting stochastic sampling as a clean measure of model improvements.The authors warn that evaluating improvements primarily through stochastic sampling may inadvertently influence architecture and training choices.Source: §4 · p. 8

5 Preconditioning and training

Noise-dependent preconditioning, revised loss weighting and noise sampling, and augmentation yield record CIFAR-10 FIDs; training improvements without augmentation also set an ImageNet-64 record. Improved training favors deterministic sampling on CIFAR-10, while ImageNet-64 still benefits from stochastic sampling.

Preconditioning

  • Noise-dependent preconditioning controls signal magnitudes and lets the network estimate clean signal, noise, or a mixture through an adaptive skip connection.Previous noise-prediction methods [37] amplify network errors by σ, motivating an alternative to direct denoiser training or fixed noise prediction. Input and target scaling enforce unit variance, the skip coefficient minimizes error amplification, and noise-conditioning scaling is chosen empirically.Source: §5 · pp. 8–9

Augmentation regularization

  • 1.79 conditional and 1.97 unconditional CIFAR-10 FIDs set new records, beating 1.85 [45] and 2.10, respectively.Augmentation regularization consistently improves results using geometric transformations borrowed from GAN training [25], applied before adding noise. Conditioning on augmentation parameters, then setting them to zero at inference, prevents augmentation leakage.Source: Table 2 · p. 9, §5 · p. 9

Loss weighting and sampling

  • Revised loss weighting and log-normal noise sampling dramatically improve FID across all tested cases when combined with the proposed preconditioning.Setting λ(σ) = 1/cout(σ)^2 balances effective loss weights and equalizes initial losses; log-normal sampling focuses training on intermediate noise levels where loss can meaningfully decrease. Preconditioning alone largely preserves FID, except for substantial VE improvement at 64×64, but makes training more robust.Source: §5 · p. 9

Stochastic sampling revisited

  • Deterministic sampling performs best on CIFAR-10 after improved training, whereas ImageNet-64 still benefits from stochastic sampling at a reduced strength.The Figure 5 comparisons use 256 steps and 511 network evaluations. On CIFAR-10, stochasticity changes from highly beneficial under the original training setup of Song et al. [49] to harmful under the proposed setup.Source: §5 · pp. 9–10, Figure 5 · p. 10

ImageNet-64

  • 1.36 FID on class-conditional ImageNet-64 improves the previous record of 1.48 [17].The model uses the unchanged ADM architecture [9], trained from scratch with config E and minimal tuning. Augmentation was omitted because overfitting was not a concern.Source: §5 · pp. 9–10

6 Conclusions

A common framework exposes diffusion models’ modular design, enabling sampler replacements and training improvements that improve benchmark results. Higher-resolution parameter tuning, stochastic sampling’s interaction with training, and societal and energy costs remain concerns.

6 Conclusions

  • 1.36 FID on ImageNet-64 established a new state of the art after training improvements, surpassing the previous 1.48 FID benchmark [17].Replacing the sampler alone improved an earlier model from FID 2.07 to 1.55.Source: §6 · p. 10
  • 35 model evaluations sufficed for new state-of-the-art CIFAR-10 results using deterministic sampling and a small network.Source: §6 · p. 10
  • The common framework exposes a modular design, enabling targeted investigation of individual components and straightforward sampler replacement in earlier models.Source: §6 · p. 10
  • Higher-resolution datasets may require parameter retuning, while the precise interaction between stochastic sampling and the training objective remains an open question.The authors believe their contributions are orthogonal to super-resolution [17] [36] [40], subspace projection [23], large-network [9], and hybrid [39] [42] extensions.Source: §6 · p. 10

Societal impact

  • Improved sample quality could amplify disinformation, stereotypes, and harmful biases when deployed in large-scale systems such as DALL·E 2.Source: §6 · p. 10
  • ∼250 MWh of electricity was consumed by the project on an in-house NVIDIA V100 cluster, illustrating diffusion training and sampling’s energy demands.Source: §6 · p. 10

Appendices · A Additional results

Additional results compare the proposed sampling and training changes with prior designs across several datasets, using generated-image comparisons and numerical summaries.

A Additional results

  • Figure 7 presents class-conditional ImageNet-64 images from a model trained from scratch using the improved training configuration.These complement the pretrained-model comparisons in Figure 6.Source: A Additional results · p. 13
  • Figure 6 compares proposed deterministic and stochastic samplers against DDIM and iDDPM [37] using the pretrained ImageNet-64 ADM model by Dhariwal and Nichol [9].Tables 3 and 4 summarize numerical deterministic and stochastic sampling results across datasets, previously plotted against NFE in Figures 2 and 4.Source: A Additional results · p. 13
  • Figures 8–10 compare the proposed samplers and training configurations with those of Song et al. on unconditional and class-conditional CIFAR-10.Figures 8 and 9 cover unconditional generation; Figure 10 covers class-conditional generation.Source: A Additional results · p. 13
  • Figure 11 extends comparisons with the samplers and training configurations of Song et al. to FFHQ [27] and AFHQv2 [7].Source: A Additional results · p. 13
  • The comparisons reuse the same latent codes x0 within each dataset and scenario across training configurations and ODE choices.This matching is used to ease visual comparison.Source: A Additional results · p. 13
  • Figure 12 shows generated-image quality at different NFE settings under deterministic sampling.Source: A Additional results · p. 13

B Derivation of formulas … C.3.2 iDDPM time step discretization

This section develops a unified framework for diffusion dynamics, connecting score matching to denoising and deriving sampling, preconditioning, and training correspondences for previous methods. It also justifies practical ODE evaluation, derives SDEs with prescribed marginals, and explains stepping errors, precision limits, and noise-range constraints when reusing pretrained models.

B.1 Original ODE / SDE formulation from previous work

  • Song et al. use linear-drift SDEs with VP- or VE-specific coefficients and a probability flow ODE sharing the same marginal distributions.The marginals are obtained by integrating the perturbation kernels over the initial data distribution.Source: §B.1 · p. 13

B.2 Our ODE formulation (Eq. 1 and Eq. 4)

  • The probability flow ODE is expressed directly through σ(t) and s(t), with every realization a reparameterization of one canonical ODE.Changing σ(t) reparameterizes time, while changing s(t) reparameterizes the state; setting s(t) = 1 recovers Eq. 1. The underlying p(x; σ) is the data distribution smoothed with i.i.d. Gaussian noise.Source: §B.2 · pp. 21–22

B.3 Denoising score matching (Eq. 2 and Eq. 3)

  • Denoising score matching yields a unique, closed-form ideal denoiser for finite datasets, with the derivation recovering the connection between scores and denoising.The loss can be minimized independently at each noisy x, and the closed-form denoiser is practical to compute for small datasets.Source: §B.3 · pp. 22–24

B.4 Evaluating our ODE in practice (Algorithm 1)

  • Algorithm 1 evaluates the scaled ODE by rewriting its score through denoising and replacing the ideal denoiser with the trained model.Backsubstituting the unscaled variable as x/s(t) yields the expressions matching lines 4 and 7 of Algorithm 1.Source: §B.4 · pp. 24–25

B.5 Our SDE formulation (Eq. 6)

  • The SDE derivation first identifies the PDE governing Gaussian-smoothed data densities, then uses Fokker–Planck to recover compatible SDEs and parameterize Eq. 6.This construction begins with the desired marginal evolution rather than a particular drift and diffusion.Source: §B.5 · p. 25

B.5.1 Generating the marginals by heat diffusion

  • κ(t) = σ(t) dσ(t)/dt is the heat-equation diffusivity that generates the desired Gaussian-smoothed marginals.The derivation works in the Fourier domain, starting from q(x, 0) = pdata(x).Source: §B.5.1 · pp. 25–26

B.5.2 Derivation of our SDE

  • The SDE family preserves the prescribed marginals while controlling noise replacement; zero diffusion recovers the probability flow ODE.β(t) scales noise replacement with σ(t), removing and injecting noise at the same rate. Anderson’s time reversal, as stated by Song et al., only changes the middle term’s sign.Source: §B.5.2 · p. 27

B.6 Our preconditioning and training (Eq. 8)

  • Preconditioning gives unit-variance inputs and targets, minimizes network-error amplification, and makes effective L2 training weights uniform across noise levels.Zero-initialized output weights make Fθ = 0 and the expected loss 1 at every noise level.Source: §B.6 · pp. 27–29

C.1.2 VP preconditioning; C.1.3 VP training

  • VP scores and losses from Song et al. are rewritten as denoiser preconditioning and a noise-weighted training objective.The resulting formulation identifies c_skip, c_out, c_in, and c_noise, together with the training noise distribution p_train and loss weighting λ.Source: §C.1.2 · p. 31, §C.1.3 · pp. 31–32

C.1.1 VP sampling; C.1.4 VP practical considerations

  • The VP reimplementation fixes incorrect Euler multipliers and final-step overshoot or undershoot in Song et al., avoiding their associated residual-noise artifacts.The original sampler uses −1/N instead of (ϵ_s − 1)/(N − 1), producing severe noise at N = 128; uniform sampling times use β_min = 0.1, β_max = 20, and ϵ_s = 10^-3. The 62-million-parameter DDPM++ cont. (VP) checkpoint supports noise levels σ ≈ [0.001, 152], encompassing the preferred sampling range [0.002, 80].Source: §C.1.1 · p. 30, §C.1.4 · p. 32

C.2.1 VE sampling in theory

  • VE theory uses no signal scaling, but its derived noise schedule does not fulfill the schedule intended by Song et al..The derived schedule is nevertheless consistent with their reported perturbation kernel.Source: §C.2.1 · pp. 32–33

C.2.2 VE sampling in practice

  • The implemented VE sampler of Song et al. is exactly an Euler iteration of the unified ODE under appropriate sampling choices.Their CIFAR-10 parameters σ_min = 0.01 and σ_max = 50 must both be multiplied by 2× when converting images from [0, 1] to [−1, 1].Source: §C.2.2 · pp. 33–34

C.2.3 VE preconditioning; C.2.4 VE training

  • VE preprocessing is moved outside the network, adjusted to [−1, 1], and paired with a training loss expressed in the unified denoiser form.Explicit c_skip, c_out, c_in, and c_noise coefficients replace embedded preprocessing and postprocessing; the rewritten loss identifies p_train and λ.Source: §C.2.3 · p. 34, §C.2.4 · p. 34

C Reframing previous methods in our framework; C.2.5 VE practical considerations

  • Double-precision sampling improves VE accuracy at high step counts by reducing round-off errors, while network evaluation remains single precision.The unified setup uses Fθ, images in [−1, 1], and consistent x and σ definitions. The 63-million-parameter NCSN++ cont. (VE) checkpoint supports σ ≈ [0.02, 100], requiring σ_min = 0.02 until training improvements permit σ_min = 0.002 in configurations E and F.Source: C Reframing previous methods in our… · p. 29, §C.2.5 · p. 35

C.3.1 DDIM ODE formulation

  • The deterministic DDIM sampler of Song et al. matches Euler integration of the unified ODE with s(t) = 1 and σ(t) = t.The derivation relates DDIM’s normalized-noise predictor to the clean-signal denoiser, assuming both are ideal in the L2 sense.Source: §C.3.1 · p. 35

C.3.2 iDDPM time step discretization

  • The cosine schedule of Nichol and Dhariwal [37] becomes a descending-noise time-step recurrence initialized by u_M = 0.Ho et al. [16] use a linear β_t schedule instead. The iDDPM implementation uses cosine offset s = 0.008, omits division by f(0), and clamps β_t to 0.999 to prevent singularities.Source: §C.3.2 · pp. 36–37

C.3.3 iDDPM preconditioning and training · C.3.4 iDDPM practical considerations · D Further analysis of deterministic sampling

iDDPM preconditioning matches VP preconditioning, and its main training loss follows the formulation used by Song et al.. Reusing the pretrained ImageNet-64 checkpoint requires adapting deterministic and stochastic sampling to its discrete noise levels.

C.3.3 iDDPM preconditioning and training

  • iDDPM uses VP-equivalent preconditioning and the same main training loss as Song et al., with uniformly sampled discrete noise levels.Nichol and Dhariwal [37] use λ(σ) = 1/σ^2 for Lsimple and also include the secondary loss Lvlb.Source: §C.3.3 · p. 38

C.3.4 iDDPM practical considerations

  • 296 million trainable parameters and 1000 supported noise levels characterize the pretrained ImageNet-64 “ADM (dropout)” checkpoint from Dhariwal and Nichol [9].Its supported noise levels range approximately from 20291 to 0.0064, constraining where the network can be evaluated.Source: §C.3.4 · p. 38
  • DDIM resampling skips the first eight supported noise levels because they exceed the preferred σmax = 80, explaining the original-versus-reimplemented sampler differences.The authors modify the flexible resampling scheme of Nichol and Dhariwal [37] with j0 = 8; Song et al. instead require N to divide 1000.Source: §C.3.4 · p. 38
  • Deterministic sampling rounds each scheduled noise level to its nearest supported counterpart and sets σmin = 0.0064.This suffices because Algorithm 1 evaluates the denoiser only at the scheduled noise levels before the terminal step.Source: §C.3.4 · p. 38
  • Stochastic sampling rounds each temporarily increased noise level to the nearest supported value before evaluating the network.Algorithm 2 replaces its perturbed time with the supported noise level closest to ti + γi ti.Source: §C.3.4 · p. 38
  • Discrete-noise adaptations enable direct reuse of the pretrained iDDPM network in both samplers, using its noise prediction and ignoring its covariance prediction.The model outputs both ϵθ and Σθ, as described in [37]; only ϵθ is used.Source: §C.3.4 · p. 38

D.1 Truncation error analysis and choice of discretization parameters · D.2 General family of 2nd order Runge–Kutta variants · E Further results with stochastic sampling

Truncation-error analysis supports noise-dependent discretization with Heun’s method, but image-generation FID favors ρ = 7 over the numerically balanced ρ = 3. The authors retain Heun’s α = 1 despite a slight experimental advantage for α = 1.1.

D.1 Truncation error analysis and choice of discretization parameters

  • ρ = 7 is adopted across experiments because emphasizing lower noise levels improves FID beyond the relatively good results at ρ = 3.This deliberately increases numerical error at high noise levels to reduce it at low noise levels.Source: Figure 13 · p. 39, §D.1 · p. 40
  • RMSE ∈[0.0030, 0.0045] remains almost constant with Heun’s method at ρ = 3, making this combination close to optimal for numerical error.Heun’s correction accounts for changing derivatives, giving cubic local-error convergence at the cost of one additional network evaluation per step.Source: Figure 13 · p. 39, §D.1 · p. 40
  • RMSE ≈0.56 occurs at σ_i ≤0.5 for Euler sampling with uniform noise spacing h_σ = 1.25, supporting smaller steps at lower noise.Low-noise errors vary negligibly across sampled latents, suggesting no benefit from adapting the timestep schedule separately for each sample.Source: Figure 13 · p. 39, §D.1 · p. 39
  • RMSE ∼0.03 remains relatively high for Euler’s method even when ρ = 2 approximately balances errors across noise levels.Increasing ρ reduces error below σ = 10 while increasing it above σ = 10; the desired RMSE is well below 0.01 at low step counts.Source: Figure 13 · p. 39, §D.1 · p. 40
  • Local approximation errors accumulate globally, motivating solver and timestep choices that minimize the largest local truncation error.For the Lipschitz-continuous networks considered, ||e_N|| ≤ E max_i ||τ_i||, where E depends on N, the endpoint times, and the Lipschitz constant.Source: §D.1 · pp. 38–39
  • A polynomial warp defines the noise schedule, using σ_min = max(σ_lo, 0.002), σ_max = min(σ_hi, 80), and a final zero-noise step.The schedule resamples w(z) = z^ρ, with endpoints determined by the model’s supported noise range.Source: §D.1 · p. 39

D.2 General family of 2nd order Runge–Kutta variants

  • The second-order Runge–Kutta family uses α to control the additional gradient’s evaluation point and weight, with equal computational cost across variants.α = 1 gives Heun’s method, α = 1/2 gives midpoint, and α = 2/3 gives Ralston. Their approximation errors differ according to the underlying function’s geometry.Source: §D.2 · p. 40
  • α = 1.1 performed slightly better experimentally, but α = 1 (Heun) was retained because overshooting lacks justification and its general benefit is uncertain.Heun also supports networks trained only at discrete noise levels: its additional gradient evaluation occurs exactly at the next scheduled timestep.Source: §D.2 · p. 41

E.1 Image degradation due to excessive stochastic iteration · E.2 Stochastic sampling parameters

Excessive stochastic iteration degrades images in noise-dependent ways, motivating restrictions at low noise levels and noise inflation. Combining the corrections gives the best observed FID, with sampling parameters selected through case-specific grid search.

E.1 Image degradation due to excessive stochastic iteration

  • Fixed-noise Langevin iteration adds and removes equal noise, but theoretical convergence toward p(x; σ) requires the denoiser to induce a conservative vector field.Figure 14 holds the noise level fixed during repeated iterations and displays the denoised outputs.Source: §E.1 · p. 41, Figure 14 · p. 42
  • Below approximately 0.2 noise, images begin oversaturating at 2k iterations and subsequently become fully corrupted; Stmin > 0 prevents stochastic sampling at very low noise levels.Repeated iteration degrades the image distribution in all tested cases, but the failure mode depends on the dataset and noise level.Source: §E.1 · p. 41, Figure 14 · p. 42
  • Snoise = 1.000 produces increasingly abstract, colorless images under extensive high-noise iteration; Snoise > 1 counteracts this tendency but not low-noise oversaturation or corruption.The distinct failure modes suggest multiple sources of degradation; their root causes remain unresolved.Source: §E.1 · p. 41, Figure 14 · p. 42
  • At fixed NFE, each tested case and correction combination generally has an optimal amount of stochasticity, beyond which FID deteriorates.The experiments vary Schurn to control stochasticity.Source: §E.1 · p. 41
  • Enabling all heuristic corrections yields the best FID across tested Schurn values, although the relative importance of Snoise and Stmin,tmax depends on the case.Figure 15 evaluates pre-trained networks from Song et al. and Dhariwal and Nichol [9] using N = 256 steps and NFE = 511.Source: §E.1 · pp. 41, 43, Figure 15 · p. 43

E.2 Stochastic sampling parameters

  • Schurn, Stmin, Stmax, and Snoise are selected by grid search, with optimal settings depending on the experimental case.Better understanding of degradation may enable more direct ways to address it.Source: §E.2 · p. 43

F Implementation details · F.1 FID calculation · F.2 Augmentation regularization

The section details implementation validation, repeated FID measurement, and conditional geometric augmentation designed to combat overfitting. Augmentation conditioning supplies auxiliary training tasks while allowing unaugmented sampling without observed geometric leakage.

F Implementation details

  • The new codebase was extensively tested to match previous samplers, pretrained models, architectures, training configurations, and evaluation from implementations, [9], and [26].Experiments used PyTorch 1.10.0, CUDA 11.4, and CuDNN 8.2.0 on NVIDIA DGX-1 systems with eight Tesla V100 GPUs each.Source: F Implementation details · p. 43

F.1 FID calculation

  • 50,000 generated images are compared against all available real images for FID [15], without augmentation.Evaluation uses the Inception-v3 model supplied with StyleGAN3 [26], a direct PyTorch translation of the original TensorFlow model. Results match the FID implementations of Dhariwal and Nichol [9] and Karras et al. [26].Source: §F.1 · p. 44
  • Three FID evaluations are performed per experiment, reporting the minimum to reduce the impact of random variation typically around ±2%.Figures 4, 5b, 13c, and 15 also highlight the difference between the highest and lowest achieved FID.Source: §F.1 · p. 44

F.2 Augmentation regularization

  • 6 geometric transformations combat denoiser overfitting through conditional augmentation; color corruption and image-space filtering were consistently harmful for diffusion-based models.The pipeline builds on augmentation concepts originally proposed for GANs by Karras et al. [25].Source: §F.2 · p. 44
  • 8 random parameters define a combined geometric transformation, while a 9-dimensional conditioning vector informs the denoiser about the applied augmentation.Each image is independently transformed before noise addition using a homogeneous 2D matrix and the implementation of [25], with 2× supersampled high-quality Wavelet filters. Enablement probabilities are 12% for CIFAR-10 and 15% for FFHQ and AFHQv2, except always-enabled x-flips; disabled augmentations receive zero parameters.Source: §F.2 · p. 44, Table 6 · p. 44
  • Augmentation conditioning creates auxiliary distribution-modeling tasks that diversify training samples, prevent individual-sample overfitting, and appear to benefit the main denoising task.The authors speculate that these benefits arise because denoising is similar across augmentation settings.Source: §F.2 · pp. 44–45
  • 100% augmentation probability produced no observed geometric leakage when sampling with zero augmentation conditioning.Zero conditioning represents unaugmented images. Always enabling the conditional x-flip augmentation while disabling dataset x-flips retains augmentation benefits without generating mirrored text or logos.Source: §F.2 · p. 45

F.3 Training configurations · F.4 Network architectures · F.5 Licenses

The training configurations separate hyperparameter and capacity adjustments from improved preconditioning, noise distribution, loss weighting, and augmentation regularization. Architectural extensions add class and augmentation conditioning to DDPM++ and NCSN++, while class-conditional ImageNet-64 uses unchanged ADM from Dhariwal and Nichol [9].

F.3 Training configurations

  • 200 million training images per model were used, selecting minimum-FID snapshots under deterministic sampling at NFE = 35 or NFE = 79, depending on resolution.Config A reproduces Song et al.; config B uses 8 GPUs, batches of 512 or 256, no gradient clipping, a CIFAR-10 learning rate of 0.001 ramped over 10 million images, and EMA half-life of 0.5 million images. Dropout is grid-searched at 1% increments; snapshots are saved every 2.5 million images, with training taking approximately 2 days for CIFAR-10 and 4 days for FFHQ and AFHQv2.Source: §F.3 · p. 45
  • Approximately 56 million parameters at 32×32 and 62 million at 64×64 remain after config C rebalances network capacity.Removing 4×4 layers and doubling capacity at 16×16 slightly reduces parameter counts; the former mainly contributed to overfitting, while the latter were critical for high-quality results.Source: §F.3 · p. 46
  • Configs D, E, and F introduce improved preconditioning, noise distribution and loss weighting, and augmentation regularization, respectively.Other hyperparameters remain unchanged from config C; config F is the final improved setup.Source: §F.3 · pp. 45–46
  • Approximately 2500 million training images over two weeks were used for ImageNet-64, which required considerably longer training than the other datasets.Training used 32 NVIDIA Ampere GPUs, batch size 4096, learning rate 0.0001, EMA of 50 million images, and the architecture and dropout of Dhariwal and Nichol [9], without augmentation because overfitting was not a concern. Parameters were stored in FP32 and network evaluation used FP16 except in embedding and self-attention layers, where FP16 occasionally caused stability issues.Source: §F.3 · p. 46

F.4 Network architectures

  • Config F makes VP and VE otherwise identical except for their respective DDPM++ and NCSN++ architectures, both proposed by Song et al..DDPM++ uses box filtering and positional noise encoding inherited from DDPM [16], whereas NCSN++ uses bilinear filtering [3] and random Fourier features. NCSN++ also adds residual connections from the input image to each encoder block.Source: §F.4 · p. 46
  • Optional class and augmentation inputs extend DDPM++ and NCSN++ by adding their projected feature vectors elementwise to the noise-level conditioning vector.Class labels are one-hot encoded and scaled before a fully connected layer; augmentation parameters enter their fully connected layer as-is.Source: §F.4 · pp. 46–47
  • Approximately 296 million trainable parameters comprise the unchanged ADM architecture of Dhariwal and Nichol [9] used for class-conditional ImageNet-64.Compared with DDPM++, ADM uses 3 rather than 4 residual blocks per resolution, 768 rather than 256 channels at the lowest resolution, and 22 rather than 6 self-attention layers. It also uses multi-head attention, including 12 heads at the lowest resolution; the precise impact of architectural choices remains open.Source: §F.4 · p. 47

F.5 Licenses

  • Dataset and pretrained-model licenses vary, with ImageNet licensing explicitly described as unclear.CIFAR-10 uses MIT, FFHQ [27] Creative Commons BY-NC-SA 4.0, and AFHQv2 [7] Creative Commons BY-NC 4.0. Pretrained CIFAR-10 models by Song et al. and Inception-v3 by Szegedy et al. use Apache V2.0, while the ImageNet-64 model by Dhariwal and Nichol [9] uses MIT.Source: §F.5 · p. 47

Related papers

Loading 2206.00364v2…