PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing and Improving the Training Dynamics of Diffusion Models

Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, Samuli Laine

arXiv:2312.02696v2cs.CVcs.AIcs.LGcs.NEstat.ML

TL;DR

Diffusion-model training remains challenging under highly stochastic losses. The paper redesigns ADM layers for magnitude control and reports ImageNet-512 FID of 1.81 with guidance 1.2.

  • Problem

    Diffusion training dynamics remain challenging under stochastic losses, with effective learning rates tending to decay unequally across layers.

    Source: NVIDIA · p. 1, §2.3 · p. 4

  • Method

    They redesign ADM layers to control activation, weight, and update magnitudes, removing group normalization and using one-quarter as many pixel-normalization layers.

    Source: §2.4 · p. 5

  • Results

    1.81 ImageNet-512 FID with guidance 1.2, down from 1.91 and setting a new dataset record.

    Source: §4 · p. 8

  • Takeaways & Limitations

    The improved denoiser is a drop-in ADM replacement, and less-entangled training aspects should make local architecture changes easier.

    Source: §5 · p. 8

    Whether the magnitude-focused methodology benefits other diffusion architectures or application areas remains an open question.

    Source: §5 · p. 8

Abstract

from arXiv · show

Diffusion models currently dominate the field of data-driven image synthesis with their unparalleled scaling to large datasets. In this paper, we identify and rectify several causes for uneven and ineffective training in the popular ADM diffusion model architecture, without altering its high-level structure. Observing uncontrolled magnitude changes and imbalances in both the network activations and weights over the course of training, we redesign the network layers to preserve activation, weight, and update magnitudes on expectation. We find that systematic application of this philosophy eliminates the observed drifts and imbalances, resulting in considerably better networks at equal computational complexity. Our modifications improve the previous record FID of 2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic sampling. As an independent contribution, we present a method for setting the exponential moving average (EMA) parameters post-hoc, i.e., after completing the training run. This allows precise tuning of EMA length without the cost of performing several training runs, and reveals its surprising interactions with network architecture, training time, and guidance.

NVIDIA · 1. Introduction

The introduction presents diffusion models as a popular, versatile approach to image synthesis and describes generation as repeated denoising from pure noise.

1. Introduction

  • Diffusion models have become widely popular for high-quality image synthesis from text prompts, example images, and other inputs [23] [55] [75] [76] [77].Source: §1 · p. 1
  • Diffusion approaches offer versatile controls [19] [22] [53], ways to introduce novel subjects [14] [68], and applications in audio [42] [61], video [7] [24], and 3D shapes [49, 60, 63, 74].Source: §1 · p. 1
  • At a high level, diffusion models turn pure noise into a novel image through repeated image denoising.Source: §1 · p. 1

NVIDIA

The paper diagnoses unstable diffusion-model training and redesigns ADM layers to balance expected weights, activations, gradients, and updates. It reports improved ImageNet-512 results and introduces post-hoc EMA tuning.

NVIDIA

  • Diffusion training is difficult because stochastic signals span noise levels, noise realizations, and conditioning, while small intermediate-step errors can snowball into image-quality losses.The network must estimate average clean images across these varied inputs.Source: NVIDIA · pp. 1–2
  • The authors modify ADM without changing its overall structure, standardizing expected weights, activations, gradients, and updates to reduce imbalances and improve learning.They target predictable, even responses to parameter updates amid interactions among hyperparameters, network design, and training setups.Source: NVIDIA · p. 2
  • FID reached 1.81 with guidance and 1.91 without guidance, versus prior 2.41 and 2.99, using a drop-in ADM replacement and fast deterministic sampling.The final model also surpasses the previous state of the art with a 5× smaller model, measured in gigaflops per evaluation.Source: Figure 1 · p. 1, NVIDIA · p. 2
  • Post-hoc EMA reconstructs networks with arbitrary EMA profiles from preintegrated weight snapshots, enabling efficient decay tuning after training.The method addresses the difficulty of tuning EMA decay, whose effects become apparent only near convergence.Source: NVIDIA · p. 2

2. Improving the training dynamics

The section progressively modifies the ADM architecture and training procedure to control activation, weight, and update magnitudes, culminating in a 2.56 FID exploration model.

2.1. Preliminary changes

  • Preliminary tuning and architectural changes reduce FID from 8.00 to 6.96 by addressing noise-level loss imbalance, simplifying the network, and stabilizing attention.The changes include hyperparameter tuning and disabling 32×32 self-attention; reciprocal raw-loss scaling lowers FID from 8.00 to 7.24, while streamlining and cosine attention lower it to 6.96 and enable 16-bit math.Source: §2.1 · pp. 2–3

2.2. Standardizing activation magnitudes

  • CONFIG D eliminates activation-magnitude drift and improves FID from 6.96 to 3.75 by normalizing learned-layer weights before use.ADM’s residual paths allow activation growth despite block-level group normalization; the normalization divides each layer’s output by the expected scaling derived from its weight norm.Source: §2.2 · pp. 3–4

2.3. Standardizing weights and updates

  • CONFIG E preserves activation and weight magnitudes and improves FID from 3.75 to 3.02 by controlling weight growth and effective learning rate.Forced weight normalization keeps weight vectors at unit variance, while inverse square root learning-rate decay is introduced because constant learning rates no longer induce convergence.Source: §2.3 · p. 4

2.4. Removing group normalizations (CONFIG F)

  • CONFIG F replaces all group normalization layers with one-quarter as many pixel normalization layers and improves FID from 3.02 to 2.71.The authors hypothesize that pixel normalization counteracts correlations that violate CONFIG D’s standardization assumptions; they also remove two embedding/output components and move residual-block resampling to the main path.Source: §2.4 · p. 5

2.5. Magnitude-preserving fixed-function layers (CONFIG G)

  • The magnitude-preserving CONFIG G design achieves FID 2.56 and is described as highly competitive with state of the art despite modest computational complexity.It balances residual additions and skip concatenations, compensates for magnitude changes in fixed-function layers, and adds learned gains where needed.Source: §2.5 · p. 5
  • CONFIG G adds learned gains at the network output and conditioning signals, allowing dropout to be disabled during training without ill effects.The output gain is zero-initialized, and conditioning gains make conditioning disabled at initialization while learning its strength in each encoder/decoder block.Source: §2.5 · p. 5

3. Post-hoc EMA

The section introduces post-hoc EMA selection and a power-function averaging profile, then uses EMA-length sweeps to examine how preferred averages vary with architecture, training time, and classifier-free guidance. The analysis also shows that post-hoc reconstruction can support new EMA profiles without requiring them to be specified before training.

3. Post-hoc EMA

  • The method enables dense post-training sampling of EMA lengths, revealing their differing effects across training configurations without requiring an EMA profile to be chosen beforehand.The authors maintain two power-function averages during training and store snapshots periodically for reconstruction.Source: §3 · p. 5, §3.2 · p. 6, §3.3 · p. 6

3.1. Power function EMA profile

  • Power-function EMA profiles give zero weight to random initialization and scale their averaging window with training time, addressing drawbacks of long exponential averages.The profile uses a time-dependent decay; its width can be reported as relative standard deviation σrel, with 10% corresponding to σrel = 0.10.Source: §3.1 · pp. 5–6

3.2. Synthesizing novel EMA profiles after training

  • Post-hoc reconstruction fits a desired EMA profile by least squares to stored averages and combines them linearly, allowing profiles to be chosen during or after training.The method is not limited to power-function profiles; it can use one stored average per snapshot, but with much lower accuracy than using two. Figure 4 reports reconstruction error decreasing experimentally on the order of O(1/n^4), with a few dozen snapshots sufficient for virtually perfect reconstruction.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3. Analysis

  • In CONFIG B, per-tensor EMA changes improve FID by up to 10% (7.24 to ∼6.5), while CONFIG G tensors agree on a sharper optimum and deviations are detrimental.Across configs B–G, the globally optimal EMA length varies, and its optimum narrows toward CONFIG G.Source: §3.3 · p. 6, Figure 5 · p. 7
  • The optimal EMA length slowly shifts toward relatively longer averages as training progresses.EMA length is defined relative to training duration, and the analysis reports interactions with network architecture, training time, and classifier-free guidance.Source: §3 · p. 5, §3.3 · pp. 6–7

4. Results

On ImageNet-512, EDM2 sets a new 1.81 FID record with modest guidance; on ImageNet-64, scaling yields 1.33 FID, and results show EMA length depends on guidance and training conditions.

4. Results

  • Dropout helps larger configurations showing overfitting (M–XXL) but harms smaller ones (XS, S), so it is enabled only for the larger models.Overfitting is identified when training loss decreases while validation loss and FID increase.Source: §4 · p. 7
  • FID 1.81 with guidance strength 1.2 improves on EDM2’s 1.91 without guidance and sets a new ImageNet-512 record.The paper reports this result for its largest model.Source: Table 2 · p. 7, §4 · p. 8
  • Without guidance, EDM2-S reaches FID 2.56 versus VDM++’s 2.99, while scaling EDM2 improves FID to 1.91.The authors also report that the results are more significant in terms of model complexity (Figure 1).Source: Table 2 · p. 7, §4 · p. 7

Guidance

  • Optimal EMA length varies strongly with guidance strength in EDM2-S, and non-optimal EMA may partly explain prior gaps between unguided and guided results.Figure 6 plots FID against EMA length at different guidance strengths; the authors describe the prior-results explanation as a postulate.Source: §4 · p. 7, Figure 6 · p. 8

Low-cost guidance

  • An XS unconditional model suffices to guide even the XXL conditional model, and using it can reduce guidance’s typical 2× computational overhead.A larger unconditional model did not improve results; all Table 2 configurations use an XS-sized unconditional model.Source: §4 · p. 8

ImageNet-64

  • FID 1.58 improves on EDM’s 2.22 at similar model complexity on ImageNet-64, and scaling further reaches 1.33.This is close to RIN’s 1.23 FID with stochastic sampling; stochastic sampling uses 1000 versus 63 NFE and entails tuning effort and computational cost.Source: Table 3 · p. 8, §4 · p. 8

Post-hoc EMA observations

  • Preliminary EMA observations suggest optimal EMA length decreases as learning rate or model capacity increases, and as dataset complexity decreases.FID remains relatively stable over roughly a 2× range of tref, so choosing αref and tref within the right ballpark may suffice; the authors characterize these findings as anecdotal.Source: §4 · p. 8

5. Discussion and future work · A. Additional results · A.1. Generated images

The improved denoiser is intended as a drop-in ADM replacement, and its less-entangled training makes local architecture changes easier. The discussion highlights future work on other architectures and applications, while post-hoc EMA makes previously costly studies more feasible.

5. Discussion and future work

  • The improved denoiser was designed as a drop-in replacement for ADM, with the hope that it will see widespread use in large-scale image generators.Source: §5 · p. 8
  • With training aspects less entangled, local architecture modifications should be easier to make without breaking other parts, enabling further U-Net structure and balance studies.Source: §5 · p. 8
  • Whether magnitude-focused methods benefit other diffusion architectures, including RIN [31] and DiT [58], or non-diffusion applications remains an open question.The passage notes that such work has received relatively little attention outside ImageNet classifiers [8] [9].Source: §5 · p. 8
  • Post-hoc EMA reduced the cost of producing some plots from a thousand GPU-years to a GPU-month.Source: §5 · p. 8
  • Cheap-to-produce EMA data may enable new studies of EMA’s role in diffusion models and principled EMA-length choices, potentially per layer or parameter.Source: §5 · p. 8

A.1. Generated images

  • Figure 7 presents hand-selected images from the largest (XXL) ImageNet-512 model without classifier-free guidance; Figures 25–27 show uncurated class images with guidance strength selected per class.Source: §A.1 · p. 12

A.2. Quality vs. compute · A.3. Guidance vs. unconditional model capacity · A.4. Learning rate vs. EMA length

The model’s efficiency gains persist across several cost measures, while guidance needs only a small unconditional model and post-hoc EMA tuning reduces sensitivity to the learning-rate decay choice.

A.2. Quality vs. compute

  • Across parameter, sampling, and training costs, the model reaches the same quality faster and improves achievable quality significantly.The sampling-cost gains are especially large because the model uses few score function evaluations (NFE).Source: §A.2 · p. 12
  • Early cleanup in CONFIG B improves convergence and execution speed, and Inception score, KID, and Recall are largely consistent with FID.The paper standardizes on FID because of its popularity.Source: §A.2 · p. 12
  • For XXL ImageNet-512, 10% dropout allows steady convergence where no dropout permits overfitting, while dropout hurts smaller models but helps larger ones.For larger models, dropout comes with slightly slower initial convergence.Source: §A.2 · p. 12

A.3. Guidance vs. unconditional model capacity

  • A very small unconditional model reduces guided-diffusion computational cost by almost 50%, while a large unconditional model is not useful; very short EMAs typically work best with guidance.The comparison uses an XXL-sized ImageNet-512 conditional model and guidance strength 1.2.Source: §A.3 · p. 12, Table 4 · p. 12

A.4. Learning rate vs. EMA length

  • With post-hoc EMA, learning-rate decays tref ∈[30k, 160k] yield FIDs within 10% of optimum, despite a sweet spot at tref = 70k.The passage says sweeping EMA lengths post hoc drastically reduces the importance of choosing the exact learning-rate decay.Source: §A.4 · p. 12
  • With EMA length fixed at 13%, varying tref raises FID by as much as 72% in the tested range.Source: §A.4 · p. 12

A.5. Fréchet distances using DINOv2 … B.1. EDM baseline (CONFIG A)

The appendix compares FID and FDDINOv2 choices and documents activation/weight magnitude measurements and the ADM-in-EDM baseline. Its results show that the preferred EMA length and guidance strength depend on the quality metric.

A.5. Fréchet distances using DINOv2

  • DINOv2 features align better with human preferences than InceptionV3, motivating a Table 2 counterpart measured on 50,000 generated and all 1,281,167 available real images.Generated class labels are sampled uniformly, and each 50,000-image sample is evaluated once because random variation between runs is small.Source: §A.5 · p. 12, Table 5 · p. 15
  • 19% is FDDINOv2’s unguided CONFIG G EMA optimum; at guidance 1.4, FDDINOv2 prefers 14% EMA while FID prefers 2%.Each metric judges the other’s optimum poorly, underscoring the need to tune EMA for the chosen quality metric.Source: §A.5 · pp. 12, 14, Figure 14 · p. 17
  • 1.9 vs. 1.4 is the preferred guidance strength for FDDINOv2 vs. FID, and CONFIG B improves FDDINOv2 at least as much as the observed FID drop.FID considers guidance strength 1.9 excessive.Source: §A.5 · p. 14, Figure 14 · p. 17

B. Architecture details

  • Appendix B presents configuration-specific architecture diagrams and associated hyperparameters for the architectural changes introduced in Section 2.Figures 16–22 illustrate the configurations.Source: B. Architecture details · p. 15

B.1. EDM baseline (CONFIG A)

  • 219 training iterations with batch size 4096 train the EDM-framework ADM baseline on 2,147.5 million images, using a 50M-image EMA half-life.For ImageNet-512, the baseline uses a Stable Diffusion VAE to encode images into 64×64×4 latents; the latents are normalized to zero mean and standard deviation 0.5 before training.Source: §B.1 · pp. 15–16
  • The baseline denoiser follows EDM preconditioning around an ADM U-Net, with noise- and class-conditioned residual blocks, encoder-to-decoder skip connections, and self-attention in blocks operating at 32×32 resolution and below.Source: §B.1 · pp. 15–16

A.6. Activation and weight magnitudes

  • Figure 15 tracks dimension-weighted activation and weight magnitudes across U-Net resolution buckets, using maximum and mean aggregation with qualitatively similar behavior.Activations are measured over 4,096 training samples; unit-normal activations and weights are scaled to expected magnitude 1 regardless of tensor dimensions.Source: §A.6 · pp. 14–15, Figure 15 · p. 18

B.2. Minor improvements (CONFIG B) · B.3. Architectural streamlining (CONFIG C) · B.4. Magnitude-preserving learned layers (CONFIG D)

CONFIG B adjusts training and loss weighting, CONFIG C streamlines the architecture, and CONFIG D makes learned layers preserve activation magnitudes. Together, the changes target more consistent training behavior across noise levels, parameters, and network layers.

B.2. Minor improvements (CONFIG B)

  • Halving batch size to 2048, doubling αref to 0.0002, lowering β2 to 0.99, and removing 32×32 self-attention improved convergence, stability, and training speed.These changes had their largest impact early in training, when the network reconfigures for the task, but helped somewhat near the end as well.Source: §B.2 · p. 18
  • CONFIG B replaces traditional EMA with a power-function averaging profile and stores two averages per snapshot for high-quality post-hoc reconstruction.This supports post-hoc tuning of EMA length without running multiple training runs.Source: §B.2 · p. 19
  • For VAE latents, CONFIG B sets Pmean = −0.4 and Pstd = 1.0, shifting the log-normal noise-level distribution toward the higher-noise in-between training region.The RGB-image settings were Pmean = −1.2 and Pstd = 1.2.Source: §B.2 · p. 19
  • A trainable continuous uncertainty function u(σ) dynamically weights the EDM loss across noise levels, adapting as their losses change during training.The method treats each noise level as a task; at its optimum, the weighting scales gradients by the reciprocal of the corresponding noise-level loss, equalizing contributions across noise levels and time.Source: §B.2 · pp. 19–21

B.3. Architectural streamlining (CONFIG C)

  • CONFIG C removes biases and learned group-normalization scales, along with other bias-like constructs, and adds a constant-one input channel to mitigate potential loss of input-scale information.The changes aim to make the remaining trainable parameters behave more uniformly; the authors report no practical issue from removing biases.Source: §B.3 · p. 22
  • Cosine attention makes attention weights depend only on query and key directions, curbing uncontrolled weight growth and enabling 16-bit floating point throughout the self-attention block.CONFIG C replaces group normalization before the query/key convolutions with pixelwise feature-vector normalization after them.Source: §B.3 · p. 23
  • CONFIG C gives zero-initialized layers and class embeddings the same uniform initialization as other layers, and replaces ADM’s timestep embedding with Fourier features.The Fourier features are computed from cnoise(σ); their frequencies and phases are fixed after initialization.Source: §B.3 · pp. 23–24

B.4. Magnitude-preserving learned layers (CONFIG D)

  • CONFIG D normalizes each learned weight vector to unit length so layers preserve activation magnitude; it initializes weights from N(0, 1) and raises αref to 0.0100 from 0.0002.Under the stated assumption of uncorrelated, equal-standard-deviation inputs, the normalization preserves input standard deviation; the initialization also gives Adam updates roughly equal impact across the model early in training.Source: §B.4 · pp. 24–25

B.5. Controlling effective learning rate (CONFIG E) · B.6. Removing group normalizations (CONFIG F)

CONFIG E constrains weight magnitudes to control effective learning rates and replaces implicit drift-driven decay with explicit scheduling. CONFIG F removes group normalization and makes targeted architectural changes, including encoder input normalization and several simplifications.

B.5. Controlling effective learning rate (CONFIG E)

  • CONFIG E counters raw-weight norm growth that can cause layerwise effective-learning-rate drift by enforcing fixed norms, making the effective rate proportional to α.Weight norms grow under standard gradient descent and can also grow with Adam; the drift is visible in CONFIG C.Source: §B.5 · pp. 26–27
  • With forced normalization eliminating implicit decay, CONFIG E explicitly uses inverse square root learning-rate decay, beginning after tref iterations.Earlier configurations’ constant-rate schedule is the special case tref = ∞.Source: §B.5 · p. 27
  • Forced normalization is paired with standard weight normalization so Adam estimates updates in the tangent plane rather than being distorted by a gradient component later erased by renormalization.Without standard normalization, Adam can produce updates smaller than intended and of uncontrolled magnitude.Source: §B.5 · pp. 27–28, Figure 23 · p. 27
  • Raw weights are constrained to norm √Nj while evaluation weight normalization scales them to norm 1, making Adam’s relative update magnitude independent of layer size.This removes an implicit dependence between learning rate and layer size.Source: §B.5 · p. 28
  • Standard weight normalization also supports EMA: averaged raw weights are automatically renormalized upon use, preserving the fixed-magnitude constraint without changing the averaging.Source: §B.5 · p. 28

B.6. Removing group normalizations (CONFIG F)

  • CONFIG F removes all group normalization after activation-magnitude drift is addressed; the authors report no obvious downsides and improved decoder FID, possibly because output-pixel scale matters to the loss.The motivation is that image-wide normalization can entangle localized representations.Source: §B.6 · p. 28
  • CONFIG F retains slight pixelwise feature-vector normalization at encoder-block inputs, hypothesizing that it counteracts correlations and helps standardize encoder and decoder magnitudes.The authors attribute decoder standardization presumably to U-Net skip connections.Source: §B.6 · pp. 28–29
  • CONFIG F also unifies resampling on the main path, moves the encoder 1×1 convolution, removes the final SiLU, and removes an embedding-network fully connected layer; these changes are roughly FID-neutral.Source: §B.6 · p. 29

B.7. Magnitude-preserving fixed-function layers (CONFIG G) … C.4. Power function EMA profile

The sections extend magnitude preservation to fixed-function layers and develop a post-hoc EMA method for choosing averaging profiles after training. They introduce a power-function profile that scales with training time and assigns no weight to the initial model state.

B.7. Magnitude-preserving fixed-function layers (CONFIG G)

  • CONFIG G extends magnitude preservation to fixed-function layers, aiming to standardize network inputs and preserve standardized activations through each operation.The resulting activation means are reported as surprisingly close to the ideal.Source: §B.7 · p. 29
  • CONFIG G treats branch balance as an independent hyperparameter, rather than tying contributions to channel counts.The reported best settings are t = 0.5 for concatenation and t = 0.3 for encoder, decoder, and self-attention sums; learned scalar gains allow selected activations to deviate from magnitude 1 when needed.Source: §B.7 · pp. 30–31

C. Post-hoc EMA details

  • Post-hoc EMA selects an averaging profile after training by linearly combining pre-averaged models stored during the run.The proposed power-function profile automatically scales with training time and assigns zero contribution at t = 0.Source: C. Post-hoc EMA details · p. 31

C.1. Definitions

  • The EMA framework represents averaged weights as an expectation over a normalized response function, whose mean indicates delay and standard deviation correlates with averaging length.The derivation treats weights and response functions as continuous in time.Source: §C.1 · p. 31

C.2. Traditional EMA profile

  • Traditional exponential EMA places probability mass from before training at t = 0, potentially giving the initial, task-untrained weights considerable influence when λ is not much smaller than training time.The profile has an infinite historical tail, so this mass placement corresponds to initializing accumulated EMA weights with the network’s initial weights.Source: §C.2 · pp. 31–32

C.3. Tracking the averaged weights during training

  • During training, averaged weights are updated by linearly interpolating them toward the current weights using β(tn).For exponential EMA, β(tn) is constant.Source: §C.3 · p. 32

C.4. Power function EMA profile

  • The power-function profile uses self-similar response functions with zero weight at t = 0, so its averaging length grows in proportion to training time.The amount of averaging is controlled by γ; when parameterized by relative standard deviation, γ > 0 requires σrel < 0.2886, and β(tn) varies with tn.Source: §C.4 · pp. 32–33

C.5. Synthesizing novel EMA profiles after training

The section presents a post-training method for synthesizing novel EMA profiles from stored snapshots, avoiding the need to track every candidate profile during training. It fits snapshot weights to match a desired response function and reports nearly perfect reconstruction over a specified range of profile lengths.

C.5. Synthesizing novel EMA profiles after training

  • Novel EMA weights are approximated by a weighted average of stored snapshots because their precise value cannot be recovered from the snapshots alone.The weights are chosen so the resulting approximate response function matches the desired response function as closely as possible.Source: §C.5 · p. 33
  • The method chooses snapshot weights by minimizing L2 response-function distance, yielding the linear system Ax = b and solution x = A^-1 b.The matrix and vector elements are inner products between the snapshot response functions and the target response function.Source: §C.5 · pp. 33–34
  • For power-function EMA profiles, analytical inner products make the implementation considerably faster and more accurate than Monte Carlo integration.Each response function is determined by its associated (t, γ) pair.Source: §C.5 · p. 34
  • The solver handles multiple target response functions simultaneously, returning a matrix with targets as columns and snapshots as rows.The NumPy implementation accepts snapshot and target (t, γ) values as lists.Source: §C.5 · p. 34
  • Across training runs, snapshots of two averages stored every 8 million images enabled nearly perfect reconstruction for σ_rel ∈ [0.015, 0.250].The tracked averages corresponded to σ_rel = 0.05 and σ_rel = 0.10; runs stored 160–512 pre-averaged models using 16-bit floating point.Source: §C.5 · p. 34

D. Implementation details · D.1. Sampling

The implementation uses EDM’s second-order deterministic sampler with fixed noise and step settings, and defines guidance by combining conditional and unconditional denoisers. The paper also reports the implementation environment and points to model details and released code.

D. Implementation details

  • Experiments used the public EDM [37] codebase, NVIDIA A100-SXM4-80GB GPUs, and 32 GPUs per training run and 8 per evaluation run.The software stack was Python 3.9.16, PyTorch 2.0.0, CUDA 11.8, and CuDNN 8.9.4.Source: D. Implementation details · p. 34
  • Table 6 lists details of the main models featured in Tables 2 and 3, and the implementation and pretrained models are publicly available [37].For ImageNet-512, EDM2-S is the same as CONFIG G in Figure 22.Source: D. Implementation details · p. 34, Table 6 · p. 35

D.1. Sampling

  • All experiments use EDM’s second-order deterministic sampler, with σ(t) = t, s(t) = 1, σmin = 0.002, σmax = 80, and ρ = 7 [37].The sampler is Algorithm 1 in [37].Source: §D.1 · pp. 34–35
  • The authors found N = 32 sampling steps sufficient for high-quality results on ImageNet-512 and ImageNet-64, without extensive sweeps over N.Source: §D.1 · p. 35
  • Following Imagen [70], guidance combines conditional and unconditional denoisers as ˆD = wDθ(x; σ, c) + (1 − w)Du(x; σ), with w = 1 disabling guidance and w > 1 strengthening it [70].Source: §D.1 · p. 35
  • NFE counts each evaluation of ˆD during sampling and is unchanged by the guidance weight w.Source: §D.1 · p. 35

D.2. Mixed-precision training … E. Negative societal impacts

These appendices describe training and evaluation protocols, define the scope of the per-layer EMA sweep, and acknowledge potential societal harms and energy costs of diffusion models.

D.2. Mixed-precision training

  • Training stores trainable parameters in FP32, casts them to FP16 for evaluation, and processes activations in FP16 except in the embedding network and associated per-block linear layers.Models train as well with FP16 as FP32 when loss is appropriately scaled; to guard against rare overflows, gradients replace NaN and Inf with 0 and block-end activations are clamped to [−256, +256]. CONFIGS A–B also use FP32 self-attention in the baseline architecture.Source: §D.2 · pp. 35–36

D.3. Training data

  • ImageNet images are resized by their shorter edge using bicubic interpolation, center-cropped to the desired square resolution, and used without horizontal flips or other augmentation, following ADM [13].Source: §D.3 · p. 36

D.4. FID calculation

  • FID uses 50,000 generated images and all available real images without augmentation; each experiment reports the minimum of three evaluations to reduce variation, typically around ±2%.The protocol follows EDM [37] and uses the pre-trained Inception-v3 model provided with StyleGAN3 [36].Source: §D.4 · p. 36

D.5. Model complexity estimation

  • For ImageNet-512 XXL, aten::_convolution accounts for 545.50 Gflops, while aten::mul, aten::div, aten::linalg_vector_norm, and aten::matmul account for 1.68, 1.62, 1.54, and 1.43 Gflops, respectively.Complexity is estimated by tracing model evaluation operations; estimates for prior work are best-effort when public values are unavailable, and the authors believe their estimates are accurate within 10%.Source: §D.5 · p. 36

D.6. Per-layer sensitivity to EMA length

  • The Figure 5b EMA-length sensitivity sweep covers selected encoder, decoder, and mapping-layer weight tensors, excluding biases, group-norm scales, and learned affine gains.Source: §D.6 · p. 36

E. Negative societal impacts

  • Diffusion models can contribute to disinformation, stereotypes, and harmful biases, while training and sampling consume substantial electricity that may contribute to climate change.The authors caution that their improvements to image quality could further amplify these issues, despite efficiency gains.Source: E. Negative societal impacts · p. 36

Related papers

Loading 2312.02696v2…