Elucidating the Exposure Bias in Diffusion Models

1 INTRODUCTION

• Diffusion models learn through a noise or score prediction objective and generate samples by progressively removing noise from random initial vectors.The paper situates diffusion models alongside GANs, VAEs, and flow-based models.Source: §1 · p. 1
• Exposure bias is the mismatch between ground-truth training inputs and previously generated sampling inputs, which can produce prediction discrepancies, error accumulation, and sampling drift.The paper identifies this issue in the long iterative sampling chain used by diffusion models.Source: §1 · p. 1
• The authors report that exposure bias lacks in-depth exploration, including a proper metric and explicit error analysis, and analytically model the sampling distribution with prediction error.Their analysis finds a larger variance in the practical sampling distribution than in the ground-truth distribution at each step, highlighting the training–sampling difference.Source: §1 · p. 1
• The paper proposes δ_t to measure exposure bias through the variance difference between training and sampling, and introduces training-free, plug-in Epsilon Scaling to alleviate it.The introduction also discusses potential solutions to exposure bias.Source: §1 · p. 2
• Experiments span diffusion frameworks, deterministic and stochastic sampling, and conditional and unconditional generation; the authors report FID improvements without affecting recall or precision.They also report that Epsilon Scaling moves the sampling trajectory toward the training trajectory, reducing exposure bias.Source: §1 · p. 2
• The authors characterize their experiments as demonstrating Epsilon Scaling’s generality and wide applicability across diffusion architectures.They describe it as a training-free, plug-in method that significantly improves sample quality.Source: §1 · p. 2

2 RELATED WORK

• Diffusion research has advanced through unified score-based and DDPM formulations and the EDM design space, while distillation, Restart sampling, and fast ODE samplers address slow reverse sampling (Xu et al., 2023a; Zhao et al., 2023).The cited work also spans conditional generation and applications in image, audio, object-shape, and time-series generation.Source: §2 · p. 2
• Prior exposure-bias work includes training regularization, sampling-time-step manipulation, and training-time optimization of a cumulative-error upper bound (Li et al., 2023a).The authors identify explicit sampling-distribution analysis, a metric, and root-cause analysis as areas still lacking illuminating research.Source: §2 · p. 2

3.1 SAMPLING DISTRIBUTION WITH PREDICTION ERROR

• The analysis models the reverse step using a posterior conditioned on predicted x0, approximates prediction uncertainty as Gaussian, and compares the resulting sampling distribution with the training distribution.The distribution analysis is also generalized to multi-step sampling and derived for DDIM; the reported variance gap is unchanged when the Gaussian prediction mean is not x0.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 EXPOSURE BIAS DUE TO PREDICTION ERROR

• Prediction error makes sampling variance exceed training variance in a single step; errors accumulate, increasing multi-step variance error toward t=1 and producing sampling drift.The paper concludes that manipulating the sampling noise schedule cannot alleviate exposure bias.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 METRIC FOR EXPOSURE BIAS

• The proposed δt metric quantifies exposure bias through the variance difference between the training distribution and the multi-step sampling distribution.For a trained model, δt exhibits a strong correlation with FID.Source: §3.3 · p. 4

3.4 SOLUTION DISCUSSION

• Potential remedies include more accurate prediction networks and data augmentation, but the discussed approaches require retraining and expensive parameter searches.The paper therefore asks whether exposure bias can be alleviated during sampling without retraining.Source: §3.4 · pp. 4–5

4.1 EPSILON SCALING

• Sampling predictions have a larger L2 norm than training predictions, consistent with sampling inputs lying around training inputs with larger variance.The authors interpret this as an inaccurate vector field with longer vectors near the training input.Source: §4.1 · p. 5
• Scaling down the sampling prediction by λ(t) is intended to counter its over-predicted magnitude and move sampling inputs closer to training inputs.The method is a plug-in and adds no computational load to the original sampling process.Source: §4.1 · p. 5

4.2 THE DESIGN OF SCALING SCHEDULE

• The scaling schedule λ(t) is derived by solving Eq. 10 after measuring ΔN(t), which reflects accumulated prediction errors from earlier sampling steps.The error is assumed to propagate linearly over the sampling chain.Source: §4.2 · pp. 5–6
• Because ΔN(t) is fit by a quadratic over t ∼ (5, T), solving Eq. 10 yields a linear schedule, but most experiments use a uniform λ(t) for ease and near-optimal performance.The authors exclude timesteps near t = 0 from fitting and search for schedule parameters instead of measuring ΔN(t) in simulations.Source: §4.2 · p. 6

5 RESULTS

• The evaluation focuses on at most 100 sampling steps, reports FID from 50k generated samples, and states that Epsilon Scaling does not affect precision or recall.FID computation follows Dhariwal & Nichol (2021); precision and recall results are reported in Appendix A.9.Source: §5 · p. 6

5.1 MAIN RESULTS ON ADM

• 2.17 FID: ADM-ES outperforms ADM across datasets and sampling steps and beats prior stochastic samplers, while sampling 5-fold to 20-fold faster than EDM and Improved SDE.On FFHQ 128×128, ADM-ES improves FID over ADM by 7.75, 16.65, and 34.52 under 100, 50, and 20 sampling steps, respectively.Source: Table 3 · p. 6, Table 2 · p. 6, §5.1 · p. 7

5.2 EPSILON SCALING ALLEVIATES EXPOSURE BIAS

• Epsilon Scaling yields lower δt at the end of sampling and moves the sampling trajectory closer to the training vector field.The reported lower δt is at t = 1; the vector-field result is conditional on the network norm being locally monotonic around the input.Source: §5.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7

5.3 RESULTS ON DDIM/DDPM

• On CelebA, Epsilon Scaling improves DDPM performance by 47.7%, 63.1%, and 60.7% at 20, 50, and 100 sampling steps, with gains also reported for DDIM and CIFAR-10.The reported improvement is smaller for DDIM than DDPM.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 RESULTS ON EDM

• EDM-ES improves CIFAR-10 FID from 3.81 to 2.80 unconditionally and from 3.74 to 2.59 conditionally with the 35-step Euler sampler.Consistent improvements are reported across conditional settings and ODE solver types; the reported gain is larger for Euler than Heun.Source: §5.4 · p. 8, Table 5 · p. 8

5.5 RESULTS ON LDM

• On CelebA-HQ, LDM-ES achieves 15.68 FID at 20 steps, almost half that of LDM, and 7.36 FID at 100 steps; improvements are also reported on FFHQ and DiT.The LDM experiments use a DDPM sampler on high-resolution datasets.Source: §5.5 · pp. 8–9, Table 6 · p. 9

5.6 QUALITATIVE COMPARISON

• With the same random seed and 100-step FFHQ 128×128 sampling, ADM-ES refines overexposure, underexposure, coarse backgrounds, and detail defects in ADM samples.The authors report that both models reach the same or similar modes from the common starting point.Source: §5.6 · p. 9

• Figure 7 compares qualitative samples from ADM in the first row and ADM-ES in the second row.Source: Figure 7 · p. 9
• The paper analytically shows the difference between the training distribution and sampling distribution in diffusion models.Source: §6 · p. 9
• Epsilon Scaling refines the sampling trajectory by explicitly scaling the prediction vector, without training.Source: §6 · p. 9
• Extensive experiments demonstrate Epsilon Scaling as a generic solution whose simplicity supports a wide range of diffusion applications.Source: §6 · p. 9

• The appendix presents the full derivation of Eq. 3 by substituting expressions for x_t+1 and the training sample x_t.Source: A APPENDIX · p. 13
• The derivation treats the noise variables as independent standard Gaussian samples and identifies the resulting distribution’s mean as √¯α_t x_0.Source: A APPENDIX · p. 13
• The remaining derivation focuses on the variance and simplifies its expression across two successive forms.Source: A APPENDIX · p. 13

• The derivation obtains qθ(ˆx_t−1|x_t+1) by iteratively plugging together the two consecutive DDPM sampling steps.Source: §A.2 · p. 14
• The resulting distribution has mean √¯α_t−1 x_0, after applying Taylor’s theorem and simplifying the preceding expression.Source: §A.2 · p. 14
• Iterative plugging also yields a distribution for ˆx_t given x_T with mean √¯α_t x_0 and variance larger than (1 −¯α_t)I.Source: §A.2 · p. 14

• DDIM shares DDPM’s loss and marginal training distribution, but its posterior is derived under a non-Markovian diffusion process.Source: §A.3 · p. 15
• The derivation models uncertainty in the network’s predicted x_0 as a Gaussian distribution, rather than assuming a prediction without error.Source: §A.3 · p. 15
• With σ_t = 0 for DDIM, the appendix substitutes the forward-diffusion expression and computes the sampling distribution’s mean and variance.Source: §A.3 · pp. 15–16
• The appendix compares the mean and variance of the DDIM sampling distribution qθ(ˆx_t|x_t+1) with the training distribution q(x_t|x_0).Source: §A.3 · p. 16, Table 8 · p. 16
• The DDIM sampling variance is larger than the training variance, indicating exposure bias in the sampler.Source: §A.3 · p. 16

A.4 PRACTICAL VARIANCE ERROR OF qθ(ˆxt|xt+1) AND qθ(ˆxt|xT)

• Single-step variance error is measured with access to the ground-truth input x_t at each sampling step, unlike multi-step measurement.The experiments use CIFAR-10 and ADM.Source: §A.4 · p. 16
• The procedure samples x_0 and noise, forms x_t, and records the residual after subtracting the ground-truth mean from the predicted previous sample.It estimates variance from 50,000 iterations and compares that estimate with the training variance to compute Δ_t.Source: §A.4 · p. 17

A.5 METRIC FOR EXPOSURE BIAS

• The procedure initializes from x_T, iteratively predicts ˆx_{t−1}, and records its residual from the ground-truth mean.It estimates variance over 50,000 iterations and defines Δ′_t as the estimated variance minus the training variance.Source: §A.5 · p. 17
• The residual ˆx_{t−1} − √ᾱ_{t−1}x_0 isolates the stochastic term used for the variance estimate.The section reports using N = 50,000 samples to compute the variance estimate.Source: §A.5 · p. 17

A.6 CORRELATION BETWEEN EXPOSURE BIAS METRIC AND FID

• The authors report a strong empirical correlation between δ_t and FID, and δ_t also tracks input quality at intermediate steps, which declines sharply near t = 1 as δ_t rises.Figure 8 presents the FID–δ_1 relationship on CIFAR-10 with 20-step sampling.Source: §A.6 · pp. 17–18, Figure 8 · p. 17

A.7 FID RESULTS ON PFGM++ BASELINE

• Epsilon Scaling yields significant FID improvements on PFGM++, while its effective λ(t) range is wider there than on the EDM baseline.Tables 9 and 10 report CIFAR-10 results for PFGM++ with D=128 and D=2048, respectively.Source: §A.7 · p. 18, Table 9 · p. 18, Table 10 · p. 18

A.8 FID RESULTS UNDER T ′ > 100

• For sampling steps T′ > 100, the section reports Epsilon Scaling results using λ(t) = b on Analytic-DDPM and ADM baselines.The cited tables cover CIFAR-10; their captions do not provide result values.Source: §A.8 · p. 18, Table 11 · p. 18, Table 12 · p. 18

A.9 RECALL AND PRECISION RESULTS

• Across five datasets, ADM-ES has higher recall and slightly lower precision than ADM, with minor overall differences.The comparison uses 100-step sampling.Source: §A.9 · p. 19, Table 13 · p. 19

A.10 EPSILON SCALING PARAMETERS: k, b; A.11 INSENSITIVITY OF λ(t)

• The suggested parameter procedure searches a uniform schedule coarsely to finely, then fits a linear schedule while preserving its total scaling; reported FID gains are insensitive across a wide λ(t) range.The optimal b generally decreases as the sampling step count increases; tables provide schedule parameters and FID comparisons.Source: §A.10 · p. 19, §A.11 · p. 19, Table 14 · p. 19

A.12 EPSILON SCALING ALLEVIATES EXPOSURE BIAS

• On CIFAR-10 and FFHQ 128×128, Epsilon Scaling consistently yields smaller δ_t and moves the sampling trajectory toward the training vector field.The replicated experiments use ADM with 20-step sampling; Figures 9 and 10 show a smaller δ_t at t = 1 and closer sampling and training ∥ϵθ∥₂.Source: §A.12 · p. 21, Figure 9 · p. 21, Figure 10 · p. 21

A.13 FID RESULTS ON DIT BASELINE

• Epsilon Scaling consistently improves DiT sample quality across sampling steps on ImageNet 256×256, supporting exposure bias in both UNet and Transformer backbones.The reported comparison is between DiT and DiT-ES under uniform λ(t) = b.Source: §A.13 · p. 21, Table 19 · p. 22

A.14 QUALITATIVE COMPARISON

• Qualitative comparisons report improved sample quality across illumination, colour, object coherence, and background details for ADM-ES and LDM-ES.The comparisons cover CIFAR-10, LSUN tower, ImageNet, and CelebA-HQ using 100-step sampling.Source: §A.14 · p. 22, Figure 11 · p. 22, Figure 12 · p. 22