Input Perturbation Reduces Exposure Bias in Diffusion Models

1. Introduction

• The paper frames DDPMs as a high-quality generative approach whose sampling proceeds through a sequence of denoising steps from noise.Unlike methods that synthesize a sample in one step, DDPMs use a long generation chain.Source: §1 · p. 1
• DDPM inference can accumulate prediction errors because training uses ground-truth inputs while inference uses previously generated samples.The paper relates this training–inference mismatch to exposure bias in autoregressive generation.Source: §1 · p. 1
• Input perturbation adds noise to training inputs to simulate inference-time prediction errors and smooth the prediction function.The perturbed input is used while keeping the ground-truth prediction target fixed.Source: §1 · p. 2
• The paper also proposes regularizers that encourage the prediction function to be Lipschitz continuous, making predictions vary less between neighboring inputs.These alternatives aim to improve robustness to inference-time errors.Source: §1 · p. 2
• On FFHQ 128×128, DDPM-IP using only 80 sampling steps generates lower FID scores than state-of-the-art ADM using 1,000 steps, corresponding to more than 12.5× sampling acceleration.Source: §1 · p. 2
• Across common benchmarks, DDPM-IP improves generation quality and reduces both training and inference time.The authors report that it is more efficient and effective than explicit minimization of the Lipschitz constant; they also describe it as requiring no additional training overhead and being straightforward to add to existing DDPMs.Source: §1 · p. 2

2. Related Work

• The paper investigates DDPM training–testing discrepancy rather than proposing an application-specific prediction network, with a solution intended to work across applications and denoising architectures.The cited applications include image, text-to-image, audio, categorical, and time-series generation.Source: §2 · p. 2
• Prior acceleration approaches include DDIM sampling with fewer inference steps, distillation into progressively faster networks, and applying DDPMs in a compressed latent space.The distillation approach requires training multiple networks.Source: §2 · pp. 2–3
• Other approaches accelerate generation through partially parallelized sampling, continuous diffusion steps, non-Gaussian reverse processes, or higher-order deterministic sampling.The paper says its approach is orthogonal to previous work and can potentially be used jointly with most of it.Source: §2 · p. 3

3. Background

• A DDPM forward process starts from a real image and iteratively adds Gaussian noise over T diffusion steps until the image is completely noisy.The setup assumes image pixels are scaled to [−1, 1] and uses a prefixed noise schedule.Source: §3 · p. 3
• Given x0, the forward process obtains xt using the noise schedule and a Gaussian noise vector.The network predicts the noise vector rather than the mean of the forward-process posterior.Source: §3 · p. 3
• Standard training samples x0, t, and noise, computes xt, and updates the network using an L2 noise-prediction loss.In the stated objective, xt and the noise are ground-truth terms, while the network prediction is ϵθ(xt, t).Source: §3 · p. 3
• Standard sampling starts from Gaussian noise and repeatedly uses the network prediction to compute the previous sample, returning x̂0 after the reverse steps.The sampling algorithm iterates from T down to 1.Source: §3 · p. 3

• Training conditions the prediction network on ground-truth samples, while inference conditions it on samples generated at the previous step, creating a training–inference discrepancy.The paper relates this discrepancy to exposure bias in autoregressive text generation.Source: §4 · p. 3
• Longer reverse processes produce higher FID scores, indicating greater accumulated error as the number of reverse steps increases.The experiment used ADM trained with 1,000 steps on ImageNet 32×32 and measured FID using 50k samples.Source: §4 · p. 4, Table 1 · p. 4
• Most of ADM’s best FID results occur with 100–300 sampling steps, which the authors attribute to a trade-off between Gaussian-step approximation and accumulated prediction errors.The models were trained with 1,000 diffusion steps.Source: §4 · p. 4

5.1. Regularization with Input Perturbation

• DDPM-IP perturbs the training input but keeps the original noise vector as the prediction target, simulating inference-time errors without changing the target.The added noise is applied asymmetrically to the input rather than being used in the ground-truth target.Source: §5.1 · p. 4, §5.2 · p. 4

5.2. Discussion

• DDPM-IP and DDPM-y use the same input distribution but different prediction targets, and DDPM-y performs worse than standard DDPM.DDPM-y changes the noise distribution in the standard training protocol to match the distribution of DDPM-IP inputs.Source: §5.2 · pp. 4–5

5.3. Estimating the Prediction Error

• Prediction errors are empirically modeled as Gaussian, and the method uses constant γ = 0.1 rather than estimating a step-dependent schedule through two-stage training.The curves for CIFAR10 and ImageNet 32×32 are close; the authors choose a constant value for simplicity, despite noting that a DDPM-specific value could improve quality.Source: §5.3 · p. 5, Figure 2 · p. 5

5.4. Regularization based on Lipschitz Continuous Functions

• The alternative Lipschitz approach regularizes the prediction function through gradient penalty or weight decay, with gradient penalty requiring one forward and two backward passes per training step.Both alternatives use the original training algorithm and add a regularization term to the L2 loss.Source: §5.4 · pp. 5–6

6.2. Main results

• The main comparison evaluates unconditional image generation across five datasets, using FID and sFID with γ fixed at 0.1.Models use T=1,000 training steps; tests with fewer steps use respacing.Source: §6 · p. 6, §6.2 · p. 6

6.1. Evaluation of the Different Proposed Solutions

• On CIFAR10, all three tested regularizers outperform the ADM baseline in image quality, while ADM-IP obtains the best FID and sFID.ADM-GP is too slow to scale to larger datasets, so the authors do not recommend it.Source: §6.1 · p. 6, Table 2 · p. 6
• ADM-IP produces significantly lower exposure bias than ADM, with a larger difference for longer sampling sequences.Source: §6.1 · p. 6

6.2. Main results

• ADM-IP beats ADM on FID and sFID across datasets and T′≤T; CelebA 64×64 reaches 1.27 FID, a new state of the art for unconditional generation.The comparison uses models trained with T=1,000 steps; the reported CelebA result used T′=900.Source: §6.2 · pp. 6–7, Table 3 · p. 7
• ADM-IP converges faster during training on all five datasets and, versus ADM’s 1,000-step baseline, achieves better or equivalent FID with 60 or 80 steps except CelebA, which requires 200, yielding 5x–16.7x inference speed-up.On FFHQ, ADM reaches FID 14.52 at convergence after 420K iterations, while ADM-IP reaches FID 8.81 after 60K iterations, a 5.71-point improvement and 7x training speed-up.Source: §6.2 · p. 7
• ADM and ADM-IP have no significant difference in recall or precision, indicating no detected effect on sample diversity.Source: §6.2 · p. 7
• DDIM-IP consistently improves FID over DDIM across sampling-step counts and both η settings, with larger gains at fewer steps.With η=0.5, the FID gain is 7.16 at 10 steps versus 0.89 at 1,000 steps; with η=0, the 10-step gain is 3.67.Source: §6.2 · pp. 7–8

7. Conclusions

• DDPM-IP improves image quality and reduces training and inference time through input perturbation that models prediction errors.The method does not require changes to the network architecture or loss function.Source: §7 · p. 8
• ADM and ADM-IP take the same time per training iteration and per sampling step, according to the Table 4 caption.The caption identifies the table as a comparison of training and testing acceleration but does not provide the acceleration values.Source: Table 4 · p. 9
• The CIFAR10 comparison evaluates DDIM and DDIM-IP models trained with 1,000 steps and tested with at most 1,000 steps.The supplied caption gives the comparison scope but not the results.Source: Table 5 · p. 9
• DDPM-IP was tested only on images, although the authors state that their method has no domain-specific assumptions and presume it may apply more broadly.The broader applicability is an expectation, not a reported evaluation.Source: §7 · p. 9
• Experiments used small-resolution images; evaluation on larger images and corresponding larger backbones is left for future work.FFHQ 128×128 was the largest resolution tested, and the authors note their best results were obtained there.Source: §7 · p. 9

A.1. Exposure Bias Analysis

• The measurement runs deterministic reverse diffusion from x_t to x̂_0 using ADM trained with T = 1,000 on ImageNet 32×32.It removes the stochastic term and measures average pixel-wise L1 distance from x_0; with pixels normalized to [−1, 1], this distance is at most 2.Source: §A.1 · p. 12
• For each sampled x_0 and t, the procedure computes x_t, reverses the trajectory, and accumulates the normalized L1 distance into the estimate for t.The estimate is averaged over the number of measurements recorded for each t.Source: §A.1 · p. 12
• The deterministic estimate shows that error accumulates greatly as the number of reverse diffusion steps increases.The accompanying figure visualizes image pairs at different trajectory lengths; the table caption notes the estimate is upper-bounded by 2.Source: §A.1 · p. 12, Table 6 · p. 12

A.2. Distribution of the Perturbed Input

• The derivation uses the fact that a linear combination of independent Gaussian variables is Gaussian distributed.The passage states this general result before applying it to y_t.Source: §A.2 · p. 12
• For a given x_0, y_t is a linear combination of x_t and ξ, which are independent Gaussian variables.The derivation substitutes y_t, x_t, and ξ into the linear-combination result to obtain its distribution.Source: §A.2 · pp. 12–13

A.3. Ablation Study: Input Perturbation is not Equivalent to Using a Different Noise Variance

• DDPM-y uses the standard DDPM algorithm with the noise distribution of y_t, while DDPM-IP is the comparison method.The section tests whether input perturbation is equivalent to changing the noise distribution.Source: §A.3 · p. 13
• DDPM-y performs worse than DDPM in the reported comparison.The passage identifies Table 7 as the source of this result.Source: §A.3 · p. 13
• The Table 7 comparison covers CIFAR10 and varies the number of reverse diffusion steps.The caption names DDPM, DDPM-y, and DDPM-IP as the compared methods.Source: Table 7 · p. 13

A.4. Recall and Precision

• ADM and ADM-IP showed no significant Recall or Precision difference; ADM-IP was slightly better on CIFAR10 32×32 and FFHQ 128×128, while ImageNet metrics were not reported due to memory limits.The comparison used 50,000 samples per dataset and model with 1,000 sampling steps.Source: §A.4 · p. 13, Table 8 · p. 14

A.5. Gaussian Prediction Error

• After standardizing pixelwise errors using all pixels from 10K images at 100 uniformly sampled timesteps (stride 10), Shapiro–Wilk rejected normality in only a small minority of cases, supporting almost-isotropic Gaussian errors.The confidence level was 95%, and the null hypothesis was rejected when p < 0.05.Source: §A.5 · p. 14

A.6. Relation between the Lipschitz Constant Minimization and the Weight Decay Minimization

• For a single-layer ReLU network, the Frobenius norm approximates the spectral-norm Lipschitz constant, so minimizing its squared norm corresponds to weight decay.The derivation uses the bound on the ReLU derivative and a first-order Taylor approximation.Source: §A.6 · pp. 14–15

A.7. Hyperparameters

• The ADM and ADM-IP experiments largely followed Dhariwal and Nichol’s settings, using AdamW, mixed precision with 32-bit weights and optimizer state, and an EMA rate of 0.9999.Training used NVIDIA V100 GPUs; DDIM and DDIM-IP were trained on CIFAR10 for 1,600K iterations with batch size 128.Source: §A.7 · p. 16, Table 9 · p. 16, Table 10 · p. 16

A.8. Qualitative Comparison between ADM and ADM-IP

• FID was 2.76 versus ADM’s 2.99 on CIFAR10 and 2.72 versus 3.53 on ImageNet 32×32; qualitative comparisons also found ADM-IP samples usually comparable or better.ADM-IP additionally avoided the overexposure and loss of background detail observed in ADM’s FFHQ 128×128 samples.Source: §A.8 · pp. 16–17

A.9. Additional Qualitative Results for ADM-IP

• Additional ADM-IP samples were shown for CIFAR10, ImageNet 32×32, LSUN tower 64×64, CelebA 64×64, and FFHQ 128×128, using each dataset’s best sampling-step count.FID was 2.68 versus ADM’s 3.39 on LSUN tower 64×64, 1.31 versus 1.60 on CelebA 64×64, and 2.98 for ADM-IP on FFHQ 128×128.Source: §A.9 · pp. 17–19