PaperlayerView exact revision on arXiv

Source-linked AI summary

Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps

Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, Marie-Francine Moens

arXiv:2305.15583v8cs.CV

TL;DR

Diffusion-model inference uses model-predicted states, unlike training on corrupted ground-truth samples, so prediction errors can accumulate. The paper proposes Time-Shift Sampler, which adjusts inference timesteps without retraining. It reports consistent improvements across sampling methods, including a 44.49% FID improvement for F-PNDM on CIFAR-10 at 10 steps.

  • Problem

    Training uses corrupted ground-truth samples, while inference generates from white noise without access to ground-truth distributions, creating a discrepancy associated with accumulating prediction errors.

    Source: §1 · p. 2

  • Method

    Time-Shift Sampler uses the generated samples' approximated variance to choose a better-coupled next timestep during inference, without retraining the model.

    Source: §1 · p. 2

  • Results

    A 44.49% improvement yields FID 3.88 for F-PNDM on CIFAR-10 with 10 sampling steps, alongside consistent improvements across sampling methods and benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The sampler can be integrated into DDPM, DDIM, and high-order solver methods with minimal additional computation.

    Source: §1 · p. 2, §3 · p. 6

    The method relies on the assumption that an alternate timestep may couple better with the network-predicted next state than the original next timestep.

    Source: §3 · p. 5

Abstract

from arXiv · show

Diffusion Probabilistic Models (DPM) have shown remarkable efficacy in the synthesis of high-quality images. However, their inference process characteristically requires numerous, potentially hundreds, of iterative steps, which could exaggerate the problem of exposure bias due to the training and inference discrepancy. Previous work has attempted to mitigate this issue by perturbing inputs during training, which consequently mandates the retraining of the DPM. In this work, we conduct a systematic study of exposure bias in DPM and, intriguingly, we find that the exposure bias could be alleviated with a novel sampling method that we propose, without retraining the model. We empirically and theoretically show that, during inference, for each backward time step $t$ and corresponding state $\hat{x}_t$, there might exist another time step $t_s$ which exhibits superior coupling with $\hat{x}_t$. Based on this finding, we introduce a sampling method named Time-Shift Sampler. Our framework can be seamlessly integrated to existing sampling algorithms, such as DDPM, DDIM and other high-order solvers, inducing merely minimal additional computations. Experimental results show our method brings significant and consistent improvements in FID scores on different datasets and sampling methods. For example, integrating Time-Shift Sampler to F-PNDM yields a FID=3.88, achieving 44.49\% improvements as compared to F-PNDM, on CIFAR-10 with 10 sampling steps, which is more performant than the vanilla DDIM with 100 sampling steps. Our code is available at https://github.com/Mingxiao-Li/TS-DPM.

1 INTRODUCTION

Exposure bias arises because inference feeds model-predicted states where training uses corrupted ground-truth samples, allowing prediction errors to accumulate. The paper proposes Time-Shift Sampler, which selects better-coupled timesteps during inference without retraining and improves sampling results.

1 INTRODUCTION

  • Training uses corrupted ground-truth samples, whereas inference starts from white noise without access to ground-truth distributions, so prediction errors can accumulate across sampling steps.The paper frames this training–inference discrepancy as exposure bias, analogous to that in autoregressive generative models.Source: §1 · p. 2
  • Time-Shift Sampler adjusts the next timestep according to the current generated samples' approximated variance, searching within a surrounding window and retaining original steps below a cutoff.The method is designed to alleviate exposure bias without retraining.Source: §1 · p. 2
  • The paper theoretically and empirically studies exposure bias in diffusion models and introduces a sampler that integrates into existing methods with minimal additional computation.The introduction notes that prior perturbation-based mitigation requires retraining the diffusion model.Source: §1 · p. 2
  • A 44.49% improvement yields FID 3.88 for F-PNDM on CIFAR-10 with 10 sampling steps.The authors also report consistent, significant improvements over various sampling methods on image-generation benchmarks.Source: §1 · p. 2

2 INVESTIGATING EXPOSURE BIAS IN DIFFUSION PROBABILISTIC MODELS

This section introduces the forward and backward processes of diffusion models, then examines exposure bias through sample-variance distributions and prediction errors during sampling.

2.1 BACKGROUND: DIFFUSION PROBABILISTIC MODELS

  • The forward process corrupts data with Gaussian noise under a Markov transition, while training a time-dependent network to predict noise supports the backward denoising process.The background presents DDPM training and sampling procedures.Source: §2.1 · pp. 2–3

2.2 THE EXPOSURE BIAS IN DIFFUSION PROBABILISTIC MODELS

  • Across forward timesteps, the variance distribution narrows toward a range around white-noise variance, making later-stage samples more sensitive to minor prediction errors.Figure 2 examines variance distributions; the text identifies the narrowing pattern after 400 steps.Source: §2.2 · pp. 3–4, Figure 2 · p. 3
  • Prediction errors initially decrease and then accumulate as sampling progresses, and the experiments attribute the late increase to exposure bias and network errors.The section concludes that accumulated prediction error can strongly affect final results.Source: §2.2 · p. 4

3 ALLEVIATING EXPOSURE BIAS VIA TIME STEP SHIFTING

The paper frames exposure bias as a mismatch between training inputs and predicted states encountered during inference. It proposes choosing a nearby time step that couples better with the predicted next state, based on empirical evidence and a KL-divergence analysis.

3 ALLEVIATING EXPOSURE BIAS VIA TIME STEP SHIFTING

  • Exposure bias reflects a discrepancy between the network input and ground-truth state; better coupling reduces that discrepancy.The paper calls C(˜xt, t) the input couple and defines it using the distance between network input ˜xt and ground-truth state xt.Source: §3 · p. 4
  • The paper assumes a predicted next state may align better with an alternate nearby time step than with t −1.Theorem 3.1 seeks the time step by optimizing the KL divergence between the predicted state and xts, assuming t −1 is sufficiently large and ts is nearby.Source: §3 · p. 5
  • For some backward steps, nearby time steps couple better with the predicted next state than the nominal step, while nearby couplings converge near zero.Figure 4 examines DDIM with 10 steps on CIFAR-10; similar patterns were observed across other pretrained models, datasets, step counts, and time-step ranges.Source: §3 · p. 5, Figure 4 · p. 5
  • Theorem 3.1’s derivation optimizes the KL divergence and relates within-sample variance to predicted-state variance; under its stated conditions, σts ≈ σt−1.The simplification applies when t is large and current network prediction error is minimal.Source: §3 · pp. 5–6
  • Time-Shift Sampler replaces the next step with the nearby step whose variance best matches var(xt−1), then returns to conventional sampling below cutoff tc.It can be incorporated into DDIM, DDPM, and high-order solver samplers; the shift is omitted after the cutoff.Source: §3 · p. 6

4 EXPERIMENTAL SETUP

The experiments apply Time-Shift Sampler to several sampling algorithms and evaluate image quality across datasets and sampling-step counts. The setup also varies time-step selection, window size, and cutoff.

4 EXPERIMENTAL SETUP

  • Algorithm 3 searches a window of size w for the time-step variance closest to var(xt−1), applies the shift above cutoff tc, and otherwise uses conventional sampling.The algorithm takes a trained diffusion model, window size, reverse time series, and cutoff as inputs.Source: §4 · p. 6
  • Time-Shift variants are evaluated with DDPM, DDIM, S-PNDM, and F-PNDM, using uniform or quadratic time-step selection for DDIM.S-PNDM uses a second-order ODE solver and F-PNDM a fourth-order solver.Source: §4 · p. 6
  • Experiments use pretrained DDPM on CIFAR-10 and CelebA 64×64, with FID evaluating generated-image quality across 5, 10, 20, 50, and 100 sampling steps.The setup also compares DDIM-based sampling with ADM-IP using ADM as backbone.Source: §4 · p. 6
  • The experiments examine the influence of window size and cutoff values in Section 5.4.Additional experimental details are in Appendix C.Source: §4 · p. 6

5 RESULTS

Time-Shift sampling improves image-generation quality across samplers and datasets, with especially strong gains at low step counts. Its added sampling time is small in many settings, while performance depends on the window size and cutoff value.

5.1 MAIN RESULTS

  • FID=3.88 with TS-F-PNDM on CIFAR-10 at 10 steps, better than DDIM at 100 steps.The result is reported for uniform time-step selection; Time-Shift variants also improve baselines under both uniform and quadratic selection.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 DISCUSSION ON EFFICIENCY AND PERFORMANCE

  • Added computation is negligible at 5, 10, and 20 steps, while TS-S-PNDM takes 9.86% more sampling time than S-PNDM at 50 steps.For 5, 10, and 20 steps, the reported increase is 9.71%.Source: Figure 5 · p. 7, §5.2 · pp. 7–8

5.3 COMPARISON WITH TRAINING-REQUIRED METHOD ADM-IP

  • With ADM, Time-Shift improves performance by a large margin without retraining, at roughly the same sampling time as DDIM.At small step counts, ADM-IP performs much worse than original ADM; its performance improves as the step count rises.Source: Table 2 · p. 8, §5.3 · p. 8, Figure 6 · p. 8

5.4 INFLUENCE OF WINDOW SIZES AND CUTOFF VALUES

  • Window size has little effect at 10, 20, and 50 steps, but matters more at 100 steps.The authors attribute the increased sensitivity to more accurate per-step predictions with smaller step sizes.Source: Figure 7 · p. 8, §5.4 · p. 9
  • For 10 steps, a cutoff of 200 is preferred to the higher cutoff values considered.The authors suggest that larger step sizes at fewer sampling steps make more time-shift operations beneficial.Source: Figure 7 · p. 8, §5.4 · p. 9

6 RELATED WORK

Related work develops diffusion models and applies them across generation tasks, while also seeking faster sampling. This paper follows that acceleration line by proposing a method for selecting time steps.

Denoising Diffusion Probabilistic Model

  • DDPM variants include learned variance, stochastic differential equation formulations, variational diffusion, consistency, and rectified-flow models.The cited variants include methods that learn the forward process or target one-step generation.Source: §6 · p. 9
  • Diffusion models are used for controllable image, language, super-resolution, and video generation.Latent-space diffusion is also described as reducing computational time and training cost.Source: §6 · p. 9

DDPM Accelerator

  • Acceleration work includes DDIM, variance optimization, and numerical solvers, with some approaches reducing generation to 10 steps while maintaining image quality.The paper positions its time-step selection method within this line of research.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents Time-Shifted Sampler as a training-free way to mitigate exposure bias by selecting a better-aligned next timestep. The paper also identifies its two tuning parameters as an open limitation and describes appendix procedures for analyzing sample variance and prediction error.

7 CONCLUSION

  • Time-Shifted Sampler uses the timestep that most aligns with the subsequent predicted step and can be integrated with DDPM, DDIM, S-PNDM, and F-PNDM.The conclusion says the optimal next-timestep variance is theoretically derived and reports extensive experiments as evidence of effectiveness.Source: §7 · p. 9

A LIMITATION AND FUTURE WORK

  • Window size and cutoff value require tuning, and deriving their optimal values analytically is left for future work.The authors estimate the parameters from training-data statistics and note their relation to each step's noise level.Source: A LIMITATION AND FUTURE WORK · p. 13

B FIGURE DETAILS

  • For Figure 2, each image is flattened before its per-sample variance is computed to form a density plot.The example image has size 3 × 32 × 32 and is flattened to 3072 values.Source: B FIGURE DETAILS · p. 13
  • The Figure 3 analysis finds that paired-step predictions recover the original image only for t < ts, around 650 steps in the DDIM experiment.The account divides generation into an early stage without clear modes and a later stage with identifiable patterns, and uses this to divide error computation.Source: B FIGURE DETAILS · p. 13
  • Figure 4 compares predicted samples with ground-truth xt and xts using mean squared error.Source: B FIGURE DETAILS · p. 13

C ADDITIONAL EXPERIMENTAL SETUP

The setup estimates a generated state's variance from a single sample and evaluates sampling methods using shared pretrained checkpoints. It also identifies Figure 8 as the detailed MSE-computation procedure.

C ADDITIONAL EXPERIMENTAL SETUP

  • Under an assumption, the variance of xt−1 is estimated from the within-sample variance of a single xt−1 rather than many generated samples.This avoids the additional workload of generating many samples to estimate the variance.Source: C ADDITIONAL EXPERIMENTAL SETUP · p. 13
  • Evaluations use pretrained checkpoints, with DDPM architecture for sampler comparisons and ADM architecture for comparisons involving ADM-IP.The study uses DDPM checkpoints for CIFAR-10 and LSUN-bedroom, and DDIM checkpoints for CelebA.Source: C ADDITIONAL EXPERIMENTAL SETUP · p. 13
  • Figure 8 presents the detailed procedure for computing MSE.Source: Figure 8 · p. 14

D ERROR ANALYSIS

CelebA prediction errors are shown across different numbers of sampling steps, with patterns similar to those reported for CIFAR-10.

D ERROR ANALYSIS

E TRAINING-INFERENCE DISCREPANCY

The discrepancy analysis finds that a surrounding time-step window can contain a step that couples better with the predicted next state, though nearby steps’ coupling becomes identical as sampling progresses. Case studies show time shifting across the trajectory and report good-quality images with fewer than 10 sampling steps.

E TRAINING-INFERENCE DISCREPANCY

  • For a backward step t, some time step in [t −w, t +w] may couple better with the predicted next state, while surrounding steps’ coupling becomes identical as sampling progresses.The discrepancy pattern is reported across varying step counts and datasets.Source: E TRAINING-INFERENCE DISCREPANCY · p. 14

F SAMPLING TIME COMPARISON

G CASE STUDY

  • TS-DDIM shifts most time steps before cutoff 200, and the case study reports better generation quality than DDIM for a CIFAR-10 horse image.The trajectory shows shifts across both larger and smaller time steps, especially between 300 and 600, with most shifts in the intermediate range.Source: Figure 11 · p. 15, G CASE STUDY · p. 16, Figure 12 · p. 16

H QUALITATIVE EXAMPLES

  • The qualitative examples show good-quality images with fewer than 10 sampling steps across the reported methods and datasets.Figures provide examples for TS-DDPM and TS-DDIM on CIFAR-10, CelebA, and LSUN-bedroom.Source: H QUALITATIVE EXAMPLES · p. 16

I ADDITIONAL RESULTS

Additional evaluations cover datasets, sampler families, and precision-recall behavior. The authors report improved DPM-solver and DEIS performance, higher recall while maintaining DDIM-level precision, and note that LSUN-bedroom parameters were not properly tuned because of computational limits.

I ADDITIONAL RESULTS

  • LSUN-bedroom results are reported, but window size and cutoff values were not properly tuned because of computational-resource limits.The authors leave efficient parameter tuning for high-resolution images to future work.Source: I ADDITIONAL RESULTS · p. 16, Table 5 · p. 19
  • The paper compares TS-DDIM with Analytic-DPM on CIFAR-10 and CelebA, with the Analytic-DPM results taken from its paper.The supplied table caption identifies the datasets and resolutions but provides no outcome values.Source: I ADDITIONAL RESULTS · p. 16, Table 6 · p. 19
  • Additional sample figures show varying-time-step outputs for TS-DDPM and TS-DDIM on CIFAR-10, CelebA, and LSUN-bedroom.The figure captions identify examples but do not state comparative image-quality outcomes.Source: Figure 13 · p. 17, Figure 14 · p. 17, Figure 15 · p. 17
  • On CIFAR-10 with ADM, TS-DDIM tends to achieve much higher recall while maintaining DDIM’s precision level.Table 7 reports the precision-and-recall comparison.Source: I ADDITIONAL RESULTS · p. 19, Table 7 · p. 19
  • The paper evaluates DDIM against its method on ImageNet with classifier guidance and on MSCOCO val2017 for text-to-image generation.The supplied captions identify Tables 8 and 10 as performance comparisons but provide no outcome values.Source: I ADDITIONAL RESULTS · p. 19, Table 8 · p. 19, Table 10 · p. 20
  • Time shifting improves both DPM-solver and DEIS performance, although the gains are smaller than for DDIM; the authors attribute this to the solvers’ semi-linear ODE structure reducing estimation error.The comparison is on CIFAR-10.Source: I ADDITIONAL RESULTS · p. 19, Table 9 · p. 19

J.1 DERIVATION OF THEOREM 3.1

The derivation selects an optimal shifted time step by minimizing KL divergence between the predicted next-state distribution and a candidate-time distribution. It then estimates a window bound under stated assumptions, connecting those assumptions to the algorithm’s cutoff behavior.

J.1 DERIVATION OF THEOREM 3.1

  • The derivation defines the optimal time step ts as the one minimizing KL divergence between the predicted next-state distribution and the candidate-time distribution.The candidate distribution is q(xts|x0), while the predicted distribution is p( x̂t−1|x̂t).Source: §J.1 · p. 20
  • The derivation assumes sample variance is approximately its expected value because image dimensionality is usually large.For large t, a further approximation underlies the derivation; the paper links it to stopping shifts when t is small.Source: §J.1 · p. 21

J.2 ANALYTICAL ESTIMATION OF WINDOW SIZE

  • The window-size estimate relaxes the assumption that ts is close to t−1, requiring the signal difference term to be sufficiently smaller than the scaled network prediction error.The paper uses this condition to bound the window around t−1.Source: §J.2 · p. 22
  • The window bounds can be estimated from network prediction error and the predefined monotonic noise schedule, using its inverse to map schedule values to time steps.The prediction error may be estimated from a small amount of data or an error-prediction network.Source: §J.2 · p. 22

Related papers

Loading 2305.15583v8…