Source-linked AI summary
Spectral Prior for Reducing Exposure Bias in Diffusion Models
Yuya Kobayashi, Masato Ishii, Yuhta Takida, Takashi Shibuya, Yuki Mitsufuji
TL;DR
Diffusion sampling can accumulate prediction errors, with training–inference discrepancies appearing as frequency-dependent spectral or effective-SNR mismatch. Spectral Alignment (SPA) calibrates intermediate prediction spectra using offline fitting and inference-time FFT guidance, improving image quality across diverse models with 3–4% overhead.
Problem
Diffusion sampling accumulates prediction errors, while spectral mismatch between training and inference can vary with frequency, model, and timestep.
Method
SPA fits a parametric target power spectrum offline, then steers each denoising step toward it using efficient FFT-based gradient correction.
Results
3–4% computational overhead accompanies improved generation quality across pixel-space, latent diffusion, and flow-matching models.
Takeaways & Limitations
SPA is a lightweight inference-time add-on that alleviates spectral mismatch and refines image quality across varied pretrained model architectures.
Takeaways & Limitations
The method assumes a single target spectrum, although optimal priors may vary across target data distributions and conditioning information.
Abstract
from arXiv · showhide
Diffusion models typically suffer from error accumulation during iterative sampling, commonly referred to as exposure bias. We reveal systematic frequency-dependent discrepancies between training and inference, which can be interpreted as frequency-dependent SNR error. Crucially, the direction of this mismatch varies across models and timesteps, indicating that fixed correction rules do not generalize. We propose Spectral Alignment (SPA), a lightweight, guidance-based method that calibrates the power spectrum of intermediate predictions to a pre-computed prior. Our approach consists of two stages: (1) offline fitting of a parametric spectrum model from training data, and (2) inference-time guidance via efficient FFT-based gradient computation. SPA introduces minimal computational overhead (3-4\%) and is complementary to Classifier-Free Guidance (CFG). We demonstrate consistent improvements across diverse architectures, from pixel-space models (DDPM, ADM) to latent diffusion models (SD2.0, SDXL) and flow-matching models (SD3.5, FLUX). Our implementation is available at https://github.com/SonyResearch/SPA.
1 Introduction
The paper identifies model- and timestep-dependent spectral mismatch during diffusion inference and proposes Spectral Alignment (SPA), a lightweight guidance-based correction method. SPA fits a spectral prior offline, applies it during sampling, and improves generation quality across diverse architectures with 3–4% overhead.
- Spectral mismatch appears as frequency-dependent effective SNR errors whose direction varies across models and timesteps.
- SPA calibrates intermediate prediction spectra toward a precomputed timestep-dependent prior using two stages: offline fitting and inference-time guidance.The prior is fitted parametrically from training data, then used to steer samples during inference.
- The evaluation spans DDPM, ADM, latent diffusion models including SD2.0 and SDXL, and flow-matching models.
- 3–4% computational overhead accompanies SPA’s improved generation quality across tested diffusion architectures.
2 Motivation
The paper frames exposure bias as a training–inference mismatch whose spectral structure varies across frequencies, timesteps, and models. These observations motivate a data-driven spectral prior and guidance procedure rather than fixed or scalar corrections.
- The proposed method uses a precomputed spectral prior to correct mismatch during inference, including the latent variable z for latent diffusion models.
- Exposure bias is the mismatch between intermediate-variable distributions in the forward training process and reverse inference process.
- Frequency-dependent SNR is primarily determined by signal power because Gaussian noise has a flat power spectrum.
- Training–inference spectral mismatch is estimated by comparing power spectra of clean-image predictions from corresponding forward and reverse-process states.
- Mismatch patterns vary by model and timestep: ADM shows high-frequency attenuation, while Stable Diffusion 2.0 shows low-frequency attenuation.
- Fixed frequency boosts and scalar variance corrections cannot generalize across architectures, motivating a model-specific data-driven target spectrum.
3 Method
SPA characterizes intermediate prediction spectra with a timestep-dependent parametric prior, using radially averaged power spectra to model frequency behavior independently by channel.
- 3 Method: SPA fits a timestep-dependent parametric spectrum model from training data and a pretrained diffusion model as an offline, per-model process.The fitted prior is later used to guide inference toward expected spectral characteristics.
- 3 Method: For each channel, SPA computes the 2D power spectrum of the intermediate prediction and reduces it to a one-dimensional radially averaged power spectrum.The normalized spatial frequency ranges from 0 to 0.5, with 0.5 representing the Nyquist frequency.
- 3 Method: The target spectrum combines a timestep-dependent power-law decay with bias and optional linear terms to capture deviations from a pure power law.The linear term is enabled only when it significantly improves the fit, such as for SDXL and FLUX.
- 3 Method: RAPS is used instead of a direct 2D representation because its one-dimensional form simplifies visualization, regression, and fitting stability.This design choice supports easier analysis, explanation, and hyperparameter tuning.
- 3 Method: Reference spectra are collected at selected timesteps, averaged across samples, and fitted independently for each channel and timestep.The method then obtains spectra across all inference timesteps through cubic-spline interpolation of the fitted parameters.
1. Sample clean images x(i)
SPA guides each denoising step by comparing the intermediate prediction spectrum with its target prior and applying an FFT-based correction compatible with existing sampling pipelines.
- 1. Sample clean images x(i): SPA obtains reference spectra by forwarding training images, predicting intermediate clean images, extracting RAPS, averaging samples, and fitting the parametric model.Least-squares regression estimates the spectrum parameters independently for each channel and selected timestep.
- 3.3 Guidance-based Spectral Mismatch Calibration: At each reverse step, SPA computes the intermediate prediction with Tweedie’s formula, extracts its RAPS, and compares it with the target spectrum.The guidance loss uses an unweighted mean-squared error in log-spectrum space over frequency bins and channels.
- 3.3 Guidance-based Spectral Mismatch Calibration: SPA applies an asymmetric penalty that more strongly penalizes spectra falling below the target than spectra exceeding it.This choice follows the observed positive skewness of forward-process log spectra and the finding that underestimation is less desirable.
- 3.3 Guidance-based Spectral Mismatch Calibration: The inference procedure applies spectral alignment after CFG combination and before the standard denoising update.Because it operates on the combined prediction, SPA is compatible with most CFG variants and other noise-prediction modifications.
- 3.3 Guidance-based Spectral Mismatch Calibration: The method extends to latent diffusion models because their latent variables preserve spatial information and can be fitted by the same parametric power-law model.The paper reports that the power spectrum of z_t is well fitted in practice.
- 3.3 Guidance-based Spectral Mismatch Calibration: FFT-based spectrum extraction and gradient correction avoid additional neural-network evaluations, keeping the added computation minimal.Spectrum extraction costs O(D log D), radial averaging costs O(D), and backpropagation uses an inverse FFT with element-wise operations.
4 Experiments
SPA is evaluated across pixel-space, latent diffusion, and flow-matching models using distributional, reward-model, and alignment metrics. It generally improves generation quality, including difficult samples, while adding limited inference cost.
- 4.1 Experimental Setup: Experiments cover DDPM, ADM, SD2.0, SDXL, SD3.5, and FLUX.1 [dev] under model-specific sampling configurations.The study uses DDPM or DDIM samplers with 30–100 denoising steps, depending on the model.
- 4.2 Quantitative Results: SPA consistently outperforms comparison methods on ADM, improving both distributional quality and Density/Coverage.The improved Density/Coverage results indicate benefits for generation diversity, while time-shift sampling performs poorly in this setup.
- 4.2 Quantitative Results: DDPM also improves with SPA, although the gain is smaller and ϵ-rescaling performs slightly better.The authors attribute this result to inaccurate target spectra caused by weak single-step x̂0|t predictions.
- 4.2 Quantitative Results: SPA performs best among the text-to-image baselines, especially on SDXL, with an approximately 4.5% gain without degrading text alignment.HPSv3 and ImageReward are primary metrics, while CLIP Score indicates preserved alignment.
- 4.2 Quantitative Results: On FLUX.1 [dev], SPA significantly improves lower guidance scales, while its effect diminishes at w = 3.5.At w = 2.5, the win-rate is 53.3±1.5% with p = 2.3×10^-5.
- 4.2 Quantitative Results: SPA preferentially improves lower-quality samples, reaching a 54.1 ± 3.3% win-rate among the bottom 20% by HPSv3 at w = 3.5.The average gain for the bottom 5% is 0.70 at w = 2.5 and 0.19 at w = 3.5.
- 4.2 Computational Overhead: SPA adds only +3.86% overhead on SDXL and +0.08% on FLUX.1 [dev].The additional guidance steps average 0.0952 s on SDXL, and FFT cost scales favorably with latent spatial resolution.
- 4.3 Qualitative Results: Qualitatively, SPA repairs flawed object shapes and unnatural structures while preserving already decent content and refining edges, color, and detail.On SDXL, baseline methods can introduce frequency artifacts into the background; on FLUX.1 [dev], SPA selectively refines defective objects.
5 Related Work
Related work frames diffusion exposure bias as a training–inference distribution mismatch involving noise levels and accumulated prediction errors. Existing corrections include noise rescaling, schedule adjustment, frequency regulation, discriminator guidance, and autoencoder-based guidance.
- Exposure Bias in Diffusion Models: Exposure bias originated in autoregressive generation, where teacher forcing creates a train–inference mismatch that accumulates prediction errors.Diffusion models exhibit a structurally identical problem because reverse-process inputs differ from forward-process training inputs.
- Noise and SNR Corrections: Prior diffusion work interprets exposure bias through noise-level or SNR mismatch and proposes ϵ-rescaling to correct it.Subsequent approaches investigate complementary corrections to the same inference-time problem.
- Guidance-Based Corrections: CFG can reduce prediction errors but may also introduce oversaturation, unnatural contrast, and loss of fine details.This makes guidance itself a possible source of inference error.
- Guidance-Based Corrections: Discriminator Guidance refines scores with a trained discriminator but requires unstable adversarial training and extra inference computation at every denoising step.MPGD likewise incurs additional inference costs through autoencoder backpropagation, whereas combinations with SPA remain empirically unvalidated.
6 Conclusion
SPA alleviates spectral mismatch and refines image quality across diverse diffusion models with minimal computational overhead, while highlighting spectrum modeling and prior selection as ongoing considerations.
- SPA alleviates spectral mismatch and refines image quality across classic pixel-space, latent diffusion, and flow-matching models.The spectrum model fits latent diffusion predictions well and supports smooth interpolation across timesteps.
- The spectral-mismatch concept is proposed as a lens for future model-training improvements and inspection tools.
- A single target spectrum may be suboptimal because illustrations, medical images, and conditional inputs can have different frequency properties.The paper leaves alternative guidance methods and frequency- or timestep-dependent loss weighting for future work.
- The parametric spectrum model achieves R2 > 0.9 for all evaluated models, including latent diffusion models.
- Fitting smooths fluctuations in measured spectra, stabilizing the subsequent guidance phase.The regression used 10K samples before introducing the fitting step.
B Additional samples of Power Spectra During Inference
Additional analyses examine spectrum fitting, interpolation, visualization, and guidance-strength effects, while noting that spectral errors can differ across models.
- Additional spectra from Stable Diffusion 2.0 and 3.5 show model-dependent error behavior, with SD3.5 evaluated at closest rather than identical timesteps.
- SD2.0 fitting results visualize the fitted spectrum model, with only a single channel shown.
- The hyperparameter analysis varies guidance strength η and penalty intensity a as the two SPA hyperparameters.
- On DDPM, FID and KID are unimodal with respect to guidance strength η, simplifying hyperparameter selection.The analysis fixes a = 1, equivalent to ablating the penalty function, to separate η from a.
C.2 Analysis on the penalty intensity a
Penalty intensity a improves HPSv3 and visual quality on SDXL, but its effects depend on timestep, frequency band, and interaction with CFG scale.
- Figure A6 evaluates HPSv3 across penalty intensities a and CFG scales, including their reported weak interaction.
- Increasing penalty intensity a improves HPSv3 on SDXL, although the score gains gradually diminish as a increases.
- Skewness of a Spectrum: Measured spectra are often positively skewed with mean > median, especially for t > 300.
- Skewness of a Spectrum: Skewness varies across timesteps and frequency bands, motivating future timestep- and frequency-dependent penalty intensities.
- Larger a produces better object shapes, more pronounced colors, and greater sharpness and contrast in SDXL samples.
- On DDPM, Figure A5 examines the effect of η on FID and KID.
D Implementation Details of the Baseline Methods
Baseline methods were adapted to latent diffusion models with minimal changes, with each method applied at a specified stage of the inference pipeline.
- The three baseline methods were reimplemented for latent diffusion models with minimal modification.
- ϵ-rescaling is applied to the model prediction after classifier-free guidance, while time-shift sampling and wavelet regulation modify the latent before the sampler step.
E Additional Qualitative Results
Additional qualitative results show that SPA generally improves generated images while rarely causing negative changes. The effect also extends to dedistilled FLUX checkpoints and non-photorealistic artistic styles.
- The additional qualitative samples use four prompts from the MS COCO dataset across text-to-image models.
- SPA enhances the original images in most samples and rarely affects them negatively.
- The difference made by SPA tends to be larger with FLUX.1[dev]'s dedistilled checkpoint than with the official guidance-distilled version.
- SPA does not negatively affect sampling of artistic styles despite guiding samples toward a single target spectrum.