PaperlayerView exact revision on arXiv

Source-linked AI summary

A Fourier Space Perspective on Diffusion Models

Fabian Falck, Teodora Pandeva, Kiarash Zahirnia, Rachel Lawrence, Richard Turner, Edward Meeds, Javier Zazo, Sushrut Karmalkar

arXiv:2505.11278v1stat.MLcs.CVcs.LGstat.ME

TL;DR

The paper examines how DDPM’s frequency-dependent noising shapes the learned reverse process, and develops an alternate Fourier-space forward process. EqualSNR performs on par with DDPM on standard imaging benchmarks while improving high-frequency generation and performance where high frequencies are dominant.

  • Problem

    The paper asks how DDPM’s faster, earlier corruption of high frequencies affects reverse generation and whether the usual frequency hierarchy is essential.

    Source: §1 · p. 2

  • Method

    The paper analyzes frequency-specific SNR and studies a variance-preserving EqualSNR forward process that noises all frequencies at the same rate.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    EqualSNR performs on par with DDPM on standard imaging benchmarks and improves high-frequency generation and performance on datasets where high frequencies dominate.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results are relevant to generative tasks where high-frequency details are a central modelling objective, including astronomy and medical imaging.

    Source: §1 · p. 3

    The analysis notes that practical diffusion models face approximation errors under constraints such as limited intermediate steps and neural-network expressiveness.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Diffusion models are state-of-the-art generative models on data modalities such as images, audio, proteins and materials. These modalities share the property of exponentially decaying variance and magnitude in the Fourier domain. Under the standard Denoising Diffusion Probabilistic Models (DDPM) forward process of additive white noise, this property results in high-frequency components being corrupted faster and earlier in terms of their Signal-to-Noise Ratio (SNR) than low-frequency ones. The reverse process then generates low-frequency information before high-frequency details. In this work, we study the inductive bias of the forward process of diffusion models in Fourier space. We theoretically analyse and empirically demonstrate that the faster noising of high-frequency components in DDPM results in violations of the normality assumption in the reverse process. Our experiments show that this leads to degraded generation quality of high-frequency components. We then study an alternate forward process in Fourier space which corrupts all frequencies at the same rate, removing the typical frequency hierarchy during generation, and demonstrate marked performance improvements on datasets where high frequencies are primary, while performing on par with DDPM on standard imaging benchmarks.

1. Introduction

The introduction asks how DDPM’s Fourier-space noising shapes learned reverse generation. The paper studies this question for modalities whose Fourier-domain signal variance and magnitude decay rapidly with frequency.

1. Introduction

  • Fourier power-law data make DDPM corrupt high frequencies faster and earlier than low frequencies, potentially leaving fine details with greater approximation error under practical resource constraints.The paper hypothesizes that limited intermediate steps and neural-network expressiveness contribute to this error.Source: §1 · pp. 1–2
  • Because DDPM reverses this frequency ordering, it generates low frequencies first and high-frequency details conditional on them, a hierarchy whose importance the paper investigates.The introduction also poses generating high frequencies first as an alternative.Source: §1 · p. 2
  • The paper theoretically analyzes and empirically tests how DDPM’s frequency-dependent noising affects assumptions about the learned reverse process and generation quality.It proposes EqualSNR, a variance-preserving forward process that noises all frequencies at the same rate.Source: §1 · p. 2
  • EqualSNR performs on par with DDPM on standard image benchmarks while improving high-frequency generation and performance on datasets where high frequencies dominate.A classifier using two high-frequency summary statistics distinguishes DDPM-generated images from real images but performs poorly for EqualSNR.Source: §1 · pp. 2–3

2. Background: a spectral analysis of DDPM

This section analyzes DDPM’s forward process in Fourier space and uses signal-to-noise ratio to describe how corruption varies by frequency. For data with rapidly decaying Fourier variance, high frequencies are corrupted faster and earlier than low frequencies.

2. Background: a spectral analysis of DDPM

  • The section recalls DDPM’s timestep transitions and their closed-form marginal, with the schedule controlling the amount of white noise added at each step.The marginal separates a scaled signal term from a scaled noise term.Source: §2 · p. 3
  • For the modalities considered, Fourier-domain signal variance and magnitude decay by orders of magnitude with frequency, so equal-variance white noise corrupts high frequencies faster.This data property is called the Fourier power law.Source: §2 · p. 3
  • Applying the Fourier transform gives an equivalent view of the DDPM forward process because the transform is invertible.The transformed noise remains complex Gaussian with covariance proportional to the identity.Source: §2 · p. 3
  • The section defines SNR to quantify signal corruption and applies it to each frequency at each timestep.The frequency-specific SNR captures both the faster change and earlier corruption of high frequencies.Source: §2 · p. 3
  • DDPM corrupts high frequencies faster and earlier, and the paper previews that EqualSNR alleviates the resulting poor high-frequency generation quality.EqualSNR noises all frequencies at the same rate.Source: §2 · p. 4

3. Fast noising disrupts high-frequency generation: alternate forward processes

The section links faster DDPM noising of high frequencies to violations of the reverse process’s Gaussian assumption, then presents alternate Fourier-space forward processes intended to alleviate that issue.

3.1. Faster noising of high frequencies violates the Gaussian assumption in their reverse process.

  • With finite discretization, DDPM’s equal-variance noise can be large relative to high-frequency signal variance, making the reverse conditional non-Gaussian and potentially accumulating errors.A counterexample shows that adding sufficiently large noise to a mixture of separated Gaussians leaves the reverse conditional a constant distance from any Gaussian.Source: §3.1 · pp. 4–5
  • DDPM’s reverse conditional departs more from Gaussianity for high frequencies (KL 1.8 × 10−2) than low frequencies (KL 3.1 × 10−4).The comparison is for CIFAR10 at t = 1; the authors note that errors may accumulate across reverse-process timesteps.Source: Figure 3 · p. 5, §3.1 · p. 5

3.2. Alternate forward processes

  • EqualSNR assigns all frequencies the same SNR at each timestep; its reverse-process KL values are similar for low frequencies (1.0 × 10−4) and high frequencies (1.3 × 10−4).Unlike DDPM, EqualSNR removes the frequency hierarchy during generation.Source: §3.2 · p. 5
  • The framework specifies forward processes through frequency-specific SNR, with EqualSNR achieved when Σii = cCi and coordinate-wise variance preservation when c = 1.This SNR-based specification is presented as applicable to high-resolution data without modification.Source: §3.2 · pp. 5–6
  • Training scales noise and the prediction loss by Fourier-space signal variances, while adapted DDIM sampling and average-SNR calibration support comparisons across forward processes.The loss is an ELBO, and calibration matches average SNR across frequencies at each timestep.Source: §3.2 · p. 6

4. Experiments

Across standard imaging benchmarks, EqualSNR performs on par with DDPM, while improving high-frequency generation and performance on data where high-frequency information is dominant. The experiments compare frequency-specific generation quality and show that EqualSNR corrupts and generates frequencies simultaneously, unlike DDPM’s high-frequency-first noising and last-generation pattern.

Forward and reverse processes

  • DDPM noises high frequencies first and generates them last, while EqualSNR corrupts low- and high-pass images at the same rate and generates all frequencies simultaneously.Figure 4 illustrates the processes in pixel and Fourier space; the experiments use three natural-image datasets at 32 × 32, 64 × 64, and 128 × 128 resolutions.Source: §4 · pp. 6–7, Figure 4 · p. 7

Frequency-specific generation quality

  • EqualSNR matches real-data magnitudes at low and high frequencies, whereas DDPM’s high-frequency magnitudes are distinguishable even by eye.Figure 5 compares spectral magnitude profiles in decibels for low and high frequencies, with generated samples in blue and real data in red.Source: §4 · p. 7, Figure 5 · p. 8

High-frequency statistical testing

  • For high-frequency bands, DDPM’s real-versus-generated differences are detected almost 100% of the time, compared with 10–13% for EqualSNR at significance level 0.05.At significance level 0.01, EqualSNR’s rejection rate is 1–5%; low-frequency results are similar for both models. The study uses a logistic classifier on two spectral-regression statistics and reports averages over 100 runs.Source: §4 · pp. 7–8, Table 2 · p. 7

High-frequency-dominant data

  • On sparse 32 × 32 dot images, EqualSNR generates the correct number of dots and matches the data distribution more closely than DDPM.DDPM generates more points on average than the real data; Figure 6 shows pixel-intensity distributions for 1000 samples.Source: §4 · p. 8, Figure 6 · p. 8

Standard imaging benchmarks

  • EqualSNR performs on par with DDPM on standard imaging benchmarks, with Clean-FID used for the comparison.On LSUN Church, calibrated EqualSNR outperforms all DDPM variants; EqualSNR also requires fewer timesteps to saturate performance.Source: Table 1 · p. 7, §4 · p. 8

5. Related work

Related work studies how diffusion schedules shape frequency ordering, connects low-to-high generation to autoregressive behavior, and explores Fourier and wavelet representations.

Fourier space and function spaces

  • Gerdes et al. (2024) describe diffusion on general function spaces and relate the overlap of adjacent frequencies’ active times to generation without a frequency hierarchy.The paper identifies EqualSNR as implementing that hierarchy-free extreme, while noting that the concurrent work did not experiment with it.Source: §5 · p. 9

Wavelet methods

  • Wavelet methods use multiple subspaces for diffusion or image restoration, with one cited approach reporting linear growth in time complexity with image size.The linear-time observation is attributed to Guth et al. (2022).Source: §5 · p. 9

The low-to-high frequency hierarchy in diffusion models

  • Prior work characterizes diffusion as approximately autoregressive in Fourier space and identifies a gap in evidence about how essential low-to-high generation is.The section also notes applications that benefit from low-to-high frequency generation, including text-to-image and video generation.Source: §5 · p. 9

Frequency hierarchy and U-Nets

  • In U-Nets, average pooling projects toward lower-resolution Haar wavelet subspaces, nullifying noisy high-frequency components along the encoder.The passage links this structure to efficient denoising under DDPM noise.Source: §5 · p. 9

6. Conclusion

The paper analyzes the frequency-dependent noising rate and hierarchy induced by diffusion forward processes, and reports that EqualSNR improves high-frequency generation while matching DDPM’s FID on standard imaging benchmarks.

6. Conclusion

  • EqualSNR alleviates DDPM’s degraded high-frequency generation and performs on par with DDPM in FID on standard imaging benchmarks.The conclusion calls for future work to investigate SNR-governed alternate forward processes and tune them to the Fourier properties of each modality and task.Source: §6 · p. 10

Impact Statement

The theoretical passages characterize frequency corruption through SNR and covariance, contrast schedules that preserve or remove frequency hierarchy, and connect the training loss to the ELBO.

A.2. Forward Processes and their Derivation

  • EqualSNR enforces the same SNR across frequencies at each timestep, removing their hierarchy; FlippedSNR reverses DDPM’s frequency ordering.EqualSNR generates all frequencies at the same rate, whereas FlippedSNR generates high frequencies before low frequencies.Source: §A.2 · p. 14

A.3. Calibration

  • The DDPM and EqualSNR schedules are calibrated so their average SNRs match at every timestep.When parameterized by SNR, EqualSNR’s schedule is the average of DDPM’s SNR across frequencies.Source: §A.3 · p. 14

A.4. Our Training Algorithm and Connection to the ELBO

  • Minimizing the stated MSE loss at each timestep is equivalent to maximizing a standard ELBO on log pθ(y0).The passage describes the loss as aligning with the KL terms in the ELBO.Source: §A.4 · p. 14

A.5. Signal to Noise Ratio in High Dimensions

  • Equal SNR in all directions holds if and only if signal and noise covariances are proportional, while matching only diagonal entries targets equal SNR over Fourier directions.The passage distinguishes this all-directions result from the schedule’s frequency-wise matching.Source: §A.5 · pp. 15–16

A.6. Counterexample: Breaking the Gaussian Assumption

A two-mode counterexample shows that Gaussian forward transitions need not yield Gaussian reverse conditionals; the appendix also limits the scope of the real-world experiments to standard imaging benchmarks.

A.6. Counterexample: Breaking the Gaussian Assumption

  • With probability 1 − τ, the reverse conditional in the counterexample cannot be approximated by any single Gaussian beyond O(τ^18) in total variation.The example uses a two-component Gaussian mixture prior and Gaussian forward noise; the resulting reverse conditional remains bimodal.Source: §A.6 · p. 17

B. Additional experimental details and results

  • The real-world experiments investigate standard imaging benchmarks only, although other modalities also have the Fourier power law property.The largest imaging dataset is 128 × 128, below the scale of state-of-the-art generative models.Source: B. Additional experimental details and… · p. 19

B.1. Analysis 1: The reverse process learns to mirror its forward

The visualizations compare how DDPM and EqualSNR corrupt and restore Fourier frequencies, with additional examples across datasets and the full diffusion interval. They also describe how the forward and reverse visualizations are computed.

B.1. Analysis 1: The reverse process learns to mirror its forward

  • On CIFAR10, DDPM noises high frequencies very early, while EqualSNR noises all frequencies at a constant rate; each model denoises at the rate set by its forward process.The heatmaps sort frequencies from low to high and show contours at 10, 0, -10, -20, and -30 dB.Source: §B.1 · p. 19
  • Forward visualizations combine sampled image and noise under the schedules, while reverse visualizations follow stepwise denoising and estimate signal and noise for SNR.The EqualSNR forward visualization normalizes noise with C^1/2; coefficient variances are converted to decibels for display.Source: §B.1 · p. 19
  • CelebA at 64x64 and LSUN at 128x128 also show high frequencies being noised excessively early under the schedule used for Figure 2.The observation agrees with prior analysis that schedules may need rescaling with image resolution.Source: §B.1 · p. 19
  • Low- and high-pass images are formed by masking Fourier coefficients inside or outside a distance threshold from the center; Figure 7 shows the masks.Source: §B.1 · p. 20, Figure 7 · p. 20
  • Extended examples for DDPM, EqualSNR, and FlippedSNR cover t = 0 to t = T and include Fourier phase as an additional row.Source: §B.1 · p. 20

B.2. Analysis 2: Faster noising degrades high-frequency generation in DDPM; EqualSNR overcomes this

The section evaluates high-frequency generation by classifying real versus generated images from fitted frequency-magnitude features. Its reported classifier results are close to chance, with DDPM slightly better on true positives.

B.2. Analysis 2: Faster noising degrades high-frequency generation in DDPM; EqualSNR overcomes this

  • The high-frequency classifier fits a line to log-magnitudes and uses its slope and intercept to distinguish real from generated images.High classifier accuracy indicates low high-frequency generation quality.Source: §B.2 · p. 20
  • Significance is assessed over equal-sized data splits using permutation-test p-values against chance accuracy, counting correctly rejected tests across runs.The described procedure uses 100 splits and significance thresholds of 0.05 or 0.01.Source: §B.2 · p. 20
  • Both low- and high-frequency classifier mean accuracies are close to chance, and their true-positive counts are similar, with DDPM slightly better.Source: §B.2 · p. 20

B.3. Analysis 3: When high-frequency information matters: a synthetic study

The synthetic dots experiment compares calibrated DDPM and EqualSNR models on images with randomly placed white pixels. The reported examples highlight a pixel-intensity difference, and the table describes a low-frequency classifier evaluation.

B.3. Analysis 3: When high-frequency information matters: a synthetic study

  • The authors report that EqualSNR generates higher pixel intensity than DDPM; Figure 8 circles the lower-intensity images in red.Yellow denotes the highest-intensity, or white-pixel, values on the figure's color scale.Source: §B.3 · p. 20, Figure 8 · p. 21
  • The experiment samples 50,000 images with 48 to 50 randomly placed white pixels and trains for 10,000 steps with a cosine schedule using the CIFAR10 U-Net architecture.It compares EqualSNR with standard DDPM calibrated to EqualSNR.Source: §B.3 · p. 20
  • Table 3 reports low-frequency generation using mean classifier accuracy and true-positive counts over 100 runs at significance levels 0.05 and 0.01.Source: Table 3 · p. 21

B.4. Analysis 4: EqualSNR performs on par with DDPM on imaging benchmarks.

EqualSNR performs on par with DDPM on standard imaging benchmarks, using a Fourier-domain loss parametrization. The appendix also details figure construction and estimates distributions for visualizing the reverse process.

B.4. Analysis 4: EqualSNR performs on par with DDPM on imaging benchmarks.

  • Models use a U-Net and train for 800k steps with a cosine schedule and learning rate 0.0002; EqualSNR uses a Fourier-domain loss on predicted data coefficients.Results are reported for CIFAR10, CelebA, and LSUN Church at 32x32, 64x64, and 128x128 resolutions, respectively.Source: §B.4 · p. 21
  • Feeding Fourier values directly into the U-Net input did not provide competitive results.Source: §B.4 · p. 21

B.5. Further details on other figures

  • The Fourier power-law illustrations cover images, videos, audio, and protein density maps, with frequencies ordered from low to high for trend plots.The examples use different Fourier dimensions by modality and fixed-size tensors for the protein maps.Source: §B.5 · pp. 21–22

B.4. Analysis 4: EqualSNR performs on par with DDPM on imaging benchmarks.

  • EqualSNR performs on par with DDPM on standard imaging benchmarks, with Table 4 reporting torcheval FID.Clean-FID and PyTorch-FID implementations yield different score levels, highlighting implementation inconsistencies.Source: §B.4 · p. 21, Table 4 · p. 22

B.5. Further details on other figures

  • For Figure 3, q(y_t) and q(y_t−1) are approximated with Monte Carlo estimates from 5,000 samples, and posterior curves use KDE with shared bandwidth across frequencies and timesteps.The plotted conditional Gaussian uses the 70%-quantile of q(y_t−1) as a representative mean.Source: §B.5 · p. 22

B.6. Further experimental illustrations and results

This appendix adds experimental illustrations and results, including further analyses of frequency-specific forward and reverse processes. It also reports that training FlippedSNR was unsuccessful despite multiple attempts.

B.6. Further experimental illustrations and results

  • FlippedSNR failed to train across tested schedules and discretization steps; the experiments provide only partial evidence that generating high before low frequencies may be challenging or inefficient.The tested step counts were T ∈ {1000, 5000, 10000}; the resulting sample quality was deteriorated.Source: §B.6 · pp. 29, 36
  • Figure 17 compares per-frequency variances in the forward and reverse processes for DDPM, ESNR, and FlippedSNR to characterize how the learned reverse process mirrors the forward process.The variances are computed from forward push-forward distributions and reverse samples, providing a view of per-frequency variability at each timestep.Source: §B.6 · p. 31, Figure 17 · p. 31
  • On CIFAR10 images normalized to [−1, 1], DDPM is not variance-preserving in pixel space: coordinate-wise signal variance deviates from 1 and changes throughout diffusion.Figure 18 plots the pixel-space coordinate-wise signal variances diag(Cov(x0)).Source: §B.6 · p. 31, Figure 18 · p. 31
  • Additional SNR heatmaps show DDPM and EqualSNR on CelebA and LSUN Church, FlippedSNR on CIFAR10, and the FlippedSNR SNR trajectory.Figures 19–20 cover CelebA (64 × 64) and LSUN Churches (128 × 128); Figure 21 covers CIFAR10 (32 × 32).Source: §B.6 · pp. 30, 32, Figure 16 · p. 30, Figure 19 · p. 32
  • Figures 22–24 provide additional forward-process examples for DDPM, EqualSNR, and FlippedSNR, showing pixel-space and Fourier-space views alongside low- and high-pass filtered images.The examples show timesteps uniformly spaced from t = 0 to t = T.Source: §B.6 · p. 33, Figure 22 · p. 33, Figure 23 · p. 34
  • Figures 25–27 provide corresponding reverse-process examples for DDPM, EqualSNR, and FlippedSNR; the FlippedSNR model could not be trained successfully and produced deteriorated samples.The examples show timesteps uniformly spaced from t = T to t = 0.Source: §B.6 · p. 36, Figure 25 · p. 36, Figure 26 · p. 37
  • Figures 28–30 compare DDPM and calibrated EqualSNR samples on CIFAR10, CelebA, and LSUN Church using the same hyperparameters and neural architecture.Sampling uses T = 200 inference steps.Source: §B.6 · p. 39, Figure 28 · p. 39, Figure 29 · p. 40
Loading 2505.11278v1…