PaperlayerView exact revision on arXiv

Source-linked AI summary

Locality in Image Diffusion Models Emerges from Data Statistics

Artem Lukoianov, Chenyang Yuan, Justin Solomon, Vincent Sitzmann

arXiv:2509.09672v2cs.CV

TL;DR

The paper asks why diffusion models develop locality that supports generalization rather than memorization. It derives locality from dataset pixel correlations through linear denoising and shows that dataset-derived analytical sensitivities better match deep diffusion behavior than prior alternatives. The results support a primary role for architectures in capturing, rather than creating, locality patterns.

  • Problem

    The paper addresses the gap between diffusion models’ theoretical memorization behavior and their empirical ability to generate novel images, focusing on why learned locality arises.

  • Method

    The paper analyzes optimal linear denoisers and dataset covariance, relating learned sensitivity fields to high-SNR principal components and testing these predictions across architectures and datasets.

  • Results

    Dataset-derived sensitivity fields explain locality in both U-Nets and Transformers, and the resulting analytical model outperforms previous approaches across multiple datasets.

  • Takeaways & Limitations

    Diffusion-model locality emerges from training-data statistics, while neural architectures primarily capture those patterns and still influence generation quality.

  • Takeaways & Limitations

    The analysis focuses on simpler architectures, relies on second-order statistics, and assumes locality fields are constant with respect to input images.

Abstract

from arXiv · show

Recent work has shown that the generalization ability of image diffusion models arises from the locality properties of the trained neural network. In particular, when denoising a particular pixel, the model relies on a limited neighborhood of the input image around that pixel, which, according to the previous work, is tightly related to the ability of these models to produce novel images. Since locality is central to generalization, it is crucial to understand why diffusion models learn local behavior in the first place, as well as the factors that govern the properties of locality patterns. In this work, we present evidence that the locality in deep diffusion models emerges as a statistical property of the image dataset and is not due to the inductive bias of convolutional neural networks, as suggested in previous work. Specifically, we demonstrate that an optimal parametric linear denoiser exhibits similar locality properties to deep neural denoisers. We show, both theoretically and experimentally, that this locality arises directly from pixel correlations present in the image datasets. Moreover, locality patterns are drastically different on specialized datasets, approximating principal components of the data's covariance. We use these insights to craft an analytical denoiser that better matches scores predicted by a deep diffusion model than prior expert-crafted alternatives. Our key takeaway is that while neural network architectures influence generation quality, their primary role is to capture locality patterns inherent in the data.

1 Introduction

The paper argues that diffusion-model locality comes from image-dataset statistics rather than convolutional architectural bias. It develops statistical and analytical denoising models that better explain deep diffusion behavior and generalization.

  • 1 Introduction: The optimal denoiser has a closed-form solution but memorizes training images instead of generating novel samples.Its behavior differs from deep diffusion models despite minimizing the training objective.
  • 1 Introduction: Dataset statistics, not neural-network architecture, determine locality patterns in image diffusion models.The paper relates learned sensitivity fields to pixel correlations and principal components of the training data.
  • 1 Introduction: The study shows that locality can be local on self-similar datasets but nonlocal on specialized datasets such as centered human faces.The differing patterns align with high-SNR principal components of each dataset’s covariance.
  • 1 Introduction: A simple Wiener filter outperforms recent optimal-denoiser modifications, while combining dataset-derived sensitivities with Kamb and Ganguli’s model performs best across multiple datasets.The benchmark compares analytical models by how closely their generations match those of a trained deep diffusion model.

2 Preliminaries and Related Work

This section reviews diffusion training, analytical optimal denoisers, locality-based approximations, and evidence that diffusion models exhibit linear behavior. It highlights the optimal denoiser’s memorization problem and prior attempts to introduce locality or smoothing.

  • 2 Preliminaries and Related Work: Diffusion models denoise Gaussian-corrupted images by predicting the clean image from a noisy input at varying noise levels.Training samples a clean image, noise level, and Gaussian noise direction before forming x_t.
  • 2 Preliminaries and Related Work: The closed-form optimal denoiser is a kernel-weighted average over training data that approaches nearest-neighbor memorization as noise vanishes.Consequently, it generates training images rather than novel images at low noise.
  • 2 Preliminaries and Related Work: Smoothing-based approaches modify the score or empirical data distribution to encourage generalization beyond the training set.These methods address the memorization behavior of analytical denoisers through explicit smoothing.
  • 2 Preliminaries and Related Work: Prior methods attribute the gap between optimal and deep denoisers to architectural inductive biases and impose local, equivariant patch constraints.These approaches use binary or compact masks to approximate learned locality fields.
  • 2 Preliminaries and Related Work: Diffusion models show strong linear behavior, especially at high noise levels, motivating linear denoisers as analytical models.The optimal linear denoiser has the same form as the optimal denoiser for Gaussian data.

3 Deriving Denoising Sensitivity from Dataset Statistics

The paper derives denoising sensitivity from dataset covariance under a locally linear assumption. High-SNR principal components produce pass-through behavior, explaining why locality can vary from compact to highly nonlocal across datasets.

  • 3 Deriving Denoising Sensitivity from Dataset Statistics: At low noise, deep denoisers pass through coarse input structure, whereas the optimal denoiser teleports novel inputs to nearby training images.The contrast explains why the optimal denoiser fails in sampling-void regions relevant to generalization.
  • 3 Deriving Denoising Sensitivity from Dataset Statistics: High-SNR components resemble low-frequency features for natural images, but specialized face datasets produce non-translation-equivariant and non-isotropic sensitivity patterns.The principal-component basis need not resemble a Fourier basis when dataset correlations are specialized.
  • 3 Deriving Denoising Sensitivity from Dataset Statistics: The Wiener filter projects inputs onto data principal components and shrinks each projection according to its signal-to-noise ratio.It is optimal for Gaussian data and for linear denoising under the stated centered-data assumption.
  • 3 Deriving Denoising Sensitivity from Dataset Statistics: Sensitivity fields are determined by dataset statistics under local linearity and approximate projections onto high-SNR principal components.This connects the Wiener filter’s sensitivity to learned sensitivity fields across denoising architectures.
  • 3 Deriving Denoising Sensitivity from Dataset Statistics: As noise decreases, the locally linear sensitivity field shrinks toward the indicator of the output pixel.This follows from the increasing signal-to-noise ratio of each component.

4 Validation

Validation shows that sensitivity fields align across architectures with high-SNR data components, vary with dataset-specific correlations, and can be induced by modifying those statistics. The resulting analytical model outperforms the baselines in explaining trained diffusion models.

  • Shared locality across architectures: U-Nets and globally attending DiTs learn similar sensitivity fields that match high-SNR projection operators, implicating pixel correlations rather than architecture as the main source of locality.The architectures still help capture data statistics, but are not identified as the main cause of locality patterns.
  • Dataset-dependent sensitivity: Specialized face data produces structured, location-dependent sensitivity fields because its principal components reflect nonlocal correlations and lack translation equivariance.CelebA-HQ sensitivity varies between pixels near the eye and image center across noise levels.
  • Manipulating sensitivity fields: Editing pixel correlations in CIFAR-10 induces a desired sensitivity pattern in a newly trained DDPM, including nonlocal patterns, showing that locality can be shaped through data covariance.A W-shaped field emerges when the added signal dominates the noise at the relevant timesteps.
  • Analytical model: The proposed analytical approach replaces fitted patch masks with high-SNR covariance components and represents locality as a data-derived subspace projection.For constant sensitivity fields, the corresponding linear denoiser is the Wiener filter; the generalized formulation permits pixel-specific sensitivity.
  • Analytical model comparison: The analytical model outperforms all baselines, while the Wiener filter is almost always second-best across the evaluated comparisons.On CelebA-HQ, patch-based locality performs worse because it erases eyes and blurs facial features, whereas dataset-dependent locality retains them.

5 Conclusion, Limitations, and Future Work

The paper concludes that diffusion-model locality emerges from dataset statistics rather than architectural inductive biases, while noting limits from simpler architectures, second-order statistics, and constant locality fields.

  • Conclusion: Locality in diffusion models emerges from dataset statistics rather than architectural inductive biases, with U-Nets and Transformers aligning with high-SNR principal-component projections.The analysis links this behavior to linear denoisers and Wiener filtering.
  • Conclusion: The analytical model based on dataset statistics outperforms previous approaches in approximating trained diffusion models across multiple datasets.
  • Limitations: The approach focuses on simpler architectures, relies on second-order statistics, and assumes locality fields remain constant across input images.Deep diffusion networks can capture higher-order statistics, and nonlinear regimes remain to be studied.
  • Future Work: Future work targets complex architectures, higher-order statistics, and conditional generation to further explain diffusion models’ theory–practice gap.

A.3 Ours: why do we binarize the sensitivity field

The appendix shows that generalized optimal denoisers depend on the support or row space of locality operators, making binary masks sufficient for diagonal operators.

  • Generalized Locality: The generalized optimal denoiser depends only on the row space of the locality operator, interpolating between unconstrained and input-independent cases.
  • Generalized Locality: The generalized analysis represents locality through a projection onto a subspace, followed by masking in an appropriate basis.
  • Diagonal Operators: For diagonal locality operators, only whether each coefficient is zero matters, reducing the operator to its binary support.
  • Why Binarize: Binary masks are as expressive as diagonal weighting schemes because the optimal denoiser depends only on which pixels are included, not their relative weights.The nonzero diagonal values cancel in the softmax computation, so more complex weighting provides no additional benefit.

B.1 Ablation of our model

The ablations examine threshold choice and self-attention, showing a trade-off between sharpness and smoothness and a lower FID after removing self-attention.

  • Threshold Ablation: Higher binarization thresholds produce sharper but patchier images, whereas lower thresholds produce larger patches and over-smoothed images.
  • Threshold Ablation: Table 2 compares r2 and MSE correlations with the trained diffusion model across datasets and threshold values.The caption states that the best values are highlighted in bold.
  • Self-Attention Ablation: FID is 6.04 without self-attention versus 4.12 with self-attention, although generated images look qualitatively similar.
  • Self-Attention Ablation: Figure 7 compares samples from U-Nets without and with self-attention using identical initial random noise.

B.3 Low rank projection of the covariance matrix

Low-rank covariance projections smooth Wiener-filter outputs but reduce their agreement with trained diffusion models, while the proposed method remains more correlated without benefiting from projection.

  • B.3 Low rank projection of the covariance matrix: Most covariance energy lies in the first 200 singular values across all datasets, followed by a long tail of low-energy components.The analysis examines both the full singular-value spectrum and its first 100 components.
  • B.3 Low rank projection of the covariance matrix: Low-rank approximation makes Wiener-filter outputs visually smoother but reduces their correlation with trained diffusion-model predictions.The comparison evaluates different total-energy cutoffs, including the full-rank covariance matrix.
  • B.3 Low rank projection of the covariance matrix: The proposed method does not benefit from covariance projection and still achieves higher correlation than the Wiener filter.This result indicates that low-rank structure alone does not account for the performance of learned denoisers.
  • B.3 Low rank projection of the covariance matrix: Sensitivity-field comparisons depend strongly on parameterization, normalization, and noise level, with the optimal denoiser diverging at low noise.Image-parameterized fields and per-image normalization are used because noise reparameterization and joint normalization can distort the apparent locality pattern.

B.7 Quantitative results for AFHQv2 and Fashion-MNIST

Additional quantitative comparisons evaluate methods on AFHQv2 and Fashion-MNIST using metrics averaged across 128 samples.

  • B.7 Quantitative results for AFHQv2 and Fashion-MNIST: The appendix reports quantitative method comparisons on AFHQv2 and Fashion-MNIST.The accompanying table marks the best and second-best results for each comparison.
  • B.7 Quantitative results for AFHQv2 and Fashion-MNIST: All reported metrics are averaged over 128 samples.
  • B.7 Quantitative results for AFHQv2 and Fashion-MNIST: Best results are highlighted in green and second-best results in maroon.

B.8 Quantitative measure of novelty of samples

The paper evaluates whether analytical models generate samples distinct from the training set and compares their generation dynamics with a trained DDPM model.

  • B.8 Quantitative measure of novelty of samples: Novelty is quantified by the average L2 distance between generated samples and their closest training-dataset images.The table averages this distance across 128 samples.
  • B.8 Quantitative measure of novelty of samples: Figure 14 measures prediction MSE against a trained DDPM on CIFAR10 and CelebA-HQ across 128 samples.The MSE is computed for x0 predictions at each point along the 10-step trajectory.

B.9 Additional generation results

Additional generation results report novelty dynamics across five datasets using a 10-step DDIM sampling procedure.

  • B.9 Additional generation results: The additional figures provide generation results analogous to Figure 5 of the main paper.
  • B.9 Additional generation results: Figure 15 reports the L2 distance from each x0 prediction to the closest training image throughout a 10-step generation trajectory.This tracks the evolution of the paper’s novelty measure during generation.
  • B.9 Additional generation results: All generations use the DDIM sampler with 10 sampling steps and a noise-time scale from t=0 to t=1000.The scheduler is linear, with α0=10^-4 and α1000=0.02.

C.2 Training DDPM Model

The experiments train DDPM U-Nets with resolution-dependent architectures and fixed optimization settings, while the analytical model uses covariance-based preprocessing and a weighted streaming softmax. Complexity analysis shows that the proposed model scales as O(nptm), with a larger-dataset implementation matching O(nptm/k).

  • DDPM training: The DDPM U-Net uses three downsampling levels for 28-pixel images and four for 32- or 64-pixel images, with two residual blocks per level and no self-attention.The base channel width is 64 for MNIST/FashionMNIST and 128 for CIFAR10, CelebA-HQ, and AFHQ.
  • DDPM training: Training runs for 200 epochs with batch size 32, Adam learning rate 10^-4, 1000 diffusion steps, dropout 0.15, and fixed random seeds.The implementation uses a third-party PyTorch DDPM U-Net.
  • Analytical baselines: The analytical model centers each dataset, computes its covariance matrix, obtains principal components with SVD, and uses a weighted streaming softmax during denoising.The Wiener matrix is computed from the dataset covariance, while Kamb&Ganguli patch sizes are fit to trained DDPM U-Nets by maximizing score r2.
  • Algorithmic complexity: O(m2) makes the Wiener filter the most efficient method for small-resolution images, whereas the proposed model and Kamb&Ganguli’s require a dataset pass per inference step.Here, m is the flattened image resolution and n is the dataset size.
  • Algorithmic complexity: O(nptm) is the proposed model’s algorithmic complexity, while larger datasets can achieve O(nptm/k) by indexing masks per timestep and pixel.The proposed model uses distinct per-pixel mask patterns and does not assume translation equivariance.
Loading 2509.09672v2…