PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalization in diffusion models arises from geometry-adaptive harmonic representations

Zahra Kadkhodaie, Florentin Guth, Eero P. Simoncelli, Stéphane Mallat

arXiv:2310.02557v3cs.CVcs.LG

TL;DR

Diffusion denoisers can appear to escape high-dimensional density-estimation limits, but small datasets can instead produce memorization. The paper analyzes denoising and score estimation, finding strong generalization at sufficiently large sample sizes and geometry-adaptive harmonic shrinkage that is near-optimal for suitable image classes.

  • Problem

    The paper examines whether diffusion denoisers learn the continuous data density or memorize training samples, given the difficulty of high-dimensional density estimation.

  • Method

    The paper relates denoising to score-based density estimation and analyzes learned denoising functions, bases, and generalization across photographic, manifold-supported, and regular image classes.

  • Results

    Sufficiently large training sets produce strong generalization, while denoisers implement shrinkage in geometry-adaptive harmonic bases and achieve near-optimal performance on the Cα image class.

  • Takeaways & Limitations

    The findings associate diffusion denoiser inductive biases with geometry-adaptive harmonic representations that can align with photographic and structured image distributions.

  • Takeaways & Limitations

    The paper does not formally define the class of geometry-adaptive harmonic bases induced by DNN inductive biases.

Abstract

from arXiv · show

Deep neural networks (DNNs) trained for image denoising are able to generate high-quality samples with score-based reverse diffusion algorithms. These impressive capabilities seem to imply an escape from the curse of dimensionality, but recent reports of memorization of the training set raise the question of whether these networks are learning the "true" continuous density of the data. Here, we show that two DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function, and thus the same density, when the number of training images is large enough. In this regime of strong generalization, diffusion-generated images are distinct from the training set, and are of high visual quality, suggesting that the inductive biases of the DNNs are well-aligned with the data density. We analyze the learned denoising functions and show that the inductive biases give rise to a shrinkage operation in a basis adapted to the underlying image. Examination of these bases reveals oscillating harmonic structures along contours and in homogeneous regions. We demonstrate that trained denoisers are inductively biased towards these geometry-adaptive harmonic bases since they arise not only when the network is trained on photographic images, but also when it is trained on image classes supported on low-dimensional manifolds for which the harmonic basis is suboptimal. Finally, we show that when trained on regular image classes for which the optimal basis is known to be geometry-adaptive and harmonic, the denoising performance of the networks is near-optimal.

1 INTRODUCTION

The paper asks whether diffusion denoisers learn continuous image densities or memorize training samples, and shows that sufficiently large datasets yield strong generalization. It attributes this behavior to geometry-adaptive harmonic representations that support near-optimal denoising in suitable image classes.

  • Small training sets can cause diffusion models to memorize samples instead of approximating a continuous density.This overfitting raises questions about whether high-quality generation reflects learning the underlying data distribution.
  • Two denoisers trained on sufficiently large, non-overlapping datasets converge to essentially the same denoising function, indicating training-set-independent models.For the studied examples, roughly 10^5 images suffice, and generated samples have high visual quality.
  • DNN denoisers perform shrinkage in orthonormal geometry-adaptive harmonic bases whose oscillating structures follow image contours and homogeneous regions.These bases connect denoising behavior with the geometry of photographic images.
  • On low-dimensional image manifolds, denoisers approximate tangent-space projection while retaining some content in additional subspaces, revealing a bias beyond the optimal manifold basis.This tests the inductive bias where harmonic bases are suboptimal.

2 DIFFUSION MODEL VARIANCE AND DENOISING GENERALIZATION

The paper connects denoising to diffusion density estimation through score learning from noisy images. It shows that increasing dataset size moves denoisers from memorization to generalization, with train and test errors matching at 10^5 images.

  • 2.1 DIFFUSION MODELS AND DENOISING: Diffusion models estimate scores of Gaussian-blurred image densities and use those scores in an iterative reverse diffusion process.The noisy densities form a scale-space representation of the clean image density.
  • 2.1 DIFFUSION MODELS AND DENOISING: Denoisers trained by mean squared error estimate the posterior conditional mean, which recovers the score through sθ(y) = (fθ(y) − y)/σ2.The modeled density error is controlled by denoising optimality across noise levels.
  • 2.1 DIFFUSION MODELS AND DENOISING: Generalization is defined as near-zero model variance, meaning the denoiser does not overfit, while model bias measures distance from the true score.For photographic images, the optimal denoiser and true distribution are generally unknown, limiting direct numerical evaluation of sub-optimality.
  • 2.2 TRANSITION FROM MEMORIZATION TO GENERALIZATION: At N = 1, the denoiser memorizes the training image and has high test error; increasing N improves test performance while worsening training performance.This pattern marks the transition from memorization toward generalization.
  • 2.2 TRANSITION FROM MEMORIZATION TO GENERALIZATION: At N = 10^5, empirical test and train errors match across all noise levels, while large-N denoisers trained on disjoint subsets converge to the same score function.With shared initialization noise, these denoisers generate nearly identical samples.

3 INDUCTIVE BIASES

Diffusion denoisers can be analyzed through the Jacobian eigendecomposition, which reveals adaptive shrinkage in geometry-adaptive harmonic bases. These bases support efficient image approximation, explain vanishing model variance in the generalization regime, and are near-optimal for regular geometric image classes but mismatched to some data distributions.

  • Inductive biases: Diffusion models impose inductive biases through their architecture and optimization, helping constrain high-dimensional density estimation.For diffusion models, these biases arise directly in the denoiser because optimal denoising at all noise levels is equivalent to learning the correct density model.
  • Adaptive shrinkage: A denoiser’s Jacobian can be diagonalized into adaptive eigenvalues and eigenvectors, interpreting the network as shrinkage along an image-dependent basis.Both the basis vectors and shrinkage factors depend on the noisy image y.
  • Adaptive shrinkage: Minimizing denoising error trades off Jacobian rank against residual error, producing a soft projection onto a subspace approximating the posterior support.The subspace dimensionality corresponds to the Jacobian’s approximate rank and provides a local approximation of the support of p(x).
  • Optimality: The optimal denoiser has a Jacobian proportional to the posterior covariance, so its adaptive eigenvectors provide an efficient approximation of the clean image from a noisy observation.Small denoising error implies an approximately low-rank Jacobian with many small eigenvalues.
  • Geometry-adaptive harmonic bases: For face images, learned eigenvectors form geometry-adaptive harmonic bases with oscillations along contours and smooth regions, while coefficients decay rapidly in that basis.The rapid eigenvalue decay exploits the sparse representation of image content in the preserved subspace.
  • Geometry and mismatch: DNN denoisers achieve near-optimal rates on regular Cα images but are suboptimal on curved disk manifolds and perform substantially worse when face pixels are shuffled.The disk results show extra GAHB vectors beyond the manifold tangent space, whereas shuffled faces lack the locality supporting harmonic structure.

4 DISCUSSION

The paper connects diffusion-model generalization to denoiser inductive biases, finding convergence to a training-set-independent density and shrinkage in geometry-adaptive harmonic bases. These bases are near-optimal for certain regular image classes but suboptimal on low-dimensional manifolds, while their formal function class remains undefined.

  • Generalization: Diffusion models transition from memorization on small datasets to strong generalization, converging to a unique density independent of specific training samples.The data required for this transition depends on image size and complexity relative to network capacity.
  • Inductive biases: DNN denoisers perform shrinkage in geometry-adaptive harmonic bases shaped by image geometry.The learned bases are more general and flexible than bandlet bases, a specific near-optimal GAHB type.
  • Inductive biases: For Cα images, geometry-adaptive bases are known to be optimal and DNN denoisers achieve near-optimal denoising performance.
  • Inductive biases: On low-dimensional manifolds, denoisers align well with the tangent subspace but retain GAHB components in unconstrained dimensions, causing suboptimal denoising.
  • Open questions: The paper does not formally define the GAHB class induced by DNN biases, leaving its precise role across architectures and modalities open.It hypothesizes harmonic structure from convolutions and geometric adaptivity from ReLU interactions.

A EXPERIMENTAL DETAILS

The experiments use bias-free UNet and BF-CNN denoisers trained with Gaussian-noise denoising objectives across noise levels, then sample with an iterative reverse-diffusion algorithm.

  • Architectures: Experiments compare two bias-free architectures: UNet and BF-CNN.Bias-free operations remove additive constants, supporting universality across noise levels and interpretability through homogeneous transformations.
  • Architectures: UNet uses encoder-decoder blocks with spatial downsampling and upsampling, totaling 7.6m parameters.
  • Architectures: BF-CNN uses 21 convolutional layers without subsampling and has 700k parameters.
  • Training: Training minimizes mean squared denoising error for images corrupted by i.i.d. Gaussian noise with standard deviations in [0, 1].Models train for 1000 epochs with batches of size 512 and receive no noise-level input.
  • Datasets: Datasets include downsampled CelebA, LSUN bedroom, and CelebA HQ images at resolutions ranging from 32×32 to 80×80.
  • Sampling: Sampling uses an iterative denoiser-residual algorithm with h = 0.01, β = 0.1, σ0 = 1, and σ∞ = 0.05.The parameters control step size, injected noise, and initial and final noise levels.

B.1 SIMILARITY BETWEEN DATA SUBSETS

The study tests whether model-variance convergence persists across data distributions and architectures, while removing near-duplicate images from the non-overlapping subsets.

  • Dataset construction: Images with cosine similarity above 0.95 are removed before training to eliminate replicated images between subsets.
  • Generalization: With enough data, models trained on non-overlapping subsets converge to virtually the same function across data distributions and architectures.The training-set size required for the memorization-to-generalization transition depends on architecture, image size, and data distribution.

B.2.1 TRAINED ON CELEBA DATASET

At N = 10^5, samples generated by separate denoisers are nearly identical when initialized identically, although trajectory bifurcations show convergence is not perfect.

  • Converged sampling: At N = 10^5, separate denoisers generate nearly identical samples from the same initialization, indicating convergence to the same function.
  • Residual variation: The two models can still reach different final samples from the same initial point because gradients bifurcate around the middle of sampling trajectories.This prevents the cosine-similarity distribution from being a perfect delta function at 1.

B.2.2 TRAINED ON LSUN BEDROOM DATASET

On LSUN bedroom images, denoisers transition from memorizing training examples to generalizing as the training set grows. At N = 10^5, training and test performance nearly coincide, while N = 100 produces patch-based memorization.

  • At N = 10^5, training and test PSNR are almost identical, indicating that the model is no longer overfitting the training data.
  • At N = 100, generated samples are combinations of patches from training images, a previously reported form of memorization.
  • The experiments vary the subset size N from 1 to 10^5 while training on non-overlapping LSUN bedroom subsets.

B.3.1 TRAINED ON CELEBA DATASET

CelebA experiments examine the transition from memorization to generalization using BF-CNN denoisers trained on non-overlapping subsets. The subset size is varied across a broad range to assess convergence of model variance.

  • Figure 11 examines the transition from memorization to generalization for a BF-CNN denoiser trained on CelebA HQ images downsampled to 40 × 40.
  • Figure 12 compares BF-CNN denoisers trained on non-overlapping CelebA HQ subsets S1 and S2.
  • The CelebA subset size N varies from 1 to 10^4 in the model-variance convergence experiment.

B.3.2 TRAINED ON LSUN BEDROOM DATASET

LSUN bedroom experiments compare denoisers trained on separate large subsets and evaluate how generalization changes with training-set size and image resolution. At N = 20,000, generated samples from the two models are more similar to each other than to their nearest training images.

  • N = 20,000 LSUN bedroom images are split into non-overlapping subsets S1 and S2, each used to train a BF-CNN denoiser.
  • Generated samples from the two denoisers have higher cosine similarity to each other than to the closest image in their respective training sets.
  • Larger image resolutions require more training data to reach the threshold PSNR gap between training and test performance.
  • With a four-fold increase in image size, the UNet parameter count increases approximately four times.

C ADDITIONAL NUMERICAL RESULTS ON INDUCTIVE BIASES

Additional experiments test the denoisers’ inductive biases on structured image classes, low-dimensional manifolds, and spatially shuffled faces. The results show geometry-adaptive harmonic structure even when that basis is not optimal, while regular classes achieve near-optimal denoising.

  • For Cα images, trained BF-CNN denoisers achieve near-optimal performance, with empirical PSNR slopes closely matching the optimal slopes.
  • The geometric-adaptive harmonic basis contains harmonics within regions and along boundaries, with increasing frequency as the basis index k increases.
  • As contour regularity increases, the top basis vectors change across Cα images while the background regularity remains fixed.
  • For a single face image with positive intensity rescalings, optimal denoising projects noisy inputs onto the ray spanned by the normalized face image.
  • For 2D sine-wave images, low-noise optimal denoising projects onto the tangent subspace spanned by unit-frequency sine waves with a π/2 phase shift.
  • Pixel-shuffled faces preserve the optimal denoising performance of ordinary faces despite destroying locality, because unshuffling and reshuffling can surround the optimal operation.

D MATHEMATICAL DERIVATIONS

The appendix derives relationships connecting Gaussian-noise denoising, score matching, and risk estimation, then analyzes optimal basis shrinkage and its error for geometric image classes.

  • D.1 MMSE AND SCORE RELATIONSHIPS: The Miyasawa identities connect the score and its Jacobian to posterior moments under additive Gaussian noise.The derivation uses Bayes’ rule and the Gaussian measurement model to obtain first- and second-order relationships.
  • D.2 CONTROL ON KULLBACK-LEIBLER DIVERGENCE: Substituting the Miyasawa equation and score parameterization rewrites score-matching error as a denoising objective.The derivation combines sθ(y) = (fθ(y) − y)/σ2 with conditional denoising-error decomposition to prove the stated equivalence.
  • D.3 SURE OBJECTIVE: Stein’s Unbiased Risk Estimator expresses denoising risk through noisy observations and the divergence of the denoiser.The decomposition uses the Gaussian-noise variance σ2d and integration by parts.
  • D.4 OPTIMAL THRESHOLDING IN A BASIS: The fixed-basis oracle denoiser balances denoising bias from discarded signal variance against denoising variance from retained noise.Its approximation efficiency is captured by the sum of ∥x − xM∥2 and Mσ2, with the coefficient ordering depending on the image.
  • D.4 OPTIMAL THRESHOLDING IN A BASIS: For coefficients obeying ⟨x, ek⟩2 ∼ k−(α+1), the oracle MSE scales as σ2α/(α+1).The threshold index satisfies M ∼ σ−2/(α+1), making the bias and variance terms have the same order.
  • D.5 GEOMETRIC IMAGE CLASSES: Geometric Cα images are defined by piecewise α-Lipschitz behavior separated by non-tangentially intersecting Lipschitz curves, and can be generated with Fourier-domain constructions.The construction smooths random one- and two-dimensional functions, creates a mask, and combines two backgrounds.
Loading 2310.02557v3…