PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime

Hugo Latourelle-Vigeant, Sinho Chewi, Aram-Alexandre Pooladian, John Sous, Theodor Misiakiewicz

arXiv:2608.23938v1stat.MLcs.LGmath.ST

TL;DR

Diffusion models can fit finite-sample denoising objectives exactly, raising questions about how they generalize in high dimensions. This paper analyzes lazy denoising score matching and finds distinct phases separating generalization, objective interpolation, and memorization during reverse-time generation.

  • Problem

    The paper asks how overparameterized diffusion models generalize when exact denoising-objective fitting can instead produce empirical-Bayes memorization of training samples.

    Source: §1 · p. 3, §2.2 · p. 12

  • Method

    The authors analyze gradient-flow-trained denoising score matching in an inner-product-kernel RKHS under the proportional regime n ≍ d, including its reverse-time SDE.

    Source: §3.1 · p. 16, §5 · p. 27

  • Results

    The learned score passes through three phases—spectral generalization, localized-bump interpolation, and empirical-Bayes memorization—and objective interpolation does not imply memorization in generation.

    Source: §1.2 · p. 6, §5 · pp. 27–28

  • Takeaways & Limitations

    Overfitting and memorization are distinct: the model can interpolate its training objective while still generating new samples, although interpolation degrades population-level score generalization.

    Source: §5 · p. 28

    In the proportional high-dimensional regime, isotropic lazy architectures generate at most Gaussian distributions, so non-Gaussian generation requires feature learning, anisotropy, or low-dimensional structure.

    Source: §1.2.2 · p. 9

Abstract

from arXiv · show

Modern score-based generative models have achieved remarkable empirical success in high-dimensional tasks such as image, audio, and video synthesis. These models reduce distribution learning to a sequence of regression problems that, if solved exactly on finite data, would ultimately reproduce the training samples. Their ability to generalize must therefore arise from the implicit or explicit regularization during training. In this work, we develop a generative counterpart to the theory of benign overfitting and algorithmic regularization for overparameterized neural networks in the supervised lazy-training regime. We study denoising score matching in a vector-valued reproducing kernel Hilbert space with an inner-product kernel. In the proportional high-dimensional regime $n\asymp d$, we derive exact risk trajectories under gradient flow training. These trajectories exhibit three phases governed by qualitatively distinct estimators: a spectral estimator that generalizes, a pure-noise score with localized peaks that interpolate the training objective, and an empirical Bayes estimator that memorizes the data. We then analyze how these estimators combine along the reverse-time SDE and characterize the distribution of the resulting samples. The analysis reveals familiar mechanisms from supervised learning, including kernel linearization and self-induced regularization from the nonlinear part of the kernel, but also reveals a distinct phenomenology specific to generative modeling.

1 Introduction

The paper develops a high-dimensional theory of diffusion models in the lazy kernel regime, analyzing how training-time regularization shapes generalization, objective overfitting, and memorization. It studies kernel denoising score-matching dynamics, their three training phases, and reverse-time generation using operator-level and pathwise tools.

1 Introduction

  • Diffusion models reduce generation to denoising regressions across noise levels, then compose the learned scores through a reverse-time SDE initialized from approximate Gaussian noise.This reduction motivates studying both score-estimation behavior and the generated distribution directly.Source: §1 · p. 3

1.2 Summary of main results: lazy training for generative modeling

  • Lazy diffusion training exhibits three regimes: spectral estimation that generalizes, pure-noise fitting with localized interpolating bumps, and empirical Bayes estimation that memorizes the data.These regimes occur at linear, polynomial, and super-polynomial training timescales, respectively.Source: §1.2 · p. 6, §1.2.1 · pp. 7–8
  • The kernel analysis transfers supervised-learning mechanisms but changes their consequences: nonlinear kernel structure induces tube-averaging regularization and localized bumps rather than an ordinary ridge penalty.These bumps interpolate denoising residuals while remaining invisible at test time and during generation.Source: §1.2 · p. 6
  • The reverse process separates objective overfitting from memorization: vanishing training loss need not reproduce training samples, although overfitting and population-score degradation nearly coincide.At intermediate times, the model fits the training objective while generating a Gaussian law; only super-polynomial training leads to collapse onto the training set.Source: §1.2 · pp. 6–7, §1.2.1 · p. 8

1.2.2 Generation: delocalization and the Gaussian barrier

  • Lazy diffusion models generate asymptotically Gaussian distributions whose covariance is explicitly determined by the data covariance, sample ratio, self-induced regularization, and training and sampling schedules.The generated law depends on the data distribution only through its first two moments, even when the model interpolates the empirical denoising objective.Source: §1.2 · p. 7, §1.2.2 · p. 9

1.2.1 Algorithmic regularization via early stopping

  • Early stopping produces a spectral shrinkage estimator with minimum test risk at training time t* ≍ d, after which interpolation drives the learned linear component toward the pure-noise score.The nonlinear kernel channel relaxes more slowly, fitting localized per-sample bumps on timescale 1/δ_n ≍ n.Source: §1.2.1 · p. 7

1.2.2 Generation: delocalization and the Gaussian barrier

  • Along the reverse SDE, high-dimensional delocalization makes localized kernel bumps asymptotically invisible, yielding a pathwise linearization for schedules up to d^(1+c) with c < 1/6.Typical trajectories have normalized overlap O(d^-1/2) with every training sample, so learned scores become indistinguishable from their linearized surrogates.Source: §1.2.2 · pp. 8–9
  • In proportional high dimension, isotropic lazy architectures generate at most Gaussian distributions, so non-Gaussian generation requires feature learning, architectural anisotropy, or genuinely low-dimensional data structure.This is the generative analogue of the supervised kernel polynomial barrier.Source: §1.1 · p. 5, §1.2.2 · p. 9

1.3 Related work

  • Prior diffusion theory studies sampler guarantees, score-estimation rates, memorization transitions, empirical-score collapse, and exactly solvable denoising models, whereas this paper analyzes what training learns when n ≍ d.The closest prior work identifies generalization and memorization timescales, while this paper sharpens their interpretation through its three-phase picture.Source: §1.3 · pp. 9–10
  • The paper’s main advances are a universal anisotropic kernel class with non-ridge self-induced regularization, exact gradient-flow equivalents including covariance unlearning, separated interpolation and memorization timescales, and pathwise reverse-SDE analysis.These results extend prior linear or Gaussian settings and show that score-level harm can coexist with unharmed generation.Source: §1.3 · p. 10
  • The paper builds on supervised interpolation, spectral kernel linearization, self-induced regularization, and covariance-resolvent equivalents, extending kernel-matrix theory to infinite-dimensional denoising designs.Section 2 introduces kernel DSM objectives and estimators; Section 3 studies fixed-noise training dynamics and three phases; Section 4 analyzes reverse-SDE generation and its Gaussian characterization.Source: §1.3 · p. 10, §1.4 · p. 11

1.4 Notation and organization

  • The notation defines norm, trace-scale, asymptotic, probability, stochastic-domination, function-space, and divergence conventions, while distinguishing diffusion time from optimization time.Constants may depend on fixed diffusion time through ρ_t and σ_t.Source: §1.4 · pp. 10–11

2 Kernel denoising score matching

The section formulates diffusion denoising score matching as time-decoupled regression in a vector-valued RKHS with inner-product kernels. It contrasts the population Bayes denoiser with the empirical Bayes optimum and defines gradient-flow and ridge estimators through kernel operators.

2.1 Denoising score matching

  • Each noise level is modeled by a separate RKHS regression, so fixed-time problems decouple for analysis before being reassembled through the reverse SDE.In practice, a single time-conditioned network couples noise levels, unlike the simplifying assumption used here.Source: §2.1 · p. 12

2.2 The empirical objective and its memorizing optimum

  • The unrestricted empirical objective is minimized by the empirical Bayes denoiser, whose reverse-SDE score retrieves training samples as noise vanishes.The RKHS and training algorithm are therefore studied as mechanisms for avoiding memorization and approaching the population Bayes denoiser.Source: §2.1 · p. 11, §2.2 · p. 12

2.3 Inner-product kernels and the model class

  • The RKHS uses positive-definite inner-product kernels and can be universal, making it expressive enough to represent the empirical Bayes denoiser rather than preventing memorization.Its connection to wide neural networks trained in the lazy regime motivates comparison with supervised kernel theory.Source: §2.3 · p. 13

2.4 Kernel operators, gradient flow, and ridge regression

  • Kernel operators map empirical training functions to population evaluations, while the empirical denoising problem is effectively noiseless because its intrinsic noise is exponentially small in high dimensions.The risk decomposition is centered on the empirical Bayes denoiser, not the population Bayes predictor.Source: §2.4 · p. 14
  • Gradient flow applies the spectral filter 1 − e^−tλ to empirical Bayes components, fitting eigencomponents on timescale 1/λ; ridge provides a comparable shrinkage with κ_eff = d/t.This establishes the early-stopping–ridge correspondence used for intuition, comparison, and proofs.Source: §2.4 · pp. 14–15

3 Learning the score along the gradient flow trajectory

This section characterizes the three statistical phases along the gradient-flow trajectory—spectral generalization, pure-noise overfitting, and eventual memorization—in the proportional high-dimensional regime. It examines their population behavior and test-risk trajectories, with kernel linearization explaining the early phases.

3 Learning the score along the gradient flow trajectory

  • The trajectory progresses from regularized linear generalization at t ≲ d^(1+c), through polynomial-time overfitting, to empirical-Bayes memorization only at super-polynomial times.These phases occur under n/d → r ∈ (0, ∞), with concentration assumptions on the data distribution.Source: §3 · p. 15

3.1 Linearization of the kernel and localized bumps

  • Kernel operators linearize on delocalized input pairs but retain nonlinear curvature on noisy views sharing a training sample, yielding localized corrections and operator error ≺ d^-3.The nonlinear contribution induces per-sample self-regularization through conditional noise averaging rather than an identity ridge penalty.Source: §3.1 · pp. 16–17
  • The effective estimator combines a global linear predictor with localized bumps that fit each training tube; positive curvature shrinks empirical covariance dependence and accelerates overfitting.The bumps cancel training-sample biases, while larger δn shortens the generalization window and causes earlier covariance forgetting.Source: §3.1 · p. 18, §3.2 · p. 21

3.2 Phase I: linearized dynamics and generalization

  • Phase I learns covariance eigendirections spectrally, with top directions learned before bulk directions, while kernel curvature opens a slower bump-fitting channel that drives training error toward zero.The data-direction timescale is approximately d, whereas localized bumps fit on a timescale approximately δn^-1 ≈ n; early stopping near t⋆ ≍ d avoids the pure-noise overshoot.Source: §3.2 · pp. 20–21

3.3 Phase II: vanishing error and overfitting

  • High-degree kernel components construct bounded-RKHS localized interpolants that match the denoising target over the union of training tubes, not merely at finitely many labels.Driving the empirical DSM objective to zero requires interpolation in L2(bµd,n), because the objective integrates over Gaussian noise around each sample.Source: §3.3 · p. 22
  • In Phase II, training error vanishes while population risk plateaus at the rescaled pure-noise score’s risk, with overfitting confined to exponentially thin neighborhoods of training tubes.The plateau is determined by the pure-noise score rather than covariance fitting, and its benignity depends on the noise level ρ^2.Source: §3.3 · p. 22
  • Throughout the polynomial-time window, further training changes the estimator only inside localized tubes, leaving population risk unchanged; high noise makes overfitting benign, whereas low noise makes it detrimental.The plateau is close to the best predictor at high noise but exceeds the zero estimator’s risk at low noise.Source: §3.3 · p. 22

3.4 Phase III: the memorization endpoint

  • At super-polynomial times, assuming positive high-degree kernel coefficients, gradient flow converges in training-sample L2 to the empirical Bayes denoiser, although unprocessed test risk may diverge from tail instability.A projection-based post-processing step prevents these instabilities while preserving the empirical Bayes interpolation limit.Source: §3.4 · pp. 22–23
  • The empirical Bayes denoiser’s test risk is twice Phase II’s interpolation plateau because winner-take-all nearest-training-point behavior adds a second, near-orthogonal error component.At fresh noisy points, the empirical score selects a single nearest training sample; its sample and signal errors have approximately equal size, while Phase II contains only the signal component.Source: §3.4 · p. 23

4 Generation: the implicit bias of the reverse process

In the early-stopped regime, reverse diffusion avoids the localized nonlinear bumps that memorize training samples and is pathwise equivalent to a linearized process. Consequently, generated samples are approximately Gaussian, with covariance determined by the empirical covariance and the training-time schedule.

4.1 The learned reverse process and its linearized approximation

  • Conditionally on the training data, the linearized reverse process is Gaussian because it has linear drift and Gaussian initialization.Its score has the form eS_t(x) = eΨ_t x/σ_t, with eΨ_t bounded, negative semidefinite, and data-dependent.Source: §4.1 · pp. 24–25

4.2 Delocalization and pathwise linearization

  • The reverse sampler remains nearly orthogonal to every training sample, so nonlinear overfitting is invisible and the trained-score process is indistinguishable from its linear surrogate.This pathwise linearization holds with high probability under the stated kernel regularity assumptions and deterministic piecewise-continuous training schedule.Source: §4.2 · p. 25
  • The pathwise result applies when the training time satisfies sup_t t_t ≲ d^(1+c) for 0 ≤ c < 1/6, although this restriction is described as a proof artifact.The stated range already includes regimes where the training loss vanishes polynomially in d, and broader validity is expected for Phase II score estimators.Source: §4.2 · p. 25

4.3 The Gaussian generated distribution

  • The generated distribution is approximately Gaussian with limiting covariance ¯Σ_t0, obtained by composing the Marchenko–Pastur-type map from population covariance to empirical covariance with the covariance-flow ODE.Every fixed deterministic projection is asymptotically Gaussian with covariance ¯Σ_t0.Source: §4.3 · pp. 26–27
  • An early-stopping schedule that is suboptimal for the DSM objective can nevertheless produce a slightly smaller KL divergence when the reverse process is stopped early.This demonstrates that score-matching performance and generation quality need not be optimized by the same schedule.Source: §4.3 · p. 27

5 Conclusion and future directions · A Ridge-regularized denoising score matching

The paper explains how lazy denoising score matching separates generalization, overfitting, and memorization, while showing that reverse diffusion can still generate new Gaussian samples. It identifies extensions involving sample-size scaling, feature learning, structured data, coupled noise levels, and ridge-regularization regimes.

5 Conclusion and future directions

  • Lazy score matching passes through spectral generalization, pure-noise interpolation, and eventual empirical-Bayes memorization phases under gradient flow.The three phases are characterized through precise train and test-risk asymptotics in the proportional high-dimensional regime.Source: §5 · pp. 27–28
  • Overfitting does not imply memorization: interpolation occurs exponentially earlier, degrades population score estimation, yet can still yield genuinely new samples.The learned score combines a global linear component with localized training-sample corrections that reverse diffusion typically avoids, producing an effectively linear Gaussian SDE.Source: §5 · p. 28
  • Beyond n ≍ d, the authors propose a polynomial-sample hierarchy in which higher-degree kernel components add non-Gaussian corrections and higher-order moments.They identify extending delocalization arguments to higher-degree scores as a key test of lazy training’s capabilities beyond feature learning.Source: §5 · p. 28
  • At proportional sample sizes, capturing non-Gaussian structure requires feature learning or architectural inductive bias, motivating analyses that connect learned features to the reverse sampler.Such analyses could determine which non-Gaussian statistics are acquired first and their sample and optimization costs.Source: §5 · p. 28
  • Extending the theory to data supported on or near low-dimensional manifolds is necessary to characterize how support geometry changes memorization-generalization interactions.The current assumptions exclude these structured data settings.Source: §5 · p. 28
  • Practical diffusion models require coupling across noise levels because they use one time-conditioned network rather than independent RKHS models at each level.Understanding this parameter sharing is identified as an important direction for future analysis.Source: §5 · p. 28

A Ridge-regularized denoising score matching

  • The ridge-regularized estimator’s moderate-penalty, polynomially vanishing-penalty, and ridgeless regimes correspond respectively to spectral learning, training-loss interpolation, and memorization.Under calibration κ = d/t, these regimes align with the three phases described in Section 3.Source: A Ridge-regularized denoising score… · p. 37

A.1 Moderate regularization … C Linearization of the kernel and localized bumps

The appendix develops the deterministic-equivalent, concentration, moment, and resolvent tools used to analyze ridge DSM, including its homogeneous polynomial decomposition. It then examines vanishing-regularization overfitting, pure-noise behavior, and kernel linearization with localized bumps.

A.1 Moderate regularization

  • Moderate regularization yields explicit train and test-risk equivalents through kernel linearization, with nonlinear effects entering only through the shift δn in χκ.The ridge estimator is stable for κ = Ωd(d−c), while the irreducible training error is exponentially small; the bounds remain non-vacuous for κ ≥ dc with c > −1/6.Source: §A.1 · pp. 37–38

A.2 Vanishing regularization: overfitting and memorization

  • For polynomially vanishing regularization, localized bumps approximately interpolate the empirical Bayes denoiser, while the test estimator approaches −u/σ and its population risk reaches the pure-noise plateau.The ridge training error vanishes, with localized-bump interpolation error Od(d−k0+2) and RKHS norm Od(d), over the stated polynomial range of κ.Source: §A.2 · p. 38
  • The ridgeless limit agrees with infinite-time gradient flow at fixed n and d, sharing its limiting test error after the specified mild post-processing.The shared limiting test error has doubled risk relative to Theorem 3.Source: §A.2 · p. 38

B.1 Consequences of the Logarithmic Sobolev Inequality

  • The technical preliminaries derive LSI-based concentration, moment, and sample-covariance bounds for the target, noisy, and tensor-polynomial quantities used later.These results extend LSI consequences to µd and control exponential overlaps, tensor moments, and empirical tensor averages with high probability.Source: §B.1 · pp. 39–41

B.2 Deterministic equivalents for the resolvent matrix

  • The resolvent analysis constructs first- and second-order deterministic equivalents uniformly when the effective regularization is bounded below, using concentration and differentiation identities.Squared resolvents arise as derivatives of first-order equivalents, while higher-order derivative bounds support the second-order approximation.Source: §B.2 · pp. 44–45
  • The appendix develops deterministic equivalents and an outside-support anisotropic local law for the empirical resolvent under the stated assumptions.The deterministic equivalent is characterized through a fixed-point equation, with uniform high-probability control away from the limiting spectral support.Source: §B.2 · p. 47

B.3 Power series decomposition of the RKHS

  • Analytic inner-product kernels induce an orthogonal RKHS decomposition into active homogeneous-polynomial degrees, with functions represented by a linear term and symmetric higher-order tensors.The RKHS norm is expressed through the corresponding feature-space weights, and higher-order components are controlled using Hermite tensors and Wiener-chaos isometries.Source: §B.3 · pp. 47–49

C Linearization of the kernel and localized bumps

C.1 Linearization of the kernels and Bayes denoiser

The section proves that the empirical and population kernels linearize around effective kernels with small operator-norm errors, and establishes properties of the Bayes denoiser under high-dimensional concentration. It also shows that the induced regularization is non-negative and characterizes when it vanishes.

C.1 Linearization of the kernels and Bayes denoiser

  • The empirical kernel linearizes with off-diagonal and diagonal operator errors each of order d^-3/2 under the proportional high-dimensional regime.The off-diagonal and diagonal contributions are controlled separately, yielding the first linearization result with very high probability.Source: §C.1 · p. 52
  • The corresponding population kernel also differs from its effective linearized form by an operator error of order d^-3/2.A Taylor expansion around the linear term, together with moment and growth bounds, controls the population-kernel remainder uniformly over the index i.Source: §C.1 · p. 52
  • The Bayes-denoiser analysis conditions on high-probability training-data and typical-noise events, using overlap bounds and n proportional to d to control the resulting norm.The proof isolates a dominant denominator term and shows that polynomial prefactors are exponentially dominated in sufficiently large dimension.Source: §C.1 · p. 53
  • If h(0) or h′′(0) is nonzero, kernel linearization gains additional constant-function contributions that can be handled separately without changing the main results.The constant empirical-mean and constant-function components are isolated as separate contributions.Source: §C.1 · pp. 53–54
  • The self-induced regularization δn = (h(ρ2τΣ) − ρ2τΣ)/n is non-negative under the kernel assumptions.Positive definiteness implies h(u) ≥ u for positive u, which yields δn ≥ 0 at u = ρ2τΣ.Source: §C.1 · p. 54
  • Equality h(u0) = u0 at one positive point forces h(t) = t for every real t, so vanishing regularization at such a point characterizes the linear kernel.The argument uses positive definiteness and a Schur-complement limit around the equality point.Source: §C.1 · p. 54

C.2 Construction of the localized bumps · D Proofs of the kernel ridge regression results · D.1 Proof of Theorem 6

The paper constructs localized polynomial bumps that nearly interpolate the training points while controlling their RKHS norms, then proves the kernel ridge regression risk equivalents through linearization, resolvent bounds, and explicit estimator representations. Under the stated assumptions and fixed diffusion time, these arguments establish the asymptotic training and test-error formulas of Theorem 6.

C.2 Construction of the localized bumps

  • The localized bump polynomial has degree 2k0, contains no monomials below degree k0, equals one at the target point, and is flat through order k0 at both endpoints.Here k0 = 2m + 6, and the flatness properties underpin the localized construction.Source: §C.2 · p. 55
  • Because k0 ≥ 6, the localized bumps lie in H≥3, and their mean interpolation matrix is asymptotically identity with operator-norm error ≺ d^-2.The construction therefore yields an invertible expected interpolation matrix with a uniformly bounded inverse.Source: §C.2 · pp. 55–57
  • For n ≍ d, high-degree bump cross-terms vanish asymptotically, while diagonal Gram contributions remain controlled, yielding the required RKHS-norm bound.The off-diagonal overlaps decay as d^-k/2 for active degrees k ≥ 6.Source: §C.2 · pp. 57–58

D Proofs of the kernel ridge regression results

  • The proof section assumes fixed diffusion time t > 0, so σ and ρ are positive constants, and derives the asymptotic risk equivalents stated in Theorems 6 and 7.The argument proceeds through intermediate linearization and operator-control results.Source: D Proofs of the kernel ridge… · p. 59, §D.1 · p. 59

D.1 Proof of Theorem 6

  • Under Assumptions 1, 2, and 3, Lemma 14 gives ∥K∥op ∨ ∥Keff∥op ≺ d−1; with resolvent norm bounds, ∥K bG∥op ≤ ∥K∥op∥bG∥op ≺ 1/κ.Source: §D.1 · pp. 59–60
  • The linearized estimators are represented as linear functions of the data and noise, with their effective-kernel action reduced to finite-dimensional block kernel matrices and resolvents.Lemma 16 supplies the representations used to evaluate both training and test risks.Source: §D.1 · p. 61
  • Combining the linearization bounds with first- and second-order deterministic equivalents of the resolvent yields the asymptotic training and test-error expressions in Theorem 6.The proof sums coordinate-wise errors over an orthonormal basis and absorbs exponentially small terms into polynomially small remainders.Source: §D.1 · pp. 62–63
  • When h(x) = x, the effective and original kernels coincide, so the second part of Theorem 6 follows by the same linearization argument.In this case K = Keff and bK = bKeff.Source: §D.1 · p. 63

D.2 Proof of Theorem 7

The proof constructs localized RKHS bumps interpolating the empirical Bayes denoiser, then compares the ridge estimator against them. This yields training-loss and norm bounds, controls linear coefficients, and shows the nonlinear residual vanishes before averaging the risk over an orthonormal basis.

Localized bump construction

  • Theorem 7 is proved by constructing n localized bumps around the samples that interpolate the empirical Bayes denoiser with small training error and RKHS norm.The ridge estimator is compared against this localized bump function to control its risk and coefficients.Source: §D.2 · p. 63

Ridge comparison

  • Ridge optimality and the condition d−k0+2 ≲κ imply both a small training loss, Lv(bfκ,v; 0) ≺κ, and an RKHS norm bound, ∥bfκ,v∥2_H ≺d.These bounds follow because the ridge estimator’s regularized risk is no larger than that of the localized bump function.Source: §D.2 · p. 63

Component control

  • Decomposing the estimator into linear and higher-order components, the proof shows the nonlinear residual vanishes in L2(πd) on a high-probability event.The argument uses RKHS norm decomposition, Hermite coefficients, Parseval’s identity, and bounds requiring κ ≲d−ϵ.Source: §D.2 · pp. 63–64

Risk conclusion

  • After bounding the linear deviation and cross-terms using Cauchy–Schwarz and the covariance decomposition, the theorem follows by averaging expected risk over an orthonormal basis.The covariance relation used is Σσ = ρ2Σ + σ2Id.Source: §D.2 · p. 64

E Proofs of the gradient flow results

This section proves the asymptotic risk equivalents in Theorems 1 and 2 for fixed diffusion time t > 0, using arguments similar to Appendix D.

E Proofs of the gradient flow results

  • The proofs establish the asymptotic risk equivalents stated in Theorems 1 and 2 under fixed diffusion time t > 0.The arguments follow methods similar to those used in Appendix D.Source: E Proofs of the gradient flow results · p. 65

E.1 Proof of Theorem 1 · E.2 Proof of Theorem 2 · F The interpolation limit

The appendices establish Theorems 1 and 2 by linearizing the training and test errors, deriving exact spectral representations, and transferring ridge-regression bounds to gradient flow. They also set up the fixed-dimensional interpolation-limit analysis as time tends to infinity.

E.1 Proof of Theorem 1

  • Theorem 1 follows by approximating nonlinear training and test errors with linearized counterparts whose errors are uniformly controlled over time.Replacing target noise with the linear signal incurs an exponentially small error, while kernel and effective-kernel differences are bounded through operator-norm estimates.Source: §E.1 · pp. 65–66
  • The linearized errors reduce to finite-dimensional matrix expressions, enabling their asymptotic analysis through random matrix theory.Effective operators preserve linear functions, and their action is represented in two-dimensional spectral blocks of the covariance matrix.Source: §E.1 · pp. 66–68
  • The spectral profiles and linearized denoiser admit explicit eigenvalue-based formulas, including continuous extensions when the two block eigenvalues coincide.The profiles are expressed using the eigenvalues γ+(x) and γ−(x) of the two-dimensional blocks.Source: §E.1 · p. 68
  • Theorem 1 is completed by combining the linearization and spectral bounds to obtain high-probability approximation guarantees for training and test risks.The stated bounds hold with probability at least 1 − C d^-D under the appendix assumptions.Source: §E.1 · pp. 68, 71

E.2 Proof of Theorem 2

  • Lemma 21 compares gradient flow with ridge regression by setting the ridge parameter to κ = d/t and comparing their spectral shrinkage and norms.The comparison uses the empirical-kernel eigendecomposition and weighted spectral measure, with inequalities relating exponential and ridge filters.Source: §E.2 · pp. 71–72
  • Theorem 2 follows because gradient flow inherits the ridge estimator’s training-loss and RKHS-norm bounds under the parameter correspondence κ = d/t.After applying Lemma 21, the remainder of the proof proceeds identically to the ridge-based argument for Theorem 7.Source: §E.2 · p. 72

F The interpolation limit

  • The interpolation-limit appendix studies the gradient flow estimator with fixed dimension and sample size as gradient-flow time tends to infinity, while noting that analogous arguments apply to ridge regression.The diffusion time remains fixed and positive, so ρ and σ stay between zero and one.Source: F The interpolation limit · p. 73

F.1 Convergence of the test error under truncation

The proof establishes RKHS density under the relevant Gaussian-smoothed measures, then shows gradient flow converges to the empirical Bayes denoiser while test error is asymptotically bounded below by Bayes risk. After truncation onto a compact convex set, the test error converges to the empirical Bayes denoiser’s risk.

Density of the RKHS

  • The RKHS is dense in L2(bµd,n) and L2(µd), using kernel universality and finite exponential moments of both measures.The argument embeds weighted polynomials in the RKHS and proves polynomial density through Laplace-transform uniqueness.Source: §F.1 · p. 73

Gradient flow limit

  • As t →∞, the gradient flow estimator converges to the empirical Bayes denoiser in L2(bπd,n), while test error is asymptotically lower bounded by Bayes risk R( b f ⋆).The convergence follows from spectral projection onto the closure of the kernel operator’s range and the RKHS density result.Source: §F.1 · pp. 73–74

Truncated estimator

  • The truncated gradient flow estimator satisfies lim t→∞R(ΠA b ft) = R( b f ⋆).Projection non-expansiveness and boundedness on the compact convex set A yield convergence in L2(µd), implying convergence of test error.Source: §F.1 · pp. 74–75

F.2 Instability of the test error at the ridgeless limit

For a Gaussian population and a universal inner-product kernel, gradient flow can approach increasingly high-degree polynomial projections while the ridgeless-limit risk becomes unbounded. The instability follows from the mismatch between finite-sample interpolation and the non-polynomial target’s analytic structure.

Analytic obstruction

  • Polynomial projections converge locally uniformly to an entire function and in L2 to the target, forcing equality everywhere under the positive Gaussian-mixture density.The analytic continuation argument then shows the target cannot have the required finite-frequency exponential-sum structure without contradiction.Source: §F.2 · p. 78

Kernel spectrum

  • The truncated empirical kernel operator spans exactly the degree-N polynomials, but its smallest positive eigenvalue decays exponentially in N up to constants.Lemma 27 establishes both the range identity Ran(K̂_0,≤N) = P_N and matching exponential upper and lower bounds for λ_N.Source: §F.2 · pp. 77, 79, 80

Gradient-flow approximation

  • At times t_N = λ_N^-2, gradient flow shadows the degree-N polynomial projection of its target, with t_N →∞ as N increases.The smallest positive eigenvalue λ_N tends to zero, while the corresponding truncated kernel operator has range equal to the degree-N polynomial space.Source: §F.2 · pp. 80–81

Main result

  • The main result shows that, for a universal inner-product kernel, gradient-flow risk can be unbounded at the ridgeless limit almost surely.The construction uses π0 = γ1 and a kernel whose Taylor coefficients are all strictly positive, satisfying the required universality assumption.Source: §F.2 · p. 82

F.3 Test error of the empirical Bayes denoiser · G Reverse process dynamics

The empirical Bayes denoiser concentrates its weights on a single training point, yielding the theorem’s high-dimensional test-risk conclusion and matching the infinite-time gradient-flow limit. The reverse-process analysis then studies a fixed-time schedule and its linearization.

F.3 Test error of the empirical Bayes denoiser

  • In analyzing the population risk of the empirical Bayes denoiser (11), the winner-take-all lemma shows that, under Assumptions 1 and 2, its weights are asymptotically concentrated on a single training point.Source: §F.3 · p. 82
  • The winner-take-all proof relies on anticoncentration and Gaussian maximal inequalities to separate the largest empirical Bayes weight from the others.The bounds are uniform in the test input and use n ≍ d with ρ/σ bounded below.Source: §F.3 · pp. 83–84
  • The resulting empirical Bayes denoiser has the theorem’s stated high-dimensional population-risk behavior, with its leading contribution asymptotically equal to τΣ and the cross term negligible.The argument uses a fresh test pair, concentration, maximal inequalities, Tr(Σ) ≲ d, and n ≍ d.Source: §F.3 · pp. 84–85
  • The empirical Bayes estimator’s risk agrees asymptotically with the long-time gradient-flow risk under Assumption 5.For fixed d, n, and training realization, lim t→∞ R(e f_t) = R(b f⋆), and this identity combines with the high-dimensional asymptotic.Source: §F.3 · p. 85

G Reverse process dynamics

  • The reverse-process section analyzes the dynamics and linearization of the reverse diffusion process over a fixed interval [t0, T].The noise schedule t ↦ t_t is deterministic and piecewise continuous, with 0 < t0 < T fixed.Source: G Reverse process dynamics · p. 86
  • The reverse-process setup assumes that the process ext solves equation (33) under the specified deterministic schedule.This defines the process studied throughout the section.Source: G Reverse process dynamics · p. 86

G.1 Properties of the linearized reverse process · G.2 Uniform linearization of the empirical kernel operator

The linearized reverse process admits a finite-dimensional representation that makes it conditionally Gaussian and enables uniform boundedness and delocalization results. Along its random delocalized path, the empirical kernel and feature map can be uniformly linearized, with sample-dependent diagonal corrections essential for the sharpest remainder bound.

G.1 Properties of the linearized reverse process

  • The linearized kernel preserves an (n+d)-dimensional subspace and has an explicit finite-dimensional matrix representation despite sample-dependent diagonal corrections.Unlike the common-correction operator, the sample-corrected operator is not a spectral function of the empirical covariance, but it preserves the larger invariant subspace eV.Source: §G.1 · p. 86
  • Conditionally on the training data, the linearized reverse process is a centered Gaussian process with covariance uniformly bounded in diffusion time and dimension.The finite-dimensional representation yields the Gaussian-process characterization, while the covariance bound holds uniformly in t and d.Source: §G.1 · pp. 87–88
  • Under Assumptions 1, 2, and 6, the reverse process is uniformly bounded and delocalized, including its norm and overlaps with every training sample.The bounds are established conditionally on the training data for every fixed moment, with the overlap and norm controls obtained through Gaussian-process chaining and Borell–TIS arguments.Source: §G.1 · pp. 87, 89

G.2 Uniform linearization of the empirical kernel operator

  • Along the delocalized reverse path, the nonlinear feature map is well-approximated by its linearized feature map under Assumptions 1, 2, and 6.The approximation is uniform along the random path generated by the reverse process and is controlled using Taylor expansions together with exponential-moment bounds.Source: §G.2 · pp. 89–90
  • The empirical kernel operator admits uniform-in-time bounds for both off-diagonal and diagonal linearization remainders.The off-diagonal remainder is strong enough for subsequent analysis, whereas the diagonal operator-norm bound requires additional structure on the invariant subspace eV.Source: §G.2 · pp. 91–92
  • Uniformly over t ∈[t0, T], the sample-dependent correction cancels the extra term and yields a d−2 bound, whereas the common correction yields only d−3/2.Source: §G.2 · pp. 93–94

G.3 Proof of Theorem 4

The proof controls the empirical score’s deviation from its linearized approximation uniformly in time, then transfers this control to the reverse-SDE path laws through stopping, Girsanov’s theorem, and lower semicontinuity of relative entropy.

Proof of Theorem 4

  • A uniform-in-time score-linearization bound along the linearized reverse process controls the KL divergence between the empirical and linearized reverse-SDE path laws.The argument combines the score decomposition with operator-norm, contraction, Bessel, and variation-of-parameters estimates.Source: §G.3 · pp. 94, 96
  • Under the stated assumptions, the linearization estimate holds with very high probability uniformly over t ∈ [t0,T] and unit vectors v.The proof concludes the estimate using positivity and operator-norm bounds together with a lower bound on σ_t.Source: §G.3 · pp. 94, 96
  • The linearized reverse SDE has a unique global strong solution, while the empirical reverse SDE is handled locally and its explosion behavior is controlled by stopping-time estimates.The linearized drift is globally Lipschitz with linear growth; for the empirical process, Gronwall’s inequality rules out explosion when the relevant energy integral is finite.Source: §G.3 · pp. 96–97
  • After stopping both processes, Girsanov’s theorem bounds the relative entropy of their path laws, and passage to the limit uses lower semicontinuity of relative entropy.The stopping times converge to the full interval for the linearized process and to the explosion time or terminal time for the empirical process.Source: §G.3 · p. 97

G.4 Proof of Theorem 5

Theorem 5 is proved by comparing the linearized reverse process with an auxiliary spectral linearized process and then applying a deterministic-equivalent analysis to its covariance. The argument yields the theorem’s covariance control under Assumptions 1, 2, and 6, uniformly along the reverse-time schedule with high probability.

Auxiliary spectral process

  • The proof introduces a spectral linearized process by replacing the kernel dynamics with the time-dependent effective empirical operator, yielding an explicit linear shrinkage field.This auxiliary process is designed to approximate the original linearized reverse process while retaining a spectral representation.Source: §G.4 · p. 97

Covariance comparison

  • The covariance of the linearized reverse process eΣt is close to that of the spectral linearized process ˇΣt.The comparison uses the covariance ODEs, their difference, integration with a common terminal condition, operator-norm bounds, and backward Gronwall’s inequality; the operator-norm difference is also bounded by 1.Source: §G.4 · pp. 97–99

Deterministic equivalent

  • The spectral covariance ODE diagonalizes in the eigenbasis of the population covariance, enabling random-matrix tools to establish a deterministic equivalent.The deterministic-equivalent statement holds for every deterministic positive semidefinite test matrix with nuclear norm at most 1, with probability at least 1 − C d^-D.Source: §G.4 · pp. 99–100

Conclusion of the proof

  • Theorem 5 follows by combining the covariance comparison from Lemma 36 with the deterministic-equivalent estimate from Lemma 37.Uniformity over the reverse-time schedule is verified through contour bounds and continuity conditions under Assumption 6.Source: §G.4 · pp. 99–100

Related papers

Loading 2608.23938v1…