PaperlayerView exact revision on arXiv

Source-linked AI summary

First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Raphaël Urfin, Tony Bonnaire, Giulio Biroli, Marc Mézard

arXiv:2609.35377v1cs.LGcond-mat.dis-nn

TL;DR

The paper asks how diffusion models trained on finite datasets can generate novel samples before collapsing onto their training data. It analyzes NTK Gram-matrix spectra and finds that repeated noise realizations create a distinct memorization bulk, separating generalization and memorization timescales.

  • Problem

    How highly overparameterized score networks generate novel, high-quality samples without simply memorizing a finite training set remains unclear.

    Source: §1 · p. 1

  • Method

    The paper analyzes the NTK Gram-matrix spectrum for noisy training data with repeated noise realizations and studies linear and polynomial sample-complexity regimes.

    Source: §1 · pp. 2–3

  • Results

    The spectrum separates into a large-eigenvalue generalization bulk and a small-eigenvalue memorization bulk, with the latter producing a parametrically longer memorization timescale and persisting in tested realistic settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Repeated noise realizations create sample-specific memorization modes, and targeted L2 regularization of that spectral part suppresses memorization in feature-learning U-Nets.

    Source: §1 · p. 3

    The analysis assumes the lazy regime, fixed diffusion time t, and finitely many noise realizations frozen during training; rigorous treatment of feature learning remains open.

    Source: §4 · p. 9

Abstract

from arXiv · show

Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed---exactly, and at any width---by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample ($m$ noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for $m=1$. The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size $n$. We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear ($n \asymp d$) and polynomial ($n \asymp d^k$) sample complexities, and prove through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank $r$ tunes the generalization--memorization transition, and an $L_2$ penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.

1 Introduction

The paper explains how diffusion models can learn global structure before memorizing finite training data by tracing training dynamics to the NTK Gram-matrix spectrum. Repeated noise realizations create a low-eigenvalue memorization bulk, separating its timescale from generalization; the paper analyzes and tests this account.

1 Introduction

  • The analysis fixes diffusion time t, a standard theoretical simplification, and studies the memorization transition as a small-t phenomenon.The paper reports a sweep over t in Section 3.Source: §1 · p. 2
  • The NTK Gram matrix on noisy training data governs score-function dynamics, making its spectrum the object that reveals the separation between generalization and memorization timescales.The NTK evaluated on training-point pairs forms the Gram matrix that closes the training-set dynamics.Source: §1 · pp. 1, 3
  • Repeated noise realizations create a memorization bulk of small eigenvalues, distinct from the large-eigenvalue generalization bulk already present at m = 1.The two bulks encode sample-specific noise directions and global features of the target distribution, respectively.Source: §1 · p. 3
  • In lazy high-dimensional settings with linear and polynomial sample complexity, the spectral hierarchy associates generalization with learning global features and delays memorization parametrically in training-set size n.The paper characterizes both sample-complexity regimes analytically.Source: §1 · p. 3
  • For m > 1, the memorization bulk contributes Θ(1) bias, unlike the vanishing bias in the supervised-learning case m = 1.This bias–variance result is derived in the linear regime.Source: §1 · p. 3
  • On CelebA Convolutional NTKs and finite-width U-Nets, the two-part spectrum persists beyond the lazy regime, while rank truncation tunes the transition and targeted L2 regularization suppresses memorization.The authors report the regularization result for feature-learning U-Nets.Source: §1 · p. 3

2 Analytical results

In the lazy high-dimensional limit, the Gram spectrum separates modes that learn population structure from modes associated with sample-specific noise. This separation explains the distinct learning timescales and persists across linear and polynomial sample-complexity regimes.

2.1 Linear regime: structure of the Gram matrix

  • In the linear regime, a mode of eigenvalue λ is learned on timescale τλ ∼ nm/(dλ), so the spectral gap separates learning timescales.The analysis takes n, d → ∞ at fixed ψn = n/d, with m = O(1), and uses a high-dimensional lazy-kernel setting.Source: §2 · p. 4, §2.1 · p. 4
  • Repeated noise realizations add an intra-sample correlation term that separates the Gram spectrum into a generalization bulk and a memorization bulk.The generalization bulk is already present at m = 1, whereas the memorization bulk emerges for m > 1.Source: §2.1 · pp. 4–5
  • The generalization modes align with data-covariance directions, while memorization modes align with noise components distinguishing repeated views of each sample.Their eigenvalue scales are Θ(nm/d) and Θ(m), respectively, producing a memorization timescale parametrically larger in n.Source: §2.1 · p. 5

2.2 Linear regime: bias–variance decomposition

  • At the ridgeless limit, the test bias persists and diverges as t → 0; optimal early stopping makes bias and variance vanish with sample complexity at fixed t.When the ridge falls to O(1), it no longer damps memorization modes, which source the persistent bias.Source: §2.2 · p. 6

2.3 Polynomial scaling

  • For dk ≪ n ≪ dk+1, each degree p has Θ(dp/p!) generalization eigenvalues of scale Θ(nm/dp), alongside a memorization bulk of Θ(dk/k!) eigenvalues of scale Θ(m).The memorization bulk is absent at m = 1.Source: §2.3 · p. 6
  • Polynomial-regime generalization sub-bulks are learned degree by degree on τp ∼ dp−1, while memorization occurs on τmem ∼ n/d.At any finite k, memorization and overfitting are partial; full reproduction of training samples requires n to grow faster than every fixed power of d.Source: §2.3 · p. 6

3 Experiments

CelebA experiments reveal a two-bulk Gram spectrum whose truncation controls the generalization–memorization transition, and the spectral structure and regularization effect persist in a feature-learning U-Net.

3 Experiments

  • Experiments use grayscale 32 × 32 CelebA images, a flow-matching interpolant, and a CNN with a closed-form infinite-width CNTK; a finite-width U-Net tests feature learning.Truncated top-r Gram modes define the velocity-field estimator; FID and a nearest-neighbor criterion assess image quality and memorization.Source: §3 · p. 7

3.1 Lazy regime: Gram matrix for real data

  • CelebA’s CNTK spectrum has a generalization bulk scaling with n and a second component independent of n, although the bulks may merge for power-law image statistics.The generalization bulk contains n eigenvalues, rather than the d predicted for sub-Gaussian data in the proportional regime.Source: §3.1 · pp. 7–8
  • The generalization threshold is essentially independent of n, while the memorization threshold scales linearly with n; rescaling rank by n collapses the fmem curves.FID, fmem, and train/test losses jointly track the transition as truncation rank increases.Source: Figure 4 · p. 8, §3.1 · p. 8

3.2 Feature-learning regime and training-time-dependent NTK

  • 0 fmem over 10,000 training steps with dynamically updated n-th-eigenvalue regularization, versus ≈0.5 unregularized, without sacrificing test loss.The regularization progressively smooths the second spectral bulk.Source: Figure 5 · p. 8, §3.2 · p. 9
  • In the feature-learning U-Net, the two-bulk spectrum and gap near r ≈ n persist during training, including as fmem rises from 0 to 0.86.Eigenvalue amplitudes shift non-monotonically, while the spectral gap sharpens for τ > 0.Source: Figure 5 · p. 8, §3.2 · p. 9

4 Conclusion and limitations

The paper concludes that repeated noise realizations create a memorization bulk distinct from the generalization bulk, with evidence extending to feature-learning U-Nets. Its analytical conclusions remain bounded by lazy-regime assumptions, and rigorous analysis beyond that regime remains open.

4 Conclusion and limitations

  • The Gram spectrum separates generalization and memorization: repeated noise realizations create the memorization bulk, which contributes a Θ(1) bias absent in supervised learning.The generalization bulk yields bias and variance that vanish with sample complexity; the two bulks also appear in CelebA CNTKs and finite-width feature-learning U-Nets.Source: §4 · p. 9
  • The analysis assumes a lazy regime, fixed diffusion time, and finitely many noise realizations frozen during training; rigorous treatment of feature learning remains open.The gap between predicted and observed numbers of generalization eigenvalues also motivates more realistic data models.Source: §4 · p. 9

Models

The appendix reviews how NTK dynamics reduce to kernel flow in the lazy regime, then gives kernel constructions and regularization results used in the analysis. It also describes the fully connected and convolutional kernel recursions.

A.1 Introduction; A.2 Lazy regime and linearization; A.3 Vector-valued outputs and decomposable kernels

  • In the infinite-width lazy regime, the NTK becomes deterministic and constant, and training is equivalent to linear regression on frozen initialization features.For score matching, the vector-valued output kernel can factor into a scalar kernel and the identity, reducing spectral analysis to the scalar kernel.Source: §A.1 · p. 16, §A.2 · p. 16, §A.3 · pp. 16–17

A.4 Kernel gradient flow

  • The training-set score dynamics close through the NTK Gram matrix, whose gradient-flow spectral filter learns larger-eigenvalue modes before smaller ones.A mode of eigenvalue λ is learned on timescale τλ ∼ nm/(dλ); finite training time acts approximately as a hard spectral cutoff.Source: §A.4 · pp. 17–18

A.4.1 The role of regularization

  • An L2 penalty shifts the Gram matrix by γI, yielding kernel ridge regression at convergence and suppressing low-eigenvalue modes between the memorization and generalization bulks.The cited regularization window is m ≪ γ ≪ ψnm.Source: §A.4.1 · p. 18

A.5 Two-layer fully-connected network

  • For two-layer fully connected networks, the infinite-width NTK combines the conjugate kernel with the first-layer tangent kernel and becomes a dot-product kernel for inputs of norm squared d.For ReLU, the associated Gaussian integrals have a closed form.Source: §A.5 · pp. 18–19

A.6 Multilayer fully-connected network

  • For multilayer fully connected networks, coupled layerwise covariance and tangent-kernel recursions determine the conjugate kernel and NTK.Homogeneous activations such as ReLU yield closed-form recursive expressions.Source: §A.6 · p. 19

A.7 Convolutional neural networks

  • For convolutional networks, covariance and NTK tensors evolve through layerwise recursions, with patch averaging preventing the NTK recursion from unrolling into a product of layerwise factors.The scalar NTK follows by applying the readout to the final-layer tensor.Source: §A.7 · p. 20

A.8 Kernels used in the numerical experiments

The numerical experiments use specified two-layer, convolutional, and finite-width U-Net kernels. The appendix relates the OU and rectified-flow formulations by time reparametrization and input scaling, and connects their score and velocity targets by an affine map.

A.8 Kernels used in the numerical experiments

  • CelebA uses a closed-form infinite-width convolutional NTK; feature-learning experiments use a finite-width U-Net empirical NTK, while analytical experiments also use two-layer ReLU and dual tanh kernels.For the two-layer ReLU kernel, the activation is not odd and a rank-one outlier is removed; the tanh activation is odd and has μ0 = 0.Source: §A.8 · pp. 20–21

B.1 Time reparametrization; B.2 Substitution rule

  • OU and RF interpolate between data and a standard Gaussian using different parameterizations; they are related by a time change and global input rescaling, and RF results at s correspond to OU at t(s) with kernel f_s.Source: §B.1 · p. 21, §B.2 · p. 21

B.3 Relation between score and velocity field

  • The OU score target and rectified-flow velocity target are related by a one-to-one affine map, so the Gram-matrix spectral analysis controls both training dynamics.Source: §B.3 · p. 22

C.1 Effect of the architecture and training dynamics

The paper situates its analysis among work on sequential learning and diffusion-model generalization, then contrasts its finite-noise Gram-matrix setting with related studies. Its high-dimensional analysis assumes sub-Gaussian data and smooth inner-product kernels, with additional structure required for polynomial sample complexity.

C.1 Effect of the architecture and training dynamics

  • Prior work describes sequential learning of data-distribution statistics in linear, two-layer, and random-features models, including learning timescales in Bonnaire et al. [15].For quadratic loss, their parameter-space Hessian and the sample-space Gram matrix share the same non-zero spectrum up to normalization.Source: §C.1 · p. 22, §C.2 · p. 22

C.2 Discussion of Bonnaire et al. [15]

  • The present analysis reads the generalization–memorization separation from the Gram spectrum, whose two bulks imply τmem/τgen ∼ ψn.Bonnaire et al. [15] and Favero et al. had identified the long generalization window; the present work studies the Gram matrix directly.Source: §C.2 · pp. 22–23

C.3 Concurrent work: Latourelle-Vigeant et al. [48]

  • Unlike Latourelle-Vigeant et al. [48], this work studies finite noise repetitions and extends the spectral picture to n ≍ d^k, where generalization splits into k sub-bulks while memorization remains Θ(m).Their analysis uses m = ∞ and an infinite-dimensional kernel operator, and is confined to n ≍ d.Source: §C.3 · p. 23

C.4 Discussion of Han et al. [36]

  • Han et al. [36] analyze a two-layer ReLU network across noise levels and derive a minimax-optimal generalization bound with early stopping, whereas this work studies a fixed noise level and Gram spectrum.Their sampling uses each clean sample once, corresponding here to m = 1; their m denotes network width, not noise repetitions.Source: §C.4 · p. 23

D.1 Assumptions

  • The analysis assumes x = Σ^1/2z with independent, zero-mean, unit-variance sub-Gaussian entries and studies smooth inner-product kernels K(x,y) = f(x⊤y/d).For polynomial sample complexity, it additionally assumes a diagonal representation in an orthogonal-polynomial basis; examples include spherical dot-product kernels and asymptotically degree-diagonal isotropic Gaussian kernels.Source: §D.1 · pp. 23–24

D.2 Notation

  • The notation defines N = nm, the noise-level covariance Σt = e^-2tΣ + ∆tId, and sample complexity ψn = n/d.It also defines λt = e^-2tλ + ∆t and the rescaled ridge ˜γ = γ/(mψn).Source: §D.2 · p. 24

D.3 Proof of the linear equivalent of the Gram matrix

The proof replaces the nonlinear Gram matrix with a linear equivalent whose spectrum approaches the original in the high-dimensional proportional regime. It controls within-cluster, diagonal, and across-cluster contributions and bounds the residual in operator norm.

D.3 Proof of the linear equivalent of the Gram matrix

  • For n/d → ψn and fixed m = O(1), the Gram matrix is asymptotically equivalent in spectrum to Glin.Every eigenvalue differs from its corresponding linear-equivalent eigenvalue by o(1), and isolated eigenvalues separated by a fixed gap are tracked individually.Source: §D.3 · pp. 24–25
  • The proof separates diagonal, within-cluster, and across-cluster overlaps, expanding the kernel around their respective means to form the linear equivalent.The resulting coefficients include µI, µB, µ1, and µ0.Source: §D.3 · pp. 24–25
  • The residual terms are controlled in operator norm, including the centered quadratic-overlap matrix, whose operator norm is o(1).The proof concludes by summing the four contributions and applying Weyl’s inequality.Source: §D.3 · p. 25

D.4 Spectrum of the Gram matrix

The appendix derives the Gram-matrix spectral distribution through self-consistent equations for its Stieltjes transform and characterizes distinct spectral components in the large-data regime. The replica calculation uses a replica-symmetry assumption, while the accompanying figures compare analytical predictions with finite-dimensional spectra.

D.5 Replica computation of the Stieltjes transform of the Gram matrix

  • The spectral distribution is characterized through self-consistent equations for the Stieltjes transform and an auxiliary order parameter.The replica calculation relies on replica symmetry, which the text notes is known to give exact results for traces of rational functions of random matrices.Source: §D.4 · pp. 25–26, §D.5 · p. 27

D.6 Asymptotic spectrum of the Gram matrix

  • The appendix presents an empirical eigenvalue histogram alongside an analytical prediction and compares linear-equivalent with nonlinear NTK spectra in finite dimension.Figure 6 uses d = 100 and m = 5; Figure 7 uses d = 128 and m = 4.Source: Figure 6 · p. 26, Figure 7 · p. 26

D.4 Spectrum of the Gram matrix

  • In the large-data regime, the spectral density decomposes into four components, with repeated-noise eigenvalues in a memorization bulk scaling as Θ(m).For each eigenvalue λ of Σ, the theorem identifies associated Gram-matrix eigenvalues using λt = e^-2tλ + ∆t.Source: §D.4 · pp. 26–27

D.6.1 Derivation from the replica equations

The replica-equation derivation resolves the spectrum into a null-space peak and two nonzero bulks. The memorization bulk has Θ(m) eigenvalues, whereas the generalization bulk has Θ(ψnm) eigenvalues.

D.6.1 Derivation from the replica equations

  • The spectrum contains a zero-eigenvalue peak with nm − n − d eigenvalues, corresponding to the finite-rank null space of G.The derivation locates the memorization density through the pole of the Stieltjes-transform integrand.Source: §D.6.1 · p. 31
  • A bulk at Θ(m) contains d eigenvalues, with its density given by the pushforward of ρΣ under the memorization-eigenvalue map.The derived location is ω = (m − 1)µB∆t/λt when µI = 0.Source: §D.6.1 · p. 31
  • A bulk at Θ(ψnm) contains d eigenvalues and matches the spectral density of the population kernel operator for the noisy distribution Pt.The derivation matches this component to a Bai–Silverstein equation.Source: §D.6.1 · pp. 30, 32

D.6.2 Derivation with linear algebra arguments

The derivation reduces the linearized Gram matrix to small subspaces associated with covariance eigenmodes, yielding distinct generalization and memorization eigenvectors. It also identifies spectral atoms that do not contribute to the estimated score.

D.6.2 Derivation with linear algebra arguments

  • For each covariance eigenmode, the Gram matrix reduces to a coupled subspace whose eigenvectors separate into generalization and memorization modes.The derivation also identifies an orthogonal direction with an unshifted eigenvalue and two atoms that do not contribute to the estimated score.Source: §D.6.2 · pp. 32–34
  • The nontrivial eigenvalues arise from a 2 × 2 matrix for each covariance eigenmode, producing separate generalization and memorization eigenvalues.The derivation obtains the two eigenvalues from the characteristic equation and expands them in the high-dimensional limit.Source: §D.6.2 · p. 33
  • The large spectral bulk scales with ψn = n/d and depends on the signal covariance, while the smaller memorization bulk is mapped to the covariance spectrum.The calculation assumes d ≫ 1 and ψn ≫ 1 with finite m.Source: §D.6.2 · pp. 32–34
  • The generalization eigenvectors align with signal components, while memorization eigenvectors involve noise components distinguishing repeated realizations of the same sample.The linearized-kernel atoms contribute zero to the estimated score for every training time and ridge.Source: §D.6.2 · pp. 34–35
  • For the linearized kernel, both identified atoms contribute zero to the score estimator at every training time and ridge.The nonlinear remainder is controlled separately and vanishes as d grows at fixed ψn and m under the stated decorrelation condition.Source: §D.6.2 · p. 35

D.7 Derivation of the spectrum in the polynomial scaling

In the polynomial scaling, the Gram spectrum is organized by polynomial degree, with degree-dependent eigenvalue groups and additional spikes and atoms. The following bias–variance decomposition setup defines the test error relative to the exact score and separates its mean and fluctuation contributions.

D.7 Derivation of the spectrum in the polynomial scaling

  • The polynomial basis is constructed by orthogonalizing monomials, and under the stated density condition it forms an orthogonal basis of L2(P).For Gaussian data the family reduces to products of one-dimensional Hermite polynomials; for sphere-uniform data it reduces to spherical harmonics.Source: §D.7 · p. 36
  • For dk+1 ≫ n ≫ dk with m = O(1), each degree p from 1 to k contributes Θ(d^p/p!) eigenvalues of order nm/d^p.Their eigenvectors are the corresponding orthogonal polynomial features.Source: §D.7 · p. 38
  • The polynomial-regime spectrum also contains a spike at mµ>k B with n − dk/k! eigenvalues and an atom at µ>k I with nm − n − dk/k! eigenvalues.These components supplement the degree-indexed eigenvalue groups.Source: §D.7 · p. 38

D.8 Bias–variance decomposition

  • The bias–variance decomposition defines the test score error against the exact score, then separates the estimator’s mean error from its fluctuation.The cross term vanishes conditionally on the test point, and the exact-score test loss is the Bayes error of denoising at noise level t.Source: §D.8 · p. 39

D.9 Derivation of Theorem 2.2: fixed-point equations in the kernel limit

The derivation reduces the isotropic Gaussian kernel predictor to a linear map and computes its bias and variance through a fixed-point equation. In the ridgeless limit, the equations simplify, with separate channels for the average and transverse directions of the repeated noisy copies.

D.9 Derivation of Theorem 2.2: fixed-point equations in the kernel limit

  • For m = 1, only terms with a = 1 remain; a = 1 refers to the average of the m noisy copies of a clean sample, and a = 2 to the m − 1 directions orthogonal to it.Source: §D.9 · p. 40
  • Under isotropic Gaussian data, the derivation reduces the predictor to a linear map and expresses bias and variance through two channels of the noisy training data.The channels correspond to the block-average directions and the directions transverse to them.Source: §D.9 · pp. 40–41
  • The fixed-point equation determines the resolvent parameter used to compute the predictor’s moments and hence its bias and variance.The resolvent calculation uses the two channel weights and kernel constants.Source: §D.9 · pp. 40–42
  • In the ridgeless limit, the equations use a finite parameter r satisfying 1/r = 1 + R0(r), and the resulting expressions do not depend on the gauge ϱ.The ridgeless derivation takes the limit ˜γ → 0+.Source: §D.9 · pp. 40, 42

D.10 Numerical check of Theorem 2.2

This section numerically examines the ridge–early-stopping correspondence, bias and variance as noise vanishes, and the spectral structure of random-feature Gram matrices. It also studies how repeated noise copies and spectral truncation relate to memorization, measured by generated samples’ proximity to training data.

D.10 Numerical check of Theorem 2.2

  • At ˜γ = 1/τ, ridge predictions of Ltest, B2, and V track finite-dimensional gradient flow, making ridge a faithful proxy for early stopping and its timescales.The comparison uses d = 256 and ψn = 8, with 16 dataset realizations for m = 1 and 8 for m = 4.Source: §D.10 · p. 44, Figure 10 · p. 44

D.11 Small-t behavior of the bias and the variance

  • As ∆t vanishes, the memorization eigenvalue vanishes linearly while λgen remains Θ(ψnm), separating the two spectral bulks and slowing and ill-conditioning memorization modes.This small-noise behavior is the setting for the reported bias and variance scalings.Source: §D.11 · p. 45
  • At m = 1, B2 is independent of t and ψn∆tV → 1 as ψn → ∞; at optimal ridge and fixed s, B2 = 1/(1 + s)^2 and V = s/(1 + s)^2.Here s = mψn∆t, and the optimal ridge is ˜γ* = µ1/s; the combined error is B2 + V = 1/(1 + s).Source: §D.11 · p. 45
  • For m > 1 in the window 1/(mψn) ≪ t ≪ 1, s ≫ 1; when s ≪ 1, the optimally ridged estimator has B2 → 1 and V ≃ s.In the latter regime it predicts almost nothing.Source: §D.11 · p. 45
  • In the ∆t → 0, fixed-s limit, the whole ridge curve depends on ψn, m, and t through s alone, while B2 does not depend on s.The optimal ridge is ˜γ* = µ1/s.Source: §D.11 · p. 46

D.12 Link between kernels and random features

  • For p ≫ n ≍ d ≫ 1, the random-feature Gram matrix has the kernel’s linear equivalent, and its infinite-width limit is consistent with the associated kernel Gram matrix.The result keeps m = O(1), unlike Bonnaire et al. [15], who take m → ∞ first.Source: §D.12 · pp. 46, 48, 49
  • The feature-space decomposition yields d large, fast eigenvalues above a plateau of p − d slow eigenvalues pinned at µI + µB, reproducing the two-group spectrum.In the proportional regime p ≍ n ≍ d, the Gaussian Equivalence Principle supports replacing random features by a centered Gaussian matrix with matching covariance.Source: §D.12 · pp. 46, 49
  • Averaging over noise copies before forming the matrix removes the cross-copy-correlation-dependent structure that supports the memorization bulk.At m = 1, the Gram matrix has the m = ∞ Hessian’s structure; for m > 1, the plateau resolves into atoms at µI and µI + mµB and a memorization bulk.Source: §D.12 · pp. 49–50

E.1 General comments

  • Generated samples are produced by backward Euler integration of the estimated velocity field, and fmem is the fraction of 2,048 samples meeting the nearest-neighbor memorization criterion.The measure is evaluated on trajectories initialized from training noises; the passage characterizes correct population-velocity learning as fmem = 0 and memorization as fmem = 1.Source: §E.1 · p. 50

E.2 Closed-form CNTK computation and top-r eigendecomposition

  • At fixed n = 256, the first CNTK spectral bulk grows linearly with m, the second grows sub-linearly, and the memorization bulk is absent at m = 1.Figure 12 shows spectra for m from 1 to 64, averaged over 10 training-set realizations at s = 0.05.Source: Figure 12 · p. 51, §E.2 · p. 51
  • Truncating below the r-th eigenvalue acts like stopping gradient flow at training time τ ∼ nm/(dλr), making rank a proxy for the generalization-to-memorization transition.Truncation, gradient flow, and ridge differ in filter shape, but activate modes in the same order.Source: §E.2 · p. 51

E.3 U-Net training and empirical NTK on a finite-width network

This section evaluates empirical NTK estimation and spectrum-aware regularization in finite-width U-Nets, then reports computational resources and LLM use. The regularized training run avoids the late test-loss increase seen without regularization.

E.3 U-Net training and empirical NTK on a finite-width network

  • The CelebA U-Net uses three residual blocks, base width 32, and eight noise realizations per image, trained for 60,000 steps.Training uses the rectified-flow objective with Adam at learning rate 10^-4.Source: §E.3 · p. 51
  • The empirical NTK estimator avoids forming the full Jacobian by using Hutchinson-style projections and chunked kernel contractions.Gradients are computed with a single reverse-mode pass and batched over data points.Source: §E.3 · p. 51
  • With spectrum-aware regularization, the U-Net stays near its optimal test loss, while unregularized training shows a sharp late increase.Figure 13 compares training and test losses for the two dynamics; the unregularized increase occurs alongside a steep rise in memorization.Source: §E.3 · p. 52, Figure 13 · p. 52
  • A spectrum-aware Tikhonov penalty tracks the n-th empirical NTK eigenvalue and damps modes below it, with the eigendecomposition refreshed every 1,000 steps.In the linearization around initialization, the penalty corresponds to kernel ridge regression.Source: §E.3 · p. 52

E.4 Computational resources

  • The n = 2,048 truncation experiment takes about four hours on four NVIDIA H100 GPUs.Runs with n < 2,048 use one GPU, while runs with n ≥ 2,048 use four GPUs in parallel.Source: §E.4 · p. 52

F LLM usage

  • LLMs assisted with coding, copy-editing, proof checking, and numerical-verification scripts; authors independently checked derivations and reviewed edits.The authors take responsibility for the work, including any remaining errors.Source: F LLM usage · pp. 52–53
Loading 2609.35377v1…