The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization

1. Introduction

• The NTK test error can be non-monotonic deep in the overparameterized regime, with additional peaks and descents when parameters scale quadratically with dataset size.The authors report empirical evidence of triple descent for finite networks trained with gradient descent.Source: §1 · p. 2, §1.1 · p. 2

2.1. Problem Setup and Notation

• The setup uses independent Gaussian samples and labels from a wide single-hidden-layer teacher, with sample size, input dimension, and hidden width tending to infinity at fixed ratios.The ratios are φ = n0/m and ψ = n0/n1.Source: §1 · p. 2, §2.1 · p. 3

1.1. Our Contributions

• The paper derives exact high-dimensional asymptotic test-error expressions, attributes non-monotonicity to the second-layer kernel, and reports empirical triple descent for finite networks trained with gradient descent.In the noiseless case, the reported learning curve scales as Etest ∼ m^-2.Source: §1.1 · p. 2

1.2. Related Work

• Prior analyses of linear models and unstructured random features provide limited insight into neural-network behavior across parameter scales; the paper instead analyzes NTK regression.The authors note that the connection between unstructured random features and neural networks trained with gradient descent is less clear.Source: §1.2 · p. 2, §1 · p. 2

2.2. Neural Tangent Kernel Regression

• The method studies approximate kernel ridge regression with the single-hidden-layer network’s NTK, which describes wide-network gradient-descent dynamics, but the predictor may fail to describe late-time network outputs when width is not asymptotically larger than the dataset.The paper restricts its study to single-hidden-layer fully connected networks.Source: §2.2 · p. 3

3. Three Regimes of Parameterization

• The first-layer and second-layer kernels suggest transitions at p = Θ(m) and p = Θ(m²), respectively, dividing parameterization into classical, abundant, and superabundant regimes.The analysis focuses on the quadratic scaling regime, where test error can rise again as p approaches m².Source: §3 · pp. 4–5

4. Overview of Techniques

• The analysis faces Hadamard-product kernels, nonlinear matrix dependencies, and factors shared between matrix inverses and multiplicative terms.The latter prevents straightforward use of many standard random-matrix proof techniques.Source: §4 · p. 5

4.1. Simplification of First-Layer Kernel

• Off-diagonal fluctuations of order O(1/m) are negligible, allowing the first-layer kernel's Hadamard product to be eliminated while preserving its bulk statistics asymptotically.The diagonal entries are proportional to the variance of the entries of F′.Source: §4.1 · p. 5

4.2. Linearization 1: Gaussian Equivalents

• Gaussian-equivalent matrices replace nonlinear random matrices while preserving the first and second mixed moments relevant to the test error.The resulting Y_lin is equivalent to a linear teacher plus noise; setting σT(x) = x removes test-label noise without changing the conclusions.Source: §4.2 · pp. 5–6

4.3. Linearization 2: Linear Pencil

• The test error is reduced to traces involving rational functions of W1, X, and ΘF, then represented by linear pencils for trace evaluation.The traces can be evaluated using random-matrix techniques or operator-valued free probability.Source: §4.3 · p. 6

5. Asymptotic Training and Test Error

• The asymptotic traces are characterized by coupled polynomial equations in the limit with φ = n0/m and ψ = n0/n1 fixed.Proposition 1 states that these equations have unique solutions.Source: §5 · p. 6
• Theorem 1 provides an asymptotic training-error result for the specified high-dimensional limit.The supplied theorem passage introduces Etrain but does not include the full expression.Source: §5 · p. 6
• When ν = 0, generalized crossvalidation is the exact asymptotic test error for the studied problem.The theorem's test-error expression includes an ε term because test-label noise is omitted; equal noise on training and test distributions removes it.Source: §5 · p. 7

6.1. Large Width Limit

• At superabundant widths, test error scales as m^-1 for finite SNR and m^-2 without noise.The large-width limit also yields an explicit asymptotic test-error expression.Source: §6.1 · p. 7

6.2. Small Width Limit

• The small-width limit upper-bounds test loss in the abundant-parameterization regime; if it is below the large-width limit, the global minimum is at intermediate widths.The paper illustrates a configuration satisfying this sufficient condition in Fig. 4(b).Source: §6.2 · p. 7

6.3. Large Dataset Limit

• When the dataset is much larger than the width, the authors again observe very steep learning curves, similar to the large-width limit.This limit takes φ→0 with φ/ψ→0.Source: §6.3 · p. 7

6.4. Ridgeless Limit: First-Layer Kernel

• For the centered ridgeless first-layer kernel, the test-error result follows by degenerating the large-width expression, and its large-dataset expansion uses ξ/ρ→η′/ζ.The expansion is described as similar to equation (30).Source: §6.4 · p. 7

6.5. Ridgeless Limit: Second-Layer Kernel

• In the centered case ν = 0, the expression obtained by letting σW2→0 agrees with Mei & Montanari (2019); decreasing σW2 makes the kernel more like K2, suggesting K2 drives overparameterized non-monotonicity.Source: §6.5 · p. 8, Figure 3 · p. 8

• For large n1, test error has a spike near n0=m as well as a hump near n1=m; the n0 non-monotonicity disappears for small n1.The joint dependence on feature count and width shows complex behavior in the quadratically overparameterized regime.Source: §7 · p. 8
• At low SNR, learning curves show bumps near m=n0 and m=n1, where adding training data can worsen test performance.Because n0=Θ(n1), the two humps are separated by a constant factor and do not alone establish multi-scale behavior.Source: §7 · p. 8
• As σW2 decreases and the kernel becomes more like K2, the quadratic-transition hump grows toward the linear-transition spike, implicating K2 in this non-monotonicity.The relative contributions of K1 and K2 are varied through σW2.Source: §7 · p. 8

• Separate nontrivial behaviors at p=m for K2 and p=m2 for K do not prove multi-scale behavior for one kernel or guarantee it appears in finite systems.The authors provide positive empirical evidence on these questions, but distinguish it from what the theoretical results establish.Source: §8 · pp. 8–9
• Across finite-system cases, test error peaks near the linear parameterization transition and bumps near the quadratic transition.The theoretical predictions agree with kernel regression in their regime of validity, near n1=m.Source: Figure 4 · p. 9, §8 · p. 9
• The global minimum is often at p=∞, but some configurations place the optimal p between m and m2.This intermediate optimum is illustrated in Fig. 4(b).Source: §8 · p. 9
• A single-layer network trained with gradient descent also shows triple descent; its deviation from kernel regression diminishes as n1 grows.The paper attributes the expected high-dimensional deviation to changes in the NTK during training when width is not much larger than the dataset.Source: §8 · p. 9

• The analysis concerns the NTK of a single-hidden-layer neural network, while similar multi-scale phenomena in broader architectures and datasets remain a conjecture.Source: §9 · p. 9
• The NTK test error can have complex non-monotonic behavior deep in the overparameterized regime, beyond the picture captured by double descent.The authors also provide empirical evidence for additional peaks and descents in kernel ridge regression and gradient-descent-trained neural networks.Source: §9 · p. 9

S1. Simplification of the first-layer kernel

• The empirical NTK is compared with the simplified first-layer kernel through a spectral-norm bound on their difference.The simplified kernel uses ζ′ as defined in the passage and applies for sufficiently large n0.Source: S1. Simplification of the first-layer… · p. 12

S1.1. Bounding ∆

• The fluctuation term ∆ is bounded using concentration, including the matrix Bernstein inequality for a sum of independent random matrices conditional on X.The argument also uses Gaussian properties of X and assumes dimensions are on the same order.Source: S1. Simplification of the first-layer… · pp. 12–13, S1.1. Bounding ∆ · pp. 13–14

S1.2. Bounding remaining terms

• The proof bounds the remaining terms individually, including a matrix R required because the entries have nonzero mean.The bounds on ∆ and the remaining terms together establish the stated kernel approximation.Source: S1.2. Bounding remaining terms · pp. 14–15

• For expressions involving random matrices, matching low-order moments with Gaussian variables can yield tractable replacements with the same limiting behavior under concentration assumptions.The discussion frames this as an instance of universality for global measurements such as test loss.Source: S2. Gaussian equivalents · p. 15
• When the kernel matrix is inverted, the analysis replaces the NTK with a linearized version to apply the linear pencil algorithm.The first-layer kernel's Hadamard-product structure is removed by bounding its difference from its linearization; the second-layer kernel F remains to be linearized.Source: S2. Gaussian equivalents · p. 15
• The matrix Dyson equation is presented as describing limiting behavior in many cases, while the paper does not explicitly show that the kernel meets its sufficient conditions.The authors identify weak correlations between kernel entries, implied by their assumptions, as the essential ingredient.Source: S2. Gaussian equivalents · p. 16

S3.1. Decomposition of terms

• The expected training loss is decomposed using the linearized NTK, with terms linear in N0 omitted because their expectation vanishes.The traces τ1 and τ2 are computed using operator-valued free probability rather than the resolvent method used in prior work for σW2 = 0.Source: S3.1. Decomposition of terms · pp. 16–17

S3.2. Linear pencils

• Linear pencils for τ1 and τ2 are represented by a self-adjoint block matrix whose blocks enable computation of the required traces.The representation doubles the dimension of the non-self-adjoint matrix, and its blocks contain constants or specified random matrices.Source: S3.2. Linear pencils · p. 17

S3.3. Operator-valued Stieltjes transform

• The operator-valued Stieltjes transform yields equations for τ1 and τ2, with the relevant solutions selected by analyticity in the upper half-plane.The resulting system includes the polynomial equations for τ1 and τ2.Source: S3.3. Operator-valued Stieltjes… · p. 18

S4.1. Decomposition of terms

• The derivation simplifies expectations using a linear teacher and leading-order substitutions, with terms linear in ω suppressed because they vanish in expectation.The linearized NTK substitution is justified by a spectral norm bound.Source: S4.1. Decomposition of terms · pp. 19–20

S4.2. Linear pencils

• Repeated application of the Schur complement formula for block matrix inversion establishes representations for E21, E22, E31, E32, and E33; linear pencils follow from the representations for E21 and E33.Although individual Q matrices are not self-adjoint, equivalent self-adjoint representations can be constructed by doubling dimensionality.Source: S4.2. Linear pencils · pp. 20, 22, S4.3. Operator-valued Stieltjes… · p. 23

S4.3. Operator-valued Stieltjes transform

• An operator-valued Stieltjes transform yields each linear pencil’s error term through a matrix equation with a unique solution satisfying ImG ≻ 0 when ImZ ≻ 0.The block-matrix representations specify the constants and covariance entries used in the equations.Source: S4.3. Operator-valued Stieltjes… · p. 23

S4.4. Total test error

• The total test error is assembled from the preceding terms, with ν = 0 for centering and ν = 1 without centering, establishing main theorem (27).The decomposition includes W2, E22, and E33.Source: S4.4. Total test error · p. 31