Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

1 Introduction

• Prior approximation theory and practice can diverge because SGD may compute a function unlike one established by universal approximation, while model choice itself is often data-adaptive.The paper identifies a remaining gap for non-convex, potentially non-smooth training of all neural-network weights.Source: §1 · pp. 2–3
• The central question is how a neural network trained by gradient flow to stationarity represents and approximates the conditional mean f*.The paper asks what the computed function f_t says about f* under this training process.Source: §1 · pp. 2–3
• Existing neural-network/RKHS theory largely uses fixed kernels, while this work studies dynamic, data-adaptive RKHSs produced by gradient flow on both layers.The authors state that their framework considers all f* in L2 without pre-specified structural assumptions.Source: §1 · pp. 3–4

1.1 Problem Formulation

• The formulation studies population approximation of f*(x)=E[y|x] and empirical risk minimization, including when gradient-flow training interpolates the observed responses.The interpolation question concerns whether and when f_t(x_i)=y_i for each training example.Source: §1.1 · p. 5
• Extending the gradient-flow results to positive-step-size or mini-batch stochastic gradient descent is left as a separate research topic.The authors cite ReLU non-smoothness and less transparent interactions between batch size and step size in non-convex problems.Source: §1.1 · p. 5

2.1 Notations

• The notation distinguishes kernels from their associated Hilbert spaces and includes L2(Px), empirical distributions, and positive and negative parts of signed measures.These conventions support the later construction of a time-varying RKHS from neuron weights.Source: §2.1 · pp. 5–6

2.2 Preliminaries

• The signed measure ρ_t determined by neuron weights is used to construct a dynamic RKHS, while gradient-flow training induces function and RKHS sequences indexed by time.The stationary RKHS is the limiting member of this time-indexed family.Source: §2.2 · p. 6

2.3 Organization and Summary

• At any local stationarity, gradient flow learns an adaptive RKHS representation and simultaneously performs the global least-squares projection onto it.The stated approximation result applies to f* in L2(Px).Source: §2.3 · p. 7
• The paper studies representation benefits of an RKHS learned from the data and task, decomposing f* into the stationary network function and a residual.The learned representation is described as implicitly data-dependent rather than pre-specified.Source: §2.3 · pp. 7–8
• In the vanishing-regularization limit, the neural-network function is characterized as kernel ridgeless regression with an adaptive kernel.The summary identifies the resulting regression kernel as the stationary kernel.Source: §2.3 · p. 8
• For ERM, Proposition 4.2 addresses extrapolation at a new point and discusses generalization error; the result is extendable to infinitely many neurons.The summary distinguishes this ERM result from the approximation theorems.Source: §2.3 · p. 8

3.1 Gradient Flow, Projection and Adaptive RKHS

• The stationary RKHS is task-adaptive, so training learns a representation for the target without requiring prior structural knowledge about it.The paper contrasts this with choosing a suitable function class in advance.Source: §3.1 · pp. 9–10
• At any stationarity, the neural-network function is a global least-squares minimizer within the stationary RKHS, which gradient flow learns along with the function.The result also extends to the infinite-neuron case.Source: §3.1 · pp. 9–10

3.2 Representation Benefits of Adaptive RKHS

• The target residual lies in a subspace of the stationary RKHS null space characterized by the GD kernel, rather than merely in the RKHS’s orthogonal complement.The decomposition into the fitted function and residual is not an orthogonal decomposition of the stationary RKHS and its complement.Source: §3.2 · p. 12
• Because the learned basis depends on the data distribution and target, the residual can occupy a smaller subspace than with a fixed basis.The paper presents this as a representation and approximation benefit of adaptive RKHSs.Source: §3.1 · p. 10, §3.2 · pp. 12–13
• The paper also states that a similar representation result holds for empirical risk minimization, where the gap between the stationary RKHS and GD kernel can be large.The stated gap can occur even for the ERM problem.Source: §3.2 · p. 13

• In the one-neuron example, the GD-kernel null space is smaller than the stationary RKHS null space, and the paper notes that the gap can be large in other settings.The example considers empirical risk minimization with n samples and input dimension d.Source: §4 · pp. 13–14
• In the vanishing-regularization limit, the neural-network function converges to kernel ridgeless regression with an adaptive kernel learned along training.The kernel is data-adaptive rather than fixed and pre-specified.Source: §4 · p. 14
• The generalization decomposition separates adaptive representation and approximation errors from variance and bias terms associated with assuming the target lies in the population adaptive RKHS.The paper states that this decomposition holds without global optimization guarantees and may support rates adapting to the target without structural assumptions.Source: §4 · p. 16

5 Time-varying Kernels and Evolution

• Gradient descent induces a time-varying kernel from the network’s training dynamics, with a corresponding evolution equation for the neuron-weight distributions.The analysis extends to optimizing both network layers and to the population approximation problem; similar results for losses with curvature require slightly stronger conditions.Source: §5 · pp. 16–17, §5.2 · p. 20

5.1 Initialization, Rescaling and K0

• The analysis uses infinitesimal initialization and a balanced condition that persists throughout training under the specified initialization.The authors also state that the balanced condition holds at stationarity with ℓ2 regularization, even for unbalanced initialization.Source: §5.1 · pp. 17–18

5.2 Evolution of ρt

• The signed measure formed by the neurons evolves under a distributional dynamics that determines the dynamic kernel.The treatment uses rescaling to define the infinite-neuron limit; the authors distinguish their exact two-layer gradient flow from the dynamics analyzed in Mei et al. [2018].Source: §5.2 · pp. 19–20

5.3 Two RKHS: K8 and H8

• At stationarity, the residual satisfies a kernel condition, while the stationary RKHS and the gradient-descent kernel are distinct.The section compares the two adaptive RKHSs as part of the argument for the representation result.Source: §5.3 · p. 21

6 Experiments

• For NN-family targets, kernel-matrix eigenvalues generally grow during training, and more complicated targets take longer to reach stationarity.Figure 3 plots the logarithms of the sorted top 80% eigenvalues; its x-axis is eigenvalue rank and colors represent training time.Source: §6 · p. 22, Figure 3 · p. 22
• With random labels, kernels behave similarly across m=200 and 500 at fixed n, but kernel behavior and convergence differ across n=50 and 200 at fixed m.Figure 4 compares kernel spectra across the model-width and sample-size settings.Source: §6 · p. 23, Figure 4 · p. 23
• On MNIST, increasing the sample size from n=50 to 200 takes longer to reach stationarity.The experiment tracks the kernel-matrix spectrum during training for the two sample sizes.Source: §6 · p. 23

7 Main Proofs

• A stationary network function satisfies the least-squares optimality condition against every function in the relevant representation, establishing it as a minimizer.The proof states the condition as orthogonality of the residual to the image of the operator T* and identifies it with the minimizer criterion.Source: §7 · pp. 23–24
• Under the stated compact-support condition, the stationary signed measure has finite total variation, supporting the comparison of the two stationary RKHSs.The proof of the RKHS comparison invokes compactness for its final inequality.Source: §7 · p. 26
• The stationary residual lies in the kernel of the RKHS associated with K∞.The proof derives this from the vanishing inner product with K∞ applied to arbitrary L2 functions.Source: §7 · p. 26

A.1 Supporting Results

• With bounded-support initialization and the stated regularity conditions, the finite-neuron distributions have a well-defined infinite-neuron limit governed by the distributional dynamics.The result uses bounded, uniformly Lipschitz gradients of the velocity-field terms.Source: §A.1 · p. 33
• Under the rescaling, the positive and negative second-layer parameters remain tied to the norms of their corresponding first-layer parameters.The relation follows from the initialization and is stated for the evolving parameters.Source: §A.1 · p. 34
• Once a neuron’s weights reach zero, there exists a subsequent solution equivalent to training without that neuron.The construction removes the neuron from the ODE and pads the reduced solution with zero weights.Source: §A.1 · p. 34