PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning: a statistical viewpoint

Peter L. Bartlett, Andrea Montanari, Alexander Rakhlin

arXiv:2103.09177v1math.STcs.LGstat.ML

TL;DR

The paper asks why overparametrized deep-learning methods can optimize non-convex objectives and generalize despite interpolating noisy data without explicit complexity control. It surveys theoretical examples of implicit regularization and benign overfitting, including linearized neural-network settings, and finds support for these principles in specific scenarios while emphasizing unresolved extensions to realistic deep learning.

  • Problem

    Deep learning achieves near-optimal non-convex optimization and excellent prediction despite near-perfect fits to noisy data without explicit model-complexity control.

  • Method

    The paper surveys classical uniform-convergence theory and theoretical examples of implicit regularization, benign overfitting, and linear-regime neural networks.

  • Results

    The reviewed results support the two working hypotheses in specific scenarios, including tractable gradient optimization and decompositions that separate predictive and interpolation components.

  • Takeaways & Limitations

    Overparametrization can have computational benefits and need not harm generalization in the reviewed settings, with model complexity adapting to sample size and network-parameter count.

  • Takeaways & Limitations

    The analysis largely concerns linear models with data-independent features, and uniform-convergence classification analysis is sensitive to arbitrarily fine-grained function properties.

Abstract

from arXiv · show

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting. We survey recent theoretical progress that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behavior of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favorable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

1 Introduction

The paper examines why overparametrized deep-learning models can be computationally tractable and generalize well despite interpolating noisy data without explicit complexity control. It reviews theoretical examples involving implicit regularization, benign overfitting, and linearized neural-network regimes while identifying limits for realistic deep networks.

  • Deep learning methods can find near-optimal solutions to highly non-convex empirical-risk problems while achieving near-perfect fits to noisy training data and excellent prediction performance.
  • Overparametrization is conjectured to make optimization tractable by leaving models sufficiently under-constrained relative to the training constraints.
  • Gradient methods can impose implicit regularization, selecting among interpolating models rather than treating all exact fits equivalently.
  • The reviewed results do not establish optimization or generalization benefits from increasing network depth, and extending the principles to realistic deep-learning settings remains open.
  • Benign overfitting is illustrated by decomposing predictions into a low-complexity component useful for prediction and a spiky component that enables interpolation without substantially harming accuracy.
  • In a linear regime, gradient methods can efficiently solve non-convex problems, remain in the linear regime, and achieve linear convergence of empirical risk.

2 Generalization and uniform convergence

This section reviews classical risk-based analysis for rich function classes, emphasizing uniform convergence and complexity control. It explains why worst-case uniform bounds can be conservative and how faster rates and oracle inequalities can improve prediction guarantees under additional structure.

  • Prediction quality is measured by population risk, while the analysis assumes only that input-output pairs are sampled independently and identically from a fixed distribution.
  • Empirical-risk minimization balances approximation error, which decreases with richer classes, against estimation error, which generally increases with function-class complexity.
  • Uniform convergence bounds control estimation error through the maximum deviation between population and empirical risks, with Rademacher complexity characterizing the relevant loss-class complexity.
  • For sub-Gaussian losses, a law-of-large-numbers term is O(n^-1/2) with probability exponentially close to 1, but data-dependent empirical minimization requires controlling an additional biased deviation term.
  • Worst-case uniform bounds may be conservative: Rademacher complexity is generally Ω(n^-1/2), although suitably simple classes can achieve faster excess-risk rates.
  • When a class contains a perfect predictor under quadratic loss, the empirical minimizer interpolates and can have population risk bounded by a faster-rate expression with high probability.
  • Oracle inequalities select complexity levels by balancing complexity penalties against training fit, while thresholding real-valued functions remains sensitive to arbitrarily fine-grained perturbations.

2.5 Computational complexity of empirical risk minimization

Classical empirical-risk minimization can be computationally hard for neural networks, even with convex losses, while surrogate convex losses can make related linear problems efficient. Uniform-convergence analyses also require sample sizes tied to network complexity and can be highly sensitive to nonlinearities.

  • Computational complexity: Replacing a discrete loss with a surrogate convex loss enables computational efficiency for thresholded linear classifiers, even when perfect classification is unavailable.For linearly separable training data, empirical-risk minimization over linear classifiers can instead be solved through a linear program.
  • Computational complexity: Convex losses do not remove the computational difficulty of neural-network empirical-risk minimization, which remains NP-hard in several settings.This contrasts with linearly parameterized classes and convex penalties, where penalized empirical-risk minimization can reduce to quadratic programming.
  • Uniform convergence: Uniform-convergence bounds for classification scale with function-class complexity, with worst-case deviations governed by the VC-dimension.The corresponding empirical-risk-minimization excess-risk bounds are O(√(d_VC(F)/n)) in the stated results, with matching lower bounds in some distributions.
  • Uniform convergence: Neural-network VC-dimension grows at least linearly with the number of parameters, so small uniform-convergence errors require sample sizes large relative to parameter count.This requirement conflicts with the empirical success of highly overparameterized networks.
  • Uniform convergence: VC-dimension can change from a small constant to infinity after an arbitrarily small perturbation of the ReLU nonlinearity.The example uses a one-parameter two-layer network class whose perturbed activation yields infinite VC-dimension.

2.8 Real prediction

Classical complexity bounds can fail to explain benign overfitting in noisy regression, because distribution-free bounds remain bounded below by the noise level. The reviewed alternatives analyze interpolants through parameter-scale complexity and decompositions into predictive and spiky components.

  • Complexity bounds: For neural-network classes, standard complexity bounds require an interpolating predictor whose complexity grows slowly enough with n for excess-risk bounds to converge to zero.The reviewed bounds use parameter-scale notions including deviations from fixed centers, spectral norms, Frobenius norms, and related quantities.
  • Interpolation: In the noiseless case, a fixed-complexity zero-risk predictor yields a minimum-complexity interpolant whose risk converges to zero as the sample size increases.The stated faster bound is L(bf) = O((log n)^4 R̄_n^2(F_r)).
  • Interpolation: With noise, distribution-free excess-risk bounds for minimum-complexity interpolants become vacuous because they cannot decrease below a constant.The fixed-design comparison explains this floor: excess risk is at least the noise level.
  • Benign overfitting: The reviewed evidence concludes that understanding benign overfitting requires analyses exploiting distributional or algorithm-specific information beyond generic uniform-convergence bounds.Examples include random-versus-fixed-design comparisons and settings where uniform-convergence or stability bounds fail despite good prediction.

3 Implicit regularization

In overparametrized problems with many empirical minimizers, optimization methods and parametrizations implicitly select particular interpolating solutions. These choices can favor minimum-norm or margin-based solutions and may support generalization beyond uniform-convergence explanations.

  • Implicit regularization: Overparametrized models can have many empirical minimizers, so optimization algorithms introduce an implicit bias that selects solutions with particular properties.This bias is not explicitly enforced and is studied as implicit regularization.
  • Linear least squares: Gradient descent on overparametrized linear square-loss problems converges to the minimum-norm solution among an affine subspace of interpolating minimizers.The minimizers form a subspace of dimension at least d−n when d > n.
  • Linear least squares: The minimum-norm interpolant is also the λ →0+ limit of ridge regression, linking interpolation to ridgeless regression.This connection is used to analyze and compare statistical properties of these methods.
  • Classification: For separable classification, gradient descent on logistic or related exponential-tail losses approaches the ℓ2 maximum-margin direction.The iterates satisfy θt = bθ · log t + ρt with residual norm O(log log t).
  • Parametrization: Reparametrization can change implicit regularization: under infinitesimal initialization and step sizes, gradient descent may converge to the minimum ℓ1-norm solution.Analogous results for matrix-valued parameters establish convergence to minimum nuclear norm under additional assumptions.
  • Implications: Optimization and parametrization choices can select interpolating solutions whose properties support good generalization through mechanisms beyond uniform convergence.The paper presents this as a summary of the role of implicit regularization in overparametrized problems.

4 Benign overfitting

Benign overfitting arises when interpolating estimators separate prediction from overfitting: a simple component captures signal while a spiky component fits training data without harming out-of-sample accuracy under favorable conditions.

  • Local methods: 1-NN can be consistent in the noiseless case because nearest-neighbor distances shrink as n^-1/d under continuity and smoothness assumptions.With noisy responses, insufficient averaging makes 1-NN inconsistent in general, motivating methods that average over growing neighborhoods.
  • Local methods: Singular-kernel Nadaraya-Watson estimators interpolate for every bandwidth, while bandwidth alone controls the bias-variance tradeoff.The singularity at zero determines exact fitting; the bandwidth determines smoothing, so empirical loss does not reveal over- or under-smoothing.
  • Local methods: A spiked-smooth decomposition combines a smooth predictor with small data-point deviations that enforce interpolation while having negligible L2(P) effect.The construction is bf = bf0 + ∆, with ∆(xj) correcting training residuals and ∥∆∥L2(P) = o(1); high dimensions can make such spikes more benign.
  • Linear regression: Minimum-norm linear interpolants decompose into a prediction component and an interpolation component whose bias and variance depend on the covariance spectrum and effective dimension.The interpolation variance is small when the tail eigenvalues decay slowly and d − k ≫ n, making overparametrization important.
  • Kernel methods: The same decomposition is not universally benign: in low-dimensional settings, Laplace-kernel minimum-norm interpolants can remain inconsistent for every kernel scale.The interpolant cannot simultaneously estimate the regression function and create interpolation spikes with small L2(P) norm.
  • Kernel methods: High-dimensional kernels exhibit multiple-descent behavior, with small variance between eigenvalue peaks and deeper valleys as dimension increases.Variance is small when d^ι ≪ n ≪ d^(ι+1), and can be at most d^-1/2 at n = d^(ι+1/2).

5 Efficient optimization

The paper argues that overparameterization can make interpolation computationally tractable and studies when gradient flow behaves like a linearized, minimum-norm interpolator. Its theory establishes exponential convergence and identifies parameterization conditions under which neural networks are accurately approximated by linear models, while noting important limitations beyond the linear regime.

  • 5 Efficient optimization: When a model is sufficiently rich to interpolate the data, constructing an interpolator can be computationally efficient even though general empirical risk minimization is often intractable.For linear classifiers, interpolation reduces to solving n linear inequalities in polynomial time; related constraint problems also become tractable when free parameters greatly exceed constraints.
  • 5 Efficient optimization: The paper hypothesizes that modern neural networks are tractable because overparameterization simplifies optimization enough for local gradient methods to find near-optimal training fits.This hypothesis links tractability to the relationship between parameter dimension and sample size rather than only to convexity.
  • 5 Efficient optimization: The available theory is strongest in the linear regime, while the linear approximation misses important properties of practical neural networks and leaves behavior beyond that regime unresolved.The paper explicitly presents extending these insights beyond linearized training as an outstanding challenge.
  • 5 Efficient optimization: In highly overparameterized networks, gradient flow can remain close to initialization, motivating a first-order linearization whose global minimizers form an affine space.With a full-row-rank Jacobian, the minimizer set has dimension p − n, and gradient flow selects the minimizer closest to initialization in ℓ2 distance.
  • 5 Efficient optimization: Under Gaussian inputs, sub-Gaussian responses, and sufficient width, gradient flow reaches zero empirical risk exponentially fast and converges exponentially to a global minimizer.The stated conditions include md ≥ C0n log n and n ≤ d^ℓ0, with high-probability guarantees.
  • 5 Efficient optimization: With suitable width and scaling, the neural network and its linearization have similar test behavior, so the network’s limiting generalization is approximated by minimum-norm linear interpolation.For α = Θ(1), the analysis requires md ≫ n^2; for growing α, nearly minimal overparameterization md ≫ n log n suffices and the approximation error is O(1/α).
  • 5 Efficient optimization: In the infinite-width limit of two-layer networks, different normalization scalings can produce either linear or nonlinear training regimes.The analysis also identifies a non-vanishing limiting quantity and a gap between bounds when n ≫ d, leaving its sharper characterization open.

6 Generalization in the linear regime

In the linear regime, gradient-based training selects minimum-norm interpolators, while random-features and neural-tangent models can generalize through decompositions into predictive low-complexity components and interpolation-oriented components. The asymptotic analysis shows that overparametrization controls approximation, statistical error, interpolation thresholds, and equivalence between several models, under explicit assumptions.

  • Gradient descent converges to the minimum ℓ2-norm interpolator in linear models, motivating analysis of which global minima gradient-based training selects.
  • When m and n are well separated, test error is determined by min(m, n), with approximation error for m ≪ n and statistical error for n ≪ m.
  • Sufficiently small regularization, including λ = 0+, yields the same polynomial-projection behavior and therefore includes min-norm interpolation.
  • Increasing m reduces test error until m ≈ n, while widths beyond m ≥ ndδ provide limited improvement and approach the corresponding kernel method.
  • Random-features regression is essentially equivalent to kernel ridge regression with a polynomial kernel whose degree depends on the smaller of sample size and network size.
  • At the interpolation threshold m/n = 1, test error peaks and, for λ = 0+, diverges before decreasing rapidly when m/n ≫ 1.
  • For n/d ≫ 1 and m/n ≫ 1, the limiting test error is ∥P>1f ∗∥L2 because the degree-one component is fitted while higher-degree components are not learned.
  • Random-features and neural-tangent models have similar generalization when their total parameter counts are matched, although their prediction-time computation differs.

7 Conclusions and future directions

The review explains why interpolation and overparametrization can coexist with effective prediction, while identifying unresolved questions about dimension, adaptivity, and nonlinear training dynamics.

  • 7 Conclusions and future directions: Classical generalization theory does not directly explain near-zero training error alongside near-optimal test error in highly complex models.Deep learning often interpolates noisy data without explicit complexity control, yet can retain excellent prediction performance.
  • 7 Conclusions and future directions: Linearized neural-network analyses show that overparametrized networks can remain close to their initialization linearization, making gradient-based training tractable in that regime.The review presents this as a direct connection between linear-model theory and certain neural-network training regimes.
  • Simple-plus-spiky decomposition: In reviewed benign-overfitting examples, prediction decomposes into a simple component and a spiky interpolating component that need not harm test error.The simple component carries predictive value, while the spiky component enables interpolation and can have negligible effect on test loss.
  • Role of dimension: Interpolation can be sub-optimal in fixed dimensions with noise, whereas its adverse effect is expected to diminish rapidly as dimension increases.This motivates studying how deep learning may depend on effective or intrinsic dimension rather than ambient dimension.
  • Adaptive model complexity: The complexity of the predictive simple component adapts with sample size and model width, increasing with data and saturating at the number of network parameters.The review proposes investigating whether this adaptive-complexity pattern extends beyond linear models.
  • 7 Conclusions and future directions: Outstanding challenges include identifying the required overparametrization and understanding how nonlinear dynamics determine the simple and spiky components.The review also notes that the general mechanisms remain insufficiently understood beyond the analyzed settings.

A.1 Bound on the variance of the minimum-norm interpolant

This appendix develops a deterministic bound for quantities governing the minimum-norm interpolant when the sample size is no larger than the feature dimension.

  • A.1 Bound on the variance of the minimum-norm interpolant: The lemma considers any data matrix X and positive semidefinite covariance matrix Σ in the regime n ≲ d.The eigenvalues of Σ are ordered as λ1 ≥ ... ≥ λd.
  • A.1 Bound on the variance of the minimum-norm interpolant: The proof decomposes Σ into contributions from its leading k eigen-directions and the remaining directions.This spectral split underlies the subsequent deterministic estimates.
  • A.1 Bound on the variance of the minimum-norm interpolant: The argument uses Sherman–Morrison updates to control leave-one-out matrix expressions and derive the stated upper bound.The proof substitutes these intermediate bounds into the target inequality.

A.2 Exact characterization in the proportional asymptotics

The appendix characterizes the minimum-norm interpolant in proportional asymptotics under covariance, smoothness, moment, and sub-Gaussian assumptions, with high-probability bias–variance guarantees.

  • A.2 Exact characterization in the proportional asymptotics: The analysis uses the kernel matrix K and its linearized kernel K1 to study the interpolating estimator.These kernels provide the central matrix objects for the asymptotic characterization.
  • A.2 Exact characterization in the proportional asymptotics: The theorem assumes independent standardized coordinates, bounded eigenvalue-related quantities, a smooth kernel function, controlled target moments, and sub-Gaussian covariates.It applies when the dimension-to-sample ratio remains bounded above and below.
  • A.2 Exact characterization in the proportional asymptotics: The result expresses the minimum-norm interpolant through squared bias and variance terms under the stated proportional-asymptotic assumptions.The theorem provides bounds with probability at least 1 − Cn^-1/4.
  • A.2 Exact characterization in the proportional asymptotics: Strict positivity conditions on kernel derivatives serve essentially as non-degeneracy requirements.Positive definiteness already implies the corresponding derivatives are nonnegative.
  • A.2 Exact characterization in the proportional asymptotics: The target function is decomposed into an affine component and an orthogonal remainder using its mean and covariance-weighted linear coefficient.The orthogonality conditions are stated for the component beyond affine functions.

A.2.1 Preliminaries

The preliminaries establish high-probability approximations for kernel-related quantities using concentration, smoothness, Hermite expansions, and sub-Gaussian bounds.

  • A.2.1 Preliminaries: Concentration and sub-Gaussian tail bounds control quadratic forms and inner products throughout the estimates.The proof invokes Hanson–Wright-type control and union bounds over sample indices.
  • A.2.1 Preliminaries: For a nonlinear kernel, the kernel matrix is bounded below by a positive multiple of the identity with high probability.The stated condition is K ⪰ c*In with c* = βγ > 0.
  • A.2.1 Preliminaries: The preliminary lemmas yield norm and Frobenius-norm approximations for the relevant vectors and matrices.Specifically, the bounds are ∥v − v0∥2 ≤ C d^-1√log d and ∥M − M0∥F ≤ C d^-3/2 log d.
  • A.2.1 Preliminaries: The proof works on high-probability events controlling normalized norms and inner products of the data vectors.These events support the later approximations uniformly over the sample.
  • A.2.1 Preliminaries: Smoothness allows the kernel to be modified outside a neighborhood of zero, with the resulting approximation error exponentially small in dimension.The error is treated as negligible relative to the theorem’s stated terms.
  • A.2.1 Preliminaries: The kernel expansion is organized through Hermite coefficients and remainders after removing low-order terms.Taylor remainders and Hermite projections are compared to control higher-order contributions.

A.2.2 An estimate on the entries of the resolvent

This section establishes moment bounds for resolvent-related entries under sub-Gaussian design assumptions. The proof uses Taylor expansions, orthogonality, Hölder’s inequality, and Hanson–Wright bounds.

  • Assumptions: Under iid zero-mean C-sub-Gaussian rows, bounded positive semidefinite S, and centered g orthogonal to linear functions, the resolvent-entry estimates are developed.The assumptions include C^-1 ≤ n/d ≤ C and E{g(z1)^2}=1.
  • Two-index estimate: For two distinct indices, the resolvent entry is represented by an off-diagonal bilinear form divided by diagonal factors, with a correction involving its squared numerator.The displayed representation is given in Eq. (230), followed by a simplified term in Eq. (232).
  • Proof strategy: The proof conditions on the remaining rows, defines diagonal fluctuations q_i and off-diagonal terms A_ij, and expands the relevant entries in powers of A.Hanson–Wright yields d^-1/2 moment bounds for A_ij and Q_i.
  • Moment control: Orthogonality of g to linear functions removes terms with vanishing conditional contribution, while Hölder and Hanson–Wright control the remaining moments.The argument repeatedly uses bounded resolvent factors and moment estimates for A_ij and Q_i.
  • Higher-order products: The lemmas extend the estimates to products involving four distinct indices and to triples with a repeated index, under an additional 2+η moment condition on g.The corresponding bounds are stated in Lemmas A.5 and A.6.
  • Aggregate bound: Theorem A.7 aggregates these moment calculations into a high-probability bound for the full sum, holding with probability at least 1 − C d^-1/4.Its proof partitions index tuples by overlap and concludes with Chebyshev’s inequality.

A.2.3 Proof of Theorem 4.13: Variance term

The variance proof reduces the nonlinear-kernel calculation to controlled matrix approximations and a ridge-regression variance term. High-probability spectral and norm bounds support this reduction.

  • Setup: The proof begins by defining the variance through events E1 and E2 and then reduces the analysis to expectations involving the kernel matrices.The argument works under E1 ∩ E2 and uses bounded diagonal quantities.
  • Matrix control: Lemma A.8 establishes high-probability spectral control for K0 and shows that the associated quantities A_l scale proportionally to d.The bounds also control B0 through the norm of X.
  • Approximation: The approximation B − B0 has norm at most C d^-3/2 with very high probability, enabling replacement of B by its simpler proxy.Lemma A.10 supplies a further high-probability relation involving B, B0, and K1.
  • Conclusion: The variance proof concludes by combining the auxiliary lemmas with high-probability error bounds and the defined Err(n) remainder.The remainder satisfies |Err(n)| ≤ C n^-c0 with very high probability.
  • Reduction to ridge regression: The resulting variance expression is identified with the variance of ridge regression using the simple features X and ridge regularization proportional to γ.The proof invokes prior ridge-regression results to evaluate the asymptotics.

A.2.4 Proof of Theorem 4.13: Bias term

The bias proof decomposes the target into constant, linear, and nonlinear components, then shows that the main components can be analyzed through ridge-regression quantities while nonlinear and mixed terms are controlled probabilistically.

  • Response decomposition: The noiseless response is decomposed into constant, linear, and nonlinear components using orthogonality conditions.The coefficients satisfy b0 = E{f*(x)} and β0 = Σ^-1 E{f*(x)x}.
  • Bias expression: The bias is written as a quadratic expression involving K^-1, M, v, and the squared L2 norm of the target.This expression separates the matrix, cross, and target-norm contributions.
  • Constant component: The constant component contributes negligibly, allowing the analysis to focus on the d-dimensional bias associated with the linear component and the nonlinear remainder.The proof explicitly states that the constant-term contribution in f* and M0 is negligible.
  • Linear component: The linear contribution is recognized as the bias of ridge regression with linear features and linear responses, and existing ridge results provide its asymptotic control.The analysis uses the eigenstructure of Σ to express the linear bias.
  • Nonlinear component: The nonlinear contribution is bounded using matrix interpolation identities, high-probability norm estimates, and a separate lemma for R_NL.The proof also controls diagonal terms using prior results.
  • Mixed term and conclusion: The mixed contribution is handled by an analogous high-probability argument, after which Lemmas A.15–A.17 complete the bias bound.The final theorem proof combines the decomposed contributions and auxiliary estimates.

A.2.5 Consequences: Proof of Corollary 4.14

The corollary derives consequences from the spectral structure of Σ and the implicit equation defining λ*. The analysis establishes uniqueness, bounds λ*, and transfers these bounds to the bias.

  • Spectral setup: The eigenvalues λ1 ≥ ··· ≥ λd of Σ are ordered decreasingly to support the subsequent spectral analysis.The consequences are expressed in the eigenbasis of Σ.
  • Fixed-point equation: The defining equation for λ* has a unique non-negative solution because its left-hand side increases while its right-hand side decreases.The proof verifies existence by considering the limits as λ* approaches zero and infinity.
  • Eigenvalue-dependent bound: When λ* is at least a constant multiple of λk+1, the defining inequalities yield a corresponding lower bound on λ* and an explicit solution form.The comparison applies whenever λ* ≥ c*λk+1.
  • High-probability consequence: Assuming the relevant right-hand side is at most 1/2, the theorem gives a high-probability bound used to derive the corollary.The same assumption implies k ≤ n/2.
  • Bias consequence: The bias formula is evaluated in the eigenbasis of Σ by expressing β0 through its coordinates along the eigenvectors.Combining this representation with Theorem 4.13 yields the desired bias bound.

Further define

Under high-dimensional Gaussian-data assumptions, the two-layer-network analysis establishes exponential convergence of gradient flow and close agreement between nonlinear and linearized dynamics. The proof controls parameter movement, kernel conditioning, and the difference between the two flows.

  • Linearized flow: The empirical risk decreases exponentially fast to 0 at rate λ0=σ_min^2/(2n).This conclusion follows from maintaining a positive lower bound on the empirical kernel’s smallest eigenvalue.
  • Control of the flow: The proof keeps parameters close to initialization by defining a stopping radius and showing the stopping time is infinite.Kernel conditioning remains valid while the parameters stay within the radius r*=σ_min/(2L_n).
  • Control of the flow: The difference between nonlinear and linearized predictions is bounded through a differential equation and Grönwall’s inequality.The argument uses exponential decay of the linearized residual and bounds the changing empirical kernel.
  • Assumptions: The two-layer results require Gaussian covariates, sub-Gaussian responses, sufficient width md≥C0n log n, and d≤n≤d^ℓ0.The stated probability is at least 1−2 exp{−n/C0}.
  • Two-layer networks: Gradient flow converges exponentially fast to a global minimizer, with rate λ*=C1α^2d/n.The result holds with probability at least 1−2 exp{−n/C0} under the theorem’s width and dimension conditions.
  • Two-layer networks: The nonlinear model and its linearized flow remain similar on test data.The analysis compares f(θ) with f_lin(θ)=f(θ0)+Df(θ0)(θ−θ0).
Loading 2103.09177v1…