PaperlayerView exact revision on arXiv

Source-linked AI summary

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

Song Mei, Theodor Misiakiewicz, Andrea Montanari

arXiv:1902.06015v1stat.MLcond-mat.stat-mechcs.LGmath.ST

TL;DR

Mean-field dynamics approximate SGD for two-layer networks, but earlier approximation guarantees depended on parameter dimension. This paper develops stronger bounds, including a dimension-independent approximation guarantee.

  • Problem

    Earlier mean-field approximation guarantees for SGD were dimension-dependent, leaving open when distributional dynamics approximate SGD without width requirements tied to parameter dimension.

    Source: §1 · pp. 3–4

  • Method

    The paper uses a coupling argument and refined concentration-of-measure analysis to control how approximation error depends on network width.

    Source: §1 · p. 4

  • Results

    The paper establishes a dimension-independent bound for the mean-field approximation.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Correctly capturing dimension dependence is important when comparing neural networks with other learning techniques.

    Source: §1 · p. 4

    The analysis assumes a one-pass model in which each data point is visited once.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

We consider learning two layer neural networks using stochastic gradient descent. The mean-field description of this learning dynamics approximates the evolution of the network weights by an evolution in the space of probability distributions in $R^D$ (where $D$ is the number of parameters associated to each neuron). This evolution can be defined through a partial differential equation or, equivalently, as the gradient flow in the Wasserstein space of probability distributions. Earlier work shows that (under some regularity assumptions), the mean field description is accurate as soon as the number of hidden units is much larger than the dimension $D$. In this paper we establish stronger and more general approximation guarantees. First of all, we show that the number of hidden units only needs to be larger than a quantity dependent on the regularity properties of the data, and independent of the dimensions. Next, we generalize this analysis to the case of unbounded activation functions, which was not covered by earlier bounds. We extend our results to noisy stochastic gradient descent. Finally, we show that kernel ridge regression can be recovered as a special limit of the mean field analysis.

1 Introduction

The introduction frames mean-field dynamics as a distributional approximation to SGD and presents dimension-free guarantees for broader activation and noise settings, together with a short-time kernel-ridge limit.

Mean-field dynamics

  • For large N and small step size, SGD’s empirical neuron distribution is approximated by a probability distribution evolving by a PDE, independently of N.This mean-field description isolates global convergence and typical behavior while neglecting local minima and statistical noise.Source: §1 · p. 3

Dimension-free approximation

  • The paper replaces ’s dimension-dependent N ≫ D requirement with a dimension-independent approximation bound, using coupling and refined concentration to control error dependence on N.Earlier guarantees required N ≫ D even for smooth functions and well-behaved data distributions.Source: §1 · p. 4

Unbounded activations

  • The analysis extends to unbounded activations, including unbounded second-layer coefficients ai, by establishing an a priori growth bound while retaining dimension-free guarantees.The cited earlier approximation guarantee applied only to bounded activations.Source: §1 · p. 4

Noisy SGD

  • For noisy SGD, the paper proves dimension-free approximation guarantees with bounded activations, but its guarantees for unbounded activations are not dimension-free.Noise adds a Laplacian term to the PDE, yielding smoother solutions supported everywhere and converging to a unique fixed point.Source: §1 · p. 4

Kernel limit

  • In a short-time limit, the PDE is approximated by linearized dynamics equivalent to kernel ridge regression with a kernel from the initial weight distribution.Beyond short times, the dynamics are analogous to kernel boosting with a time-varying, data-dependent kernel.Source: §1 · p. 4

2 Related work

Related work connects finite-width neural networks to distributional mean-field descriptions and studies their optimization and kernel limits. This paper targets dimension-independent quantitative approximation bounds, including settings with unbounded second-layer coefficients, and clarifies how kernel and mean-field behavior emerge at different time scales.

2 Related work

  • Classical approximation theory lifts N-neuron networks to a space of probability distributions, an idea also exploited algorithmically [NS17].The lifted space is parametrized by distributions ρ.Source: §2 · pp. 4–5
  • Recent work proved that SGD converges to the infinite-dimensional evolution for sufficiently many neurons, while gave quantitative bounds motivating improved dimension scaling and unbounded second-layer coefficients [RVE18] [SS18].Global convergence was proved in special examples and for noisy SGD in; [RVE18] used homogeneity, with establishing a general result for full-support initialization and homogeneous activations, but without finite-N approximation bounds.Source: §2 · p. 5
  • Resampling-based mean-field optimization and radial-basis-function networks also achieved global or exponentially fast convergence, but their settings differ and the former lacks quantitative PDE-to-finite-N SGD bounds [WLLM18] [JMM19].The present theorems do not cover [WLLM18], though the authors suggest their approach could apply; [JMM19] constrains weights to a convex compact domain.Source: §2 · p. 5
  • Prior work proposed that wide two-layer networks perform kernel ridge regression [JGH18] [GJS+19] [DZPS18] [DLL+18] [AZLS18]; here the kernel regime is recovered as a special mean-field limit, focusing on population risk.The kernel phenomenon was also reported as generic across broad model classes [CB18a].Source: §2 · p. 5
  • For suitable initialization scaling, kernel behavior arises early and mean-field behavior at longer time scales, rather than their difference being due only to initialization [DL19]; the mean-field connection to time-varying-kernel boosting appeared implicitly in [RVE18].[DL19] attributed the difference between kernel and mean-field behavior to different initializations of the coefficients ai’s.Source: §2 · p. 5

3 Dimension-free mean field approximation

The section establishes mean-field approximation results for one-pass SGD, including a dimension-independent neuron requirement in the noiseless case and a noisy-SGD guarantee with important limitations. It applies the noiseless result to anisotropic Gaussian classification, reducing the required neuron count from O(d) to O(1).

3.1 General results

  • Under A1–A4, noiseless SGD is approximated with a neuron requirement independent of D (N≫1 for bounded T and K), although accuracy still requires ε≪1/D.The analysis assumes one-pass sampling, with each data point visited once.Source: §3.1 · pp. 5–7
  • For noisy SGD with general coefficients, Theorem 2(B) gives the first quantitative distributional-dynamics bound, but it is not dimension-free, requires T=o(log log N), and implies accuracy when N≫D.The stated difficulty is controlling coefficient growth without a deterministic bound on |a_i|.Source: §3.1 · p. 7

3.2 Example: Centered anisotropic Gaussians

  • For anisotropic Gaussian classification with s0=γd, the theorem requires O(1) neurons rather than the O(d) required in [MMN18].For good initializations and ε≤1/(C0d), it guarantees risk within η of the finite-network optimum over k∈[T/ε, 10T/ε] with probability at least 1−δ.Source: §3.2 · p. 8

4 Connection with kernel methods · A Notations •

The paper connects mean-field training to a time-varying kernel-boosting process, then shows that at large scale the dynamics can be approximated by a fixed-kernel limit related to kernel ridge regression. It also records notation used in the appendix.

4 Connection with kernel methods

  • Large-width networks can exhibit a kernel regime alongside mean-field behavior, and kernel descriptions of neural networks were studied in [JGH18] [DZPS18].The kernel regime is also a generic feature of overparameterized models under differentiability assumptions [CB18a].Source: §4 · p. 8

4.1 A coupled dynamics

  • Gradient flow for two-layer networks is kernel boosting with a time-varying kernel, whose evolution speed is controlled by α.The coupled dynamics also imply non-increasing risk, although a changing kernel makes convergence to zero difficult to analyze.Source: §4.1 · p. 9

4.2 Kernel limit of residual dynamics

  • The large-α kernel regime simplifies the dynamics by replacing evolving Hρα_t with the fixed kernel Hρ0 in the linearized residual dynamics.The motivation is that the time derivative of ρα_t is of order 1/α.Source: §4.2 · p. 9
  • With an empty null space for Hρ0, the linearized residual tends to zero; the paper emphasizes population risk becoming close to zero, unlike [CB18a]’s local-minimum result.More generally, the limiting residual is characterized by projection onto the null space of Hρ0.Source: §4.2 · pp. 9–10
  • For α ≥ t^2D^(3/2), the linearized dynamics approximates the mean-field dynamics; an independence condition on a and w yields an α-independent bound.The stated condition is |E(a)| ≤ K5/α when a and w are independent under ρ0; the theorem covers fixed and general coefficients.Source: §4.2 · pp. 9–10
  • The linearized residual dynamics can be interpreted as kernel ridge regression with kernel Hρ0 [JGH18], including for the empirical data distribution.The empirical setup uses the sample points and target vector y = [f(x1), ..., f(xn)]T.Source: §4.2 · p. 10

A Notations •

  • Appendix A defines W2 as Wasserstein distance, [N] as {1, ..., N}, and K as a generic constant depending on context-specific Ki values.It also uses [s] for the discretized time index.Source: A Notations • · p. 13

B Proof of Theorem 1 part (A) … B.5 Bound between GD and SGD

The proof bounds successive gaps from the PDE through nonlinear and particle dynamics to GD and SGD, then combines them to prove Theorem 1(A).

B Proof of Theorem 1 part (A)

  • Four interpolation bounds, proved under Theorem 1(A)'s setting, combine to establish Theorem 1(A) with probability at least 1 − e^−z²; the proof generalizes to bounded |a_i| ≤ K and nonconstant ξ(t).Source: B Proof of Theorem 1 part (A) · pp. 14–15

B.1 Technical lemmas

  • Technical lemmas establish boundedness and regularity from the assumptions, including a uniform-in-time bound for the dynamics.The stated bounds use bounded activation and labels, derivative assumptions, and the Lipschitz properties of V and U.Source: §B.1 · pp. 15–16

B.2 Bound between PDE and nonlinear dynamics

  • The PDE-to-nonlinear-dynamics gap is bounded with probability at least 1 − e^−z², with a constant depending only on K1, K2, and K3.The proof decomposes the difference into two terms and combines their bounds.Source: §B.2 · p. 16

B.3 Bound between nonlinear dynamics and particle dynamics

  • The nonlinear-dynamics-to-particle-dynamics gap is bounded with probability at least 1 − e^−z².The proof first establishes a uniform bound for individual particles, then applies Gronwall’s inequality.Source: §B.3 · pp. 17–18

B.4 Bound between particle dynamics and GD

  • The particle-dynamics-to-GD gap is bounded by a constant K.The proof uses a time-regularity bound and Gronwall’s lemma.Source: §B.4 · p. 18

B.5 Bound between GD and SGD

  • The GD-to-SGD gap is bounded with probability at least 1 − e^−z² under the stated assumptions.The proof applies Azuma–Hoeffding concentration to sub-Gaussian gradient terms, using bounded labels and predictions.Source: §B.5 · pp. 18–19

C Proof of Theorem 1 part (B) … C.5 Bound between GD and SGD

The proof establishes Theorem 1(B) by bounding the gaps between the PDE, nonlinear and particle dynamics, GD, and SGD. It handles unbounded coefficients through control of the support in the a variable and growth- and Lipschitz-type estimates.

C.1 Technical lemmas

  • Compact support in a at initialization keeps the support of ρt in a bounded on [0, T], while V and U satisfy growth and Lipschitz bounds depending on a.The support bound depends only on K1, K2, K3, K4, and T; the estimates for V and U allow their magnitudes and Lipschitz constants to grow with |a| and |a′|.Source: C Proof of Theorem 1 part (B) · p. 19, §C.1 · pp. 21–22

C Proof of Theorem 1 part (B)

  • Under the stated step-size condition, the interpolation bounds hold with probability at least 1 − e^−z^2 and combine to prove Theorem 1(B).The step-size restriction is ε ≤ 1/[K0(D + log N + z^2)e^{K0(1+T)^3}].Source: C Proof of Theorem 1 part (B) · pp. 20–21

C.2 Bound between PDE and nonlinear dynamics

  • The proof bounds the gap between the PDE and nonlinear dynamics with high probability by decomposing it into two terms controlled by Lemmas 10 and 11.The proposition states a probability of at least 1 − e^−z^2.Source: §C.2 · p. 23

C.3 Bound between nonlinear dynamics and particle dynamics

  • A high-probability bound relates nonlinear dynamics to particle dynamics.Proposition 6 states probability at least 1 − e^−z^2; the supplied passages do not expose the full bound.Source: §C.3 · p. 24

C.4 Bound between particle dynamics and GD

  • With ε ≤ 1/(K0e^{K0(1+T)^3}), Proposition 7 bounds the gap between particle dynamics and GD for every t ≤ T.The proof uses a stopping-time argument and Gronwall’s inequality to extend the bound through the interval [0, T].Source: §C.4 · pp. 25–26

C.5 Bound between GD and SGD

  • Under a dimension- and confidence-dependent step-size condition, Proposition 8 bounds the GD–SGD gap for every t ≤ T with probability at least 1 − e^−z^2.The proof uses conditional sub-Gaussian increments and Azuma–Hoeffding concentration for a martingale.Source: §C.5 · pp. 26–27

D Proof of Theorem 2 part (A) … E Proof of Theorem 2 part (B)

The proofs of Theorem 2 parts (A) and (B) control approximation by comparing coupled dynamics and combining probabilistic interpolation bounds. Part (A) treats the coefficients a_i ≡ 1 for simplicity; the authors say the general ξ(t) case follows by adaptation.

D Proof of Theorem 2 part (A)

  • Propositions 9–12 yield constants K and K0 such that the stated event holds with probability at least 1−e−z2; combining these inequalities gives the conclusion of Theorem 2(A).The following subsections prove all the above interpolation bounds under the setting of Theorem 2(A).Source: D Proof of Theorem 2 part (A) · p. 29

D.1 Technical lemmas

  • Technical lemmas bound initialization norms and Brownian noise, using sub-Gaussian estimates and Doob’s martingale inequality.The section defines maximum and average initialization norms and bounds on Brownian noise.Source: §D.1 · pp. 29–30

D.2 Bound between PDE and nonlinear dynamics

  • Proposition 9 bounds the gap between the limiting PDE and nonlinear dynamics with high probability.Its proof reuses a decomposition from Proposition 1 and bounds on the potential and term I.Source: §D.2 · p. 32

D.3 Bound between nonlinear dynamics and particle dynamics

  • Proposition 10 bounds the nonlinear-to-particle dynamics gap; coupling them with the same Brownian motion cancels the noise term.The proposition gives bounds with probability at least 1 − e^−z^2.Source: §D.3 · p. 32

D.4 Bound between particle dynamic and GD

  • Proposition 11 bounds the gap between particle dynamics and gradient descent with high probability.The proof applies Gronwall’s inequality to conclude the bound.Source: §D.4 · pp. 33–34

D.5 Bound between GD and SGD

  • Proposition 12 bounds the gradient-descent-to-SGD gap; the proof couples their noise and notes that regularization vanishes in the martingale difference.The proposition states a bound with probability at least 1 − e^−z^2.Source: §D.5 · p. 34

E Proof of Theorem 2 part (B)

  • Part (B) likewise combines interpolation bounds for four coupled dynamics to establish Theorem 2 under assumptions A1–A6.The proof is written under a simplifying restriction; the authors state that a general ξ(t) follows by adaptation.Source: E Proof of Theorem 2 part (B) · pp. 34–35

E.1 Technical lemmas

The technical lemmas control coefficient growth under Brownian noise and establish bounds along the trajectories. They show sub-Gaussian behavior for the nonlinear dynamics and high-probability control of coefficient norms.

E.1 Technical lemmas

  • The nonlinear-dynamics coefficient ā_t is sub-Gaussian with parameter Ke^Kt, obtained by decomposing it into three dependent sub-Gaussian terms.The component parameters are K², Ke^Kt, and Kt.Source: §E.1 · p. 37
  • With probability at least 1 − e^−z², the trajectory bounds include sup over s in [0,T] of ||ā_s||_1/N ≤ Ke^KT[1+z].The same proof applies to the other trajectories.Source: §E.1 · p. 37
  • Lemma 20 asserts the existence of a constant K and states a further result for t,h ≥ 0 satisfying t+h ≤ T.Source: §E.1 · p. 38

E.2 Bound between PDE and nonlinear dynamics … F.1 Equation (DD) (noiseless SGD)

The section establishes bounds across successive PDE, nonlinear, particle, gradient-descent, and stochastic-gradient dynamics, then states conditions ensuring the limiting PDE has a unique solution. It also explains how the PDE is interpreted in the weak sense.

E.2 Bound between PDE and nonlinear dynamics

  • Proposition 13 states a bound between the PDE and nonlinear dynamics, with the proof organized around two terms.The two terms are bounded separately in Lemmas 22 and 23.Source: §E.2 · pp. 39–40
  • Lemma 24 extends the concentration bound uniformly over [0, T ] using a time grid and bounds on variation within grid intervals.The passage states a probability of at least 1 −e−z2.Source: §E.2 · pp. 41–42

E.3 Bound between nonlinear dynamics and particle dynamics

  • Proposition 14 gives a high-probability bound between nonlinear and particle dynamics, with displayed terms depending on D, N, T, and z.The proposition states probability at least 1 −e−z2; its bound includes terms displayed as D log N, log3/2(NT ), and z.Source: §E.3 · p. 42

E.4 Bound between particle dynamics and GD

  • Proposition 15 gives a high-probability bound between particle dynamics and gradient descent.The stated probability is at least 1 −e−z2.Source: §E.4 · p. 45

E.5 Bound between GD and SGD

  • Proposition 16 gives a high-probability bound between gradient descent and SGD.The proof treats the stochastic term as a martingale and applies Azuma-Hoeffding’s inequality.Source: §E.5 · pp. 46–47

F.1 Equation (DD) (noiseless SGD)

  • The PDE is interpreted weakly, and under A1–A4 its solution exists and is unique for fixed coefficients and for general coefficients.For fixed coefficients, bounded Lipschitz gradients suffice; for general coefficients, the argument is adapted from [SS18, Section 4] and [Szn91, Theorem 1.1].Source: §F.1 · p. 48

F.2 Equation (diffusion-DD) (noisy SGD)

This section analyzes the diffusion-DD limiting PDE for noisy SGD, establishing existence and uniqueness of a weak solution and results on its density and regularity.

F.2 Equation (diffusion-DD) (noisy SGD)

  • The section reproduces the limiting PDE and defines its weak solution notion, which is equivalent to the earlier notion in Eq. (61) [San15, Proposition 4.2].Source: §F.2 · p. 48
  • Under conditions A1–A5, PDE (62) admits a unique weak solution on [0, ∞).The proof uses a contraction mapping on short time intervals and iterates the fixed point to extend the solution globally.Source: §F.2 · pp. 48, 50
  • For sufficiently short intervals, the solution map Φ is a contraction, enabling the fixed-point construction of the unique weak solution.The argument is iterated to obtain existence and uniqueness on [0, ∞).Source: §F.2 · pp. 49–50
  • For any t > 0, a weak solution has a density satisfying the Duhamel representation for PDE (62).The section states Duhamel’s principle and identifies the heat kernel used in the representation.Source: §F.2 · p. 50
  • If conditions A1–A6 hold and ρ0 has a density, then ρ(t, θ) is C1,2 on (0, ∞) × R^D.This means it has a continuous time derivative and continuous second-order spatial derivatives.Source: §F.2 · p. 51

F.3 The noisy PDE as a gradient flow in the space of probability distributions

The section interprets PDE (62) as a Wasserstein gradient flow and establishes a unique weak solution under stated assumptions, with uniform bounds on key quantities.

F.3 The noisy PDE as a gradient flow in the space of probability distributions

  • PDE (62) is a gradient flow of the free energy F(ρ) over probability measures on R^D with the W2 distance [MMN18, Section 10.2].Source: §F.3 · p. 51
  • Under A1, A2, A3, and A6, PDE (62) has a unique weak solution whose densities are absolutely continuous and whose M(ρt) and Ent(ρt) are uniformly bounded in time.The result assumes ρ0 ∈ K and F(ρ0) < ∞.Source: §F.3 · pp. 52–53
  • The existence proof takes the vanishing-step limit of a discretized scheme, whose step has a unique solution, to obtain a weak solution [JKO98].The scheme's uniqueness follows from convexity of the functional and strict convexity of −Ent(ρ).Source: §F.3 · pp. 52–53

G Proof of Theorem 4 … H.5 The mean field limit

The appendix traces two-layer network training from finite-neuron dynamics to the mean-field limit and then the kernel limit. It describes the distributional and residual dynamics used to connect these regimes.

G Proof of Theorem 4

  • Theorem 4’s proof treats general coefficients by bounding their support, controlling the difference between mean-field and linearized residual dynamics, and deriving Eq. (22) via Lemma 1.Source: G Proof of Theorem 4 · pp. 54–56

H The mean field limit and kernel limit

H.1 Two layers neural networks with a scale parameter α

  • The scale parameter α permits exploration of learning regimes; the appendix takes α→∞ after N→∞ to examine the crossover between mean-field and kernel behavior.The text notes that joint limits are also possible and gives [JGH18] as an example; it follows [CB18a] for the sequential limit.Source: §H.1 · p. 56

H.2 The residual dynamics in the pre-limit

  • In the finite-neuron pre-limit, residual dynamics describe prediction changes but are not self-contained because they depend on the evolving weights θt.Source: §H.2 · p. 57

H.3 The distributional dynamics in the pre-limit

  • The pre-limit distributional dynamics is equivalent to gradient flow.Source: §H.3 · p. 57

H.4 The coupled dynamics

  • The coupled pre-limit distributional and residual dynamics are random because the neurons are independently initialized from ρ0.Source: §H.4 · p. 58

H.5 The mean field limit

  • For fixed α, under some conditions the N→∞ limit has a distributional evolution, whose residual dynamics depends on a kernel determined by the evolving distribution.The mean-field residual dynamics is not a self-contained equation.Source: §H.5 · p. 58

H.6 The kernel limit · H.7 Kernel limit as kernel ridge regression · I Technical lemmas

As α grows, mean-field residual dynamics converge to linearized dynamics governed by the initialization kernel. With empirical data and suitable assumptions, the kernel-limit prediction is zero-regularization kernel ridge regression; the technical section supplies auxiliary concentration bounds.

H.6 The kernel limit

  • As α becomes large, mean-field residual dynamics converge to the linearized residual dynamics for fixed time.Theorem 4 establishes this kernel limit.Source: §H.6 · p. 58
  • The linearized residual dynamics equal continuous-time kernel boosting with kernel Hρ0, whose solution is explicit.Source: §H.6 · p. 59
  • For a strictly positive-definite kernel, residual L2 norm tends to zero over time; empirical risk also permits bounding convergence time and approximating the linearized dynamics along the trajectory.For empirical data, a strictly positive-definite kernel matrix with least eigenvalue λmin > 0 enables the convergence-time bound.Source: §H.6 · p. 59

H.7 Kernel limit as kernel ridge regression

  • Under A1–A4, empirical data, an invertible data kernel matrix, and initialization property (I), the kernel-limit prediction performs kernel ridge regression with λ = 0.Property (I) requires initialization weight a to be independent of w and to have mean zero.Source: §H.7 · pp. 59–60

I Technical lemmas

  • Lemma 30 gives an auxiliary bound for functions of i.i.d. bounded, mean-zero random vectors.The supplied passage does not include the bound’s full expression.Source: I Technical lemmas · p. 61
  • Lemma 31 states an Azuma–Hoeffding bound for an RD-valued martingale starting at zero.The supplied passage omits the bound’s precise conditions and expression.Source: I Technical lemmas · p. 61

Related papers

Loading 1902.06015v1…