Asymptotics of Wide Networks from Feynman Diagrams

• Wide-network analysis uses the large-width limit to study initialization and training dynamics, addressing the limited analytic understanding of deep networks.Source: §1 · p. 2
• Correlation functions average network outputs, products, and parameter derivatives, encompassing quantities such as Gaussian-process kernels and the Neural Tangent Kernel.Source: §1 · p. 2
• The paper adapts Feynman diagrams to bound correlation-function asymptotics, introducing a treatment for network-function derivatives absent from the original formulation.Source: §1 · p. 2
• For gradient-descent evolution, the method tightens gradient-flow bounds, extends analysis to SGD, and derives explicit first-order finite-width corrections.Source: §1 · p. 3
• Additional applications show that large-width SGD updates are linear in the learning rate and examine finite-width corrections to the Hessian spectrum.Source: §1 · p. 3
• The main result remains a conjecture without a proof covering all tested cases, and the method yields asymptotic bounds rather than width-independent coefficients.Source: §1 · p. 3

2.1 Notation

• The network notation uses equal layer widths n for simplicity, with Gaussian initialization and results extending to widths that scale linearly with n.Source: §2.1 · pp. 3–4
• The section defines correlation functions as expectations of products of derivative tensors with parameter indices paired by contractions.Source: §2.1 · p. 4

2.2 Asymptotic bounds on wide networks

• Conjecture 1 bounds a correlation function as O(n^sC) from the number of even connected components in its cluster graph, with no odd-sized components.Source: §2.2 · p. 4
• Feynman diagrams compute the Kronecker-delta sums determining large-width behavior, providing the analytic basis for the deep linear-network theorem.Source: §2.2 · pp. 4–5
• The conjecture also holds for ReLU networks with equal inputs and one-hidden-layer networks with smooth activations.Source: §2.2 · p. 5

2.3 Numerical experiments

• Numerical experiments on linear, ReLU, and tanh networks support the conjecture in all tested cases, with bounds tight in most cases.The experiments use three-hidden-layer networks across widths 27 through 2^13, averaging 1,000 initializations and fitting the exponent.Source: §2.3 · p. 5, Table 1 · p. 5

3 Applications to training dynamics

• For fixed training time, the expected NTK deviation from initialization is O(n−1) under gradient flow, improving the bound in [4].The result assumes analyticity in training time and exchange of the Taylor expansion with the large-width limit.Source: §3 · p. 6
• The same asymptotic NTK-deviation bound, Eθ[Θ(t)−Θ(0)] = O(n−1), extends to stochastic gradient descent.The derivation again relies on Conjecture 1.Source: §3 · p. 6
• The derived O(n−1) bound for kernel corrections during SGD improves the existing O(n−1/2) gradient-flow bound and is empirically tight.The method also applies when all layer widths grow simultaneously.Source: §3 · pp. 6–7

3.1 Finite width corrections

• At order O(n−1), the paper derives explicit time-dependent corrections to the network map and kernel beyond the infinite-width evolution.The infinite-width network evolution is f0(t) = y + e−tΘ(0)(f(0)−y), and Figures 1b–1c compare the correction predictions with empirical evolution.Source: §3.1 · p. 7
• Higher-order operators with more derivative tensors are increasingly width-suppressed, so operators Os with s ≥ 5 can be omitted when solving f and Θ through O(n−1).The paper gives Eθ[dO4/dt] = O(n−2) at all times for the relevant term.Source: §3.1 · p. 7

4 Feynman diagrams for deep linear networks

• The diagram family Γ(C) uses m vertices of degree d+1, typed edges for layer weights, and forced edges for contracted derivative tensors.The graphs are independent of the input values.Source: §4 · pp. 7–8
• For a single hidden layer with linear activation, the network covariance Eθ[f(x1)f(x2)] scales as O(n0), with input dependence irrelevant to the width exponent.The contributing factors are the network’s n−1/2 normalization and a summed Kronecker-delta product.Source: §4 · p. 8
• Each Feynman diagram contributes n^(lγ−m/2), because vertices contribute n−1/2 and loops contribute n.The loop factor arises from sums over products of Kronecker deltas via Isserlis’ theorem.Source: §4 · pp. 8–9
• For Eθ[f(x1)f(x2)f(x3)f(x4)], disconnected diagrams scale as O(n0) while connected diagrams scale as O(n−1), so the correlation is O(n0).Connected graphs therefore vanish faster at large width than disconnected graphs.Source: §4 · p. 9, Figure 2 · p. 9
• Contracted derivative pairs are represented by forced edges because paired derivatives produce Kronecker-delta factors equivalent to covariance factors.For the NTK correlation function, this leaves one contributing diagram and gives Eθ[Θ(x,x′)] = O(n0).Source: §4 · p. 9
• The derivative treatment is presented as novel because prior physics applications were not known to include derivatives in Feynman-diagram correlation functions.The paper proves the general theorem separately, with the displayed proof focused on one hidden layer and linear activations.Source: §4 · pp. 9–10

5 Discussion

• The method gives asymptotic behavior for ensemble averages including losses, network covariances, and the NTK through Conjecture 1.These functions recur throughout wide-network analysis.Source: §5 · p. 10
• Conjecture 1 has a complete analytic treatment for deep linear networks and is supported by analytic and empirical evidence for nonlinear and non-Gaussian networks.The paper reports that it holds in all cases tested.Source: §5 · p. 10
• The applications tighten known bounds, extend analysis to SGD and gradient flow, and derive closed-form first finite-width corrections to network evolution.The authors describe these results as a basis for systematic expansion around infinite width.Source: §5 · p. 10
• The work addresses whether the infinite-width limit models realistic deep networks by calculating the next perturbative order toward finite-width descriptions [28] [29].The paper presents this as a step toward an analytic description of finite-width networks.Source: §5 · pp. 10–11

A.1 Non-Gaussian initialization

• Uniform {±1} initialization yields good agreement with Conjecture 1’s correlation-function bounds, with tight bounds in many cases.The setup matches Table 1 except for the initialization distribution.Source: Table 2 · p. 11, §A.1 · p. 11

A.2 Experimental details

• Two-class MNIST experiments measured one randomly chosen kernel component and individual network trajectories under specified widths, steps, learning rates, samples, and initialization counts.Kernel measurements used 1024 steps, learning rate 1.0, 1000 samples per class, and 100 initializations; trajectory curves used 10 samples per class and learning rate 0.1.Source: §A.2 · p. 11
• The network input is normalized by the square root of its input dimension in the MNIST experiments.This normalization follows [4].Source: §A.2 · p. 11

B Feynman diagrams for deep linear networks

• Feynman diagrams derive asymptotic upper bounds for deep linear networks in the large-width limit.The section uses this method to prove Theorem 1.Source: B Feynman diagrams for deep linear… · p. 12

B.1 Feynman diagrams and double-line diagrams

• Isserlis’ theorem expands Gaussian parameter moments into permutation-indexed terms, each corresponding to a Feynman diagram; cross-layer covariance independence can leave only one nonvanishing diagram in simple examples.For Eθ[f(x)f(x′)] with two hidden layers, the calculation yields a single Feynman diagram.Source: §B.1 · p. 12
• Double-line diagrams resolve matrix-covariance counting by replacing each matrix covariance’s two Kronecker-delta lines with a graph whose loops count triangulation faces.U and V edges become single edges, whereas W edges become double edges; the resulting surfaces can have boundaries.Source: §B.1 · pp. 12–13
• For a deep linear correlation function, each diagram contributes O(n^sγ) with sγ determined by double-line loops and network depth, while the full function scales as O(n^s) with the maximal exponent.The rule is sγ = lγ − dm/2 and s = maxγ∈Γ(C) sγ.Source: §B.1 · p. 14

B.2 Asymptotics of deep linear networks

• For deep linear networks, connected-component and surface arguments convert diagram topology into asymptotic bounds on correlation functions.The proof uses Euler characteristic bounds for connected surfaces with at least one boundary, together with component parity constraints.Source: §B.2 · pp. 14–15

C.1 Deep networks with ReLU non-linearities

• With ReLU activations and equal inputs, the correlation-function asymptotic bound is no larger than that of the corresponding deep linear network.The argument treats binary activations as independent of weights in the relevant construction and reduces the remaining Gaussian calculation to the linear case.Source: §C.1 · pp. 15, 17
• The ReLU reduction relies on independence properties of activation masks and weights that fail for non-equal inputs.Thus the proof does not cover correlation functions with non-equal inputs.Source: §C.1 · pp. 15–16

C.2 Single hidden layer networks

• Consequently, Feynman-diagram asymptotics for sufficiently deep linear networks provide bounds for single-hidden-layer networks with smooth nonlinearities.The resulting comparison is C(x1, . . . , xm; fnl) = O(C(x1, . . . , xm; f)).Source: §C.2 · pp. 17, 19
• The single-hidden-layer analysis bounds asymptotic scaling by first maximizing index sums in reduced correlation-function terms and then bounding each summand independently of width.The maximum number of index sums is denoted rmax.Source: §C.2 · p. 18
• The auxiliary graph family Γ′(C) maps derivative tensors to vertices, weight-vector covariances to typed edges, and contractions to connecting edges.Each connected component corresponds to one remaining index sum, so the maximal number of sums equals the maximal component count.Source: §C.2 · pp. 18–19
• The maximal component count in Γ′(C) equals the corresponding deep-linear Feynman-diagram count when the comparison depth is sufficient for the derivatives.The required depth is at least the maximum derivative-tensor rank appearing in the correlation function.Source: §C.2 · p. 19

D.1 Variance asymptotics

• If Conjecture 1 gives C = O(n^sC), then the variance of the corresponding integrand is O(n^2sC), and typical initializations are also O(n^sC).Empirical tests in Table 3 found the Lemma 5 variance bound in all cases tested.Source: §D.1 · p. 20, Table 3 · p. 20

D.2 Gradient Flow

• Gradient-flow derivative bounds are established for polynomial losses by expressing time derivatives as combinations of correlation functions whose cluster graphs extend the original graph.More general losses require exchanging the large-width limit with the loss Taylor expansion.Source: §D.2 · p. 21

D.3 Stochastic Gradient descent

• After t SGD steps, a linear-activation correlation function remains O(n^sC) when Conjecture 1 holds.Each update expands derivative tensors into correlation functions whose cluster graphs are subgraphs of the original, allowing Lemma 6 to preserve the bound.Source: §D.3 · pp. 21–22
• For general activation functions, the SGD proof is incomplete because the update expansion may contain an infinite sum whose aggregate bound is not established.The paper leaves this case for future work.Source: §D.3 · p. 22

E.1 Neural Tangent Kernel

• The NTK converges in probability at large width, with variance O(n−1).Source: §E.1 · pp. 22–23
• Connected Feynman diagrams determine the NTK variance scaling, because the surviving diagrams scale as O(n−1).Source: §E.1 · p. 23
• At large width, the NTK remains constant during training in expectation up to O(n−1) corrections for deep linear networks.Source: §E.1 · p. 24

E.2 Wide network evolution is linear in the learning rate

• The same evolution argument extends to stochastic gradient descent by replacing full-training-set sums with mini-batch sums.Source: §E.2 · pp. 24–25
• Wide-network function updates under discrete gradient descent are linear in the learning rate at leading order.Source: §E.2 · pp. 24–25

E.3 Spectral properties of the NTK and the Hessian

• The top D_in Hessian eigenvalues receive corrections scaling as O(n−1/2) for one hidden layer and O(n−1) for deeper networks.Source: §E.3 · p. 25
• The remaining Hessian eigenvalues vanish at large width with the same one-layer and deeper-network scalings, corroborated by Figure 8.Source: §E.3 · p. 25, Figure 8 · p. 27
• At late times, expected traces of every Hessian power approach the corresponding NTK traces, while training further reduces eigenvalue differences.Source: §E.3 · p. 26
• In multi-class networks, the large-width NTK has repeated class-space spectral copies, implying repeated Hessian-spectrum structure under MSE loss.Source: Figure 8 · p. 27, §E.3 · p. 28

E.4 Higher-order network evolution

• Higher-order dynamics can be expanded in 1/n and explicitly computed at O(1/n), with predictions experimentally confirmed for NTK evolution.Source: §E.4 · p. 28
• Finite-width corrections arise from the initial kernel, nonlinear-in-learning-rate updates, and kernel non-constancy.Source: §E.4 · p. 28

E.4.1 Continuous time.

• An infinite ODE tower governs network-map and kernel evolution, but truncating it at s=2r yields dynamics accurate through O(n−r).Source: §E.4.1 · p. 29
• Each operator is expanded order by order in 1/n, and higher-order equations can be solved iteratively using lower-order solutions.Source: §E.4.1 · pp. 29–30
• The complete O(1/n) continuous-time correction predicts full time dependence and is experimentally confirmed for NTK evolution.Source: §E.4.1 · p. 31, Figure 10 · p. 32
• At late times, the O(1/n) correction to the NTK approaches a constant, so general-loss error evolution uses a corrected constant kernel.Source: §E.4.1 · p. 31
• For non-smooth activations such as ReLU, differentiability assumptions are suspect, so the presented experiments use smooth activations and leave full analysis open.Source: §E.4.1 · p. 31

E.4.2 Discrete time.

• Discrete-time corrections include novel terms that vanish in continuous time, while the equations remain solvable iteratively order by order in 1/n.Source: §E.4.2 · pp. 33–34