PaperlayerView exact revision on arXiv

Source-linked AI summary

Para-Gaussian Approximations of Random Deep Neural Networks

Mihai Nica, Janosch Ortmann

arXiv:2610.37388v1cs.LGmath.PR

TL;DR

Gaussian approximations of randomly initialized networks can be poor at finite width and worsen with depth. This paper constructs a Hermite-based para-Gaussian approximation and recursions for deep networks. It reports an n^-2 total-variation approximation rate and treats linear and ReLU networks.

  • Problem

    Gaussian approximations can worsen with network depth, limiting their accuracy for finite-width networks.

    Source: §1.1 · p. 2

  • Method

    The paper represents finite collections of network neurons by a para-Gaussian law with fourth-order Hermite corrections and derives recursions for its parameters.

    Source: §1.6.1 · p. 9, §1.6.3 · p. 10

  • Results

    The para-Gaussian approximation holds in total variation with rate n^-2 for networks of arbitrary depth and finite numbers of inputs and neurons.

    Source: §1.6.3 · p. 10

  • Takeaways & Limitations

    The paper provides an explicit non-Gaussian distributional approximation that accounts for dependence among neurons in finite-width networks.

    Source: §1.6.1 · p. 9, §1.6.3 · p. 10

    For ReLU networks, the Gaussian expectations do not yield a closed recursion for finitely many coefficients, and the fourth-cumulant coefficient lacks an exact formula for several inputs.

    Source: §2.3 · p. 15

Abstract

from arXiv · show

It is known that randomly initialized neural networks have Gaussian behaviour in the $n\to \infty$ limit. Unfortunately, this approximation can be poor for finite $n$ and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is \emph{slightly} non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).

1 Introduction

Gaussian behavior describes randomly initialized networks at infinite width, but its approximation can worsen with depth. The paper introduces a Hermite-based para-Gaussian law and develops methods for applying it to deep networks.

1.1 Gaussians and not-quite-Gaussians

  • The paper extends Gaussian approximations with a para-Gaussian density built from Hermite polynomials to retain non-Gaussian corrections.The paper situates this construction alongside the Edgeworth expansion around the central limit theorem for sums of i.i.d. random variables.Source: §1.1 · p. 2
  • The para-Gaussian law matches single-neuron Monte Carlo simulations more accurately than the Gaussian approximation, and the Gaussian deviation grows with depth.The illustration uses a randomly initialized ReLU network at layers 2, 4, and 8.Source: §1 · p. 1, §1.1 · p. 3

1.2 Outline and Contributions

  • The paper derives a recursion for estimating the fourth cumulant in neural networks using multivariable Hermite polynomials.This is presented as part of the layerwise method for propagating the para-Gaussian approximation.Source: §1.2 · p. 3
  • For linear networks, the paper solves the covariance and fourth cumulant exactly and tests the resulting approximation against Monte Carlo experiments.For the linear activation, the infinite-width covariance remains equal to the input covariance across layers.Source: §1.2 · p. 3
  • The paper also applies its theory to ReLU networks to obtain approximations tested against Monte Carlo experiments.The introduction identifies this alongside the exact linear-network analysis as an application of the theory.Source: §1.2 · p. 3

1.3 Total Variation Distance and Asymptotic Rates of Convergence

The section defines total-variation convergence rates and shows that, under smoothness and kernel non-degeneracy assumptions, a fourth-cumulant para-Gaussian approximation improves the rate over a Gaussian approximation for finite collections of network neurons.

1.3 Total Variation Distance and Asymptotic Rates of Convergence

  • Total-variation convergence measures the worst-case difference in expectations over measurable test functions bounded by one.The section defines an approximation rate through the decay of this distance as width n grows.Source: §1.3 · p. 4
  • For symmetric summands, Gaussian approximation has rate n^-1 while the fourth-cumulant para-Gaussian approximation has rate n^-2.The stated rates improve when all odd cumulants vanish; the fourth cumulant supplies the leading correction.Source: §1.3 · p. 4

1.4 Random fully connected neural networks on initialization and their rates of convergence

  • For finitely many neurons at a fixed layer, the para-Gaussian approximation has error O(n^-2), versus O(n^-1) for the Gaussian, under the stated smoothness and non-degeneracy assumptions.The Gaussian rate was previously established; the para-Gaussian rate uses a fourth-cumulant Hermite correction.Source: §1.4 · p. 6
  • The paper focuses on fourth-cumulant corrections, while describing higher-order 2p corrections and rate n^-p as a possible extension.The authors limit the present treatment to the fourth cumulant for concreteness and exposition.Source: §1.4 · p. 6
  • The proof attributes finite-width non-Gaussianity to fluctuations in the conditional covariance and matches the para-Gaussian correction through a Taylor expansion.Truncating after the first two terms yields the Gaussian bound, while retaining the first four terms yields the para-Gaussian bound.Source: §1.4 · pp. 6–7

1.5 Recursions for calculating the covariance and fourth cumulant

The section gives a recursive procedure for computing average covariance and a rescaled covariance tensor, which together provide the inputs for para-Gaussian approximations in deeper layers.

1.5 Recursions for calculating the covariance and fourth cumulant

  • The fourth cumulant needed for the next layer depends on covariances of the random covariance matrix Σℓ+1.The section introduces Cℓ to track these covariance terms and defines it through covariances of Σℓ.Source: §1.5 · p. 7
  • The rescaled tensor Cℓ remains order one as width grows, while the fourth cumulant itself decays like 1/n.This scaling motivates tracking Cℓ in the recursion.Source: §1.5 · p. 7
  • Expectations under the para-Gaussian law can be evaluated as Gaussian expectations with a modified integrand.The density is expressed as a Gaussian factor multiplied by Hermite-polynomial correction terms.Source: §1.5 · p. 7
  • Recursing average covariance and Cℓ provides a practical way to compute para-Gaussian approximations with error size O(1/n^2).The recursion uses quantities indexed by inputs, with neuron-level distributions obtained using exchangeability.Source: §1.5 · p. 7

1.6 Related work

The section situates finite-width corrections beyond the Gaussian limit and contrasts perturbative descriptions with an explicit para-Gaussian law for the network. It also relates the approximation’s scope to network depth and earlier work by the authors.

1.6.1 Perturbative descriptions of finite-width networks

  • Earlier finite-width work developed fourth-cumulant recursions [Yai20] [RYH22], while supplied rigorous cumulant and test-function results and single-input recursions.The paper positions its results relative to these perturbative descriptions.Source: §1.6.1 · pp. 8–9
  • The joint network law is approximated in total variation by a Gaussian density corrected with fourth-order Hermite polynomials, enabling density comparisons and bounds for arbitrary bounded test functions.The errors depend on a test function only through its supremum, including for indicator functions.Source: §1.6.1 · p. 9
  • For any width, neuron cumulants of order 2p have an explicit pairing formula in terms of cumulants of the random covariance matrix, exposing higher-order dependence between uncorrelated neurons.The formula sums over (2p − 1)!! pairings, weighted by Kronecker deltas in neuron indices and cumulants of entries of Σℓ.Source: §1.6.1 · p. 9
  • Theorem 1.18 gives recursions for mean covariance and fourth cumulants across multiple inputs and activation functions satisfying the assumptions of [Han24].Written with Hermite polynomials, their coefficients are explicit Gaussian integrals; the ReLU integrals are evaluated numerically.Source: §1.6.1 · p. 9
  • The exact linear-network solution for multiple inputs and neurons reveals a third fourth-cumulant pairing, which becomes important when ℓ/n is of order one.This pairing is invisible at first order in the 1/n recursion.Source: §1.6.1 · p. 9

1.6.2 Deep networks and the depth-to-width ratio

  • When ℓ/n is of order one, finite-cumulant descriptions no longer suffice; the linear first-order recursion departs from exact coefficients, and the ReLU 1/n recursion underestimates cumulants.The para-Gaussian law is presented as an approximation for small ℓ/n.Source: §1.6.2 · p. 10

1.6.3 Generalization of earlier work by the authors

  • This paper extends from a single neuron, one hidden layer, and at most two inputs to arbitrary depth, finitely many inputs and neurons, with a total-variation error rate n^-2 and layerwise parameter recursions.The extension includes dependence between different neurons.Source: §1.6.3 · p. 10

1.7 AI Disclosure

  • Claude Opus 5.5 assisted with writing, discovered the law-of-total-cumulance proof of Theorem 2.1, and wrote the Monte Carlo experiment and figure-generation code.The authors’ original human proof is included in Appendix B.Source: §1.7 · p. 10

2 More Results

The section gives exact cumulant formulas and applies them to linear and ReLU networks. Linear networks admit an exact fourth-cumulant recursion, while ReLU results rely on numerical evaluation of the 1/n recursions.

2.1 Exact formulas for higher cumulants

  • Odd neuron cumulants vanish, and even cumulants can be expressed through the layer’s random matrix Σℓ.The formula sums over pairings of indices.Source: §2.1 · p. 10

2.2 Linear Activation Function

  • For linear networks, the mean covariance remains K at every layer, and the fourth-cumulant tensor has an exact recursive formula.The fourth-cumulant recursion adds three covariance-pairing terms to the previous layer’s tensor.Source: §2.2 · p. 11
  • The linear-network fourth cumulant has an exact solution, and its coefficients satisfy vℓ = ℓ + O(1/n) and wℓ = 0 + O(1/n) for fixed ℓ.The exact solution includes a third pairing with coefficient wℓ, which becomes important when ℓ/n is of order one.Source: §2.2 · p. 12
  • For the linear network’s joint density, the para-Gaussian approximation and Monte Carlo agree closely in the bulk but differ most in the tails.The reported difference is near zero inside the inner ellipse and of order 0.1 in the ring between ellipses; in the tails, the para-Gaussian ratio keeps growing while the true density does not.Source: §2.2 · p. 14

2.3 ReLU Activation Function

  • For ReLU networks with distinct inputs, the covariance has a nonzero finite-width correction, and no exact fourth-cumulant formula is available for several inputs.The Gaussian expectations in the recursions are evaluated numerically; positive homogeneity lets them factor into radial moments and angular integrals.Source: §2.3 · p. 15
  • For ReLU networks, the 1/n recursion is accurate when ℓ/n is small but underestimates cumulants once ℓ/n is of order one.At ρ = 0, the recursion already gives a positive cross-neuron cumulant because ReLU layers make initially orthogonal inputs positively correlated.Source: §2.3 · pp. 15–16

3 Proof of the Asymptotic Convergence, Theorem 1.13

The proof conditions on the previous layer, where the current layer is Gaussian with a random covariance. It decomposes that covariance into its mean and fluctuations, then expands around the mean.

3 Proof of the Asymptotic Convergence, Theorem 1.13

  • Conditionally on the previous layer, Zℓ is Gaussian with covariance determined by the random matrix Σℓ.The proof approximates expectations by expanding around the mean covariance after separating Σℓ into its mean and fluctuations.Source: §3 · p. 17

3.1 gf and its derivatives

The random function g_f interpolates between the Gaussian approximation and the conditional quantity of interest. Taylor expansion around the Gaussian endpoint, together with Gaussian derivative identities, yields the para-Gaussian form and controls its error through the test function’s supremum norm.

3.1 gf and its derivatives

  • g_f(0) gives the Gaussian approximation, while g_f(1) recovers the conditional expectation of interest.Taylor expansion around t = 0 compares these endpoints.Source: §3.1 · p. 18
  • Differentiating Gaussian expectations with respect to covariance can be represented using expectations of derivatives of f.The proof justifies the derivative formula through a Gaussian diffusion and Itô’s lemma.Source: §3.1 · p. 18
  • Gaussian integration by parts transfers derivatives from f to the Gaussian density, where they become Hermite polynomials.This identifies the para-Gaussian law and makes the error estimates depend on f only through ∥f∥∞.Source: §3.1 · p. 18
  • The first derivative of g_f at zero is linear in the centered covariance fluctuation, so its expectation vanishes.The coefficients are finite and deterministic because they depend on the mean covariance.Source: §3.1 · p. 19
  • The expected second derivative depends only on the second moments of the covariance fluctuation, summarized by Tℓ.The second derivative is quadratic in the entries of Δℓ.Source: §3.1 · p. 19

3.2 Pair symmetries and alternate form of the para-Gaussian law

The section relates pair-symmetric tensors to fully symmetric fourth-order tensors through a pairing sum. This gives an alternate form of the para-Gaussian law and connects its normalization to an earlier convention.

3.2 Pair symmetries and alternate form of the para-Gaussian law

  • A prototypical pair-symmetric tensor is Cov(YaYb, YcYd), symmetric within each index pair and under exchange of the pairs.Its pairing sum groups four indices into the three possible pairs.Source: §3.2 · pp. 19–20
  • For pair-symmetric T, its pairing sum κ is fully symmetric and can be contracted against fully symmetric tensors using one-third of the pairing-sum contraction.The three pairings are {ab | cd}, {ac | bd}, and {ad | bc}.Source: §3.2 · p. 20
  • When κ is the pairing sum of pair-symmetric T, the para-Gaussian law can be written using T with normalization 1/8 instead of 1/4!.The measure is parameterized by covariance Σ and fully symmetric fourth tensor κ, which plays the role of fourth cumulants.Source: §3.2 · p. 20
  • For one neuron and one layer, the 1/8n convention matches the para-Gaussian definition in and the 1/4! Edgeworth convention.In this case, κ is the 1/n sum of the three pairings of C from [NO24, Theorem 2].Source: §3.2 · p. 21

3.3 Taylor series expansion of gf

The expected degree-two Taylor polynomial of g_f is the para-Gaussian law. Its covariance and fourth cumulants match those of Zℓ, with the first-order term vanishing and the second-order term supplying the cumulant contribution.

3.3 Taylor series expansion of gf

  • The first-order Taylor term has mean zero, while the second-order term is evaluated using the second-derivative formula.Gaussian integration by parts moves derivatives onto a Hermite factor; the pairing sum of Tℓ is κℓ.Source: §3.3 · p. 21
  • The expected degree-two Taylor polynomial of g_f at zero is the para-Gaussian law, whose covariance and fourth cumulants match those of Zℓ.Its covariance is I_m ⊗ Σ̄ℓ, and its fourth cumulant tensor is κℓ.Source: §3.3 · p. 21

3.4 Error analysis

The error analysis combines fluctuation estimates with derivative bounds to control Taylor-expansion errors, including when the interpolated covariance may become singular. It then assembles these bounds for the approximation results.

3.4.1 Estimates imported from [Han24]

  • Imported estimates give eventual positivity of the mean covariance and fluctuation moment control, including O(n^-2) third moments.Source: §3.4.1 · p. 22

3.4.2 Deterministic estimates on derivatives of Gf

  • Gaussian integration by parts transfers derivatives from the test function to Hermite polynomials, yielding bounds in ∥f∥∞ that worsen as the covariance approaches singularity.Source: §3.4.2 · p. 23

3.4.3 The Taylor remainder

  • The Taylor remainder is handled by splitting small from large fluctuations; for large ones, separate deterministic bounds avoid relying on Taylor’s theorem near singular covariance.Both cases yield a bound in powers of the fluctuation relative to the mean covariance’s minimum eigenvalue.Source: §3.4.3 · p. 24

3.4.4 The third order term

  • The third-order term is controlled by bounding Hermite-polynomial coefficients and using the fluctuation’s third-moment estimate.Source: §3.4.1 · p. 22, §3.4.4 · p. 25

3.4.5 Summary of all Taylor series error bounds

  • Corollary 3.23 summarizes Taylor-series error bounds for bounded, eight-times-differentiable test functions when n is sufficiently large.The threshold ensures the mean covariance is positive definite.Source: §3.4.5 · p. 25

3.5 Proof of Theorem 1.13

The proof compares expectations under the network law with Gaussian and para-Gaussian approximations using a Taylor expansion of a conditional Gaussian expectation. It obtains total-variation error rates of n^-1 for the Gaussian and n^-2 for the para-Gaussian approximation.

Expansion of E[f(Zℓ)]

  • Expanding the conditional Gaussian expectation around the mean covariance gives the Gaussian baseline at zeroth order, with the first-order term vanishing in expectation.Source: §3.5 · p. 26

Part (i) and Part (ii)

  • For the para-Gaussian approximation, taking the supremum over f yields total-variation error at most 6C n^-2 for all n ≥ n0, where C does not depend on f.Source: §3.5 · p. 26

Part (ii)

  • Keeping terms through order two gives exactly the para-Gaussian law, which underlies its improved total-variation rate.Source: §3.5 · p. 26

4 Derivation of recursion relations, Theorem 1.18

The section derives layer-to-layer covariance and fourth-cumulant recursions by approximating expectations of functions of finitely many neurons with the para-Gaussian law. Hermite-polynomial splitting separates contributions from one neuron and from pairs of neurons.

4.1 Estimating expectation of polynomials under para-Gaussian law

  • Expectations of continuous polynomially bounded functions of finitely many neurons are approximated by the para-Gaussian law with error of order n^-2.This covers unbounded products of activations that the bounded-test-function estimate does not handle directly.Source: §4.1 · p. 27

4.2 Spliting Hermite polynomials

  • Because the Gaussian reference covariance is block diagonal across independent neurons, multivariate Hermite polynomials split into products of neuron-specific Hermite polynomials.Source: §4.2 · pp. 27–28

4.3 Expectations of one neuron and covariances of two neurons

  • The para-Gaussian correction shifts nonlinear activation means at order 1/n, while identity activation has no fourth-order correction; cross-neuron covariance is also order 1/n.The cross-neuron term comes from second-order Hermite factors on each neuron, whereas same-neuron products can be correlated even under the Gaussian reference.Source: §4.3 · pp. 28–29

4.4 Proof of Theorem 1.18

  • The recursion’s fourth-cumulant update combines a same-neuron covariance already present under Gaussianity with contributions from the n−1 pairs of distinct neurons.Each distinct-neuron pair contributes at order 1/n through second-order Hermite polynomials.Source: §4.4 · p. 30

A Para-Gaussian theory

The para-Gaussian family is built as a Hermite-corrected Gaussian density and has useful closure and moment properties. When its density is nonnegative, it defines a probability law.

  • A para-Gaussian density is integrable, and when it is nonnegative almost everywhere it defines a probability measure.Source: A Para-Gaussian theory · p. 30
  • The projection onto a subset of variables remains para-Gaussian, with the covariance kernel and cumulants restricted to that subset.Source: A Para-Gaussian theory · p. 30
  • Any linear combination of para-Gaussian variables is itself para-Gaussian, with transformed covariance and cumulant parameters.Source: A Para-Gaussian theory · p. 31
  • For polynomials of degree less than k, expectation under the para-Gaussian law equals the corresponding Gaussian expectation.Source: A Para-Gaussian theory · p. 31

B Alternate hands-on proof of Theorem 2.1

The alternate proof derives Theorem 2.1 by expanding the layer’s characteristic function through a scalar quadratic form and matching cumulant coefficients.

Loading 2610.37388v1…