Mean Field Residual Networks: On the Edge of Chaos

Motivation

• Vanilla networks move between stable and chaotic regimes as initialization variance changes, with exponential input-distance convergence and corresponding vanishing or exploding gradients [9] [3] [11].Prior work argues that effective computation requires behavior near the edge of chaos [9].Source: §1 · p. 1

Residual architectures

• Residual networks [6] [7] add identity passthrough connections so each nonlinear layer computes a residual relative to the previous activation.The study considers fully connected residual networks without batch normalization.Source: §1 · p. 2

Theoretical dynamics

• For many nonlinearities, residual networks converge toward stable-chaotic boundary points polynomially rather than exponentially, while gradients grow subexponentially and α-ReLU gradients grow polynomially for α < 1.These dynamics let networks hover near the boundary across many layers.Source: §1 · p. 2

Initialization

• Initialization quality depends on depth: tanh residual networks are governed by average gradient explosion, whereas α-ReLU networks are governed by expressivity, making Xavier and He [5] schemes non-optimal.The paper identifies expressivity, rather than gradient behavior, as the relevant initialization criterion for α-ReLU residual networks.Source: §1 · p. 2

Scope and validation

• The theoretical and empirical results concern a simplified residual-network model without batch normalization or convolutional layers, so they may not directly describe practical MSRA residual networks [6].Experiments nevertheless predict relative trained-network performance from initialization hyperparameters across settings.Source: §1 · p. 2

Network formulation

• Vanilla feedforward networks consist of layered affine transformations with coordinatewise nonlinearities such as tanh or ReLU.The network uses activations x(l), pre-activations h(l), weight matrices w(l), bias vectors b(l), and a nonlinearity φ.Source: §2 · p. 2

Vanilla network dynamics

• Poole et al. [9] showed that normalized activation lengths and dot products converge exponentially to fixed points, causing cosine distance to forget input geometry quickly.The cosine distance is determined by the ratio of normalized squared length to normalized dot product.Source: §2 · pp. 2–3

Gradient dynamics

• Schoenholz et al. [11] found that expected normalized squared gradient norms vanish or explode exponentially with depth, with the half-life controlled by σw and σb.Their theoretical half-life correlated with the maximum depth supporting good practical performance.Source: §2 · p. 3

Related work

• Daniely et al. [3] provided finite-width, logarithmic-depth high-probability estimates but fixed variance parameters and used a framework that does not immediately generalize to residual networks.Their work was formulated through reproducing kernel Hilbert spaces.Source: §2 · p. 3

Background and contribution

• Residual networks typically yield subexponential, often polynomial, cosine-distance convergence and gradient-growth rates, unlike vanilla feedforward networks.The work identifies these dynamics as fundamentally different from the exponential behavior of vanilla networks.Source: §2 · p. 3

Architectures

• Residual networks, introduced by [6] and refined by [7], add shortcut connections; the paper analyzes fully connected reduced and full residual architectures.The full residual network additionally connects the nonlinearity to the next layer through affine weights and biases.Source: §3 · p. 3

Initialization and assumptions

• Weights and biases are independently Gaussian across layers, with fixed initialization scales, while theoretical calculations use symmetry and gradient-independence assumptions.The assumptions treat activations and gradients as coordinate-symmetric and posit independent backward weights and gradients relative to previous-layer activations.Source: §3 · pp. 3–4

Mean-field quantities

• The analysis defines activation-length quantities q(l) and input-scale p(0) as expectations over random initialization in the infinite-width limit.These quantities characterize average layerwise signal magnitudes under the mean-field setting.Source: §3 · p. 4

Input geometry

• For two inputs, correlation γ(l), metric expressivity s(l), and angular expressivity e(l) quantify Euclidean separation and normalized alignment through the network.Under equal initial input scales, these simplify to s(l) = (1 − e(l))p(l) and e(l) = γ(l)/p(l).Source: §3 · p. 4

Gradient quantities and asymptotics

• Gradient quantities χ(l) measure expected squared backpropagated gradients, while O, Θ, ˜O, ˜Θ, and ˇΘ specify the paper’s asymptotic growth conventions.The gradient framework builds on Axiom 3.2, previously used for feedforward tanh networks [11] and related to feedback alignment.Source: §3 · p. 4

Mean-field analysis

• The central-limit approximation tracks activation means and variances, yielding recurrence equations for p, q, γ, λ, and χ across architectures and nonlinearities.These recurrences govern forward quantities e and s and backward quantity χ.Source: §4 · pp. 4–5

Forward dynamics

• Residual networks converge polynomially rather than exponentially toward fixed cosine geometry for tanh and α-ReLU, preserving input-space information over practical depths.The cosine distance e converges to e* as depth grows, but its slower convergence keeps typical finite-depth networks away from the limiting stability or chaos values.Source: §4 · p. 5

Forward and backward dynamics

• For tanh and α-ReLU with α < 1, the metric geometry quantity s evolves polynomially with depth, while most considered residual architectures have subexponential gradient dynamics.Residual networks avoid vanishing χ, unlike vanilla tanh networks; ReLU is the exception for full residual-network gradient dynamics.Source: §4 · p. 5

Gradient dynamics

• Weight gradients remain subexponential or polynomial for tanh and α-ReLU, while ReLU residual weight gradients are layer-independent within O(1), uniquely preserving gradient norms.For α-ReLU with α < 1, χ(0) is polynomially related to χ(l); for ReLU ResNets, χ itself remains exponentially varying despite O(1) weight gradients.Source: §4 · p. 5

Experiments

• Theory accurately predicts which initialization performs better after training: tanh controls χ(0)/χ(L), whereas ReLU and α-ReLU maximize s without numerical instability.The comparison concerns test-set performance after training across random initializations.Source: §4 · p. 5

Kernel identities

• The α-ReLU analysis derives new kernel identities relating its associated functions to zeroth Bessel functions.These identities are given in Lemmas C.31 to C.34, extending kernels first defined in.Source: §4 · p. 5

5 Theoretical Results

• Central-limit-theorem recurrences track activation-architecture dynamics and accurately match experiments for tanh and α-ReLU residual networks.The recurrences are summarized in Tables 1 and 2, with strong theory–experiment agreement reported for tanh/FRN, tanh/RRN, and α-ReLU/FRN.Source: §5 · pp. 5–6

5.1 Tanh

• For tanh residual networks, forward quantities grow linearly, normalized correlations converge polynomially to a stable fixed point, and gradient growth admits accurate large-depth asymptotics.Specifically, p(l) is asymptotically linear in both RRN and FRN, while e(l) converges polynomially; the gradient approximation is verified for l ≥ 1000.Source: §5.1 · pp. 6–7
• For tanh/FRN, increasing ρ = σa/σv moves the network from chaotic, expressive behavior toward stable, less expressive behavior while improving gradient trainability.The convergence exponent ranges from approximately 0.36338 to 1/2 as ρ increases, with small ρ worsening gradient explosion and large ρ reducing expressivity.Source: §5.1 · pp. 7–8

5.2 α-ReLU

• For α-ReLU, α < 1 yields polynomial forward growth, whereas standard ReLU produces exponential growth and can overflow after around 100 layers when σv^2 = 1.5.The expressivity quantity s is Θ(p) for α < 1, but for ReLU it grows exponentially because e(l) approaches 1 at a polynomial rate.Source: §5.2 · p. 8
• For α-ReLU with α ∈ (3/4, 1), gradient dynamics are polynomial, with the exponent minimized at α = 3/4 where R = 9/2.On α ∈ (1/2, 1), the exponent is instead minimized at α = 2/3 with R = 4; these rates are empirically verified.Source: §5.2 · p. 8
• Random ReLU FRNs simultaneously have constant weight and v-gradient norms and exponential a- and b-gradient dynamics, while experiments favor the largest α avoiding numerical instability.The reported experiments indicate that expressivity s, rather than gradient explosion alone, determines performance.Source: §5.2 · p. 8

Tanh, vary σw

• For tanh ResNets, gradient explosion, measured by χ(0)/χ(L), determines initialization quality more strongly than expressivity.The experiments attribute this to exponentially explosive gradient dynamics versus subexponential expressivity dynamics.Source: §6 · p. 9
• Deeper tanh ResNets favor smaller σw, and the theoretical level curve log R ≈ σwL pinpoints the largest accuracy-maximizing σw at each depth.When depth or σw is small, reduced gradient explosion leaves expressivity limitations relevant, so R = 1 is not optimal.Source: §6 · p. 9

Tanh, vary σv

• Holding the leading gradient-ratio term approximately fixed, tanh FRN performance peaks around a fixed depth across ρ, confirming gradient dynamics as the main determinant.Performance also increases slightly with ρ, an unexplained effect despite ρ reducing expressivity.Source: §6 · pp. 9–10

ReLU and α-ReLU

• For ReLU and α-ReLU FRNs, the best test accuracy occurs just below numerical failure, where maximizing expressivity predicts performance better than minimizing gradient explosion.For ReLU, optimality tracks high s(L), p(L), and χ(0)/χ(L) without infinities; for α-ReLU, s(l) contours predict accuracy while gradient-ratio contours do not.Source: §6 · p. 10

Overall findings

• Across all experiments, e dynamics did not predict neural-network performance.This limits the predictive value of the forward fixed-point dynamics compared with gradient and expressivity measures.Source: §6 · p. 10

• Residual networks generally avoid exponential collapse of input geometry and gradient information in both forward and backward passes, while the theory predicts test-time performance despite modeling no training dynamics.The formalism extends prior mean field work [9] [10] [11] to residual networks.Source: §7 · p. 10
• Optimal initialization must account for residual-network depth because Xavier and He [5] schemes fail primarily through structural assumptions that break on residual architectures.Their statistical assumptions remain valid for sufficiently wide networks, but their structural assumptions do not transfer well to residual networks.Source: §7 · p. 10

Open Problems

• Open problems include analyzing combinations of skip connections, batch normalization, and convolutions, as well as recurrent and attention architectures and the accuracy of Axioms 3.1 and 3.2.The authors aim to develop mathematically grounded initialization schemes for state-of-the-art architectures.Source: §7 · p. 10

Notes

• Daniely et al. [3] established fixed points for well-behaved activations under simplified feedforward conditions, but not for residual architectures.Source: Notes · p. 12
• Tempering α-ReLU with Ψα(x) avoids the divergent derivative near x = 0, and the paper’s conclusions for ψα are expected to extend similarly.Source: Notes · p. 12
• Daniely et al. [3] called Wφ with fixed ρ = 1 the “dual function” of φ.Source: Notes · p. 12
• Floating-point error prevents e(l) − e* from converging exactly to zero, so the expected log-log visualization uses an alternative plotting choice.The more natural graph is e(l) − e* versus l − δ*, but residual numerical error would distort its appearance.Source: Notes · p. 12
• The derivations apply for α ∈ (1/2, 1], but expected gradient norms diverge at α = 1/2 and gradient variance diverges for α ≤ 3/4.Empirical values may then disagree with theory, although predictions appear to upper-bound gradient norms in Fig. A.1.Source: Notes · p. 12

A Additional Figures

• Theoretical predictions closely track empirical forward quantities p(l), γ(l), and e(l) across α, while gradient-norm accuracy holds from α=1 to α=.7 but degrades for α=.6 to α=.5.Gradient variance is especially large at α=1 and for lower α, limiting reliable average-gradient predictions.Source: Figure A.1 · p. 13
• The empirical comparisons use 20 width-1000 runs for forward quantities and 25 width-250 runs for gradient quantities, with theoretical estimates shown against empirical means and standard deviations.The α=1 forward plots use shorter ranges because typical ReLU residual networks reach NaN near layer 90.Source: A Additional Figures · p. 13, Figure A.1 · p. 13
• For α-ReLU FRNs, the predicted polynomial exponents for forward cosine-distance and backward dynamics are highly accurate for every tested α below 1.The α=1 case is excluded because exponential explosions in p(l) and χ(l) cause numerical failure, including within roughly 100 layers.Source: A Additional Figures · p. 14, Figure A.2 · p. 14

B.1.1 Reduced Residual Network

• In the reduced residual network with σw=0, p(l) grows linearly while q(l) follows the stated variance-dependent recurrence; with σw>0, p(l)/l and q(l)/(σ2_w l) both converge to 1.For tanh, the theorem also provides further asymptotic expansion terms, including a logarithmic correction.Source: §B.1.1 · p. 15
• For tanh-like nonlinearities in the reduced residual network, e(l) approaches 1 with 1−e(l)=Θ(l^-1), making s(l)=p(l)(1−e(l)) remain Θ(1) in the corresponding regime.These conclusions follow from the stated recurrence results for λ and γ under the reduced architecture.Source: §B.1.1 · p. 15
• When σw>0, the reduced residual network has γ(l)=Θ̃(l^(2/π)) and e(l) tends to 0 at the stated polynomial rate.The gradient-related quantities χ, χb, and χw are related through the variance and derivative-kernel expressions given in the theorem statements.Source: §B.1.1 · p. 16, §C.5.1 · p. 38

B.1.2 Full Residual Network

• Theorem B.9 applies to tanh-like nonlinearities under the full residual-network architecture.Source: §B.1.2 · p. 16
• For σw > 0, p(l) has the expansion b0l + b1l1/2 + b2 log l + O(1).Source: §B.1.2 · p. 16
• For tanh FRNs with σw = 0, e(l) approaches 1 with 1−e(l) = Θ(l−1), yielding s(l) = Θ(1).Source: §B.1.2 · p. 16
• For σw > 0, e(l) converges to a unique fixed point e∗̸ = 1 determined by an equation.Source: §B.1.2 · p. 17
• For any FRN nonlinearity satisfying the stated axioms and finite variance of ˙φ(ζ)2, Theorem B.12 establishes a general result.Source: §B.1.2 · p. 17
• The asymptotic expansion of χ is empirically verified for various small values of σ•s; larger values make χ behave more like exp(Θ(l)) through depth 1000.Source: §B.1.2 · p. 17, Figure B.4 · p. 18

B.2 α-ReLU

• The B.2 α-ReLU subsection continues the quantities specified in Theorem B.13, as indicated by its displayed continuation.Source: §B.2 · p. 18

B.2.1 Full Residual Network

• For α = 1, the full residual network has p(l) = Θ((1 + σ_w^2/2)^l), while for 0 < α < 1, p(l) = Θ(l^(1/(1−α))).The exponential asymptotics for α = 1 are empirically verified in Figure B.5; the polynomial behavior for α < 1 is stated in Theorem B.16.Source: §B.2.1 · p. 18, Figure B.5 · p. 19
• For α = 1, e(l) approaches 1 with 1 − e(l) asymptotic to a constant times l^-2, yielding s(l) = exp(Θ(l)); for α < 1, e(l) converges to a unique nonunit fixed point e* and s(l) = Θ(l^(1/(1−α))).For α < 1, the convergence error satisfies |e* − e(l)| = ˇΘ(l^-µ), while the conjectured µ = 1/[2(1−α)] is unproved and is discontinuous at α = 1.Source: §B.2.1 · p. 19
• The derivative variance Var(˙ψ_α(ζ)^2) diverges for α ≤ 3/4 and is finite for α > 3/4, limiting the applicability of the corresponding full-residual-network equations.Backward asymptotic dynamics are empirically verified for α < 1, and the plotted exponent reaches its maximum at α = 1 on [0.5, 1].Source: §B.2.1 · pp. 19–20

C Proofs

• The proofs establish asymptotic expansions and comparison arguments for recurrence sequences, including regimes with polynomial, logarithmic, bounded, and logarithmically corrected behavior.The preliminary lemmas provide Gaussian-kernel bounds for tanh derivatives and asymptotics for power sums; the Dynamics Zoo derives recurrence solutions using monotonicity comparisons and differential-equation analogies.Source: §C.1 · p. 22, §C.2 · pp. 23, 24, 26

C.2 Dynamics Zoo

• The dynamics analysis derives asymptotic expansions and distinct growth regimes for nonlinear difference equations, including logarithmic corrections and behavior that depends on α.For one regime, the expansion includes a leading l^(α/(1−α)) log l term; another result distinguishes α above, below, or at the threshold 1/2.Source: §C.2 · p. 27

C.3 Forward Dynamical Equations

• Forward dynamical equations give recurrences for residual-network length, correlation, and related quantities p, q, λ, and γ.For reduced residual networks, the derivation assumes an antisymmetric nonlinearity and uses large-width Gaussian independence; for full residual networks, extra weight randomization removes that antisymmetry requirement.Source: §C.3 · pp. 27–28

C.4 Backward Dynamical Equations

• Backward dynamical equations derive recurrences for gradient quantities χ and χ• under the gradient-independence assumptions.The reduced-network result requires finite variance of φ̇² for a Gaussian input, while the full-network theorem requires finite variance of φ̇(ζ)² and uses the same independence framework.Source: §C.4 · pp. 29–30

C.5.1 Forward Dynamics

• For σw > 0, p(l) is asymptotically linear and q(l)/(σw^2 l) → 1; when σw = 0, p(l) = lVφ(σ2 b) + p(0) and q(l) = σ2.These results contrast the active-weight regime with the degenerate σw = 0 case.Source: §C.5.1 · p. 30
• For tanh, the forward asymptotic expansion includes logarithmic corrections, with subleading terms of the form σw log l + O(1).The expansion is obtained recursively from V_tanh(q) = 1 − Cq^−1/2 + Θ(q^−3/2), with C = 2/π.Source: §C.5.1 · p. 31
• The tanh kernel satisfies Wφ(q, cq) = 2/π arcsin(c) − Θ(q^−1) for fixed c ∈ (0, 1), with constants uniform when c stays away from 0 and 1.For c = 0 or 1, equality holds regardless of q; Figure C.11 empirically verifies the q^−1 subleading term.Source: §C.5.1 · pp. 37–38, Figure C.11 · p. 38
• With e(0) < 1 and σw = 0, e(l) → 1 with 1 − e(l) = Θ(l^−1), so s(l) = p(l)(1 − e(l)) = Θ(1).This is the zero-weight-variance branch of the forward correlation dynamics.Source: §C.5.1 · p. 38
• For σw > 0, the forward correlation quantity satisfies γ(l) = ˇΘ(l^(2/π)) and e(l) tends to 0 at the corresponding rate.The analysis uses the instability of e = 1 under the arcsin recurrence and the asymptotic relation p(l) ∼ l.Source: §C.5.1 · pp. 38–39

C.5.2 Backward Dynamics

• The backward analysis assumes the forward expansion p(l) = b0l + b1l^(1/2) + b2 log l + O(1) and derives a corresponding logarithmic expansion for D.The displayed derivation relates the coefficients to σw and the terminal depth L, but the supplied passage does not show the complete backward theorem statement.Source: §C.5.2 · pp. 39–40

C.6.1 Forward Dynamics

• For tanh residual networks with σw > 0, p(l) has expansion b0l + b1l1/2 + b2 log l + O(1).The theorem assumes a tanh-like nonlinearity and the FRN architecture.Source: §C.6.1 · p. 40
• When σw = 0, tanh residual dynamics satisfy e(l) → 1, 1−e(l) = Θ(l−1), and s(l) = Θ(1).The same result gives λ(l) = σ2b and γ(l) = l(σ2a)+γ(0).Source: §C.6.1 · pp. 41–42
• For σw > 0 and e(0) < 1, e(l) converges to the unique fixed point e∗ ≠ 1, with |e(l)−e∗| bounded between polynomial rates Ω(l−δ∗−ε) and O(l−δ∗+ε).The convergence argument identifies e∗ as the only possible limit because the dynamics repel points from 1.Source: §C.6.1 · pp. 42–43

C.6.2 Backward Dynamics

• The backward-dynamics subsection derives its result by assuming σw > 0 and following arguments analogous to Theorems B.6 and B.7.The supplied passages do not state the resulting backward-dynamics formula.Source: §C.6.2 · p. 45

C.7 α-ReLU: Full Residual Network

• Figure C.14 verifies the leading term of Theorem C.28 for α = 0.55.The supplied passage identifies the figure’s verification target but does not provide numerical values or the theorem’s formula.Source: Figure C.14 · p. 46
• In the α-ReLU full residual network analysis, Lemma C.27 establishes the identity cα+1/cα = 2α + 1.The passage describes the computation as useful for subsequent simplifications.Source: §C.7 · p. 46

C.7.1 Forward Dynamics

• For α = 1, forward variance satisfies p(l) = Θ((1 + σ2 w/2)l), while for 0 < α < 1, p(l) = Θ(l1/(1−α)).These regimes are stated for the residual network and the α < 1 asymptotic is given by Theorem B.18.Source: §C.7.1 · pp. 47, 51
• The forward analysis also proves that Jα is increasing and convex for α > −1/2 on [0, 1], and represents Jα through the zeroth Bessel function of the second kind K0.These identities support the fixed-point and asymptotic analysis of the α-ReLU kernels.Source: §C.7.1 · p. 47
• For α ∈ [1/2, 1), Jα has an unstable fixed point at 1 and a stable fixed point e∗ ∈ (0, 1), characterized by ˙Jα(e∗) < 1.The same stable-fixed-point behavior is supported by the displayed qualitative plots and confirmed rigorously by Theorem C.35.Source: §C.7.1 · pp. 49–50
• For φ = ψ1, forward correlation approaches 1 while 1 − e(l) has the stated asymptotic decay, and s(l) = exp(Θ(l)).The result combines e(l) → 1 with the supplied asymptotic expression for 1 − e(l).Source: §C.7.1 · p. 49
• For 0 < α < 1, the correlation converges to a unique nonunit fixed point e∗, with |e∗−e(l)| = ˇΘ(l−µ) and s(l) = Θ(p(l)).The exponent is µ = (1 − ˙Jα(e∗))/(1 − α).Source: §C.7.1 · p. 51

C.7.2 Backward Dynamics

• For ψα in an FRN, Var( ˙ψα(ζ)2) diverges when α ≤ 3/4 and is finite when α > 3/4 for any zero-mean Gaussian ζ.The threshold follows from the distributional behavior of ψ−β(X) and the relation ˙ψα ∝ ψα−1.Source: §C.7.2 · p. 52
• Backward gradient correlations have distinct depth laws: for α = 1, χ(l−m) follows the stated χ(l) relation, while for α ∈ (1/2, 1), χ(l−m) = Θ(1)χ(l)(l/(l−m))R.Here R = α2 / ((1−α)(2α−1)), and the Θ(1) constants do not depend on l or m.Source: §C.7.2 · p. 52