Exponential expressivity in deep neural networks through transient chaos

• Prior work showed particular functions needing polynomially many neurons in deep networks but exponentially many in shallow ones [7] [9] [10], leaving open whether generic deep-network functions share this gap.Source: §1 · p. 1
• The paper addresses the challenge of a unifying expressivity framework applicable to arbitrary nonlinearities, generic networks, and a natural general measure of functional complexity.Earlier techniques were tied to specific nonlinearities and complexity measures, including ReLU linear regions [7], sum-product monomials, and Pfaffian topological measures.Source: §1 · pp. 1–2
• For broad classes of nonlinearities, random deep networks can produce hidden representations whose global extrinsic curvature grows exponentially with depth but not width.The paper uses Riemannian geometry and dynamical mean field theory to establish this result.Source: §1 · p. 2
• Signal propagation in large random feedforward networks undergoes an order-to-chaos transition with weight and bias statistics, and transient finite-depth evolution in the chaotic regime underlies exponential expressivity.Source: §1 · p. 2
• The paper quantitatively defines disentangling and verifies that deep random networks can transform highly curved input manifolds into flattened hidden representations.The motivating intuition is that flattened manifolds can aid simple linear readouts.Source: §1 · pp. 1–2

2 A mean field theory of deep nonlinear signal propagation

• The analysis studies random networks with independently sampled, zero-mean Gaussian weights and biases, whose weight scaling keeps each neuron’s input contribution O(1) across layer widths.This maximum-entropy ensemble imposes only means and variances, isolating depth’s contribution to typical functions.Source: §2 · p. 2
• At large layer widths, signal propagation becomes essentially deterministic, and a length map predicts how input length changes across layers.The map follows from approximating each layer’s inputs by a zero-mean Gaussian distribution.Source: §2 · p. 2
• With zero bias, weights below the critical scale shrink inputs to zero, while larger weights yield a stable nonzero length; any nonzero bias yields a stable nonzero fixed point.For larger weights without bias, small inputs expand and large inputs contract; length often converges to its fixed point within four layers.Source: §2 · p. 3
• The length map’s fixed points are intersections with the unity line and, for monotonic nonlinearities, the map is increasing and concave.The figure illustrates the map and rapid convergence for tanh networks, with theory matching simulations.Source: §2 · p. 3, Figure 1 · p. 3

3 Transient chaos in deep networks

• A length map and a correlation map predict the typical layer-wise evolution of the geometry of two inputs in a fixed large network.The pair’s geometry is represented by a 2-by-2 matrix of inner products, with the correlation map tracking the normalized off-diagonal term.Source: §3 · p. 3
• When the correlation-map slope χ1 is below 1, nearby inputs converge; above 1, the fixed point at 1 is unstable and nearby inputs separate.χ1 acts as the typical multiplicative growth or shrinkage factor for a small random perturbation across one layer.Source: §3 · p. 4
• For finite σb, increasing σw moves the network from a stable c*=1 regime, through an intermediate regime with decreasing c*, to a stable c*=0 regime.Correlations take about 20 layers to approach their fixed point, compared with about 4 layers for lengths; weights and nonlinearities decorrelate inputs while biases correlate them.Source: §3 · p. 4
• The boundary χ1(σw, σb)=1 separates ordered dynamics, where nearby points converge, from chaotic dynamics, where they separate; finite-depth feedforward networks therefore exhibit transient chaos.The logarithm of χ1 is related to the Lyapunov exponent, which is negative in the ordered phase and positive in the chaotic phase.Source: §3 · p. 4

4 The propagation of manifold geometry through deep networks

• For a circle input, chaotic sigmoidal networks are predicted to decorrelate and grow more complex, whereas ordered networks contract around a central point; simulations confirm the prediction.Figure 3 shows flatter singular-value distributions and more tangled projected curves for large weights.Source: §4 · p. 5, Figure 3 · p. 5
• Extrinsic curvature κ measures bending through the inverse osculating-circle radius, while Euclidean length LE tracks distance and Gauss-map length LG tracks tangent change, with gG = κ^2gE.For a circle, linear scaling increases length and radius, decreases curvature, and leaves LG unchanged.Source: §4 · pp. 5–6
• In the chaotic regime, nonlinear propagation can increase circle length without decreasing extrinsic curvature; in a 1000-hidden-unit network, all principal curvatures grow exponentially when backpropagating to the input layer.Source: §4 · p. 6, Figure 4 · p. 6
• This exponential growth means the curve explores many tangent directions, making hidden coordinate functions highly complex and enabling deep networks to compute exponentially complex functions on simple low-dimensional manifolds.The section illustrates this with output activities and random-function regression across networks of different depths.Source: §4 · p. 7, Figure 5 · p. 7

5 Shallow networks cannot achieve exponential expressivity

• LE ≤ N1(1 + s)R for any shallow network when the activation is monotone with bounded dynamic range R and no 1D projection of the input-curve derivative changes sign more than s times.The bound holds for any choice of W1 and b1.Source: §5 · p. 8
• For circle inputs and tanh, normalized length is at most 2√N1, whereas chaotic deep networks have normalized length that grows exponentially with depth.For the circle, s = 1; for tanh, R = 2.Source: §5 · p. 8
• Deep networks’ exponential curve-length growth drives exponential global-curvature growth, while shallow networks expand curves only as the square root of width and show minimal expressivity growth even at large widths (Figure 5D).The deep-network length expansion applies even to random deep networks.Source: §5 · p. 8

6 Classification boundaries acquire exponential local curvature with depth

• A final-layer linear classifier has a hyperplane decision boundary, whose preimage under the nonlinear feedforward map is a curved manifold in the input layer.The input boundary is the solution set of G(x0) = β · xD(x0) − β0 = 0.Source: §6 · p. 8
• Principal curvatures are eigenvalues of the normalized Hessian of the boundary equation projected onto the tangent plane.Locally, the boundary is approximated by a paraboloid whose quadratic form has the principal curvatures as eigenvalues.Source: §6 · p. 8
• A subset of decision-boundary principal curvatures grows exponentially with depth, and the curved input boundary maps to a flat boundary in the output layer.The curvatures are signed; positive and negative curvature indicate geodesics bending toward and away from the normal, respectively.Source: §6 · p. 8

7 Discussion

• In the chaotic phase, deep networks exponentially grow global curvature from input to output for simple 1D manifolds and local curvature from output to input for simple co-dimension-one manifolds.Global-curvature growth captures functions deep networks compute efficiently but shallow networks cannot; local-curvature growth quantifies the disentangling of curved input manifolds.Source: §7 · pp. 8–9
• Maximum-entropy deep networks provide a null model for signal propagation; geometry-derived metrics may help identify what is special about trained networks.The discussion also proposes weight chaos: by input–weight duality, output autocorrelations as weights change shrink exponentially with depth, as do autocorrelations across inputs.Source: §7 · p. 9
• The graph of a map from RN1 to RND is an RN1-dimensional submanifold of RN1+ND, whose geometry may reveal secrets of deep learning as it changes with depth and learning.The authors present this as a speculation about a direction for future study.Source: §7 · p. 9

A Derivation of a transient dynamical mean field theory for deep networks

• The network has D weight layers W_1, …, W_D and D + 1 activity vectors x_0, …, x_D, with N_l neurons in layer l.Each W_l is an N_l × N_{l−1} matrix, and x_l ∈ R^{N_l}.Source: A Derivation of a transient dynamical… · p. 11
• Layer activity follows x_l = φ(h_l), where h_l = W_l x_{l−1} + b_l and the scalar nonlinearity φ acts component-wise.Here b_l is the bias vector and h_l is the pattern of inputs to neurons at layer l.Source: A Derivation of a transient dynamical… · p. 11
• Weights and biases are drawn independently from zero-mean Gaussians, and scaling weight variance by 1/N_{l−1} keeps each neuron's input contribution O(1) as layer width varies.The weight variance is σ_w^2/N_{l−1}; the bias variance is σ_b^2.Source: A Derivation of a transient dynamical… · p. 11

A.1 Derivation of the length map

• The method tracks a single input’s normalized squared length as it propagates, allowing its length to grow or shrink in downstream layers.Source: §A.1 · p. 11
• For large layer width, neuron inputs are treated as Gaussian, with independence across neurons following from independently chosen incoming weights and biases.Source: §A.1 · p. 11
• The variance is computed from the input to one neuron, then the preceding-layer neuron average is replaced by an integral over a zero-mean Gaussian.Source: §A.1 · p. 11
• The iterative variance map V predicts how input length changes across layers, beginning from the initial activity-layer length q0 and relying on self-averaging at large width.Here, self-averaging means the empirical distribution across neurons in a fixed network converges to the distribution of a single neuron's inputs across random networks.Source: §A.1 · p. 11

A.2 Derivation of a correlation map for the propagation of two points

• At large width, two-input geometry is tracked by a 2 by 2 inner-product matrix, whose joint Gaussian propagation through one layer yields a correlation map.The map is expressed as C(c^l_12, q^l_11, q^l_22 | σ_w, σ_b), with correlated Gaussian variables determined by the previous layer's covariance.Source: §A.2 · p. 12
• At fixed-point input lengths q*, the correlation recursion has c(1) = 1, and its slope χ1 at 1 is obtained by differentiating the map.The derivation applies the chain and product rules and uses integration by parts.Source: §A.2 · p. 12

B Derivation of evolution equations for Riemannian curvature

• The appendix derives recursion relations for Riemannian curvature quantities.Source: B Derivation of evolution equations… · p. 12

B.1 Curvature and length in terms of inner products

• The analysis considers a translation-invariant manifold, or 1D curve h(θ) ∈ R^N, lying on a sphere of constant radius.Source: §B.1 · p. 12
• At large N, inner products of derivatives of h are expressed through derivatives of Q, using approximate translation invariance during propagation.Source: §B.1 · p. 12
• The Euclidean metric is expressed using these inner products, while extrinsic curvature is described using the velocity v(θ) and acceleration a(θ).Source: §B.1 · pp. 12–13
• On a sphere whose radius is the fixed point of the length map, the radius stays constant as the manifold propagates.Source: §B.1 · p. 13
• For such manifolds, local length and curvature propagation reduces to tracking derivatives of the correlation function c(θ) at θ = 0 under the C-map.Source: §B.1 · p. 13

B.2 Behavior of first and second derivatives under function composition

• The composition analysis assumes an even H1 with H1(0) = 0 and G(0) = 0, then uses chain and product rules to derive propagation of second and fourth derivatives.Source: §B.2 · p. 13

B.3 Evolution equations for curvature and length

• Derivative recursions yield curvature and length evolution equations, with χ2 related to the second derivative of the correlation map and that derivative equal to χ2q∗.The equations are reported as tested against experiments; χ1 is the stretch factor defined in equation (19).Source: §B.3 · p. 13

C.1 Upper bound on Euclidean length

• For non-decreasing activations with bounded range R, the maximal neuron derivative sign-change count r1 is bounded by s0, the maximal sign-change count of any input-derivative projection.The length-bound argument partitions the input parameter range into segments of constant derivative sign.Source: §C.1 · p. 14

D Simulation details

• Simulations used Keras/Theano, width 1,000 and tanh except for Figure 5C, usually chaotic initialization σb=0.3 and σw=4.0; curvature derivatives used forward-mode Theano R-op.The reported error bars are standard deviations across inputs, and larger widths reduced fluctuations in averaged quantities.Source: D Simulation details · p. 14

D.1 Details on Figure 4G: backpropagating curvature

• To measure decision-boundary curvature, the authors optimized G(xD(xl))2 with respect to input x using Adam, then evaluated Jacobians and Hessians at the resulting boundary points.Principal curvatures were computed from those derivatives.Source: §D.1 · p. 14

D.2 Details on Figure 5C-D: evaluating expressivity

• Expressivity was evaluated by regressing network outputs onto Fourier bases and measuring predicted-versus-true basis-vector angles, which serve as a proxy for bounded-frequency function prediction error.The Fourier basis was truncated at a maximum frequency ωmax.Source: §D.2 · pp. 14–15

E Additional visualization of hidden actions

• In the chaotic regime, curve radius of curvature stays constant while length grows exponentially with depth, producing exponentially many turns at depth-independent acceleration.The t-SNE view reflects the number of turns but not their radius; apparent radius decrease in t-SNE and PCA is a dimensionality-reduction artifact.Source: Figure 6 · p. 15

F A view from the function space perspective

• In the chaotic regime, a layer-2 weight perturbation separates functions, and hD(θ, Δ) roughly random-walks through function space with autocorrelation length decreasing exponentially in depth D.The family varies Δ only through layer 2; the other layers’ weights and biases are independent of Δ.Source: F A view from the function space… · pp. 15–16