PaperlayerView exact revision on arXiv

Source-linked AI summary

Exponential expressivity in deep neural networks through transient chaos

Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli

arXiv:1606.05340v2stat.MLcond-mat.dis-nncs.LG

TL;DR

The paper asks whether the expressive advantages attributed to deep networks are generic and whether manifold disentangling can be quantified beyond restricted nonlinearities. Combining Riemannian geometry with dynamical mean field theory, it shows that random networks undergo an order-to-chaos transition, with chaotic networks exhibiting exponentially growing curvature and formalized disentangling.

  • Problem

    It remains unclear whether shallow networks cannot efficiently compute generic deep-network functions and whether deep-network manifold disentangling can be formally quantified across broad nonlinearities.

  • Method

    The paper combines Riemannian geometry and dynamical mean field theory to analyze signal propagation, curvature, and representations in random deep networks.

  • Results

    In the chaotic phase, random deep networks produce global curvature that grows exponentially with depth but not width, while deep functions resist efficient shallow computation.

  • Takeaways & Limitations

    The framework quantitatively supports deep-network expressivity and manifold disentangling while providing a null model for studying signal propagation in trained networks.

  • Takeaways & Limitations

    The discussion remains exploratory about understanding deep functions through the geometry of their graphs as Riemannian manifolds.

Abstract

from arXiv · show

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitatively demonstrate the long conjectured idea that deep networks can disentangle highly curved manifolds in input space into flat manifolds in hidden space. Our theoretical analysis of the expressive power of deep networks broadly applies to arbitrary nonlinearities, and provides a quantitative underpinning for previously abstract notions about the geometry of deep functions.

1 Introduction

The introduction frames deep-network expressivity around the hypothesis that depth compactly represents functions too costly for equally wide shallow networks. It proposes Riemannian geometry and dynamical mean field theory as a framework for testing genericity and measuring functional complexity.

  • Motivation: Deep networks are hypothesized to compactly express highly complex input-space functions that shallow networks with the same neuron count cannot.This intuition is presented as a key explanation for deep networks’ success.
  • Open problems: Prior results show particular functions requiring polynomially many neurons in deep networks but exponentially many in shallow networks.The introduction asks whether these examples are rare curiosities or representative of generic deep-network functions.
  • Open problems: A second open question is whether one framework can address arbitrary nonlinearities, generic networks, and a natural general measure of functional complexity.The introduction identifies limits in prior theoretical techniques as motivating this question.
  • Approach: Combining Riemannian geometry with dynamical mean field theory, the authors analyze expressivity for very broad classes of nonlinearities.The framework is designed to attack both central problems of deep neural expressivity.
  • Main contribution: For random deep networks, hidden internal representations can have global extrinsic curvature that grows exponentially with depth but not width.This geometric result provides a quantitative route to studying functional complexity.

2 A mean field theory of deep nonlinear signal propagation

This section develops a mean field description of signal propagation in deep random networks by modeling layerwise activity lengths through an iterative variance map. In the large-width limit, the map becomes deterministic and its fixed points characterize how input lengths evolve across depth.

  • Network ensemble: Deep feedforward dynamics apply a component-wise nonlinearity to affine layer inputs, with weights and biases drawn independently from zero-mean Gaussian ensembles.The model uses xl = φ(hl) and hl = Wl xl−1 + bl, with weight variance σ2w/Nl−1 and bias variance σ2b.
  • Network ensemble: The weight scaling keeps each neuron’s input contribution O(1) as layer width changes and defines a maximum-entropy ensemble without additional structure.This scaling makes the input contribution independent of Nl−1 while constraining only weight and bias means and variances.
  • Large-width limit: For Nl ≫1, signal propagation becomes essentially deterministic, enabling mean field analysis of how input-manifold geometry changes through deep layers.The large-width limit supports analysis of the Riemannian geometry induced as manifolds propagate from the input layer.
  • Length dynamics: The iterative variance map V predicts layerwise input lengths, with fixed points q∗(σw, σb) determined by intersections of the map with the unity line.For monotonic nonlinearities, V is monotonically increasing and concave; for φ(h) = tanh(h), theoretical map predictions match network simulations.
  • Length dynamics: For σb = 0 and σw < 1, the length map has a single fixed point at q∗ = 0.This describes the subunit-weight-variance regime for monotonic nonlinearities.

3 Transient chaos in deep networks

The section develops correlation and length maps to predict how pairs of inputs evolve through deep random networks. Their correlation dynamics exhibit an order-to-chaos transition governed by the multiplicative perturbation factor χ1, with finite-depth behavior characterized as transient chaos.

  • Geometric propagation: A 2×2 matrix of layer-wise inner products captures the propagated geometry of two inputs, while length and correlation maps predict its typical evolution.The correlation map depends on the previous correlation and the two input lengths, and together with the length map provides a theoretical prediction for fixed large networks.
  • Correlation dynamics: Lengths rapidly converge to q*(σw, σb), enabling an iterative C-map whose fixed point c*=1 is stable or unstable depending on its slope at 1.The slope χ1 determines whether nearby points become more similar or separate as they propagate.
  • Perturbation growth: χ1 is the typical multiplicative growth or shrinkage factor for a random perturbation across one layer, obtained by averaging its Jacobian norm over network randomness.A perturbation u at one layer maps approximately to Ju at the next, and the averaged squared-norm ratio becomes χ1(q*).
  • Timescales and phases: Correlation dynamics take about 20 layers to approach their fixed point, whereas lengths need only 4 layers, because the C-map lies closer to the unity line.The fixed point c* and slope χ1 vary across σw and σb, producing three qualitative regions as σw increases for any fixed finite σb.
  • Order-to-chaos transition: The boundary χ1(σw, σb)=1 separates ordered dynamics, where nearby points converge, from chaotic dynamics, where they separate; finite-depth feedforward networks therefore exhibit transient chaos.The logarithm of χ1 corresponds to the Lyapunov exponent, while feedforward dynamics terminate at finite depth D.

4 The propagation of manifold geometry through deep networks

The section develops Riemannian measures for tracking how manifolds propagate through deep networks and shows that chaotic nonlinear dynamics can exponentially increase representational complexity. In this regime, networks expand manifold length while continually generating curvature, producing highly complex functions on simple input manifolds.

  • Manifold propagation: A one-dimensional input manifold propagates layer by layer, with pairwise geometry summarized by the metric q_l(θ_1, θ_2).The pairwise propagation theory applies to all pairs of points on the manifold.
  • Geometric measures: Riemannian analysis characterizes manifold complexity through tangent vectors, extrinsic curvature, Euclidean length, and Gauss-map length.The Gauss metric measures changes in the unit tangent and satisfies g_G(θ) = κ(θ)^2g_E(θ).
  • Geometric measures: Linear expansion increases Euclidean length and radius while decreasing curvature, leaving Gauss-map length unchanged.For a circle, scaling by χ makes length and radius scale by χ, curvature by χ^-1, and Gauss-map length unchanged.
  • Chaotic versus ordered propagation: In the chaotic phase, χ_1 > 1 causes the renormalized Euclidean metric to grow exponentially with depth, while nonzero χ_2 adds new curvature.Multiplicative stretching attenuates inherited curvature by 1/χ_1, but curvature is regenerated through the nonlinear map.
  • Expressivity: Exponential growth of global curvature makes hidden representations explore many tangent directions and enables exponentially complex functions over simple low-dimensional manifolds.The resulting coordinate functions become highly complex curved basis functions on the input-manifold coordinate.

5 Shallow networks cannot achieve exponential expressivity

A one-hidden-layer network has a general upper bound on curve length determined by width, activation range, and input-curve projection complexity. For circle inputs with tanh, normalized length grows only as the square root of width, whereas chaotic deep networks expand curve length exponentially with depth.

  • General upper bound: LE ≤ N1(1 + s)R for any shallow network, where R is the activation’s bounded dynamic range and s bounds projection sign changes.The bound holds for every choice of W1 and b1 under the theorem’s monotonicity and input-curve conditions.
  • Circle and tanh case: For circle inputs with tanh, normalized length satisfies ¯LE ≤ 2√N1.Here, the theorem uses s = 1 and R = 2.
  • Depth versus width: Deep networks in the chaotic regime grow ¯LE exponentially with depth, while shallow networks can expand it only as the square root of width.The exponential depth growth occurs in h space and consequently in x space.
  • Expressivity consequence: Because exponential growth of ¯LE drives exponential growth of ¯LG, shallow random networks show minimal expressivity growth even at large widths.This behavior is reported in Figure 5D.

6 Classification boundaries acquire exponential local curvature with depth

A linear decision boundary in the final layer can acquire exponentially increasing local curvature when propagated toward the input through depth. This geometric transformation enables highly complex classifications.

  • Boundary setup: A final-layer linear classifier has a hyperplane decision boundary defined by β·x_D − β_0 = 0.The boundary is N−1 dimensional in the final representation layer.
  • Curvature geometry: Local boundary curvature is characterized by geodesics spanning the normal ∇G and tangent directions, with the maximum defining the first principal curvature κ_1.Each principal curvature describes curvature along a tangent direction at a boundary point.
  • Curvature geometry: The boundary’s principal curvatures are the eigenvalues of a projected Hessian quadratic form that locally approximates it as a paraboloid.The projection operator restricts the Hessian to the tangent plane.
  • Depth-dependent result: A subset of signed principal curvatures grows exponentially with depth, making the decision boundary exponentially curved and enabling highly complex classifications.Positive and negative signs indicate whether geodesics curve toward or away from the normal vector ∇G.

7 Discussion

The discussion frames deep networks as high-dimensional nonlinear maps whose expressivity and manifold disentangling can be formally studied beyond restricted nonlinearities. It also proposes geometric analysis of random and trained networks, including the evolving geometry of their graphs.

  • Deep neural networks compute nonlinear maps between high-dimensional spaces, motivating mathematics suited to otherwise daunting spaces of maps.
  • Deep networks can efficiently compute highly expressive functions that shallow networks cannot, while disentangling curved input manifolds is formally quantified.
  • A maximum entropy distribution over deep networks provides a null model for assessing signal propagation and identifying behavior that distinguishes trained networks.
  • Riemannian metrics adapted to random networks may help explain what is special about trained networks.
  • The graph of a map from R^N1 to R^ND is an R^N1-dimensional submanifold of R^(N1+ND), whose geometry may reveal secrets of deep learning as depth and learning change.

A Derivation of a transient dynamical mean field theory for deep networks

The paper formulates deep feedforward networks through layerwise activity dynamics, combining affine weight-and-bias inputs with a component-wise scalar nonlinearity. Weights and biases are modeled as independent Gaussian variables, with weight scaling keeping each neuron's input contribution O(1) across layer widths.

  • Network setup: The network has D weight layers, activity vectors x0 through xD, and N_l neurons in layer l.The weight matrix W_l has dimensions N_l × N_{l−1}.
  • Network dynamics: Each layer computes x_l = φ(h_l) from h_l = W_l x_{l−1} + b_l for l = 1, . . . , D.The scalar nonlinearity φ acts component-wise on neuron inputs.
  • Random parameters: The synaptic weights W_l are drawn independently from zero-mean Gaussians with variance σ²_w/N_{l−1}.This distribution specifies the random-weight assumption used in the dynamical mean field formulation.
  • Random parameters: The biases are drawn independently from zero-mean Gaussians with variance σ²_b.Biases enter each layer through the affine input h_l.
  • Scaling condition: O(1) input contributions per neuron are maintained independently of the preceding layer width by the weight scaling.The scaling prevents the input contribution from growing with N_{l−1}.

A.1 Derivation of the length map

The derivation tracks how an input’s normalized squared length changes across layers by modeling neuron inputs with a self-averaging Gaussian distribution. This yields an iterative variance map V with initial condition q1 = σ2_w q0 + σ2_b that predicts length propagation.

  • Length propagation: The analysis tracks the normalized squared length of a single input point as it propagates through downstream layers.This length may either grow or shrink across layers.
  • Gaussian approximation: For large layer width, neuron inputs are modeled by a zero-mean Gaussian empirical distribution because they are sums of many independent random variables.Independence follows from independently chosen synaptic weight vectors and biases for different neurons.
  • Variance map: Replacing the neuron average with a Gaussian integral produces the iterative variance map V that predicts how input length changes across layers.The Gaussian measure is the standard measure Dz = dz √2πe−z2 2.
  • Initial condition and self-averaging: The map starts from q1 = σ2_w q0 + σ2_b, where q0 = 1/N0 x0 · x0 is the initial activity-layer length.The derivation uses self-averaging, meaning the empirical input distribution converges to the distribution of inputs as width becomes large.

A.2 Derivation of a correlation map for the propagation of two points

The section derives a correlation map for two inputs by tracking their 2 × 2 inner-product geometry through layers. In the large-width limit, joint hidden-unit activations become Gaussian, yielding a recursion for correlation and its fixed-point slope.

  • Two-point propagation: Two inputs are represented at each layer by a 2 × 2 matrix of inner products capturing their propagated geometry.This matrix describes the layer-wise propagation of x0,1 and x0,2.
  • Gaussian limit: At large N_l, the joint empirical distribution of hidden activations converges to a two-dimensional Gaussian with covariance q_l.The covariance matrix specifies the joint statistics used to propagate the two-point geometry.
  • Correlation map: Propagating this joint Gaussian distribution through one layer defines the correlation map C, parameterized by the preceding correlation and variances.The map is written as c_l12 = C(c_{l−1}12, q_{l−1}11, q_{l−1}22 | σ_w, σ_b).
  • Fixed-point dynamics: When both inputs have fixed-point length q*, setting q_l11 = q_l22 = q* and dividing by q* yields a recursion for c_l12.The map fixes c_l12 = 1, and its slope χ_1 at 1 is obtained by direct differentiation.

B Derivation of evolution equations for Riemannian curvature · B.1 Curvature and length in terms of inner products · B.2 Behavior of first and second derivatives under function composition

This section derives recursion relations for Riemannian curvature and expresses geometric quantities through derivative inner products and correlation functions. It also examines how second- and fourth-order derivatives propagate under function composition.

  • B Derivation of evolution equations for Riemannian curvature: The section develops recursion relations for Riemannian curvature quantities.
  • B.1 Curvature and length in terms of inner products: A translation-invariant manifold is represented as a 1D curve h(θ) ∈ R^N on a constant-radius sphere.
  • B.1 Curvature and length in terms of inner products: At large N, propagated inner products of derivatives remain approximately translation invariant and can be expressed through derivatives of Q.
  • B.1 Curvature and length in terms of inner products: Derivative notation uses dots for θ-derivatives, providing the inner-product formulation of extrinsic curvature.
  • B.1 Curvature and length in terms of inner products: With v(θ) = ∂θh(θ) and a(θ) = ∂²θh(θ), the extrinsic-curvature expression simplifies in terms of inner products.
  • B.1 Curvature and length in terms of inner products: The curvature simplification includes the denominator term (v · v)^3.
  • B.1 Curvature and length in terms of inner products: On a sphere of radius Nq*, the radius remains fixed, allowing gE and κ to be expressed through c(θ) = Q(θ)/q* and its derivatives at θ = 0.
  • B.2 Behavior of first and second derivatives under function composition: For even H1(∆t) with H1(0) = 0, the analysis uses its Taylor expansion to study derivative propagation under composition.

B.3 Evolution equations for curvature and length … F A view from the function space perspective

The supplement derives tested recursions governing curvature and length, bounds shallow-network manifold length, documents simulation and expressivity procedures, and analyzes how weight changes separate functions in the chaotic regime.

  • B.3 Evolution equations for curvature and length: The curvature and length evolution equations are derived from derivative recursions and expressed using the stretch factors χ1 and χ2.The derivation accounts for factors of q∗ and N and is reported as successfully tested against experiments.
  • C Upper bounds on the complexity of shallow neural representations: The shallow-network analysis asks how one hidden layer changes the length and curvature of an input manifold as a function of hidden width N1.The network uses x1 = φ(W1x0)+b and a linear readout of x1.
  • C.1 Upper bound on Euclidean length: For monotone nonlinearities with bounded dynamic range R, the Euclidean length in hidden space admits a general upper-bound analysis based on derivative sign changes.The assumptions are φ′(h) ≥ 0 and maxh φ(h) − minh φ(h) = R.
  • C.1 Upper bound on Euclidean length: The maximal hidden-layer derivative sign-change count satisfies r1 ≤ s0 because each preactivation derivative is a linear projection of the input derivative.Here s0 bounds sign changes in any one-dimensional projection of ∂θx0(θ), while r1 bounds changes for any neuron.
  • D Simulation details: Simulations used Keras and Theano, generally with width 1,000 and tanh activations; larger widths reduced fluctuations in averaged quantities.Simulation error bars were standard deviations across different inputs and h1(θ).
  • D.1 Details on Figure 4G: backpropagating curvature: Decision-boundary curvature was evaluated by optimizing G(xD(xl))2 to find boundary points, then computing the Jacobian and Hessian to obtain principal curvatures.The optimization was performed with Adam from randomly initialized data points.
  • D.2 Details on Figure 5C-D: evaluating expressivity: Expressivity was evaluated by regressing network outputs onto Fourier bases up to maximum frequency ωmax and measuring angles between predicted and true basis vectors.Fourier-component prediction error serves as a proxy for predicting bounded-frequency functions.
  • F A view from the function space perspective: In the chaotic regime, changing weights in one layer lowers the function-space correlation below one and the resulting difference amplifies through subsequent function-space evolution.For the fixed-point input circle, C1(∆) = 1, while any nonzero weight difference produces C2(∆) < 1.
Loading 1606.05340v2…