Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Andrew M. Saxe; James L. McClelland; Surya Ganguli

Source-linked AI summary

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Andrew M. Saxe, James L. McClelland, Surya Ganguli

arXiv:1312.6120v3Dec 20, 2013cs.NEcond-mat.dis-nncs.CVcs.LGq-bio.NCstat.ML

TL;DR

The paper analyzes learning dynamics in deep linear networks through exact solutions and identifies initializations that support rapid learning as depth increases. It further examines how these initialization properties relate to signal and gradient propagation in nonlinear networks.

Problem
Learning times in deep networks depend sensitively on input statistics, initial weight conditions, and network depth, motivating an analytical account of these dependencies.
Method
The paper derives exact gradient-descent dynamics for deep linear networks and analyzes decoupled connectivity modes, pretraining-based initializations, random orthogonal initializations, and nonlinear propagation regimes.
Results
With suitable initial conditions, deep linear networks can be only a finite amount slower than shallow networks; orthogonal initialization and pretraining yield depth-independent learning times, unlike scaled Gaussian initialization.
Takeaways & Limitations
Unsupervised pretraining can find rapidly learning initial conditions for tasks with the right structure, while random orthogonal conditions provide another such initialization and support dynamical isometry near the nonlinear edge of chaos.
Takeaways & Limitations
The learning-time discrepancy between scaled Gaussian and orthogonal or pretrained initializations is modest at depths around 6 and becomes larger at greater depths.

Abstract

from arXiv · show

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed can nevertheless remain finite: for a special class of initial conditions on the weights, very deep networks incur only a finite, depth independent, delay in learning speed relative to shallow networks. We show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, while scaled random Gaussian initializations cannot. We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times. We further show that these initial conditions also lead to faithful propagation of gradients even in deep nonlinear networks, as long as they operate in a special regime known as the edge of chaos.

1 General learning dynamics of gradient descent

The paper formulates learning in a three-layer linear network as coupled nonlinear weight dynamics and analyzes how input-output modes are learned. Orthogonal connectivity-mode initializations decouple the modes, yielding exact trajectories, conserved quantities, and characteristic sigmoidal learning curves.

Learning dynamics: Gradient descent on a linear network produces coupled nonlinear differential equations with up to cubic interactions in the weights.The input-output map is linear, but the weight dynamics are not.
Learning dynamics: With whitened inputs, the input-output correlation matrix contains all dataset information and determines the modes analyzed by the dynamics.The analysis uses the SVD of the input-output correlation matrix to identify independent input and output variation modes.
Mode interactions: Connectivity modes associated with the same input-output mode cooperate, while interactions between different modes create competition.The cooperative terms increase aligned mode magnitudes and drive their product toward the corresponding correlation strength.
Fixed points: Only the fixed point activating the strongest N2 modes is stable, so the converged network represents the closest rank N2 approximation to the target correlation matrix.Other fixed-point families are unstable, and the remaining fixed points are saddles rather than non-global local minima.
Decoupled dynamics: Parallel, mutually orthogonal mode initializations eliminate cross-mode interactions and preserve those directions throughout learning.Such initializations can be constructed from the SVD using an arbitrary orthogonal hidden-layer matrix.
Time course of learning: The equal-magnitude case conserves a^2 − b^2 and approaches the fixed-point manifold ab = s along a hyperbola.The origin is an unstable fixed point, whereas the product ab monotonically approaches s.
Time course of learning: Each mode learns on a timescale O(τ/sα), while its product strength follows a sigmoidal rise that can produce plateaus and sharp transitions.Here sα is the mode’s correlation strength; stronger modes learn faster.

2 Deeper multilayer dynamics

For deep linear networks, the paper identifies an invariant decoupled submanifold that permits exact analysis of depth-dependent learning. With depth-appropriate learning rates, learning remains only finitely delayed as depth tends to infinity, although the result concerns iterations rather than computational cost.

Deeper multilayer dynamics: The deeper-network analysis begins from initial conditions that yield particularly simple gradient descent dynamics.It tracks an invariant submanifold where connectivity modes can be analyzed independently.
Decoupled modes: Aligned singular-vector bases across adjacent layers prevent mode mixing and produce independently evolving connectivity modes.The construction allows hidden layers to have different sizes, including undercomplete or overcomplete layers.
Depth and learning rate: For fixed learning rate, continuous-time iteration counts tend to zero as depth approaches infinity, but this limit is not directly implementable with a finite stable step size.The paper therefore estimates a depth-dependent optimal learning rate from the Hessian.
Depth and learning rate: With the optimal learning rate, the difference between learning times for Nl = 3 and Nl = ∞ remains finite, with t∞−t3 ∼ O(s/ϵ) for small ϵ.The analysis measures learning speed by iterations, not computation; each deep-network iteration is more expensive.
MNIST verification: Deep linear networks trained on MNIST were evaluated from Nl = 3 to Nl = 100 using separately optimized learning rates.Training time was measured by the iteration reaching a threshold corresponding to nearly complete learning.
MNIST verification: Depth causes only a finite slowdown under decoupled initial conditions, and the delay decreases as the initial association strength increases.The result makes initialization of composite mode strengths central to fast deep learning.

3 Finding good weight initializations: on greediness and randomness

The paper compares greedy pretraining and random initializations as routes to depth-independent learning in deep linear networks. Greedy pretraining can produce decoupled, balanced initial conditions under a data-consistency condition, while random orthogonal weights match its rapid learning and scaled Gaussian weights do not.

Greedy pretraining: Greedy pretraining is intended to place weights near a decoupled submanifold where each connectivity mode evolves independently and learning times remain depth-independent.This requires every mode's initial composite end-to-end strength to be O(1).
Greedy pretraining: During autoencoder pretraining, the input-output correlation becomes the input correlation matrix, linking the procedure to PCA of the training inputs.The pretrained composite map approaches QMQT, with M approaching the identity during learning.
Greedy pretraining: Starting from small random weights, pretraining yields an approximately balanced factorization whose input-to-hidden mapping is W21 = R2M^1/2QT.R2 is an arbitrary orthogonal matrix, and M^1/2 supplies mode strengths near one for successful fine-tuning.
Greedy pretraining: MNIST satisfies the pretraining consistency test V11Σ11V11T ≈ diagonal, and pretrained five-layer networks learn faster than small-random networks even after accounting for pretraining time.The figure compares pretrained and random initializations, with the pretrained curve including an initial pretraining delay.
Random initialization: Random orthogonal initialization gives depth-independent learning times indistinguishable from greedy pretraining, unlike norm-preserving scaled Gaussian initialization.For orthogonal weights, W^T W = I; the scaled Gaussian scheme's learning time grows with depth on MNIST.
Random initialization: Products of scaled Gaussian matrices preserve typical vector norms anisotropically, concentrating amplification in a few singular directions while attenuating most others and weakening backpropagated gradients.Orthogonal products retain singular values equal to 1, whereas Gaussian products develop highly kurtotic singular-value spectra as depth increases.

4 Achieving approximate dynamical isometry in nonlinear networks

Random orthogonal nonlinear networks can preserve useful signal and gradient-propagation properties near the edge of chaos. The end-to-end Jacobian’s singular values depend on gain and input variance, remaining well-behaved near the critical gain even in nonlinear regimes.

Random orthogonal connectivity with gain g and saturating nonlinearities supports activity propagation whose behavior changes at a critical gain gc.For g < gc, activity decays to zero; for g > gc, it propagates indefinitely, regardless of depth.
For tanh nonlinearities, the critical gain is gc = 1, separating vanishing activity from a positive asymptotic population variance.
The end-to-end Jacobian measures input-to-output perturbation propagation, and its singular-value distribution indicates whether gradients explode or decay.
With N = 1000 neurons and Nl = 100 layers, the Jacobian spectrum varies with gain g and input population variance q1.The distribution was numerically computed for a random orthogonal nonlinear network.
Near g = 1, singular values remain O(1) across 100 layers even when input variance is far above 1 and tanh operates nonlinearly.This reflects approximate dynamical isometry in the nonlinear network.

5 Discussion

The discussion emphasizes that deep linear networks exhibit rich, analytically tractable learning dynamics despite their simple input-output maps. Appropriate initialization can make learning and gradient propagation largely depth-independent, including in nonlinear networks near the edge of chaos.

Deep linear networks display hyperbolic dynamics, plateaus, sudden performance transitions, saddle points, symmetries, conserved quantities, and independently evolving connectivity modes.
Learning times depend sensitively but computably on input statistics, initial weight conditions, and network depth.
With suitable initial conditions, very deep linear networks can be only finitely slower than shallow networks.
Unsupervised pretraining and random orthogonal initialization can provide depth-independent learning times and faithful gradient propagation under the stated conditions.

A Hyperbolic dynamics of learning

The hyperbolic-coordinate analysis gives exact learning trajectories for unequal layer-mode strengths and clarifies their learning timescales. It also extends beyond earlier matrix Riccati solutions to arbitrary depth and broader initial conditions.

Hyperbolic coordinates parameterize invariant manifolds defined by a^2 − b^2 = ±c0 and track the product ab toward the target correlation s.
The learning time is the travel time from θ0 to θf along a hyperbola, with a cutoff marking when ab approaches s within factor 1 − ϵ.
For small initial weights, learning time scales as O(τ/s), with only weak logarithmic dependence on the endpoint cutoff.Each input-output mode therefore learns on a timescale inversely proportional to its correlation strength.
The hyperbolic solutions apply to initial conditions outside the restrictions required for the earlier matrix Riccati formulation.
The analysis extends the treatment from three-layer networks to arbitrary Nl-layer networks while providing intuition about learning timescales.

B Optimal discrete time learning rates

The optimal discrete-time learning-rate analysis relates stability to network depth and the largest visited mode strength. It shows that, with suitable rate scaling, very deep networks incur only a finite delay relative to shallow ones.

The optimal first-order learning rate is determined by maximizing λ1 over the mode strengths visited during learning.
The analysis assumes a symmetric manifold with ai = aj = a and notes that its basin of attraction excludes small initial conditions.
The maximum occurs at aopt = s1/(Nl−1), linking the optimal rate to the strongest mode and network depth.
The optimal learning rate α scales inversely with λ1(aopt).
With learning rate scaled according to the optimal-depth analysis, very deep networks incur only a finite delay relative to shallow networks.

C Experimental setup for MNIST depth experiment

The depth experiment trains deep linear networks on MNIST across fifteen depths, using batch gradient descent and overcomplete hidden layers. The surrounding analysis contrasts pretrained and carefully scaled random initializations as possible sources of optimization-speed differences.

Experimental setup: 15 network depths from 3 to 100 layers were evaluated on MNIST classification.The networks mapped 784-dimensional inputs to 10-dimensional one-hot output vectors using 50,000 training examples.
Experimental setup: Batch gradient descent trained the networks on the 50,000-sample MNIST training dataset.The update rule exploited the linearity of the network to make very deep experiments computationally feasible.
Experimental setup: The hidden layers contained 1000 units, and overcompleteness was used to demonstrate applicability rather than increase representational power.
Initialization comparison: Pretraining is discussed as an optimization-speed advantage because pretrained initializations can converge faster than random initializations.The cited comparisons include Hessian-free optimization and stochastic gradient descent, including carefully chosen random schemes.
Scope: The utility of combining pretraining with momentum, convolutional architectures, dropout, and large supervised datasets remained unclear.
Scope: Pretraining may also improve generalization through a special regularization effect, but analyzing that effect in deep linear networks was outside this work’s scope.

E Learning dynamics with task-aligned input correlations

The analysis extends beyond orthogonal inputs to a structured class of input correlations aligned with the singular-vector basis of the input-output task. Under this structure, the learning dynamics can be generalized from the decoupled case.

Task-aligned correlations: Input correlations can be generalized from Σ11 = I to Σ11 = V11DV11^T.The decomposition Σ31 = U33S31V11^T supplies the relevant singular-vector basis.
Task-aligned correlations: The required structure aligns the input’s axes of variation with the axes of the input-output correlation.
Task-aligned correlations: The learning dynamics remain decoupled when the relevant weight matrices begin diagonal in this basis.

F MNIST pretraining experiment

The MNIST pretraining experiment compares small Gaussian random initialization with greedy layerwise reconstruction before supervised fine-tuning in depth-5 networks.

Initialization: Depth-5 MNIST networks used 200 hidden units per layer and were initialized either randomly or through greedy layerwise pretraining.Random weights were independently drawn from a Gaussian distribution with standard deviation 0.01.
Pretraining procedure: The pretrained initialization was created by training each layer to reconstruct the output of the next lower layer.The resulting network was then fine-tuned to predict a 10-dimensional class output.

G Analysis of Neural Dynamics in Nonlinear Orthogonal Networks

For nonlinear networks with orthogonal weights, the analysis derives a variance recursion and its stable fixed points across depth. For tanh activations, the critical gain is 1, separating vanishing activity from nonzero chaotic propagation, with theory matching depth-30 simulations.

Variance propagation: A recursion relation tracks neural population variance q_l across layers under nonlinear dynamics with orthogonal weights.The derivation uses a Gaussian approximation for activity distributions across neurons.
Variance propagation: The recursion has a stable fixed point q∞(g) obtained by solving a nonlinear fixed-point equation.
Phase transition: For g < 1, the asymptotic variance is zero, whereas for g > 1 a stable nonzero solution appears.
Theory versus simulation: The analytic fixed-point prediction matches population variances from depth-30 simulations with 1000 neurons per layer.The simulations estimate q∞ by averaging the population variance over the last five layers.
Phase transition: For tanh activations, the critical gain is g_c = 1, called the edge of chaos.Above this gain, infinitely deep networks exhibit chaotic percolating activity propagation.