Source-linked AI summary
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
TL;DR
Deep-network initialization can preserve average gradient scale without controlling worst-case Jacobian distortion, leaving the nonlinear extension of linear-network dynamical isometry unresolved. The paper uses free probability to compute full Jacobian singular-value distributions and finds that ReLU cannot achieve dynamical isometry, while orthogonal sigmoidal networks can and learn orders of magnitude faster. These results support designing networks around concentrated Jacobian spectra rather than only their second moment.
Problem
The paper addresses how Jacobian singular-value distributions and dynamical isometry extend from deep linear networks to nonlinear networks.
Method
The authors use free probability theory to analytically compute the entire singular-value distribution of deep networks' input-output Jacobians across depth, initialization, and nonlinearity.
Results
Orthogonal sigmoidal networks achieve dynamical isometry and learn orders of magnitude faster than ReLU networks, whereas ReLU networks cannot achieve dynamical isometry.
Takeaways & Limitations
Controlling the entire Jacobian singular-value distribution, rather than only its second moment, is an important design consideration for deep learning.
Takeaways & Limitations
The analysis is presented for feedforward networks, with extensions to skip connections and convolutional architectures left for future work.
Abstract
from arXiv · showhide
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is $O(1)$ is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condition that all singular values of the Jacobian concentrate near $1$ is a property known as dynamical isometry. For deep linear networks, dynamical isometry can be achieved through orthogonal weight initialization and has been shown to dramatically speed up learning; however, it has remained unclear how to extend these results to the nonlinear setting. We address this question by employing powerful tools from free probability theory to compute analytically the entire singular value distribution of a deep network's input-output Jacobian. We explore the dependence of the singular value distribution on the depth of the network, the weight initialization, and the choice of nonlinearity. Intriguingly, we find that ReLU networks are incapable of dynamical isometry. On the other hand, sigmoidal networks can achieve isometry, but only with orthogonal weight initialization. Moreover, we demonstrate empirically that deep nonlinear networks achieving dynamical isometry learn orders of magnitude faster than networks that do not. Indeed, we show that properly-initialized deep sigmoidal networks consistently outperform deep ReLU networks. Overall, our analysis reveals that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning.
1 Introduction
Weight initialization strongly affects deep-network training by governing gradient propagation. Dynamical isometry strengthens critical initialization by requiring all Jacobian singular values, not merely their mean square, to remain near 1.
- Appropriately scaled Gaussian weights can prevent gradients from exploding or vanishing exponentially, a condition necessary for reasonable learning speeds.
- The mean squared Jacobian singular value preserves a randomly chosen error vector's norm on average but provides no worst-case guarantee.
- Dynamical isometry requires every Jacobian singular value to remain near 1, preserving error-vector norms and angles during backpropagation.
- In deep linear networks, orthogonal initialization achieves dynamical isometry and depth-independent learning times, unlike random Gaussian initialization.
- For nonlinear networks, the paper asks how Jacobian spectra depend on depth, initialization, and nonlinearity, and whether isometry improves learning speed beyond preventing gradient explosion or vanishing.
2 Theoretical Results
The theory computes deep-network Jacobian spectra and identifies how initialization and nonlinearity determine conditioning. At criticality, ReLU networks remain non-isometric, whereas orthogonal hard-tanh networks can approach isometry by operating mostly in their linear regime.
- 2.2 Review of signal propagation: χ = 1 is the critical initialization line where the mean squared singular value of J equals 1, avoiding exponentially vanishing or exploding gradients.χ > 1 gives exponential gradient explosion, while χ < 1 gives exponential gradient decay.
- 2.3 Free probability, random matrix theory and deep networks.: Free probability computes the full singular-value density of the input-output Jacobian from products of random matrices.The analysis also uses the mean squared singular value, maximum singular value smax, and eigenvalue-distribution variance to quantify conditioning.
- 2.4 Linear networks: Deep linear Gaussian networks develop anisotropic spectra with extended tails, and their maximum eigenvalue scales as λmax ∼ e^L, indicating poor conditioning.
- 2.5.1 Gaussian weights: For Gaussian initialization, ReLU and hard-tanh networks cannot prevent linear growth of conditioning measures with depth, even at criticality.Thus critical Gaussian initialization fails to produce dynamical isometry for these nonlinearities.
- 2.5.2 Orthogonal weights: Orthogonal hard-tanh is much better conditioned than orthogonal ReLU at criticality, matching theoretical spectra with empirical simulations.
- 2.5.2 Orthogonal weights: For orthogonal weights, ReLU remains non-isometric, whereas hard-tanh can achieve dynamical isometry by choosing small q∗ so that p(q∗) approaches 1.Here p(q∗) is the fraction of neurons in the linear regime, allowing orthogonal hard-tanh networks to mimic orthogonal linear networks.
3 Experiments
Experiments show that dynamical isometry is associated with substantially faster learning, including sublinear depth scaling and improved performance when isometry persists during training.
- Experiments: Optimal training time grew sublinearly with depth L, while the optimal learning rate scaled as O(1/L).For orthogonal tanh networks, the reported training-time scaling is O(sqrt(L)).
- Experiments: Smaller q∗, associated with increased dynamical isometry in J, enabled faster training by allowing a larger optimal learning rate.This comparison was conducted at fixed depth L = 200.
- Experiments: Orthogonal tanh networks with small σ2_w learned several orders of magnitude faster than ReLU networks, robustly across optimizers.The initialization effect was larger than the optimizer choice in the tanh setting where dynamical isometry helped most.
- Experiments: Dynamical isometry present at initialization persisted for some time during training, and its persistence was associated with faster learning and better generalization.The experiments tracked singular-value evolution, test accuracy, and generalization accuracy across initial q∗ values.
- Experiments: The q∗ that best preserved dynamical isometry during training was nonzero rather than the perfectly isometric q∗ = 0.The optimal nonzero q∗ also produced the fastest learning and best generalization accuracy.
4 Discussion
The paper concludes that controlling the full Jacobian singular-value distribution enables dynamical isometry in deep nonlinear networks and can substantially improve learning performance.
- 4 Discussion: Free probability theory analytically computes the entire Jacobian singular-value distribution as a function of depth, initialization, and nonlinearity.The analysis identifies which combinations enable dynamical isometry.
- 4 Discussion: ReLU networks lose dynamical isometry, whereas orthogonal sigmoidal networks can maintain O(1) maximum Jacobian singular values with depth.Gaussian sigmoidal networks instead have maximum singular values that grow linearly with depth.
- 4 Discussion: On CIFAR-10, orthogonal sigmoidal networks learned orders of magnitude faster than ReLU networks and had training times growing as the square root of depth.The advantage persisted across SGD, momentum, RMSProp, and ADAM.
- 4 Discussion: Tightly concentrating the full Jacobian singular-value distribution around 1 can let very deep sigmoidal networks outperform ReLU networks.Controlling only the second moment can leave significant performance advantages unused.
- 4 Discussion: The methods are proposed for future extension to architectures such as skip-connected and convolutional networks.The discussion also suggests explicitly optimizing dynamical isometry in architecture searches.
Supplemental Material Resurrecting the sigmoid in deep learning through
The supplemental material is titled “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice.”
- Supplemental Material Resurrecting the sigmoid in deep learning through: The supplemental material carries the paper title “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice.”
1 Theoretical results
The theoretical-results supplement develops S-transform identities and matrix-product constructions used to analyze Jacobian spectra.
- 1 Theoretical results: Result 1 gives the S-transform for JJT.
- 1 Theoretical results: The S-transform depends only on distribution moments, which can be expressed through traces invariant under cyclic permutations.This invariance is used to relate products formed in different cyclic orders.
- 1 Theoretical results: The matrices QL = JJT and its cyclically permuted counterpart have equal S-transforms.Identically distributed weight matrices then support the subsequent simplification.
- 1 Theoretical results: Products of Gaussian random matrices with variance σ2_w are treated as an example of the S-transform framework.
- 1 Theoretical results: The supplement uses Wishart moments and their generating function to derive the relevant transform expressions.It also records the identity matrix’s S-transform as SI = 1.