Source-linked AI summary
Deep Information Propagation
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, Jascha Sohl-Dickstein
TL;DR
Randomly initialized networks need to propagate information through depth to remain trainable, but existing architectural design lacks a general guiding theory. The paper uses mean-field analysis of signal and gradient propagation to identify depth scales, then relates them to training across datasets and initialization regimes. It finds that trainability is tied to propagation depth, criticality allows arbitrarily deep networks in principle, and dropout removes the critical point that enables this divergence.
Problem
A general theory that informs neural-network architecture and initialization choices has lagged behind the development of increasingly deep networks.
Method
The paper develops mean-field models for forward signal propagation and backpropagated gradients in randomly initialized, fully connected networks, and tests predictions by training networks on MNIST and CIFAR10.
Results
Depth scales control information propagation and trainability, while the ordered and chaotic phases correspond to vanishing and exploding gradients.
Takeaways & Limitations
Networks can support very large trainable depths near the order-to-chaos transition, whereas dropout removes the critical point and imposes an upper bound on trainable depth.
Takeaways & Limitations
The framework applies directly only to bounded activations and fully connected feed-forward networks, not unbounded activations or structured architectures such as convolutional networks.
Abstract
from arXiv · showhide
We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these depth scales diverges. Thus arbitrarily deep networks may be trained only sufficiently close to criticality. We show that the presence of dropout destroys the order-to-chaos critical point and therefore strongly limits the maximum trainable depth for random networks. Finally, we develop a mean field theory for backpropagation and we show that the ordered and chaotic phases correspond to regions of vanishing and exploding gradient respectively.
1 INTRODUCTION
The paper develops depth scales for signal and gradient propagation in randomly initialized networks, connecting these scales to trainability, criticality, and dropout. It tests this connection across network depths and datasets and argues that sufficiently deep training requires proximity to the order-to-chaos transition.
- Several characteristic depth scales emerge naturally and control signal propagation in random neural networks.
- The correlation depth scale ξc diverges at the order-to-chaos boundary, generally across architectural choices such as activation function.
- The gradient mean-field model identifies a duality between forward signal propagation and backpropagated gradients.
- Ordered and chaotic phases correspond respectively to vanishing and exploding gradients, with gradient covariance controlled by the same scale limiting correlated forward signals.
- Training experiments on MNIST and CIFAR10 support trainability when network depth is not much larger than ξc.
- Asymptotically deep networks should be trainable when initialized sufficiently close to the order-to-chaos transition, whereas dropout removes that critical point and limits trainable depth.
2 BACKGROUND
The background formulates signal propagation in wide, untrained, fully connected networks through a Gaussian mean-field approximation. It tracks variances and correlations across depth, yielding fixed points and an order-to-chaos critical line.
- The mean-field formalism studies fully connected, untrained, feed-forward networks with random weights and biases and nonlinearity φ.
- The approximation replaces each layer’s pre-activations with Gaussians matching their first two moments.
- Single-input signal evolution is described by a variance recurrence relation that can have a fixed point q∗ as depth grows.
- Two-input propagation tracks covariance and correlation, with c_l measuring the correlation between inputs after l layers.
- Figure 1 presents the ordered-chaotic phase boundary and exponential depth behavior on a log-scale.
- The correlation c∗ = 1 is stable when χ1 < 1 and unstable otherwise, defining ordered and chaotic phases separated by χ1 = 1.
3 ASYMPTOTIC EXPANSIONS AND DEPTH SCALES
Mean-field recurrences yield two exponential depth scales governing propagation from a single input and correlations between inputs. The correlation scale diverges at the order-to-chaos transition, whereas dropout removes that divergence and reduces propagation depth.
- Vanilla networks: Two natural depth scales arise in mean-field random networks, governing distinct modes of signal propagation.ξq describes information about a single input, while ξc describes correlations between two inputs.
- Asymptotic analysis: Asymptotic expansions around fixed points produce exponential convergence over depth scales ξq and ξc.The recurrences are expanded about their fixed points, yielding residuals that decay exponentially with depth.
- Criticality: ξc diverges at any order-to-chaos transition when the relevant fixed points exist.The divergence follows because the transition occurs at χ1 = 1.
- Empirical validation: The measured depth scales agree well with theoretical predictions, and ξc shows the predicted divergence at criticality.The correlation scale is consistently longer than the single-input scale ξq.
- Dropout: Dropout eliminates the sharp order-to-chaos transition because c∗ = 1 is no longer a fixed point for any ρ < 1.The asymptotic correlation decreases as dropout increases, and ξc no longer diverges.
- Dropout: With dropout, increasing the dropout rate drastically reduces the depth of signal propagation and further decreases the correlation depth at fixed σ2_w.For a single input, dropout instead acts as an effective rescaling of the weight variance by σ2_w/ρ.
4 GRADIENT BACKPROPAGATION
The paper develops a mean field account of backpropagation in random networks, identifying a duality between forward signal propagation and gradient behavior. Gradients vanish in ordered networks, explode in chaotic networks, and exhibit experimentally supported depth-scale predictions.
- Mean field theory: Backpropagation has a duality with forward signal propagation, with gradient recurrences governed by the forward quantity χ1 in constant-width networks.The analysis assumes independently drawn forward and backward weights and focuses on constant layer width.
- Gradient regimes: The gradient depth scale ξ∇ = −log χ1 can be positive or negative, yielding characteristic depths |ξ∇| for vanishing or exploding gradients.At criticality, ξ∇ diverges and gradients are predicted to remain stable regardless of depth.
- Gradient regimes: Gradients vanish in the ordered phase, remain stable at criticality, and explode in the chaotic phase.These regimes correspond respectively to χ1 < 1, χ1 → 1, and χ1 > 1.
- Experimental validation: For 240-layer random networks on MNIST, gradient norms show exponential vanishing or explosion across many orders of magnitude.The experiments vary initial weight variance and use a cross-entropy loss.
- Experimental validation: Measured gradient depth scales agree quantitatively with mean field predictions, with deviations near criticality attributed primarily to finite-size effects.The theory is compared with depth scales obtained from exponential fits to experimental gradient data.
- Gradient covariance: The depth scale controlling correlated forward signal propagation also controls how long covariance between gradients of similar inputs survives.This connects gradient consistency across samples to the same propagation scale.
5 EXPERIMENTAL RESULTS
Experiments test whether mean field propagation scales predict trainability across depths, variances, datasets, optimizers, and dropout settings. Networks train when depth is not much larger than ξc, while dropout imposes a finite upper bound.
- Trainability and depth: Trainability requires input and gradient information to propagate through the full network, with network depth L not much larger than ξc.The resulting hyperparameter constraint is presented as data independent.
- Experimental setup: Training experiments span depths 10 ≤ L ≤ 300, initial weight variances 1 ≤ σ_w^2 ≤ 4, MNIST and CIFAR10, and SGD or RMSProp.Learning rates vary with depth and optimizer according to the experimental setup.
- Experimental results: Training reaches 100% training accuracy, with test accuracies of 98% on MNIST and 55% on CIFAR10 across the reported conditions.The study evaluates trainability rather than optimizing test accuracy.
- Trainability and depth: Networks appear trainable when L ≲ 6ξc, although the paper notes that the origin of this factor remains to be understood.At L = nξc, signal is attenuated by a factor e^n, and the experiments overlay multiples from n = 1 to 6.
- Trainability and depth: Networks closer to criticality train more quickly, with the relationship showing no obvious dependence on dataset, training duration, or minimizer.The authors conclude that the resulting bounds on trainable hyperparameters are universal within these experiments.
- Dropout: Dropout rates ρ = 0.99, 0.98, and 0.94 limit maximum trainable depth to about L = 100, while ξc accurately predicts the trainable region.The experiments report good agreement between theoretical predictions and training accuracy under dropout.
6 DISCUSSION
The discussion frames information flow through inputs and gradients as central to training random networks and links covariance propagation to Gaussian-process kernels. It also proposes practical initialization ideas while delimiting the theory’s architectural scope.
- Core implications: Training depends crucially on propagating information about inputs and gradients through the network’s full depth.At the order-to-chaos transition, input correlations can propagate infinitely far, supporting sufficiently deep networks near criticality.
- Core implications: The framework predicts which hyperparameters permit training and implies an upper trainable-depth bound when dropout removes the critical point.This extends beyond simply advocating near-critical initialization.
- Gaussian-process connection: Propagation of covariance between inputs corresponds to situations in which the associated infinitely wide Gaussian Process is well defined.The covariance q^l_ab is identified with the kernel of the induced Gaussian Process.
- Practical directions: The authors suggest pre-training schemes that favor information flow, including local layer-wise criteria or batch-normalization settings maximizing the covariance depth scale.These are proposed as possible directions rather than established methods.
- Scope and limitations: The framework does not directly apply to unbounded activations such as rectified linear units and currently targets fully connected feed-forward networks.Extensions are needed for structured weight matrices such as those in convolutional networks.
- Open hypothesis: The paper hypothesizes that behavior near the fully connected mean field critical point may control a broad class of neural network topologies.This is explicitly presented as a bold hypothesis grounded in an analogy to renormalization near critical points.
7 APPENDIX
The appendix derives asymptotic recurrence relations around fixed points for variance and covariance dynamics, showing exponential relaxation over characteristic depth scales. The covariance analysis treats c∗<1 and c∗=1 separately before establishing a common recurrence result.
- Two-input covariance: The covariance between two inputs is analyzed by expanding its recurrence relation around the fixed point c∗.The derivation considers separate cases for c∗<1 and c∗=1.
- Two-input covariance: Convergence of the covariance-related quantity to q∗ occurs much more quickly than convergence of ql.
- Two-input covariance: For c∗<1, the asymptotic recurrence reduces to the c∗=1 result, allowing equation 4 to be used for all c∗.The common recurrence is associated with χ1, the stability condition for the ordered phase.
7.3 VARIANCE OF AN INPUT WITH DROPOUT
With dropout, the variance of a single input is described by a modified recurrence relation for signal propagation through the network.
- Variance with dropout: Dropout changes the variance dynamics of a single input through a recurrence relation parameterized by dropout rate ρ.
- Variance with dropout: The dropout variance derivation recalls the recurrence relation for the network pre-activations.
7.4 COVARIANCE OF TWO INPUTS WITH DROPOUT
The appendix formulates covariance propagation under dropout by tracking two signals subject to separate independent dropout masks.
- Covariance with dropout: The covariance between two signals is analyzed when the signals use separate i.i.d. dropout masks.
- Covariance with dropout: The covariance recurrence is derived directly using expectations over the dropout variables.
7.5 THE LACK OF A c∗= 1 FIXED POINT WITH DROPOUT
This section develops mean-field expressions for gradient magnitudes under dropout and uses i.i.d. weight and Gaussian pre-activation assumptions in the derivation.
- Gradient propagation: Mean field theory expresses the expected magnitude of the gradient with a recurrence relation.
- Gradient propagation: The derivation uses the i.i.d. structure of the weights to replace sample expectations with analytic expectations.
- Gradient propagation: The calculation assumes pre-activations in each layer are i.i.d. Gaussian.
7.7 MEAN FIELD BACKPROPAGATION
The paper develops a mean field treatment of backpropagation, analyzing gradient variance and covariance in randomly initialized networks. It uses independence and short-ranged depth-scale approximations to derive these quantities.
- Mean field backpropagation: Mean field theory analyzes the recursion for the variance of backpropagation errors across network depth.The derivation computes the variance directly and applies a mean field approximation.
- Mean field backpropagation: The variance analysis uses a short-ranged depth scale to approximate q_l by its fixed-point value q∗.This approximation supports replacing depth-dependent quantities with their fixed-point behavior.
- Gradient covariance: Mean field theory predicts how the covariance between gradients generated by two different inputs scales.The calculation proceeds analogously to the variance derivation.
- Gradient covariance: Independence of the weights in the forward and backward passes allows the relevant expectation to factorize.The argument relies on independently chosen weights and mean field sampling assumptions.
- Further experimental results: Additional experiments examine how training time, optimizer, and dataset affect training accuracy.The supplied figures show MNIST accuracy after several SGD or RMSProp training durations.