Source-linked AI summary
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
Boris Hanin
TL;DR
Deep neural networks can suffer exploding or vanishing gradients, and the paper studies this problem rigorously for randomly initialized fully connected ReLU networks. It derives finite-width and finite-depth Jacobian statistics, showing that gradient variance depends exponentially on β, the sum of reciprocal hidden-layer widths, while introducing annealed and quenched formulations.
Problem
Exploding and vanishing gradients create derivatives that are very large for some parameters and very small for others, complicating gradient-based optimization.
Method
The paper derives exact finite-depth, finite-width Jacobian moment formulas and analyzes annealed and quenched EVGP criteria for randomly initialized fully connected ReLU networks.
Results
The empirical variance of squared input-output Jacobian entries is exponential in β, the sum of reciprocal hidden-layer widths; large β corresponds to wildly fluctuating gradients.
Takeaways & Limitations
Architecture determines whether the EVGP occurs under the paper’s criteria, and equalizing hidden-layer widths is advantageous for avoiding it under fixed budgets.
Takeaways & Limitations
The analysis is restricted to ReLU activations, independent zero-centered nonzero weights, fixed inputs, and does not directly extend to convolutional, residual, recurrent, or orthogonal-weight networks.
Abstract
from arXiv · showhide
We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, given by the sum of the reciprocals of the hidden layer widths. When beta is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.
1 Introduction
The paper rigorously analyzes exploding and vanishing gradients in randomly initialized fully connected ReLU networks, focusing on finite-depth and finite-width effects. It identifies the sum of reciprocal hidden-layer widths as the key architectural quantity governing gradient fluctuations and early training dynamics.
- 1 Introduction: The exploding and vanishing gradient problem occurs when loss derivatives are very large for some parameters and very small for others.Such variation can make SGD increments either meaningless or imprecise.
- 1 Introduction: The authors derive exact formulas for joint even moments of input-output Jacobian entries at finite depth and width.These formulas are stated as a main contribution in Theorem 3.
- 1 Introduction: The empirical variance of gradients is exponential in β, the sum of reciprocals of hidden-layer widths.A large β is associated with wildly fluctuating gradients and slow early training.
- 1 Introduction: For fixed depth and neuron or parameter budgets, equal hidden-layer widths minimize β and are advantageous for avoiding the EVGP.The equal-width condition is characterized through special cases of the power-mean inequality.
- 1 Introduction: An MNIST experiment reports that β summarizes how quickly fully connected networks begin training.Figure 1 plots mean epochs over 100 runs to reach 20% accuracy against the sum of reciprocal hidden-layer widths.
- 1 Introduction: The analysis is limited to ReLU activations, independently initialized zero-centered nonzero weights, fixed inputs, and non-orthogonal initialization strategies.The conclusions do not directly carry over to convolutional, residual, or recurrent networks.
2 Relation to Prior Work
The paper complements infinite-width mean field analyses by rigorously quantifying finite-width gradient fluctuations in fully connected ReLU networks. It introduces annealed and quenched, non-spectral formulations of the EVGP and relates them to Jacobian-entry moments and architecture.
- 2 Relation to Prior Work: Prior work studies Jacobian singular values, including average singular value near one and the stronger dynamical-isometry condition.Those analyses consider infinite-width and then infinite-depth limits.
- 2 Relation to Prior Work: This paper quantifies finite-width effects rather than assuming networks are essentially infinitely wide.It leaves aside different nonlinearities and initialization methods beyond independent weights.
- 2 Relation to Prior Work: The authors replace the singular-value formulation with two non-spectral EVGP notions called annealed and quenched.Both are defined through joint moments of input-output Jacobian entries.
- 2 Relation to Prior Work: In the infinite-width limit, Theorem 1 implies that all ReLU networks avoid the EVGP under both the quenched and annealed senses.These senses are weaker than dynamical isometry but stronger than requiring only average singular value one.
- 2 Relation to Prior Work: The quenched formulation concerns empirical variance across all input-output Jacobian entries for a fixed weight and bias realization.Large typical variance means derivatives with respect to different trainable parameters can span several orders of magnitude, producing inefficient SGD updates.
- 2 Relation to Prior Work: At the ReLU edge of chaos, the results provide a rigorous computation of finite-width 1/n_j corrections to gradient statistics.This connects the paper’s finite-width analysis to mean field theory.
3 Defining the EVGP for Feed-Forward Networks
The paper defines exploding and vanishing gradients through fluctuations in input-output Jacobian entries, distinguishing annealed variation across initializations from quenched variation within one initialization. It relates these entrywise definitions to spectral formulations and characterizes when ReLU architectures avoid each problem.
- Entrywise versus spectral definitions: The EVGP is studied through fluctuations of the input-output Jacobian entries Z_p,q, rather than only through its singular values.These entries directly represent partial derivatives of outputs with respect to inputs and contribute to gradient propagation.
- Entrywise versus spectral definitions: The paper notes that Jacobian condition numbers capture worst-case stretching or compression, but their relevance is unclear when layer dimensions vary with depth.This motivates analyzing the joint distribution of gradient-related Jacobian entries across network parameters.
- Entrywise versus spectral definitions: The entrywise approach focuses on the Jacobian factor in weight-gradient moments, while activations from the preceding layer provide a second factor.The paper treats fluctuations in the Jacobian factor as its measure of the EVGP.
- Entrywise versus spectral definitions: At finite depth and width, the average singular value of the Jacobian equals 1 for all considered random ReLU networks, so this criterion alone does not detect their entrywise EVGP.The paper connects average singular values to Jacobian-entry moments while distinguishing this condition from its fluctuation-based definitions.
- Annealed EVGP: The annealed EVGP concerns fluctuations of typical partial derivatives across random weights and biases, affecting whether one learning rate can be selected a priori.Annealed avoidance requires bounded fluctuations around a constant mean value.
- Quenched EVGP: The quenched EVGP concerns empirical variance across Jacobian entries within a fixed realization of weights and biases.For the ReLU architectures studied, quenched avoidance occurs if and only if annealed avoidance occurs.
5 Notation and Main Results
The paper defines randomly initialized fully connected ReLU networks and studies input-output Jacobian gradients through exact moment calculations. Its main results characterize exploding and vanishing gradients through architecture-dependent width and depth effects.
- Random network notation: Random networks are fully connected feed-forward ReLU nets whose weights and biases are randomized independently by layer-specific distributions.The network function is defined from layerwise neuron inputs and outputs, with hidden-layer widths n_j and depth d.
- Assumptions and scope: The analysis assumes atomless bias distributions, although this condition can be removed under a restriction on depth relative to exponential width growth.The authors do not pursue the resulting slightly messier formulation because the results are not meaningfully different.
- Gradient statistics: The gradient quantities Z_p,q are entries of the input-output Jacobian, measuring output-component derivatives with respect to input components.The paper studies their moments and the empirical variance of their squares across input-output neuron pairs.
- Main theoretical results: Theorem 1 shows that the fourth moment of Z_p,q is exponential in the sum of reciprocals of hidden-layer widths.The theorem provides the principal width- and depth-dependent gradient fluctuation result for fixed inputs.
- Main theoretical results: Theorem 2 characterizes the empirical variance of the squared Jacobian entries and gives a criterion for avoiding exploding and vanishing gradients in the quenched sense.The variance is computed across all M = n_0 n_d input-output neuron pairs.
- Main theoretical results: Theorem 3 gives exact expressions for even gradient moments as sums over ordered tuples of paths through the network.These expressions depend on moments of the layerwise weight distributions and support the preceding gradient-statistics results.
A Proof of Theorem 3
Theorem 3 is proved by expanding Jacobian-related quantities over path collections and evaluating expectations using symmetry, independence, and layerwise Markov structure.
- Auxiliary lemma: Lemma 1 supplies the key averaging identity for symmetric independent random variables under the condition ψ(t) + ψ(−t) = 1.The lemma assumes that X has an atomless distribution and that the variables are independent and symmetric.
- Symmetry and parity: Symmetric weight distributions eliminate terms with odd moments, so only path collections satisfying even-multiplicity conditions contribute.This parity condition is applied repeatedly to path collections and their shared neuron transitions.
- Path expansion: The proof represents network computations through paths connecting input and output neurons, with hidden-layer neuron sequences determining each path.The path expansion follows directly from the chain rule.
- Expectation decomposition: Layerwise conditional independence and the tower property reduce the expectation to successive calculations over activations, weights, and biases.The network computation is treated as a Markov chain with respect to the layers.
- Completion of the proof: Theorem 3 follows after separating path collections according to their parity structure and showing that the required identity holds either directly or because both sides vanish.In the latter case, symmetry makes the relevant odd moments zero.
B Proof of Theorem 1
Theorem 1 is established by applying Theorem 3 to path collections, relating their weights to layerwise path overlaps, and bounding contributions according to how often paths coincide.
- Application of Theorem 3: Theorem 3 converts the relevant Jacobian moment expressions into expectations over uniformly selected collections of paths between fixed input and output neurons.The path collections are conditioned on their common endpoints.
- Symmetry simplification: Symmetry of the weight distribution removes terms from distinct initial paths because the first moment vanishes.The proof uses μ_1 = 0 to eliminate those contributions.
- Second-moment structure: Path collections are grouped by the number k of hidden layers at which two paths coincide, enabling separate lower and upper bounds.The number of such collections is counted combinatorially across the hidden layers.
- Moment bounds: The contribution of each path pair is bounded using the overlap-dependent quantity X_d, whose lower and upper estimates produce the bounds in (12).The lower-bound argument uses X_d ≥ 2k+1, while the upper-bound argument proceeds analogously.
- Higher moments: For higher moments, only collections whose layerwise path multiplicities are even contribute, and their weights are controlled by a fixed constant depending on K.The argument extends the second- and fourth-moment analysis to K ≥ 3.
- Conclusion: The proof concludes by combining the path-counting estimates and moment bounds to establish Theorem 1.The final step explicitly closes the theorem after deriving the required estimates.
C Proof of Theorem 2
Theorem 2 is proved by comparing path collections through a bijection and analyzing how endpoint coincidences and hidden-layer overlaps affect their contributions.
- Path comparison: The proof fixes input and output neurons and compares collections of four paths with corresponding collections that share the same hidden-neuron sequences.The comparison uses a bijection between the two path representations.
- Moment selection: Symmetry forces odd moments of the weight distributions to vanish, restricting nonzero terms to collections with even path multiplicities across layer transitions.This parity restriction is used in the expansion of the relevant expectations.
- Case analysis: The proof separates cases according to whether paired paths have distinct or coinciding output neurons and then bounds each case using earlier estimates.The two cases are explicitly indexed by qm1 ≠ qm2 versus coinciding outputs.
- Contribution comparison: When path collections are modified, their layerwise factors satisfy comparison inequalities that control the resulting contribution relative to the original collection.The argument includes inequalities such as C_d(Γ) ≥ 2C_d(¯Γ).
- Completion: Counting the relevant endpoint configurations and combining the case bounds yields the stated estimates of Theorem 2.The final upper bound follows by dropping a negative term and applying the upper bound from (12).