Source-linked AI summary
How to Start Training: The Effect of Initialization and Architecture
Boris Hanin, David Rolnick
TL;DR
Deep ReLU networks can fail to begin training because mean activation lengths explode or vanish, or because activation-length variance grows exponentially. The paper rigorously analyzes these failures across fully connected, convolutional, and residual architectures, then tests its criteria empirically. It finds that critical initialization and residual scaling prevent the first failure, while sufficient width controls the second in fully connected networks and residual networks avoid it once the first is prevented.
Problem
The paper addresses why deep ReLU networks fail to start training, focusing on exploding or vanishing mean activation length and exponentially growing activation-length variance.
Method
The authors rigorously analyze layerwise activation length scales at initialization for fully connected, convolutional, and residual ReLU networks, and evaluate the predictions empirically.
Results
Critical variance 2/fan-in prevents exploding or vanishing mean activation lengths, while residual-module rescaling prevents FM1 and consequently FM2; fully connected networks require sufficiently wide architectures to control FM2.
Takeaways & Limitations
Correct initialization and architecture allow much deeper networks to start training, whereas many popular initializations fail the paper’s criteria.
Takeaways & Limitations
The residual-network analysis assumes residual-module depths and widths are uniformly bounded.
Abstract
from arXiv · showhide
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly weighting the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.
1 Introduction
The paper identifies two early-training failure modes: exponentially changing mean activation length and exponentially growing variance across layers. It analyzes how initialization and architecture prevent these failures in fully connected and residual networks.
- FM1 is exponential growth or decay of the mean length scale in the final layer with depth.
- FM2 is exponential growth of the empirical variance of length scales across layers with depth.
- Avoiding both failure modes can allow deeper networks to start training without taking longer than shallower networks.
- Correct weight variance prevents FM1 in fully connected and convolutional networks, while residual networks require correctly weighted residual modules.
- For fully connected and convolutional networks, wider layers prevent FM2; constant-width networks need width to grow approximately linearly with depth.
- For residual networks, FM2 cannot occur once residual modules are weighted to avoid FM1, largely independently of architecture.
- FM2 can impede early training because extreme activation sizes may exceed machine precision or make backpropagated updates ineffective or excessively large.
2 Related Work
The paper builds on work about ResNet initialization, mean-field dynamics, normalized ReLU initialization, finite-width effects, and activation statistics. It distinguishes its contribution through rigorous finite-width analysis of activation length scales.
- Prior ResNet studies examined initialization heuristics and mean-field forward and backward dynamics as functions of residual-module weights.
- Earlier work showed that information can propagate in infinitely wide ReLU networks under appropriately normalized independent weight distributions.
- This paper differs from those studies by rigorously computing finite-width effects for randomly initialized networks.
- Related research studied finite-size corrections, Gaussian pre-activation distributions, and how hidden-layer activation means and variances affect training onset.
- The sum of reciprocals of layer widths was previously studied for gradient variance and is shown here to determine activation-size variance in fully connected ReLU networks.
- Earlier work also reported better convergence time for wider networks.
3 Results
The results show that initialization controls mean activation-length stability, while architecture controls activation-length variance in fully connected networks. Residual scaling prevents both failure modes under the paper’s conditions, and the predictions track early-training performance.
- FM1 and initialization: The mean normalized output length equals the input length when symmetric weights have variance 2/fan-in; higher or lower variance causes exponential growth or decay.This establishes the initialization criterion for avoiding FM1 in fully connected ReLU networks.
- Empirical validation: Experiments associate stable output lengths with faster initial training and show that the reciprocal-width sum closely predicts training effectiveness across fully connected architectures.Figure 1 varies initialization, while Figure 3 varies architecture with He normal initialization to isolate FM2.
- Residual networks: In residual networks, the mean activation-length scale grows exponentially with the sum of residual-module scales, so module weighting is required to avoid FM1.The residual recursion explicitly combines modules using scales ηℓ, and the mean length depends on their cumulative contribution.
- FM2 and architecture: For fully connected networks, empirical training speed is predicted by the sum of reciprocals of hidden-layer widths, which determines FM2 severity.The mean empirical activation-length variance grows exponentially in this sum, providing a quantitative account of why deep, narrow networks are difficult to train.
- FM2 and architecture: Width order does not affect FM2, whereas networks with a lower sum of reciprocal widths train faster than networks with the same depth and higher sums.Networks with equal reciprocal-width sums showed similar early-training dynamics despite differing width permutations and architectural details.
- Residual networks: Residual-network activation-length variance is exponential in the sum of module scales, but FM2 does not occur once FM1 is avoided, assuming individual modules avoid FM2.This distinguishes residual architectures from fully connected and convolutional networks, where FM2 can persist despite avoiding FM1.
4 Notation
This section formalizes fully connected ReLU networks by specifying their dimensions, layers, parameters, activations, and random initialization distributions.
- The networks studied are fully connected feed-forward networks with ReLU activations and depth d.
- The jth hidden layer has width n_j, while n_0 denotes the input dimension.
- The network function is determined by its weights and biases, with preact^(j) and act^(j) representing the inputs and outputs of each layer's nonlinearity.
- The initialization distributions are symmetric around zero, and the weight variance at layer j is 2/n_(j−1).
- Random networks draw weights and biases independently from layer-specific distributions μ^(j) and ν^(j), respectively.
5 Formal statements
The formal results characterize when activation means and variances remain controlled in fully connected and residual ReLU networks. Fully connected networks can exhibit FM2 depending on widths, whereas residual networks avoid it once residual scales prevent FM1.
- Fully connected networks: Theorem 5 analyzes the mean and variance of the normalized squared activation length Md in fully connected ReLU networks.The theorem applies to fixed input, layer widths, random weights, and biases under the paper’s stated assumptions.
- Fully connected networks: Under condition (4), Md converges pointwise almost surely to a finite random variable, and if Σj 1/nj < ∞, convergence also holds in L2 with finite limiting variance.The supplied theorem statements connect almost-sure convergence to condition (4) and finite limiting variance to summability of reciprocal widths.
- Fully connected networks: FM2 occurs in fully connected networks when the empirical variance of activation lengths grows exponentially with Σj 1/nj.The result is stated as a corollary of Theorem 5; equal widths give β = (d − 1)/n.
- Fully connected networks: For equal hidden widths nj = n, the relevant exponent is β = (d − 1)/n, so increasing width reduces the depth-dependent reciprocal-width sum.The supplied passages state that E[d Var[M]] is exponential in β for equal-width hidden layers.
- Residual networks: In residual networks, the mean squared activations are uniformly bounded in the number of modules L if and only if the residual scales ηℓ form a convergent series.Theorem 6 further states that, once this condition holds and supℓηℓ < 1, activation-variance moments are uniformly bounded in L.
- Residual networks: Once the residual-scale condition holds, both the mean and the empirical variance of residual activation lengths remain uniformly bounded as L grows.The theorem’s bounds have implied constants depending on K but not on L.
6 Conclusion
The paper rigorously analyzes layerwise activation length scales at initialization across fully connected, convolutional, and residual ReLU networks. It identifies initialization and architecture conditions that control mean and variance, and reports that these quantities predict early training dynamics.
- Initialization and architecture: Correct initialization preserves mean activation length in fully connected and convolutional networks, while residual networks require appropriately rescaled residual modules.The paper associates these choices with well-behaved mean length scales.
- Initialization and architecture: Fully connected networks require sufficiently wide architectures to control variance of layerwise length scales, whereas residual networks require no further condition beyond mean control.This distinction is stated as the paper’s main architectural conclusion.
- Empirical implications: Empirical results show that both mean and variance of activation length scales are strong predictors of early training dynamics.The conclusion presents this as an empirical finding of the paper.
- Scope and future work: The paper proposes extending the analysis to sigmoidal activations, recurrent networks, non-i.i.d. initializations, and joint activation distributions over several inputs.These are identified as future directions rather than established results.
A Proof of Theorem 5
The proof of Theorem 5 uses martingale structure, symmetry, and moment bounds to control normalized squared activation lengths in fully connected ReLU networks. Iterated inequalities then yield convergence and variance estimates across depth.
- Martingale setup: Md is treated as a submartingale with respect to the filtration generated by weights and biases through layer d.Measurability follows because the input is fixed and Md depends on activations through layer d.
- Moment calculations: Symmetrizing layerwise expressions uses symmetric weight and bias laws, independence from prior-layer information, and zero weight means.The proof also uses Var[w(d)α,β] = 2/nj−1.
- Convergence: A normalized version of Md is shown to be a martingale, and condition (4) permits Doob’s convergence theorem to establish a finite almost-sure limit.Uniform L1 boundedness is used to invoke pointwise martingale convergence.
- Bounds: Iterating lower and upper conditional bounds produces the theorem’s depth-dependent estimates for the mean and variance of Md.The proof combines the lemma’s bounds with an estimate on the reciprocal-width sum.
B Proof of Corollary 2
The proof of Corollary 2 studies the empirical variance of squared activation lengths using the martingale property of bias-free activations. Equal-width specialization shows exponential dependence on the reciprocal-width sum.
- Variance analysis: With zero biases, the squared activation lengths Mj form a martingale, enabling analysis of their empirical variance across layers.The proof fixes the network, input, depth, and hidden widths before studying d Var[M].
- Variance analysis: The variance bounds inherit exponential dependence on the sum of reciprocal layer widths.The proof invokes the mean bounds from Theorem 5 to establish this dependence.
- Equal-width networks: For equal widths nj = n, the sum Σj 1/nj becomes (d − 1)/n, making the depth-width trade-off explicit.This specialization establishes the lower bounds in the corollary; upper bounds are obtained similarly.
C Proof of Theorem 6
The section proves estimates for activation behavior in fully connected ReLU networks by tracking layerwise activations and exploiting martingale properties. The proof concludes by combining these estimates with Jensen’s inequality.
- The network assumptions include random weights, zero biases, and hidden-layer widths n0, . . . , nd.
- Positive homogeneity of zero-bias ReLU networks and norm preservation are used in the derivation.
- The proof analyzes activations layer by layer, denoting the input by x(0) and the activations at layer j by x(j).
- The squared activation quantity is a martingale, making x(j) a supermartingale.
- Jensen’s inequality supplies an intermediate bound, and combining it with equation (13) completes the proof.