PaperlayerView exact revision on arXiv

Source-linked AI summary

The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions

George Philipp, Dawn Song, Jaime G. Carbonell

arXiv:1712.05577v4cs.LGcs.CV

TL;DR

The paper asks whether exploding gradients are a numerical quirk removable by gradient rescaling or reflect inherently difficult optimization, and whether avoiding gradient pathologies entails unavoidable tradeoffs. The authors introduce the GSC to assess exploding gradients, analyze their causes and tradeoffs in MLPs, and use the residual trick to view arbitrary networks as residual networks. The authors report that popular MLP architectures can exhibit GSC-defined exploding gradients that limit effective depth, while architectures avoiding them can exhibit collapsing domains or extreme pseudo-linearity.

  • Problem

    The paper asks whether exploding gradients are a numerical quirk removable by gradient rescaling or reflect inherently difficult optimization, and whether avoiding gradient pathologies entails unavoidable tradeoffs.

    Source: §1 · pp. 2–3

  • Method

    The authors introduce the GSC to assess exploding gradients, analyze their causes and tradeoffs in MLPs, and use the residual trick to view arbitrary networks as residual networks.

    Source: §1 · p. 3

  • Results

    The authors report that popular MLP architectures can exhibit GSC-defined exploding gradients that limit effective depth, while architectures avoiding them can exhibit collapsing domains or extreme pseudo-linearity.

    Source: §1 · pp. 3–4, §9 · p. 31

  • Takeaways & Limitations

    The authors argue that eliminating untrainable, potentially unnecessary non-orthogonality can avoid both pathologies to a large degree, while skip connections simplify network mathematics.

    Source: §1 · p. 3, §7.5 · p. 28, §9 · p. 31

    Large-k k-diluted functions are close to linear, which can damage representational capacity; in randomly initialized compositions, dilution disappears slowly.

    Source: §7.1 · p. 26

Abstract

from arXiv · show

Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the *collapsing domain problem*, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that *any neural network is a residual network*, we devise the *residual trick*, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

1. Introduction

The paper challenges the view that stabilizing activations or gradients alone resolves gradient pathology, and asks whether exploding gradients signal an inherently difficult optimization problem. It introduces a metric and analysis of exploding gradients, their tradeoffs, and why skip connections may help.

1. Introduction

  • The paper introduces the gradient scale coefficient (GSC), a measure intended to assess pathological exploding gradients robustly to network scaling and layer width.The authors propose that the metric can help standardize research on gradient pathology.Source: §1 · p. 3
  • Popular MLP architectures can exhibit exploding gradients despite techniques intended to combat them, and normalization layers may exacerbate the problem.The paper also argues that stabilizing forward activations is not sufficient to avoid exploding gradients.Source: §1 · p. 3
  • Exploding gradients measured by the GSC indicate difficult optimization and limit the effective depth of MLPs, making very deep MLPs effectively much shallower.The authors argue this is not merely a numerical issue solved by rescaling gradient components.Source: §1 · p. 4
  • The paper explains why exploding gradients may occur even when forward activations do not explode, identifying this as a fundamental challenge for constructing very deep trainable networks.Source: §1 · p. 4
  • The collapsing domain problem can arise in architectures that avoid exploding gradients and can be at least as damaging to training.Source: §1 · p. 4
  • Skip connections reduce gradients in general deep architectures, an effect the authors trace to the Pythagorean equation; they argue ResNets’ mathematical simplicity helps explain their performance at great depths.The paper presents the residual trick as a way to view arbitrary networks as residual networks.Source: §1 · pp. 3–4

2. Exploding gradients defined - the gradient scale coefficient

The section distinguishes exponential Jacobian growth from gradient pathology that makes training difficult, then defines the GSC to assess relative gradient growth. It also states the analysis’s scope and the metric’s rescaling invariance.

2.1 Notation and terminology

  • Nominal depth counts all layers, while compositional depth counts parametrized layers; the paper’s notation also tracks layer and parameter Jacobians along gradient flow.Source: §2.1 · pp. 4–5

2.2 The colloquial notion of exploding gradients

  • Exponential Jacobian growth alone does not establish training difficulty: layer and parameter rescaling can preserve predictions and trainability while producing arbitrarily growing Jacobians.The paper therefore seeks a definition of exploding gradients distinct from exponential Jacobian growth.Source: §2.2 · pp. 5–6

2.3 The gradient scale coefficient

  • The GSC combines a Jacobian norm with forward-activation lengths to measure relative backward-gradient size; the qm norm captures expected vector-length impact under uniformly random orientation.The authors define exploding gradients through exponential growth of the error GSC, while noting that no objective pathological threshold exists.Source: §2.3 · pp. 6–7

(See section D.5 for details.)

  • The analysis focuses on MLPs with fully connected linear layers and unparametrized layers, studies training error rather than generalization, and shows the GSC is invariant to prediction-preserving multiplicative rescaling.Source: (See section D.5 for details.) · p. 7

3. The prevalence of exploding gradients - gradients explode despite bounded activations

The authors find that several popular MLP architectures exhibit exploding gradients in the randomly initialized state, including architectures using normalization or SELU. Other tested architectures avoid this behavior but carry tradeoffs discussed in later sections.

3. The prevalence of exploding gradients - gradients explode despite bounded activations

  • Four of seven tested architectures—batch-ReLU, layer-tanh, batch-tanh, and SELU—show near-linear GSC growth in log-space, corresponding to gradient explosion.The experiments use networks with 50 linear layers and width 100; the plotted metrics are evaluated at random initialization.Source: §3 · pp. 8–9, Figure 1 · p. 8
  • Normalization can introduce exploding gradients in the tested comparisons: batch-ReLU versus ReLU, and layer-tanh or batch-tanh versus tanh.Source: §3 · p. 9
  • Controlling forward-activation size does not by itself reduce the GSC, which is invariant to multiplicative rescaling; techniques may still affect it by changing the network’s mathematical properties.Source: §3 · p. 9
  • ReLU, layer-ReLU, and tanh do not exhibit exploding gradients in these experiments, but the authors note this apparent advantage has costs explored later.Source: §3 · p. 9
  • Individual layers have small effects on the GSC, while linear layers tend to shrink gradients relative to activations and normalization layers can counterbalance that shrinkage.Source: §3 · p. 9
  • Increasing width slightly decreases the explosion rate, while oscillating layer widths leave approximate exponential GSC growth at a rate similar to constant-width counterparts.Source: §3 · pp. 9–10

4. The impact of exploding gradients - exploding gradients limit depth

The paper argues that exploding gradients can make effective depth prohibitively costly to train, and uses the residual-network framework to analyze that limit. Its experiments on exploding MLPs support the predicted depth limitations and show that batch-normalization noise can further impair training.

4.1 Notation and terminology - ResNet

  • In a ResNet, each block combines a fixed skip block with a residual block, and effective depth measures how many residual blocks can substantially co-adapt.The cited account relates limited effective depth to behavior resembling an ensemble of relatively shallow networks.Source: §4.1 · p. 10, §4.2 · p. 10

4.3 The residual trick

  • The residual trick expresses any network as blocks with fixed skip functions and residual functions, allowing ResNet analyses based on small residual Jacobians to apply to arbitrary networks.With skip functions set to each layer’s initial function, residual functions begin as zero functions.Source: §4.3 · p. 11

4.4 Notation and terminology - residual network

  • The paper distinguishes architectural ResNets from residual networks, which are arbitrary networks represented using skip and residual functions.This terminology separates the computational use of skip connections from the mathematical representation used for analysis.Source: §4.4 · p. 11

4.5 Theoretical analysis

  • For a network with explosion rate r, the theory bounds useful relative updates by the inverse GSC, so achieving larger effective depth requires exponentially many updates in depth.The lower bound is independent of nominal network depth when the theorem’s condition-dependent constants remain fixed.Source: §4.5 · p. 12
  • Under the theorem’s conditions, reaching effective depth λ in an MLP with explosion rate r and intercept c takes at least c′cr^λ⁴ gradient-based updates.If updates to convergence are bounded, effective depth is bounded as well.Source: §4.5 · p. 12

4.6 Experiments

  • Best-step-size experiments found that lower layers generally require smaller relative updates, and architectures with smaller updates tended to attain lower effective depth.The measured effective depth rose most sharply early in training, then changed little as error continued to fall.Source: §4.6 · pp. 14–15
  • Across the four 51-layer exploding architectures, lower gradient explosion tracks lower final error, while none reaches its compositional depth under the study’s effective-depth estimate.The experiments used batch-ReLU, layer-tanh, batch-tanh, and SELU on CIFAR10.Source: §4.6 · pp. 12, 15

4.7 A note on batch normalization and other sources of noise

  • Under minibatch training, batch-ReLU stayed above 89% training error; the authors attribute this to interaction between exploding gradients and batch-normalization noise.They note that techniques such as running averages can reduce batch normalization’s dependence on the current minibatch.Source: §4.7 · p. 16

5. The origins of exploding gradients - quadratic vs geometric means

The paper asks why exploding gradients arise even when forward activations are stable. It relates gradient scale to the quadratic mean of Jacobian singular values, contrasting it with the geometric-mean quantity associated with information propagation, and concludes that avoiding explosion can involve tradeoffs.

5. The origins of exploding gradients - quadratic vs geometric means

  • The paper attributes layer-wise GSC values above 1 in networks with stable forward activations to a mismatch between gradient scale and information propagation.The authors identify information loss or constrained Jacobians as strategies used by ReLU and ResNet, while noting practical drawbacks.Source: . (See section E.2 for details.) · p. 18
  • The GSC depends on the quadratic mean of absolute Jacobian singular values, whereas information propagation is governed by their geometric mean.The quadratic mean is at least the geometric mean, with the difference depending on how spread out the singular values are.Source: . (See section E.2 for details.) · p. 18

. (See section E.2 for details.)

  • The authors caution that avoiding exploding gradients in current practice involves tradeoffs, while leaving open architectures that might exploit conditions such as constant width or a probability density function.The theoretical analysis relies on several conditions, including constant width and the presence of a probability density function.Source: . (See section E.2 for details.) · pp. 18–19

6. Exploding gradient tradeoffs - the collapsing domain problem

Avoiding exploding gradients can introduce domain bias, creating a tension because both pathologies can hamper training. In ReLU and layer-ReLU networks, domain bias is associated with pseudo-linearity and low effective depth.

6. Exploding gradient tradeoffs - the collapsing domain problem

  • Domain bias contracts latent representations around a point, reducing Jacobian eigenvalues and the gradient scale coefficient when output length is held constant.The paper calls this contraction-induced activation bias “domain bias.”Source: §6 · p. 19
  • ReLU and layer-ReLU show increasing domain bias with depth, as pre-activation variation diverges from quadratic expectation and bias explains a growing fraction of the signal.The authors identify these as the two biased architectures.Source: §6 · p. 19
  • In ReLU networks, biased pre-activations can make nonlinearities approximate zero or identity functions, so sufficiently deep pseudo-linear networks lose nonlinear representational capacity.Pseudo-linearity takes hold substantially after layer 10 and completely after layer 25 in the two biased architectures.Source: §6 · pp. 19–20
  • Domain bias reduces useful update sizes and effective depth; ReLU and layer-ReLU achieve effective depths significantly below their compositional depths and comparable to exploding-gradient architectures.The authors also find that final CIFAR10 errors are substantially higher than those of exploding architectures, except for batch-ReLU.Source: §6 · pp. 20–21
  • Avoiding exploding gradients and collapsing domains involves an inherent tension because either pathology can severely hamper training.The authors identify the discrepancy between geometric and quadratic means of layer-wise Jacobian singular values as the source of this tension.Source: §6 · p. 21

7. Exploding gradient solutions - ResNet and the orthogonal initial state

On CIFAR10, ResNets generally improve relative update sizes, error, and effective depth over corresponding vanilla networks. The paper attributes their gradient reduction to dilution and orthogonality, while noting limits to how much dilution can be used.

7. Exploding gradient solutions - ResNet and the orthogonal initial state

  • K-dilution reduces GSC growth beyond the dilution factor: the reduction is by k^2 + 1, which the authors connect to ResNets’ ability to train at great depths.Theorem 4 attributes the effect to adding an uncorrelated linear transformation; with orthogonality, the Pythagorean theorem reduces gradient growth.Source: §7 · p. 23
  • In experiments, skip-connected ResNets show lower gradient growth than corresponding vanilla architectures, and layer-ReLU ResNets show slower growth of domain bias.The measured gradient reduction is checked against the rate predicted by theorem 4.Source: §7 · p. 23
  • On CIFAR10, ResNets generally achieve higher relative update sizes, lower error, and higher effective depth than corresponding vanilla networks, with layer-SELU the stated exception.The gradient reduction persists throughout training.Source: §7 · p. 25

7.1 The limits of dilution

  • Dilution has a limit: large k makes a function close to linear, and under theorem 5’s conditions, removing dilution can require O(k^2) blocks.The authors conclude that gradient reduction achievable without catastrophic pseudo-linearity is limited.Source: §7.1 · p. 26, (See section E.5 for details.) · p. 26

7.2 Choosing dilution levels

  • Dilution levels are controllable: scaling skip and residual blocks can set desired levels, and the authors argue users need not increase dilution with depth.Moving normalization layers between blocks is another proposed way to keep dilution constant.Source: §7.2 · pp. 26–27

7.3 On the relationship of dilution, linear approximation error and the standard deviation of absolute Jacobian eigenvalues

  • The authors relate dilution to linear approximation error and Jacobian singular values, reporting that tanh avoids exploding gradients through extreme pseudo-linearity but performs poorly.They describe dilution as squared in its effect on signal-to-noise ratio and connect it to reductions in gradient growth.Source: §7.3 · p. 27

7.4 The orthogonal initial state

  • The residual trick frames arbitrary networks as residual networks; ResNets are mathematically simpler because their initial functions are closer to orthogonal transformations, and approximate orthogonal initial states can reduce gradients.The paper also reports that looks-linear initialization (LLI) applied to ReLU outperforms ResNet in its CIFAR10 experiments; an orthogonal initial state alone is not enough without sufficiently nonlinear residual functions.Source: §7.4 · pp. 27–28, §7.5 · p. 28

7.5 The power of initial vs residual dilution

  • Initial-only dilution schemes perform better than leaky-ReLU schemes that dilute both initial and residual functions, but tools such as skip connections and LLI do not resolve the pathology tension.The authors argue that non-orthogonality in initial functions requires explicit justification.Source: §7.5 · pp. 28–29

8. Related work

The related work situates exploding gradients among other measures and analyses of deep-network trainability. The authors distinguish their general analysis from prior architecture-specific results and question whether some proposed stability strategies add benefit beyond dilution.

8. Related work

  • Mean-field studies analyze infinitely wide and deep networks at random initialization, distinguishing order and chaos regimes associated with gradient vanishing or explosion under specified conditions.The authors say their work differs from and extends this line of research.Source: §8 · p. 30
  • Prior dynamical-systems work derives ResNet stability strategies; the authors question whether these offer significant additional gradient-growth reduction beyond dilution.They cite dilution’s capacity to suppress gradient growth and its slow disappearance with depth.Source: §8 · p. 30
  • The paper relates explosion and domain bias to gradient shattering, while distinguishing gradient growth relative to forward activations from growth caused by activation scale.It also notes that divergence of Jacobian singular values can make gradient direction depend on dominant eigenvectors rather than labels, slowing training.Source: §8 · pp. 30–31
  • The authors position their work as extending Balduzzi et al. (2017): they challenge the claim that exploding gradients were largely overcome, give a general training-difficulty argument using effective depth, and analyze beyond ReLU.They focus particularly on very deep batch-ReLU MLPs examined by that work.Source: §8 · p. 30
  • The authors question the motivation for treating covariate shift as pathological, arguing that layerwise changes and co-adaptation are part of how deep networks train.They note that they know of no rigorous definition of covariate shift or explanation of why it is undesirable.Source: §8 · p. 31

9. Conclusion

The paper argues that MLPs face a tradeoff among exploding gradients, collapsing domains, and pseudo-linearity, while more orthogonal initial functions, including ResNet-like architectures, can reduce these pathologies. Its recommendations emphasize monitoring gradient scale and effective depth, controlling dilution, and avoiding the assumption that greater compositional depth is always better.

9. Conclusion

  • Many MLPs exhibit exploding gradients, while alternatives can exhibit collapsing domains or extreme pseudo-linearity; both pathologies can lower effective depth.The paper attributes the tradeoff to a discrepancy between geometric and quadratic means of layer-wise Jacobian singular values.Source: §9 · p. 31

Practical Recommendations

  • Orthogonal initial functions, including ResNet-like architectures, can reduce GSC growth and domain collapse during training.The paper says skip connections help in general, while normalization, initialization, and optimizer changes do not generally address GSC-relative gradient explosion or collapsing domains.Source: Practical Recommendations · pp. 32–33
  • Extreme compositional depth may be suboptimal: preventing gradient explosion can require dilution that reduces effective depth and makes layers nearly linear.With a fixed parameter budget, the paper notes that width may be a better investment than extreme depth.Source: Practical Recommendations · p. 33

Implications for deep learning research

  • Exploding gradients indicate an inherently difficult optimization problem, rather than a numerical quirk that simple gradient rescaling can solve.The paper states that these gradients limit effective depth, making very deep MLPs effectively shallower.Source: §9 · p. 31, Implications for deep learning research · p. 33
  • GSC is presented as a robust benchmark for gradient explosion, with a rigorous link to training difficulty and robustness to rescaling, layer width, and individual layers.The paper also recommends adjusting step sizes as GSC grows and controlling dilution to trade off gradient growth and representational capacity.Source: Implications for deep learning research · p. 33, Practical Recommendations · p. 33
  • The residual trick treats any neural network as a residual network, allowing ResNet-specific tools such as effective-depth theory to apply to arbitrary networks.The paper argues that skip connections simplify the network mathematically.Source: Implications for deep learning research · p. 33
  • Different layerwise step sizes substantially affect training success and can change conclusions about deep-network behavior.The paper warns that studies omitting layerwise or global step-size comparisons may reach inaccurate conclusions.Source: Implications for deep learning research · p. 33

A. Further discussion

The discussion contrasts the paper's GSC-based account of gradient pathology with mean-field analyses based on activation correlations. It argues that gradient and domain pathologies harm training, and that layerwise step sizes and general ResNet structure matter to the resulting comparisons.

A.1 Mean field theory - exploding gradients / collapsing domain vs order / chaos

  • The GSC yields different explosion, vanishing, and stability regions than the quadratic mean of gradient entries, including stability at every weight scale for constant-width, bias-free ReLU MLPs.For these networks, forward-activation correlations converge sub-exponentially at all weight scales, matching GSC stability.Source: §A.1 · p. 38
  • Width variation can reduce the quadratic mean of gradient entries, but the authors' analysis suggests it does not reduce GSC growth effectively.They note that the training-difficulty connection they establish concerns an exploding GSC, not an exploding quadratic mean.Source: §A.1 · p. 38
  • Gradient explosion and domain bias can directly harm training, extending mean-field work that primarily relates gradient behavior to activation-correlation regimes.The cited mean-field work associates order with vanishing gradients and, for tanh MLPs, chaos with explosion when forward activation lengths converge.Source: §A.1 · p. 38
  • Allowing layerwise step sizes changes architecture comparisons: SELU and layer-tanh performed best among the reported vanilla architectures, while edge-of-chaos ReLU, layer-ReLU, and tanh had higher training classification error.The authors contrast this with prior experiments favoring edge-of-chaos architectures and attribute the difference to allowing step sizes to vary by layer.Source: §A.1 · p. 39
  • The authors extend prior findings about reduced pathologies in specific ResNets to general ResNet architectures.Their stated contribution concerns the generality of the effect, rather than a particular GSC reduction level.Source: §A.1 · p. 39

A.2 ResNet from a dynamical systems view

This discussion reports how initialization and residual structure affect gradient growth, while distinguishing the paper's MLP analysis from open questions about other architectures and domain collapse. It also describes formal and computational approaches for defining and estimating effective depth.

A.2 ResNet from a dynamical systems view

  • Skew-symmetric initialization slightly reduces gradients, while −CTC initialization grows them faster and correlated successive blocks weaken dilution; scaling residual blocks by 1/√B is reported sufficient to bound GSC at arbitrary depth.The authors did not test the exact architectures proposed in related dynamical-systems work and call for further evaluation.Source: §A.2 · p. 40

A.3 Exploding and vanishing gradients in RNNs

  • Feedforward networks differ from RNNs because each layer is meant to modify its signal, whereas LSTM and GRU gates can leave latent-state neurons unchanged across time steps.The authors therefore describe managing exploding gradients in feedforward networks as arguably more difficult.Source: §A.3 · pp. 40–41

A.4 Open research questions and future work

  • The analysis and experiments focus on MLPs, and the authors expect orthogonal initialization's effective-depth gains to be limited as residual functions grow and dilution falls.They identify extending principled analysis to biases, convolutional layers, and recurrent layers as future work.Source: §A.4 · pp. 41–42

B. Further terminology, notation and conventions

  • The notation section assumes differentiable networks for results in sections D and E, while defining Gaussian and orthogonal initialization conventions.The stated extensions include almost-surely differentiable and directionally differentiable networks such as SELU and ReLU.Source: B. Further terminology, notation and… · p. 43

C.1 Formal definition

  • Effective depth is defined through the largest residual-Jacobian count whose removed-gradient contribution exceeds threshold h, plus one for the co-adapting residual function.The paper defines λ-contribution by comparing parameter trajectories with and without λ-residual gradient terms.Source: §C.1 · pp. 43–44

C.2 Computational estimate

  • The computational estimate assumes layer Jacobians increase terms approximately equally and sums estimated λ-contributions across batches and updates; the authors expect it to overestimate actual effective depth.The estimation algorithm assumes SGD with one global or custom layerwise step sizes.Source: §C.2 · p. 44, §13 · p. 45

13 end

  • For ResNets, the estimator writes each layer and residual block as an initial function plus a residual function, then modifies its accumulated contribution as it crosses a skip block.The skip connection and initial functions are approximated by the whole-block effect, while residual-function effects are bounded separately.Source: §13 · p. 45, §12 · p. 46

12 end

  • The ResNet estimate adjusts the accumulated λ-contribution at block crossings to account for both the skip path and the residual path.The modification is nested within the vanilla-network estimation algorithm.Source: §13 · p. 45, §12 · p. 46

C.3 Discussion

The discussion identifies cases where effective depth can misrepresent a network’s functional depth or training behavior. Its estimate is also sensitive to choices such as initial step size and layer definitions.

C.3 Discussion

  • A linear MLP can have effective depth much greater than 1 while remaining equivalent to a depth-1 network.Thus, effective depth alone does not always reflect functional depth.Source: §C.3 · p. 46
  • A weight-resampling step can significantly increase effective depth even when the resulting algorithm is equivalent to gradient descent.The measure can rise without a meaningful change to the training procedure.Source: §C.3 · p. 46
  • Unnecessarily large initial step sizes can worsen outcomes even if later reduced, while still yielding a much higher effective depth.The measure is susceptible to the initial step size.Source: §C.3 · p. 46
  • Effective depth can change when layers are redefined, although the network function and gradient-based training course remain unchanged.For example, a ReLU operation may be grouped with its preceding linear transformation.Source: §C.3 · p. 46

D.1 Proposition 1

The appendix gives a proposition showing that networks with the same predictions and error can exhibit arbitrarily different Jacobian growth. It also relates the quadratic mean norm to random-vector effects and describes when GSC is invariant or decomposes across layers.

D.1 Proposition 1

  • Under the proposition’s conditions, a network can be transformed to have arbitrarily large Jacobian growth while preserving its depth, training outcome, and predictions.The transformed network can be trained in the same number of updates with identical computation when the scaling transformation is applied; the stated Jacobian result excludes one index case.Source: §D.1 · pp. 47, 49

D.3 Proposition 3

  • For an m × n matrix, the qm norm is the matrix L2 norm divided by √n.Source: §D.3 · p. 50

D.5 Proposition 5

  • Radially symmetric parameter initialization includes Gaussian and orthogonal weight-matrix initialization, covering two popular initialization strategies.Source: §D.5 · p. 54

D.6 Proposition 6

  • GSC is invariant under multiplicative rescalings that preserve network predictions and error, but this does not cover every practical scale-control transformation.Weight scaling in ReLU networks is covered when rescaling factors multiply to 1; adding normalization layers or rescaling tanh or SELU networks without normalization is not covered.Source: §D.6 · p. 55

D.7 Proposition 7

  • Assuming the relevant Jacobian norms, GSC across multiple layers is approximately the product of the adjacent-layer GSC values.Source: §D.7 · p. 55

E.1 Theorem 1 - exploding gradients limit depth

Theorem 1 relates exploding gradients to limits on effective depth under stated assumptions about training updates and residual terms. The paper notes that its training-time bound is too loose to be practically informative.

E.1 Theorem 1 - exploding gradients limit depth

  • Theorem 1 bounds the first-order approximation of the relative λ-contribution at layer lN over T updates by h.Making h small is sufficient for the paper’s effective-depth estimate; the bound on T is exponential in λ and independent of L and N.Source: §E.1 · pp. 60–61
  • The theorem assumes exploding gradients, bounded updates, and that large singular values in layer-wise Jacobians do not compound disproportionately in λ-residual terms.The authors describe this residual-term condition as the strongest condition and note it tolerates all such terms having the same orientation.Source: §E.1 · p. 60
  • The algorithm condition is not satisfied theoretically by RMSprop or Adam, although denominator ϵ regularization prevents arbitrary gradient scaling in practice.Source: §E.1 · pp. 60–61
  • The theorem’s training-time bound is too loose for practical use and would imply trainable depths greater than those observed in practice.Source: §E.1 · p. 61

E.2 Theorem 2 - why gradients compound

Theorem 2 supports a layerwise decomposition of GSC under specified assumptions, with an approximate extension to practical MLPs. The argument uses normalized activation lengths and orthogonality properties to control how gradients propagate.

E.2 Theorem 2 - why gradients compound

  • In the constructed layer-normalized MLP, LOlayer operations make input length immaterial and enforce constant output length and orthogonality conditions.With Gaussian or orthogonal initialization, the passage says the resulting network fulfills the theorem’s conditions.Source: §E.2 · p. 64
  • For practical MLPs, approximately equal activation lengths across inputs allow a pseudo-LOlayer approximation under which GSC decomposes approximately across layers.The passage attributes the similar activation lengths to the law of large numbers.Source: §E.2 · pp. 64–65
  • The qm norm’s denominator should reflect the input space’s intrinsic dimensionality, which in this theorem is one less than the number of matrix columns.Source: §E.2 · p. 65

E.3 Theorem 3 - why gradients explode

The theorem presents two results using H and E to bound gradient-related quantities, with different strengths and assumptions. Its scope is limited to MLPs preserving nominal and intrinsic dimensionality.

E.3 Theorem 3 - why gradients explode

  • The H-based result is more widely interpretable and does not require independence, while the E-based result is tighter by avoiding Jensen’s inequality.The E-based bound remains meaningful when a small region has zero singular values, unlike the H-based bound.Source: §E.3 · p. 66
  • The entropy-based result uses independence between the input density and the Jacobian, whereas the stated H-based result does not require that condition.The text describes obtaining this independence by replacing E_X with E_X,θ under specified hypersphere and random-orientation conditions.Source: §E.3 · p. 66
  • The theorem’s stated results apply to MLPs whose layer functions preserve both nominal and intrinsic dimensionality.The theorem can be extended to layer functions that change nominal dimensionality while preserving intrinsic dimensionality.Source: §E.3 · p. 66

E.4 Theorem 4 - skip connections reduce the gradient

The appendix analyzes skip connections as a way to reduce gradient growth, then extends dilution results to compositions and describes Taylor approximations and looks-linear initialization. The results rely on stated assumptions that may hold only approximately in practical networks.

E.4 Theorem 4 - skip connections reduce the gradient

  • Theorem 4 relates gradient growth before and after a skip connection, with practical MLP ResNets expected to show similar reductions when its conditions hold approximately.The analysis assumes uncorrelated residual and skip functions and gradients; this assumption is not fully guaranteed for backward evaluations.Source: §E.4 · pp. 66–68

E.5 Theorem 5 - how dilution compounds

  • Theorem 5 states that composing functions diluted in expectation yields a composition that is itself diluted in expectation.The result assumes independent components, specified forms for the skip matrices, and scale-symmetric decomposable residual functions; one condition is generally only approximate in practice.Source: §E.5 · pp. 69, 72

F. Taylor approximation of a neural network

G. Looks-linear initialization

  • Looks-linear initialization groups neighboring ReLU neurons in pairs so they effectively compute the identity, yielding an approximately orthogonal initial state.The stated construction requires even-dimensional ReLU layers.Source: G. Looks-linear initialization · pp. 73–74

H. Experimental details

The experiments span static Gaussian-noise measurements, CIFAR10 weight-space color maps, and CIFAR10 training, with protocols specifying architectures and step-size selection. They compare scaling and clipping variants by end-of-training error and describe joint step-size scaling based on iterative trials.

H.1 Architectures used

  • Vanilla models are fully connected MLPs, while ResNets use 25 six-layer or 50 three-layer blocks; skip connections are identity or Gaussian-projection, with an orthogonal projection for CIFAR10 width reduction.Source: §H.1 · pp. 74, 76

H.2 Protocol for Gaussian noise experiments

  • Gaussian-noise experiments evaluated untrained networks on 100 datasets of 10,000 points per architecture and measured regular gradients, sampling ResNet gradients between blocks.Source: §H.2 · p. 77

H.3 Protocol for color map experiments

  • Color-map experiments varied one layer’s weights over a sphere, propagated a CIFAR10 image through each configuration, and colored the weight sphere by the normalized three-output prediction.Output colors encode which of the three neurons is largest; purer colors correspond to more confident predictions under a hypothetical three-class softmax.Source: §H.3 · p. 78

H.4 Protocol for CIFAR10 experiments

  • CIFAR10 training used standardized training data and SGD with shared or layer-specific step sizes; shared-step runs selected the best of 21 starts under a 5-epoch, 3x decay schedule.Training stopped after 500 epochs or 11 step-size reductions, whichever came first; the selected starting step size minimized final training classification error.Source: §H.4 · p. 78

H.4.1 Selecting custom step sizes

  • Custom step sizes were estimated by pretraining selected layers, then choosing each layer’s one-epoch step size under relative-update limits of 0.01 for exploding architectures and 0.001 otherwise.Pretraining reduced noisy, spurious estimates; the selection stage froze other layers and rejected step sizes causing relative updates of 0.1 or higher.Source: §H.4.1 · pp. 79–80
  • Optional clipping stopped the step-size search when error rose by 0.1%, while log-scale regression smoothing adjusted layerwise step sizes to reduce noise.Clipping was used only when it improved end-of-training error; smoothing replaced noisy relative update sizes with their best linear fit.Source: §H.4.1 · p. 80
  • The scaling search increased a shared layer-wise step-size constant by 1.5 after each one-epoch trial and stopped when any layer’s relative update reached 0.1.Each trial’s epoch was rewound before the next trial.Source: §H.4.1 · pp. 80–81
  • Joint parameter updates were approximately 51 times larger than single-layer selection updates, so a scaling constant below 1 was usually needed; clipping was used for four named architectures.Clipping applied to vanilla tanh, vanilla layer-tanh, ResNet layer-tanh, and looks-linear layer-ReLU.Source: §H.4.1 · p. 81
  • For each architecture, three completed networks—with no clipping, scaling-stage clipping, or clipping in both stages—were compared, and the lowest-error network was presented.For single-step-size training, 21 end-of-training error values were compared.Source: §H.4.1 · p. 81
Loading 1712.05577v4…