PaperlayerView exact revision on arXiv

Source-linked AI summary

The Shattered Gradients Problem: If resnets are the answer, then what is the question?

David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, Brian McWilliams

arXiv:1702.08591v2cs.NEcs.LGstat.ML

TL;DR

Deep learning had largely addressed vanishing and exploding gradients, yet deep feedforward networks remained inferior to resnets. This paper identifies shattered gradients, analyzes their depth-dependent correlations, and introduces LL-init. It finds exponential correlation decay in feedforward networks, slower decay with skip-connections, and preliminary evidence that LL-init enables very deep networks without skips.

  • Problem

    Despite effective initialization and batch normalization, sufficiently deep standard architectures underperform resnets, revealing an unresolved gradient problem distinct from vanishing and exploding gradients.

    Source: §1 · p. 1

  • Method

    The paper analyzes gradient correlations in a minimal rectifier-network model, supports the analysis with fully connected and convolutional experiments, and proposes the LL-init initialization.

    Source: §1.1 · p. 1

  • Results

    Gradient correlations decay exponentially with depth in feedforward networks but sublinearly in resnets, while LL-init prevents shattering in preliminary experiments.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Shattered gradients help explain why deep feedforward networks are difficult to train despite effective initialization and batch normalization, while skip-connections and LL-init preserve gradient structure.

    Source: §6 · p. 8

    The analysis applies at initialization, and understanding how shattering changes during training remains an open problem.

    Source: §1 · p. 1

Abstract

from arXiv · show

A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new "looks linear" (LL) initialization that prevents shattering, with preliminary experiments showing the new initialization allows to train very deep networks without the addition of skip-connections.

1. Introduction

The paper identifies shattered gradients as a depth-related problem distinct from vanishing and exploding gradients: correlations decay until deep feedforward gradients resemble white noise. Skip-connections reduce this shattering, while LL-init offers an alternative initialization-based solution.

Motivation

  • Careful initialization and batch normalization address vanishing and exploding gradients, yet sufficiently deep resnets still outperform standard architectures.The paper argues that this gap motivates a distinct gradient problem.Source: §1 · p. 1

The shattered gradients problem

  • Shattered gradients become increasingly uncorrelated with depth, progressively erasing the spatial structure needed by optimization methods.This can make minibatch averaging resemble integration over white noise and make nearby inputs produce unstable gradients.Source: §1 · p. 1, §1.1 · pp. 2–3
  • In a minimal scalar-to-scalar network with 200 rectifier neurons per hidden layer, the authors isolate gradients for analysis and visualization.The laboratory model uses inputs on a one-dimensional grid to expose gradient structure.Source: §1.1 · pp. 1–2

Empirical and theoretical observations

  • Shallow-network gradients resemble brown noise, whereas deep feedforward gradients resemble white noise; Donsker’s theorem explains the shallow-network limit.The observed transition is quantified with autocorrelation functions, whose decay is theoretically exponential in feedforward networks.Source: §1.1 · p. 2

Skip-connections

  • Resnet gradients retain more correlation than equivalent feedforward gradients, and β-rescaling makes even a 50-layer network’s ACF resemble brown noise.The analysis attributes this preservation to skip-connections combined with suitable rescaling.Source: §1.1 · p. 3

Alternative initialization

  • Preliminary experiments show LL-init permits training networks of approximately 200 layers without skip-connections, with performance comparable to a resnet.The paper also reports empirical shattering in convnets for real data and reduced minibatch-average gradients with depth.Source: §1.2 · p. 3

2. Observations on batch normalization

Batch normalization improves the efficiency of rectifier nonlinearities by balancing activation patterns, but in feedforward networks it also destroys spatial structure that resnets partly preserve.

Activation statistics

  • With batch normalization, neurons are active for half their inputs on average and are co-active for 1/4 of distinct input pairs.The coactivation rate matches unbiased coin-flip behavior.Source: §2 · p. 4
  • Without batch normalization, co-active proportions increase with depth, suggesting increasingly redundant neuronal responses.Resnets with batch normalization show the same activation and coactivation behavior as feedforward networks in this comparison.Source: §2 · p. 4

Activation efficiency

  • By depth 50, feedforward neurons without batch normalization are often always active or always inactive, whereas batch normalization makes most active for half the inputs.Always-active neurons behave linearly, while always-inactive neurons contribute nothing, so batch normalization improves rectifier efficiency.Source: §2 · p. 4

Trade-off

  • Batch normalization increases nonlinear efficiency at the cost of spatial structure in feedforward networks, while resnets preserve some structure through deep layers.Contiguity histograms show broad activation runs in resnets but fragmented sections in batch-normalized feedforward nets.Source: §2 · p. 4

3. Analysis

The analysis characterizes shattered gradients by studying gradient correlations across datapoints and depth, showing how architecture and rescaling affect variance and whitening.

Feedforward networks

  • Gradient correlations in feedforward rectifier networks decay exponentially with depth as 1/2^L, even when initialization preserves gradient variance.The analysis uses a path-weight basis to reduce covariance calculations to counting co-active paths and rescaling factors.Source: §3 · p. 4, §3.1 · p. 5
  • Feedforward gradients whiten because the number of paths grows exponentially faster with depth than the fraction of co-active paths.The analysis assumes typical datapoints activate half the neurons and share one quarter of activations per layer, though this does not hold for every pair.Source: §3 · p. 4, §3.1 · p. 5

Residual networks

  • Unnormalized resnets have gradient variance growing as 2^L, while gradient correlation still decays exponentially with depth.The result identifies exploding variance and exponentially decaying correlation as separate problems in resnets without batch normalization.Source: §3.2 · pp. 5–6

Rescaling in Resnets

  • Batch normalization changes correlation decay from 1/2^L to 1/√L in resnets, and β-rescaling further slows whitening.For a 100-layer resnet with β = 0.1, the model predicts typical correlation 0.1, versus correlation indistinguishable from zero in a feedforward net.Source: §3.3 · p. 6

Highway networks

  • Modified highway networks with scalar gates can achieve a depth-independent gradient-correlation structure, although contributions increasingly approach the identity as depth grows.The highway analysis studies γ-rescaling and derives variance and correlation behavior under its assumptions.Source: §3.4 · p. 6

4. Gradients shatter in convnets

Experiments on CIFAR-10 convnets show that gradient structure degrades faster with depth in feedforward networks than in resnets. Relative effective rank grows faster and average gradient norms decay more rapidly without skip-connections, while resnet gradients remain more structured.

Experimental setup

  • CIFAR-10 convnet experiments compare equivalent feedforward and resnet architectures at depths 2, 4, 10, 24, and 50, with batch normalization in both.The resnets use one skip-connection for every two convolutional layers.Source: §4 · p. 6

Measurements

  • Figure 5 measures gradient covariance, relative effective rank, and average gradient norms across minibatches.Covariance matrices use a single minibatch, while relative effective rank and average norms are averaged over 30 minibatches.Source: §4 · pp. 6–7, Figure 5 · p. 7

Gradient structure

  • Relative effective rank grows much faster with depth in networks without skip-connections than in resnets.For resnets, β slows the effective-rank growth as predicted by theorem 3.Source: §4 · p. 7

Gradient averaging

  • Average coordinatewise gradient norms decay much more rapidly with depth in feedforward networks because increasingly whitened minibatch gradients average out their signal.The effect is much less pronounced in resnets.Source: §4 · p. 7

Conclusion

  • Overall, resnet gradients remain more structured than feedforward gradients and therefore do not vanish when averaged within a minibatch.The results extend the earlier analysis to networks with convolutional layers.Source: §4 · p. 7

5. The “looks linear” initialization

The looks-linear initialization combines rectifier networks with a linear-like initialization intended to prevent gradient shattering. Preliminary CIFAR-10 experiments found that it enabled substantially deeper feedforward networks, with performance comparable to a resnet at the maximum tested depth.

Motivation

  • Linear networks avoid shattered gradients but lack the expressivity needed to be useful, motivating a rectifier initialization that looks linear.Source: §5 · p. 7

Construction

  • LL-init initializes rectifiers with mirrored block structure so their outputs are linear at initialization.The paper implements this idea with concatenated rectifiers, or CReLUs.Source: §5 · p. 7
  • The initialized output ceases to be linear when weight updates cause the two blocks to diverge.Source: §5 · p. 7

Alternative

  • A PReLU-based alternative sets its parameter a = 1 at initialization, but preliminary experiments suggest the CReLU-based LL-init is more effective.The reason for this difference is unclear.Source: §5 · p. 7

Experiments

  • The LL-init was evaluated with orthogonal weights in very deep CIFAR-10 networks to test whether preventing shattering improves training.The experiment was designed to compare concatenated rectifier networks that were and were not shattered at initialization.Source: §5 · pp. 7–8, §5.1 · p. 8

Results

  • At maximum depth 198, the standard ReLU network performed below the 40% linear baseline, whereas feedforward LL-init performed comparably to a resnet.Each plotted point is the mean of 10 trained models; standard ReLU and CReLU performance worsened steadily with depth.Source: §5.1 · p. 8

6. Conclusion

The paper characterizes shattered gradients as increasingly uncorrelated gradients in randomly initialized deep networks and identifies skip-connections and LL-init as approaches that reduce or prevent shattering. It also notes unresolved questions about training dynamics and possible hybrid architectures.

Background

  • Rectifier networks gain exponentially many linear regions with depth, but this also proliferates discontinuities in their gradients.Source: §6 · p. 8

Shattered gradients

  • In randomly initialized deep networks, gradients in deeper layers become increasingly uncorrelated, creating a previously unnoticed problem distinct from vanishing and exploding gradients.The paper links shattering to difficulties faced by optimization methods and minibatch gradient averaging.Source: §6 · p. 8

Remedies

  • Skip-connections with suitable rescaling change correlation decay from exponential in feedforward architectures to sublinear in resnets.Source: §6 · p. 8
  • LL-init is presented as an alternative initialization that does not shatter gradients, while combining it with skip-connections is proposed as a future direction.Source: §6 · p. 8

A1. Backprop and Brownian Motion

The appendix connects gradients in a minimal rectifier network to random walks and Brownian motion. Under Glorot-style initialization, the suitably reparameterized derivative converges weakly to Brownian motion as width grows.

Setup

Brownian motion

Donsker’s theorem

Proposition

Derivation

  • As the input changes, the derivative becomes a random walk whose jumps occur at Gaussian-sampled points after neurons are ordered by their biases and parameterized by the active-neuron fraction.Source: A1. Backprop and Brownian Motion · p. 10

A2. The Karhunen-Loeve theorem

The Karhunen-Loeve theorem represents a centered stochastic process using eigenfunctions of its covariance operator. The resulting coefficients are uncorrelated and have variances given by the corresponding eigenvalues.

A3. Details of the Analysis

The analysis treats neural-network outputs and gradients as random processes indexed by datapoints. It then uses path-weights as a covariance-orthogonal basis, paralleling Fourier-style expansions.

Neural functional analysis

Path-weight basis

A3.1. Covariance structure of path-sums

The analysis models gradients as sums of active path-weights and computes their covariance by exploiting path orthogonality and counting co-active paths. It extends this framework to residual paths, batch normalization, and depth-dependent variance behavior.

Feedforward path-sums

Scope of the analysis

Residual path-sums

Depth and normalization

Related papers

Loading 1702.08591v2…