PaperlayerView exact revision on arXiv

Source-linked AI summary

The Shattered Gradients Problem: If resnets are the answer, then what is the question?

David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, Brian McWilliams

arXiv:1702.08591v2cs.NEcs.LGstat.ML

TL;DR

Deep learning had largely addressed vanishing and exploding gradients, yet deep feedforward networks remained inferior to resnets. This paper identifies shattered gradients, analyzes their depth-dependent correlations, and introduces LL-init. It finds exponential correlation decay in feedforward networks, slower decay with skip-connections, and preliminary evidence that LL-init enables very deep networks without skips.

  • Problem

    Despite effective initialization and batch normalization, sufficiently deep standard architectures underperform resnets, revealing an unresolved gradient problem distinct from vanishing and exploding gradients.

  • Method

    The paper analyzes gradient correlations in a minimal rectifier-network model, supports the analysis with fully connected and convolutional experiments, and proposes the LL-init initialization.

  • Results

    Gradient correlations decay exponentially with depth in feedforward networks but sublinearly in resnets, while LL-init prevents shattering in preliminary experiments.

  • Takeaways & Limitations

    Shattered gradients help explain why deep feedforward networks are difficult to train despite effective initialization and batch normalization, while skip-connections and LL-init preserve gradient structure.

  • Takeaways & Limitations

    The analysis applies at initialization, and understanding how shattering changes during training remains an open problem.

Abstract

from arXiv · show

A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new "looks linear" (LL) initialization that prevents shattering, with preliminary experiments showing the new initialization allows to train very deep networks without the addition of skip-connections.

1. Introduction

The paper identifies shattered gradients as a depth-related problem distinct from vanishing and exploding gradients: correlations decay until deep feedforward gradients resemble white noise. Skip-connections reduce this shattering, while LL-init offers an alternative initialization-based solution.

  • Motivation: Careful initialization and batch normalization address vanishing and exploding gradients, yet sufficiently deep resnets still outperform standard architectures.The paper argues that this gap motivates a distinct gradient problem.
  • The shattered gradients problem: Shattered gradients become increasingly uncorrelated with depth, progressively erasing the spatial structure needed by optimization methods.This can make minibatch averaging resemble integration over white noise and make nearby inputs produce unstable gradients.
  • The shattered gradients problem: In a minimal scalar-to-scalar network with 200 rectifier neurons per hidden layer, the authors isolate gradients for analysis and visualization.The laboratory model uses inputs on a one-dimensional grid to expose gradient structure.
  • Empirical and theoretical observations: Shallow-network gradients resemble brown noise, whereas deep feedforward gradients resemble white noise; Donsker’s theorem explains the shallow-network limit.The observed transition is quantified with autocorrelation functions, whose decay is theoretically exponential in feedforward networks.
  • Skip-connections: Resnet gradients retain more correlation than equivalent feedforward gradients, and β-rescaling makes even a 50-layer network’s ACF resemble brown noise.The analysis attributes this preservation to skip-connections combined with suitable rescaling.
  • Alternative initialization: Preliminary experiments show LL-init permits training networks of approximately 200 layers without skip-connections, with performance comparable to a resnet.The paper also reports empirical shattering in convnets for real data and reduced minibatch-average gradients with depth.

2. Observations on batch normalization

Batch normalization improves the efficiency of rectifier nonlinearities by balancing activation patterns, but in feedforward networks it also destroys spatial structure that resnets partly preserve.

  • Activation statistics: With batch normalization, neurons are active for half their inputs on average and are co-active for 1/4 of distinct input pairs.The coactivation rate matches unbiased coin-flip behavior.
  • Activation statistics: Without batch normalization, co-active proportions increase with depth, suggesting increasingly redundant neuronal responses.Resnets with batch normalization show the same activation and coactivation behavior as feedforward networks in this comparison.
  • Activation efficiency: By depth 50, feedforward neurons without batch normalization are often always active or always inactive, whereas batch normalization makes most active for half the inputs.Always-active neurons behave linearly, while always-inactive neurons contribute nothing, so batch normalization improves rectifier efficiency.
  • Trade-off: Batch normalization increases nonlinear efficiency at the cost of spatial structure in feedforward networks, while resnets preserve some structure through deep layers.Contiguity histograms show broad activation runs in resnets but fragmented sections in batch-normalized feedforward nets.

3. Analysis

The analysis characterizes shattered gradients by studying gradient correlations across datapoints and depth, showing how architecture and rescaling affect variance and whitening.

  • Feedforward networks: Gradient correlations in feedforward rectifier networks decay exponentially with depth as 1/2^L, even when initialization preserves gradient variance.The analysis uses a path-weight basis to reduce covariance calculations to counting co-active paths and rescaling factors.
  • Feedforward networks: Feedforward gradients whiten because the number of paths grows exponentially faster with depth than the fraction of co-active paths.The analysis assumes typical datapoints activate half the neurons and share one quarter of activations per layer, though this does not hold for every pair.
  • Residual networks: Unnormalized resnets have gradient variance growing as 2^L, while gradient correlation still decays exponentially with depth.The result identifies exploding variance and exponentially decaying correlation as separate problems in resnets without batch normalization.
  • Rescaling in Resnets: Batch normalization changes correlation decay from 1/2^L to 1/√L in resnets, and β-rescaling further slows whitening.For a 100-layer resnet with β = 0.1, the model predicts typical correlation 0.1, versus correlation indistinguishable from zero in a feedforward net.
  • Highway networks: Modified highway networks with scalar gates can achieve a depth-independent gradient-correlation structure, although contributions increasingly approach the identity as depth grows.The highway analysis studies γ-rescaling and derives variance and correlation behavior under its assumptions.

4. Gradients shatter in convnets

Experiments on CIFAR-10 convnets show that gradient structure degrades faster with depth in feedforward networks than in resnets. Relative effective rank grows faster and average gradient norms decay more rapidly without skip-connections, while resnet gradients remain more structured.

  • Experimental setup: CIFAR-10 convnet experiments compare equivalent feedforward and resnet architectures at depths 2, 4, 10, 24, and 50, with batch normalization in both.The resnets use one skip-connection for every two convolutional layers.
  • Measurements: Figure 5 measures gradient covariance, relative effective rank, and average gradient norms across minibatches.Covariance matrices use a single minibatch, while relative effective rank and average norms are averaged over 30 minibatches.
  • Gradient structure: Relative effective rank grows much faster with depth in networks without skip-connections than in resnets.For resnets, β slows the effective-rank growth as predicted by theorem 3.
  • Gradient averaging: Average coordinatewise gradient norms decay much more rapidly with depth in feedforward networks because increasingly whitened minibatch gradients average out their signal.The effect is much less pronounced in resnets.
  • Conclusion: Overall, resnet gradients remain more structured than feedforward gradients and therefore do not vanish when averaged within a minibatch.The results extend the earlier analysis to networks with convolutional layers.

5. The “looks linear” initialization

The looks-linear initialization combines rectifier networks with a linear-like initialization intended to prevent gradient shattering. Preliminary CIFAR-10 experiments found that it enabled substantially deeper feedforward networks, with performance comparable to a resnet at the maximum tested depth.

  • Motivation: Linear networks avoid shattered gradients but lack the expressivity needed to be useful, motivating a rectifier initialization that looks linear.
  • Construction: LL-init initializes rectifiers with mirrored block structure so their outputs are linear at initialization.The paper implements this idea with concatenated rectifiers, or CReLUs.
  • Construction: The initialized output ceases to be linear when weight updates cause the two blocks to diverge.
  • Alternative: A PReLU-based alternative sets its parameter a = 1 at initialization, but preliminary experiments suggest the CReLU-based LL-init is more effective.The reason for this difference is unclear.
  • Experiments: The LL-init was evaluated with orthogonal weights in very deep CIFAR-10 networks to test whether preventing shattering improves training.The experiment was designed to compare concatenated rectifier networks that were and were not shattered at initialization.
  • Results: At maximum depth 198, the standard ReLU network performed below the 40% linear baseline, whereas feedforward LL-init performed comparably to a resnet.Each plotted point is the mean of 10 trained models; standard ReLU and CReLU performance worsened steadily with depth.

6. Conclusion

The paper characterizes shattered gradients as increasingly uncorrelated gradients in randomly initialized deep networks and identifies skip-connections and LL-init as approaches that reduce or prevent shattering. It also notes unresolved questions about training dynamics and possible hybrid architectures.

  • Background: Rectifier networks gain exponentially many linear regions with depth, but this also proliferates discontinuities in their gradients.
  • Shattered gradients: In randomly initialized deep networks, gradients in deeper layers become increasingly uncorrelated, creating a previously unnoticed problem distinct from vanishing and exploding gradients.The paper links shattering to difficulties faced by optimization methods and minibatch gradient averaging.
  • Remedies: Skip-connections with suitable rescaling change correlation decay from exponential in feedforward architectures to sublinear in resnets.
  • Remedies: LL-init is presented as an alternative initialization that does not shatter gradients, while combining it with skip-connections is proposed as a future direction.

A1. Backprop and Brownian Motion

The appendix connects gradients in a minimal rectifier network to random walks and Brownian motion. Under Glorot-style initialization, the suitably reparameterized derivative converges weakly to Brownian motion as width grows.

  • Setup: The appendix uses Brownian motion as the limiting object for a suitably reparameterized neural-network derivative.
  • Brownian motion: Brownian motion has continuous sample paths, independent increments over disjoint intervals, and covariance E[B_tB_s] = min(t, s).
  • Donsker’s theorem: Donsker’s theorem states that normalized sums of independent mean-zero, unit-variance variables converge weakly to Brownian motion on [0, 1].
  • Proposition: With zero-mean variance σ^2 = 1/N initialization, the derivative of the network converges weakly to Brownian motion as N →∞.
  • Derivation: As the input changes, the derivative becomes a random walk whose jumps occur at Gaussian-sampled points after neurons are ordered by their biases and parameterized by the active-neuron fraction.

A2. The Karhunen-Loeve theorem

The Karhunen-Loeve theorem represents a centered stochastic process using eigenfunctions of its covariance operator. The resulting coefficients are uncorrelated and have variances given by the corresponding eigenvalues.

  • A centered stochastic process is characterized by its covariance function and associated integral operator.
  • When the covariance kernel is continuous, Mercer’s theorem supplies an orthonormal eigenbasis with associated eigenvalues.
  • The Karhunen-Loeve theorem expands the process in this covariance-derived eigenbasis.
  • The expansion coefficients have zero mean, are pairwise uncorrelated, and have variances equal to the eigenvalues.
  • The representation is understood under uniform convergence in the mean with respect to t.

A3. Details of the Analysis

The analysis treats neural-network outputs and gradients as random processes indexed by datapoints. It then uses path-weights as a covariance-orthogonal basis, paralleling Fourier-style expansions.

  • Neural functional analysis: Functional analysis expands functions as linear combinations of basis functions satisfying an orthonormality relation.
  • Neural functional analysis: The Karhunen-Loeve transform applies the same orthogonal-expansion idea to random processes, with principal component analysis as a special case.
  • Neural functional analysis: Random initialization makes network outputs random variables indexed by datapoints, and gradients form a similarly indexed random process.
  • Path-weight basis: Path-weights provide an orthogonal basis under the covariance inner product, analogous to sin, cos, and exp in Fourier analysis.

A3.1. Covariance structure of path-sums

The analysis models gradients as sums of active path-weights and computes their covariance by exploiting path orthogonality and counting co-active paths. It extends this framework to residual paths, batch normalization, and depth-dependent variance behavior.

  • Feedforward path-sums: Gradients are represented as sums of products of weights along paths, with each path contribution zeroed when any neuron is inactive.
  • Feedforward path-sums: A path-weight is the product of weights along a sequence of neurons, and paths are uncorrelated unless they coincide exactly.
  • Feedforward path-sums: Activation configurations determine active and co-active paths, allowing gradient covariance to be computed from path counts.
  • Scope of the analysis: The theorems target large-scale exponential-versus-sublinear behavior, while their precise quantities depend on the setup.
  • Residual path-sums: Residual paths specify which layers are skipped and which are traversed, with standard feedforward paths recovered when no layers are skipped.
  • Depth and normalization: Without batch normalization, each additional layer doubles the contribution to gradient variance in the stated model.
  • Depth and normalization: The analysis tracks how variance and covariance change with depth, including rescaling from batch normalization and large-depth approximations.
Loading 1702.08591v2…