PaperlayerView exact revision on arXiv

Source-linked AI summary

Collapse of Deep and Narrow Neural Nets

Lu Lu, Yanhui Su, George Em Karniadakis

arXiv:1808.04947v2stat.MLcs.LG

TL;DR

The paper addresses why theoretically effective deep, narrow ReLU networks can fail during training. It combines numerical experiments with theoretical analysis and collapse-probability estimates, finding convergence to erroneous constant states and identifying design strategies that can reduce or prevent collapse.

  • Problem

    Approximation theory does not ensure successful training, and deep, narrow networks can encounter collapse despite their representational advantages.

  • Method

    The paper analyzes collapse theoretically, estimates its probability, verifies it numerically, and tests initialization, normalization, and dropout strategies.

  • Results

    Deep, narrow ReLU networks converge with high probability to erroneous mean, partial-mean, or median states depending on the loss; collapse probability increases with depth and narrowness.

  • Takeaways & Limitations

    Batch normalization and SELU can prevent collapse, whereas orthogonal initialization, LSUV, weight normalization, and dropout do not reliably avoid it.

  • Takeaways & Limitations

    The theoretical assumptions include connected input space and initialization with independently sampled weights and biases from zero-mean Gaussian distributions in the stated analysis.

Abstract

from arXiv · show

Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives. We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse. We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states. Finally, we examine different ways of initialization and normalization that may avoid the collapse problem. Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

1 INTRODUCTION

Deep networks can approximate some functions with far fewer neurons than shallow networks, but theoretical approximation results do not ensure successful training. Deep-network optimization is difficult, and this paper focuses on collapse in deep, narrow networks as a distinct bad-local-minimum problem.

  • Approximation theory: Universal approximation theorems establish representability but do not specify the network size needed for a prescribed accuracy.For a three-dimensional C1 function with accuracy 10^-6, one cited estimate requires 10^18 neurons, compared with 1000 for a C6 function.
  • Deep versus shallow networks: Deep networks may outperform shallow networks of comparable size, according to theoretical results on depth-dependent approximation efficiency.One example contrasts a 3-layer network with 10 neurons per layer against a 1-layer network with 30 neurons.
  • Approximation theory: Shallow networks may require exponentially more neurons than deep networks for comparable ReLU approximation accuracy.For a target accuracy ϵ, multilayer networks may use Θ(log(1/ϵ)) layers and O(poly log(1/ϵ)) neurons, whereas shallower networks may require Ω(poly(1/ϵ)) neurons.
  • Training difficulty: Sigmoid and tanh networks are difficult to train because they suffer from vanishing gradients, especially across many neurons or layers.The passage links this difficulty to properties of these activations and their Hessians.
  • Training difficulty: Training outcomes need not match approximation theory because neural-network training is NP-hard and can be affected by bad local minima, singularities, and saddle points.The paper identifies collapse in deep and narrow networks as a particular class of bad local minima.

2 COLLAPSE OF DEEP AND NARROW NEURAL NETWORKS

Numerical tests show that deep, narrow ReLU networks often collapse to constant mean or partial-mean outputs across target regularities and dimensions. The collapsed state can depend on the loss: MSE yields means, whereas MAE yields medians.

  • Numerical demonstrations: Deep, narrow networks collapse to mean or partial-mean values for one- and two-dimensional targets with C0, C∞, and L2 regularity.The tests include scalar and vector-valued targets, including y(x)=|x|, x sin(5x), and 1{x>0}+0.2 sin(5x).
  • Numerical demonstrations: Approximately 90% of 10-layer, width-2 ReLU networks collapsed to the mean for y(x)=|x|, across listed initializers and optimizers.The tested optimizers included first- and second-order methods.
  • Collapse mechanism: When collapse occurred, the last-layer bias was usually the target mean while preceding layers acted as a zero function, causing gradients to vanish.The same collapse pattern was observed for targets with C∞ and L2 regularity.
  • Numerical demonstrations: For two-dimensional vector targets, a 10-layer ReLU network with width 4 showed very high-probability mean collapse and low-probability partial-mean collapse.The tested target was y(x)=[|x1+x2|, |x1−x2|].
  • Effect of loss: Collapse occurred under both MSE and MAE, with MSE producing the target mean and MAE producing the target median.The collapse itself was reported as independent of the loss function, while the resulting constant depended on the loss.

3 INITIALIZATION OF RELU NETS

The section analyzes how ReLU activations and random initialization propagate activation lengths through layers, emphasizing symmetric initialization and variance choices that preserve scale while leaving collapse possible.

  • Initialization effects: Symmetric random initialization can cause deep and narrow ReLU networks to collapse with high probability, despite avoiding exploding or vanishing mean activation lengths.The paper distinguishes stable activation-length propagation from the separate collapse problem.
  • Network dynamics: The feed-forward dynamics applies an affine transformation followed by a component-wise activation at each hidden layer.The layer weights and biases determine the affine map before activation.
  • Length propagation: The network propagates normalized squared pre-activation lengths recursively from one layer to the next under i.i.d. Gaussian weights and biases.The recursion is stated for ReLU and requires infinite widths for other activation functions.
  • Variance-preserving initialization: Setting weight variance to 2 and bias variance to zero preserves the expected squared activation length across layers.This is the variance choice associated with He normal initialization.

4 THEORETICAL ANALYSIS OF THE COLLAPSE PROBLEM

The theoretical analysis connects zero-function initialization in ReLU networks to constant outputs, vanishing gradients, and collapse probabilities that increase with depth and narrowness. It derives probability formulas and a practical depth–width safety guide.

  • Collapse mechanism: Under the stated assumptions, if some layer is identically zero, gradient-based training eventually produces a constant function.With L2 loss the constant is the target mean, while with L1 loss it is the target median when that median exists.
  • Collapse mechanism: When a network is constant, gradients with respect to earlier-layer weights and biases vanish, preventing those parameters from updating under the relevant loss.The result includes all-order gradients in layers up to the zero layer and, for the mean state, any parameter under L2 loss.
  • Initialization probability: A one-layer ReLU network with independently symmetric nonzero initialization outputs zero for a fixed input with probability (1/2)^dout, apart from the stated all-zero special case.This lemma supplies the building block for multilayer collapse-probability estimates.
  • Initialization probability: For multilayer networks with symmetric weights and zero biases, the fixed-input zero-output probability is 1 − Π_l(1 − (1/2)^N_l) when the last layer also uses ReLU.If the last layer does not use ReLU, the corresponding product stops at layer L−1.
  • Probability estimates: The fixed-input probability is an upper bound for whole-network collapse, while width-2 networks admit a separate transition-matrix estimate and general cases use numerical simulations.The simulations use 1 million random initializations per probability estimate and report the same collapse probability across symmetric distributions.
  • Practical design guide: A 10-layer network has about 1% collapse probability at width 10, above 10% at width 5, and above 60% at width 3.The paper presents these values as a practical safe-region guide and reports closer agreement between numerical and theoretical results as depth increases.

5 TRAINING DEEP AND NARROW NEURAL NETWORKS

The paper evaluates techniques for training deep and narrow ReLU networks, showing that collapse risk grows with depth and decreases with width, while some normalization methods can prevent it.

  • Initialization: Orthogonal initialization produces collapse probabilities close to, but slightly lower than, symmetric initialization and does not eliminate collapse.Figure 7 compares the two initialization schemes across widths and depths.
  • Collapse risk: Collapse probability increases as ReLU networks become deeper and narrower.Figure 6A compares theoretical, approximated, and numerical collapse probabilities across layers and widths.
  • Safe design: The safe operating region specifies the maximum layer count at each width for keeping collapse probability below 1% or 10%.The region below the blue line is identified as safe for network design.
  • Normalization: Batch normalization and SELU are examined as normalization methods, whereas layer, instance, and group normalization are excluded for the low-dimensional settings considered.The exclusions reflect the paper’s stated input-dimension and per-sample normalization constraints.
  • Dropout: Dropout fails to solve collapse and may increase zero activations by inducing sparsity.The simulations report that collapse remains unresolved under dropout.

6 CONCLUSION

The paper concludes that deep and narrow ReLU networks can collapse to erroneous constant states during training, and combines theoretical analysis, numerical verification, and design guidance to characterize and mitigate this problem.

  • 6 CONCLUSION: Deep and narrow ReLU networks may converge to erroneous means, partial means, or medians of target functions despite their approximation properties.The observed outcome depends on the loss and was demonstrated for functions with C0, C∞, and L2 regularity.
  • 6 CONCLUSION: The analysis proves that a constant network output entails a zero-output layer and vanishing gradients in all preceding layers.The proof proceeds through Lemma 1, Corollary 2, and Lemma 3.
  • 6 CONCLUSION: The paper derives collapse-probability estimates, verifies them numerically across depths and widths, and constructs a practical safe-region diagram for network design.The estimates include a more precise result for deep networks of width 2.
  • 6 CONCLUSION: Orthogonal initialization and LSUV cannot avoid collapse, whereas batch normalization and SELU can prevent it; weight normalization and dropout fail.These conclusions summarize the paper’s tested mitigation methods.
  • 6 CONCLUSION: Once these conditions hold, the network converges to a loss-dependent constant, including the target mean under L2 loss and the median under L1 loss.The constant state prevents gradient-based updates in earlier layers.

F PROOF OF COROLLARY 6

The proof establishes that gradients vanish under L2 loss in the specified constant-output settings, including both restricted cells and their complement.

  • F PROOF OF COROLLARY 6: Gradients also vanish outside the union of the restricted sets under the L2 loss.This completes the second case of the proof.
  • F PROOF OF COROLLARY 6: Gradients vanish on each restricted set Ki because the network is constant there and the L2 loss is zero.The proof separately considers inputs in each Ki.
  • F PROOF OF COROLLARY 6: Therefore, gradients vanish for the entire domain when using the L2 (MSE) loss.The conclusion combines the two cases.

G PROOF OF LEMMA 7

The proof computes the probability that a ReLU output is zero under symmetric random initialization by applying symmetry to each neuron and multiplying across outputs.

  • G PROOF OF LEMMA 7: Each ReLU output is written as ReLU(wi · x + bi), with the bias included in an augmented input vector.The augmented weight vector has dimension din + 1.
  • G PROOF OF LEMMA 7: Under a symmetric initialization, each preactivation has probability 1/2 of being nonpositive, so each ReLU output is zero with probability 1/2.The proof uses the symmetry of the augmented weight distribution.
  • G PROOF OF LEMMA 7: For dout outputs, the probability that the complete output vector is zero is (1/2)^dout.The result follows by multiplying the per-output probabilities.

H PROOF OF THEOREM 8

The proof recursively computes the probability that the final-layer output is nonzero under ReLU activation, yielding a product over layers. Without final-layer ReLU, this probability instead equals the preceding-layer probability.

  • With ReLU in the last layer, the nonzero-output probability factors through the preceding layer and the conditional probability at the last layer.The proof uses the law of total probability and the conditional factor supplied by Lemma 7.
  • The resulting probability is the product ΠL across the network layers.
  • Without ReLU in the last layer, the nonzero-output probability remains ΠL−1.

I PROOF OF PROPOSITION 9

The proof distinguishes whether the final layer uses ReLU and identifies the single-layer perceptron as a trivial boundary case. For networks with at least two layers and no final-layer ReLU, the preceding-layer state determines the result.

  • The proof separately analyzes the probability that the final-layer output is zero when the last layer uses ReLU.
  • When the last layer has no ReLU and L ≥2, the proposition applies under the stated multi-layer condition.
  • For L = 1, the network is a single-layer perceptron, treated as a trivial case.

J PROOF OF PROPOSITION 10

The proof models the hidden-layer configurations as 16 cases and derives their layer-to-layer probability transitions from ReLU behavior and random coefficient configurations. It then propagates the resulting distribution across layers.

  • With zero biases and width-two hidden layers, the network output is zero at zero input and linear in the input on either sign region.This structural property underlies the case-by-case probability analysis.
  • Each hidden-layer case can induce all 16 configurations in the next layer, whose probabilities are organized in a transition matrix.The entry Pji denotes the probability of next-layer case j given current case i.
  • For the analyzed cases, some configurations occur with probability 1/4, while other cases have probability 0.
  • In one configuration, all 16 cases have probability 1/16.
  • The layer distributions are obtained from the first-hidden-layer distribution and repeated application of the transition structure.
Loading 1808.04947v2…