PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Very Deep Networks

Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber

arXiv:1507.06228v2cs.LGcs.NE

TL;DR

Stacked nonlinear transformations in conventional feedforward networks impair activation and gradient propagation, making it difficult to investigate very deep networks across problems. Highway networks use LSTM-inspired adaptive gates to create paths for information to flow across many layers without attenuation. The 100-layer highway network performed more than 2 orders of magnitude better than a similarly-sized plain network; 50/100-layer highway networks performed similarly to 10/20-layer networks.

  • Problem

    Stacked nonlinear transformations in conventional feedforward networks impair activation and gradient propagation, making it difficult to investigate very deep networks across problems.

    Source: §1 · p. 2

  • Method

    Highway networks use LSTM-inspired adaptive gates to create paths for information to flow across many layers without attenuation.

    Source: §1 · p. 2

  • Results

    The 100-layer highway network performed more than 2 orders of magnitude better than a similarly-sized plain network; 50/100-layer highway networks performed similarly to 10/20-layer networks.

    Source: §3.1 · p. 4

  • Takeaways & Limitations

    Highway networks provide a way to examine how much computation depth a given problem needs, which is difficult to do with plain networks.

    Source: §5 · p. 8

    The reported results cover only a simplified block design, although pilot experiments also succeeded with a more complex LSTM-like block.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

1 Introduction & Previous Work

Conventional deep networks are difficult to train because activations and gradients propagate poorly, while LSTM-inspired adaptive gates create information paths across many layers. The resulting highway networks can be trained directly with SGD, even at extreme depths, and generalize well to unseen data.

1 Introduction & Previous Work

  • Depth has supported ImageNet top-5 accuracy rising from ∼84% to ∼95% [2] [3], and recurrent networks can represent n-bit parity with 3 units and 5 weights.The passage also notes that deep networks can represent some function classes more efficiently than shallow ones.Source: §1 · p. 1
  • Prior approaches to deep-network training include optimizer and initialization improvements [15] [17], competitive activations [20], skip connections [23] [2] [24], teacher-student distillation [25], and layer-wise training.Layer-wise training can aid credit assignment but is described as less attractive than direct training.Source: §1 · p. 1
  • Conventional feedforward networks often propagate activations and gradients poorly through stacked nonlinear transformations, making the benefits of very deep networks hard to investigate.The passage describes this as a training problem, though perhaps less fundamental than vanishing gradients in standard recurrent networks.Source: §1 · p. 2
  • LSTM-inspired adaptive gates create information highways that let information flow across many layers without attenuation.The architecture modifies very deep feedforward networks to make information flow across layers easier.Source: §1 · p. 2
  • Highway networks can be trained directly with SGD at extreme depths, unlike increasingly hard-to-optimize plain networks, and they generalize well to unseen data.Networks with limited computational budgets can also be trained in a single stage after conversion to highway networks.Source: §1 · p. 2

2 Highway Networks

Highway layers use a transform gate to blend a learned transformation with the input, allowing information to pass through or be transformed. Negative transform-gate bias initializes the network toward carrying inputs and supports training very deep networks.

2 Highway Networks

  • A highway layer combines H(x, WH) weighted by transform gate T(x, WT) with the input weighted by 1 − T(x, WT).The carry gate is set to 1 − T; when T is 0 the layer passes its input, and when T is 1 it applies H.Source: §2 · p. 2

2.2 Training Deep Highway Networks

  • Plain networks became harder to optimize as depth increased, while highway networks with up to 100 layers could still be optimized well.Figure 1 compares training curves for the best hyperparameter settings and mean performance among the top 10 of 100 settings at each depth.Source: Figure 1 · p. 3

2.1 Constructing Highway Networks

  • For dimensionality changes, the study uses a plain layer; convolutional highway transforms share weights and use matched receptive fields with zero-padding.The input can alternatively be sub-sampled or zero-padded to change dimensionality.Source: §2.1 · p. 3

2.2 Training Deep Highway Networks

  • Negative transform-gate bias initializes highway networks toward carry behavior, and pilot experiments found SGD did not stall in networks with more than 1000 layers.The paper suggests biases of -1, -2, and -3 for convolutional highway networks of approximately 10, 20, and 30 layers, respectively.Source: §2.2 · p. 3
  • The reported results use a simplified block design, although pilot experiments also succeeded with a more complex LSTM-like block unrolled in time.The paper reports results only for the simplified form.Source: §2.2 · p. 3
  • CIFAR-10 convolutional highway networks trained in a single stage without hints matched or exceeded the performance of two-stage hint-trained fitnets.The comparison is reported for test-set accuracy; no specific accuracy values are given in the supplied passage.Source: Table 2 · p. 4

3 Experiments

The experiments test whether highway networks remain trainable as depth increases and assess their classification performance on MNIST and CIFAR datasets. They report that deep highway networks avoid the degradation seen in plain networks and can be trained in a single stage at FitNet-comparable scales.

3.1 Optimization

  • Depth comparisons trained thin plain and highway networks with roughly 5000 parameters per layer across 10, 20, 50, and 100 total layers.Both network types were randomly searched over 100 runs for hyperparameters on MNIST, with normalized initialization; highway networks additionally varied the initial transform-gate bias.Source: §3.1 · p. 4
  • A 100-layer highway network performed more than 2 orders of magnitude better than a similarly-sized plain network, while deep highway networks matched shallower ones and converged faster.Plain networks’ performance degraded with increased depth despite their increased capacity; 50- and 100-layer highway networks performed similarly to 10- and 20-layer networks.Source: §3.1 · p. 4

3.2 Pilot Experiments on MNIST Digit Classification

  • Two 10-layer convolutional highway networks on MNIST achieved test performance competitive with state-of-the-art methods using much fewer parameters.Each architecture used nine convolutional layers followed by softmax; widths were 16 and 32 filter maps.Source: §3.2 · p. 5

3.3.1 Comparison to Fitnets

  • Highway A and Highway B achieved similar or higher test accuracy than Fitnet A and Fitnet B, and a 32-layer highway network with ∼1.25M parameters outperformed the earlier teacher network [20].The highway networks were trained in a single SGD stage at parameter and operation counts comparable to FitNets, whereas the cited prior work required hint-based two-stage training for deeper networks.Source: §3.3.1 · p. 5

3.3.2 Comparison to State-of-the-art Methods

  • CIFAR-10 and CIFAR-100 experiments used global contrast normalization, small translations, mirroring, a 1×1 convolutional layer, and global average pooling.Hyperparameters from the preceding section were reused, and results are tabulated in Table 3; the supplied passages do not report its accuracy values.Source: §3.3.2 · p. 5, Table 3 · p. 5

4 Analysis

The analysis finds that highway gates route transformations selectively and in an input- and class-dependent way, while layer contributions vary with dataset complexity: many MNIST layers are idle, but early CIFAR-100 layers matter.

4 Analysis

  • Transform-gate biases mostly decreased during training, yet strong negative biases made gates selective rather than shutting them down; CIFAR-100 biases increased with depth as mean activity fell.The biases were initialized to -2 for MNIST and -4 for CIFAR-100; single-example gate activity was sparse, especially for CIFAR-100, and also to a lesser extent for MNIST.Source: §4 · p. 6, Figure 2 · p. 6
  • Block outputs mostly stay constant across layers, with most changes occurring in the first ≈15 MNIST layers and ≈40 CIFAR-100 layers, illustrating information highways.Figure 2 visualizes these output patterns for the 50-hidden-layer networks.Source: §4 · p. 6, Figure 2 · p. 6

4.1 Routing of Information

  • For each sample, only a few blocks transform, but different blocks serve different samples; class-mean gate activity also differs, including within MNIST’s first 15 layers.MNIST digits 0 and 7 show substantial differences within the first 15 layers, while CIFAR-100 classes 0 and 1 show sparser differences spread across all layers; gating therefore contributes to computation, not only training.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Layer Importance

  • Lesioning shows that MNIST error rises when early layers are removed, but layers 15–45 have little effect; removing any of CIFAR-100’s first ≈40 layers noticeably degrades performance.Each layer was tested by setting its transform gates to 0 and evaluating the full training set. About 60% of MNIST layers did not contribute to the final result, whereas the CIFAR-100 pattern suggests the network can use all its layers on a complex task.Source: §4.2 · p. 7

5 Discussion

Very deep highway networks can be trained with simple gradient descent because their architecture routes information through input-responsive multiplicative gates, without requiring a particular transformation or specialized initialization. Their depth also supports useful computation and lets researchers examine how much computation a problem needs.

5 Discussion

  • Alternative approaches have limitations: competitive routing improves credit assignment [38] but struggles beyond ≈20 layers despite careful initialization [17], while initialization can be difficult and deep supervision can hurt thin deep networks [24] [25].Source: §5 · p. 7
  • Highway networks train very deep models with simple gradient descent, using input-responsive multiplicative gates rather than fixed skip connections, without requiring a specific transformation or suitable initialization scheme.The transformations can be complex convolutional or recurrent operations; gating adds parameters to route information.Source: §5 · p. 7
  • Deep, narrow highway networks can match or exceed wide, shallow maxout networks, indicating that their many layers perform useful computations.Closing each layer’s transform gates during evaluation allows its contribution to be assessed, helping reveal the computation depth needed for a problem (Figure 4).Source: §5 · p. 8, Figure 4 · p. 8

A Highway Networks Implementation

The implementation builds a Highway operation from transformations and gates, using element-wise operations in its forward pass. In this paper, the carry gate is set to 1 − T for simplicity, and backpropagation requires derivatives only for x, T, and H.

A Highway Networks Implementation

  • The implementation represents neural networks as chained simpler operations, with learned parameters for transformations H, T, and C.H, T, and C are assumed to be defined for input x.Source: A Highway Networks Implementation · p. 10
  • H may be affine with a nonlinear activation or take convolutional, recurrent, or other forms, while T and C are typically gates mapping to (0, 1).Source: A Highway Networks Implementation · p. 10
  • The Highway operation is defined using x, T, H, and C, and its forward pass uses element-wise multiplication and addition.Source: A Highway Networks Implementation · p. 10
  • For simplicity, the paper sets C = 1 − T.Source: A Highway Networks Implementation · p. 10
  • Because the Highway operation adds no parameters of its own, backpropagation computes derivatives only for x, T, and H.Source: A Highway Networks Implementation · p. 10
  • Plain networks become harder to optimize as depth increases, while highway networks remain optimizable through 100 layers.The comparison used SGD with momentum; the figure reports training curves for the best hyperparameter settings and mean performance of the top 10 of 100 settings.Source: Figure 5 · p. 11

Related papers

Loading 1507.06228v2…