Source-linked AI summary
Training Very Deep Networks
Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber
TL;DR
Very deep networks are difficult to optimize despite their representational advantages. The paper introduces highway networks, which use LSTM-inspired adaptive gates to facilitate information flow across layers, and reports direct SGD training with strong optimization and generalization results.
Problem
Very deep networks offer representational advantages, but conventional feed-forward architectures remain difficult to train because activations and gradients propagate poorly across stacked nonlinear transformations.
Method
Highway networks add LSTM-inspired adaptive transform and carry gates that regulate information flow across many feed-forward layers.
Results
Highway networks can be trained directly with SGD, including very deep models, and experiments show strong optimization, generalization, and layer-utilization behavior.
Takeaways & Limitations
The architecture supports studying extremely deep and efficient networks while allowing the contribution of individual layers to be examined.
Takeaways & Limitations
The reported results use a much-simplified highway block rather than the more complex LSTM-like block that also succeeded in pilot experiments.
Abstract
from arXiv · showhide
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.
1 Introduction & Previous Work
Deep networks can represent some functions more efficiently, but stacking layers makes activations and gradients propagate poorly. Highway networks address this with LSTM-inspired adaptive gates that create attenuated-information-free paths and enable direct SGD training of very deep models.
- Motivation: Deep networks can represent certain function classes far more efficiently than shallow networks.The paper illustrates this with recurrent networks, Boolean circuits, and modern neural networks.
- Motivation: Increasing depth makes conventional feed-forward networks harder to train because stacked nonlinear transformations impair activation and gradient propagation.This difficulty limits investigation of the benefits of very deep networks.
- Highway Networks: Highway networks use LSTM-inspired adaptive gating to create information highways across layers, allowing information to flow without attenuation.The architecture modifies very deep feed-forward networks rather than replacing their transformations.
- Contribution: The primary contribution is demonstrating that extremely deep highway networks can be trained directly with stochastic gradient descent, unlike increasingly difficult-to-optimize plain networks.The paper also reports direct single-stage training for deep networks with limited computational budgets.
2 Highway Networks
A highway layer combines a learned transformation with the input through complementary gates, enabling behavior between transformation and direct carry paths. The paper discusses its construction, training initialization, and deep-network experiments.
- Highway Layer: A plain layer applies a nonlinear transformation H to its input, whereas a highway layer additionally computes transform and carry gates.The transformations may be feedforward, convolutional, recurrent, or other forms.
- Highway Layer: With C = 1 − T, the highway output is y = H(x, WH)·T(x, WT) + x·(1 − T(x, WT)).T controls how much transformed output is used, while the complementary term carries the input forward.
- Construction: Convolutional highway layers use weight-sharing and local receptive fields for both H and T, with zero-padding matching their feature-map sizes to the input.The same-sized receptive fields are used for the block-state and transform-gate computations.
- Training: Negative transform-gate bias initialization favors carry behavior and was sufficient for training very deep networks across several H initializations and activation functions.Pilot experiments reported no SGD stall beyond 1000 layers; suggested convolutional biases are -1, -2, and -3 for depths around 10, 20, and 30.
- Scope: The reported pilot results use a much-simplified highway block despite successful experiments with a more complex LSTM-like block design.The simplification limits what architecture was evaluated in the reported results.
3 Experiments
The experiments compare highway and plain networks across depths and tasks, focusing on optimization and generalization under controlled training settings. Highway networks remained trainable at greater depths and performed competitively with fewer parameters.
- 3.1 Optimization: Optimization experiments compared plain and highway networks across depths using normalized initialization and controlled parameter counts.The MNIST study used 9, 19, 49, or 99 added layers and approximately 5000 parameters per layer.
- 3.1 Optimization: 100-layer highway networks performed more than 2 orders of magnitude better than similarly sized plain networks on MNIST training loss.Their performance remained similar to that of 10- and 20-layer highway networks, while deeper plain networks degraded.
- 3.1 Optimization: Highway networks consistently converged significantly faster than plain networks in the MNIST optimization experiments.
- 3.3.2 CIFAR experiments: CIFAR-10 and CIFAR-100 evaluations used typical data augmentation and compared convolutional highway networks with results from recent studies in similar settings.
- 3.2 Generalization: 10-layer convolutional highway networks on MNIST achieved test performance competitive with state-of-the-art methods using much fewer parameters.The evaluated architectures used 9 convolutional layers followed by a softmax output, with widths of 16 or 32 filter maps.
- 3.3.1 Comparison to Fitnets: Highway A and Highway B matched or exceeded the test accuracy of corresponding Fitnet architectures while using comparable parameter and operation counts in single-stage SGD training.A 32-layer highway network with approximately 1.25M parameters also outperformed the earlier teacher network.
4 Analysis
The analyses show that highway gates route information selectively across depth, with class-dependent activity and different layer-usage patterns for MNIST and CIFAR-100.
- Routing of Information: Most block outputs remain constant across many layers, while most changes occur in approximately the first 15 MNIST layers and first 40 CIFAR-100 layers.These stripe patterns visualize information highways through the network.
- Routing of Information: Transform-gate activity is sparse for individual samples, especially on CIFAR-100, although many gates are active on average across samples.Different samples therefore use different blocks for transformation.
- Routing of Information: Mean transform-gate activity differs between classes: MNIST differences concentrate within the first 15 layers, whereas CIFAR-100 differences are sparser and distributed across all layers.The gating system therefore participates in computation rather than merely easing optimization.
- Layer Importance: Lesioning any early MNIST layer significantly raises error, but layers 15–45 have nearly no effect on final performance.About 60% of MNIST layers do not learn to contribute to the final result.
- Layer Importance: Removing any of the first approximately 40 CIFAR-100 layers noticeably degrades performance, indicating that the deeper network uses substantially more layers for this task.The authors contrast this with MNIST, where many layers remain idle.
5 Discussion
Highway networks combine adaptive multiplicative routing with direct trainability by simple gradient descent, while enabling examination of how much depth a problem uses.
- Discussion: Very deep highway networks can be trained directly with simple gradient descent because of their architecture, without requiring a specialized initialization scheme.The property is not tied to a particular nonlinear transformation.
- Discussion: Adaptive multiplicative gating routes information differently for different inputs, unlike fixed skip connections.The gating mechanism requires additional parameters but permits input-dependent routing.
- Discussion: Highway networks can match or exceed wide, shallow maxout networks, indicating that closed transform gates do not make the architecture ineffective.The experiments also support evaluating individual layer contributions directly.
- Discussion: Figure 4 uses single-layer gate closure to assess how much computation depth a given problem requires.This layerwise evaluation is presented as difficult to perform with plain networks.
A Highway Networks Implementation
Highway operations combine a nonlinear transform with gated input pathways, using element-wise operations and, in this paper, a complementary carry gate. Experiments compare optimization across depths and show that highway networks remain easier to optimize than plain networks.
- Gating functions: The transform and carry functions T and C are interpreted as gates and should typically map inputs to values in (0, 1).H can use affine transformations with nonlinear activations such as tanh or rectified linear functions.
- Highway operation: A Highway operation combines transformed input H with gated pathways from x using element-wise multiplication and addition.The transformations H, T, and C are defined for input x and learned parameters.
- Gating functions: For simplicity, the implementation sets the carry gate to C = 1 − T, coupling transformed and carried information.This complementary formulation is the paper’s chosen Highway operation configuration.
- Backward pass: Backpropagation through the Highway operation requires derivatives of x, T, and H because the operation introduces no additional parameters of its own.The backward pass computes only the derivatives associated with these existing quantities.
- Optimization across depth: Plain networks become much harder to optimize as depth increases, whereas highway networks up to 100 layers can still be optimized well.Figure 5 compares optimization of plain and highway networks at various depths using SGD with momentum.