Source-linked AI summary
Highway Networks
Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber
TL;DR
Increasing depth can improve neural-network capability, but it makes optimization difficult. The paper introduces highway networks, which use learned gates to regulate information flow across layers. Preliminary experiments show that these networks can be trained to hundreds of layers with simple SGD, while optimization remains largely unaffected through 100 layers.
Problem
Increasing network depth improves neural-network capability but makes training very deep networks difficult.
Method
Highway networks use learned gating mechanisms to regulate information flow and create paths across several layers without attenuation.
Results
900-layer highway networks were optimized with simple SGD with momentum, and optimization was not hampered as depth increased to 100 layers.
Takeaways & Limitations
Training extremely deep networks enables studying the impact of depth on complex problems and using activation functions lacking robust initialization schemes.
Abstract
from arXiv · showhide
There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on "information highways". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.
1. Introduction
Deep networks offer important representational and empirical advantages, but increasing depth makes optimization difficult. Highway networks address this challenge with learned gates that regulate information flow, enabling very deep optimization and maintaining training behavior as depth increases.
- Motivation: Network depth has been central to recent supervised-learning breakthroughs, including ImageNet top-5 accuracy rising from ∼84% to ∼95%.The cited ImageNet comparison attributes the improvement to ensembles of deeper architectures and smaller receptive fields.
- Motivation: Deep networks can represent some function classes exponentially more efficiently than shallow networks and may provide computational and statistical efficiency.
- Problem: Optimization becomes considerably more difficult as networks deepen, motivating specialized initialization, staged training, and auxiliary losses.
- Contribution: Highway networks use learned gates to regulate information flow, creating paths across layers where information can flow without attenuation.These paths are called information highways, and the architecture is inspired by Long Short Term Memory recurrent networks.
- Results: 900-layer highway networks were optimized with simple SGD with momentum in preliminary experiments.
- Results: Through 100 layers, highway optimization was virtually independent of depth, while traditional networks degraded as depth increased; comparable CIFAR-10 architectures matched test accuracy without a pre-trained teacher.
2. Highway Networks
Highway layers reparameterize a nonlinear transform and input carry path with learned gates, allowing behavior to vary smoothly between transformation and direct passage. A negative transform-gate bias supports training very deep networks across varied transforms and activation functions.
- Gated layer formulation: A highway layer combines a nonlinear transform H(x, WH) with the input x through transform and carry gates.The general form is y = H(x, WH)·T(x, WT) + x·C(x, WC).
- Gated layer formulation: Setting C = 1 − T makes the output interpolate between a plain transformed layer and an identity-like input pass-through.When T = 0, the layer carries x; when T = 1, it produces H(x, WH).
- Gated layer formulation: Transform gates regulate how much output comes from transformation versus carrying the input, while highway blocks compute both a block state and gate output.The block output is yi = Hi(x)·Ti(x) + xi·(1 − Ti(x)).
- Architecture and implementation: Highway layers require matching dimensionality among x, y, H(x, WH), and T(x, WT); this study changes dimensionality with a plain layer before stacking highway layers.Convolutional versions use weight sharing, local receptive fields, and zero-padding to match feature-map sizes.
- Training very deep networks: Plain deep-network training can stall initially without initialization preserving forward- and backward-signal variance, and that initialization depends on H.Highway layers instead initialize transform-gate biases negatively, initially favoring carry behavior.
- Training very deep networks: Negative-bias initialization was sufficient for learning in very deep networks across various zero-mean WH initializations and activation functions.The authors describe this as significant because effective initialization may be unavailable for many choices of H.
3. Experiments
Experiments test whether highway networks remain optimizable as depth increases and whether they generalize competitively against FitNets. Across these comparisons, highway networks preserve optimization and achieve similar or higher CIFAR-10 accuracy through direct backpropagation.
- 3.1. Optimization: Training-set cross-entropy was measured on MNIST to assess optimization without conflating it with generalization.Plain and highway networks used the same architecture and varying depths, with approximately matched parameter counts.
- 3.1. Optimization: Highway networks showed no apparent degradation with depth, whereas plain networks degraded substantially as depth increased.The 100-layer highway network achieved a final result about one order of magnitude better than the 10-layer highway network and comparable to the 10-layer plain network.
- 3.2. Comparison to Fitnets: Highway networks were compared with FitNets on CIFAR-10 using random translations, directly testing supervised generalization.FitNets required a two-stage procedure involving soft targets from a pretrained teacher network.
- 3.2. Comparison to Fitnets: Highway 1 and Highway 4 obtained similar or higher test accuracy than corresponding FitNet architectures.The comparisons used highway networks with parameter and operation counts comparable to FitNets.
- 3.2. Comparison to Fitnets: A 19-layer highway network with ~1.4M parameters and a 32-layer network with ~1.25M parameters performed similarly to Romero et al.'s teacher network.These thinner, deeper networks were trained directly using backpropagation.
4. Analysis
The analysis examines optimization across depth and the internal behavior of highway gates. It shows that highway networks remain optimizable at depth while selectively routing information through layers.
- Internal gating: Transform-gate biases became more negative during training in many units, while CIFAR-100 biases increased with depth and inversely tracked average gate activity.The authors interpret strong negative biases as promoting selectivity rather than simply shutting gates down.
- Optimization: Plain networks become harder to optimize as depth increases, while highway networks remain well optimized through 100 layers.Figure 1 compares networks trained with SGD with momentum using the best settings found by random search.
- Information highways: Most block outputs remained constant across many layers, with most changes occurring in approximately 10 MNIST layers and 30 CIFAR-100 layers.The authors hypothesize that the difference reflects the greater complexity of CIFAR-100.
- Information highways: The gating mechanism passes information almost unchanged through many layers and selectively routes it according to current input patterns.This behavior is presented as evidence that gating is used in trained networks, not only to ease optimization.
5. Conclusion
The conclusion presents highway networks as an architecture for training extremely deep models with simple SGD. Their gating mechanism supports information flow across depth and permits further study of deep networks and alternative activation functions.
- Conclusion: Training very deep networks remains difficult, especially without substantially increasing total network size.The conclusion frames highway networks as addressing this difficulty through learned information routing.
- Conclusion: Highway networks enable training extremely deep networks with simple SGD, while plain architectures become increasingly difficult to train as depth grows.The experiments report that highway optimization is not hampered through 100 layers.
- Conclusion: The ability to train extremely deep networks opens the possibility of studying depth on complex problems without restrictions.The paper also identifies future work on understanding learning in highway networks.
- Conclusion: Highway networks can use activation functions that may suit particular problems even when robust initialization schemes are unavailable.This possibility is stated as a consequence of the architecture's training behavior.