Source-linked AI summary
Highway and Residual Networks learn Unrolled Iterative Estimation
Klaus Greff, Rupesh K. Srivastava, Jürgen Schmidhuber
TL;DR
The paper questions whether Highway and Residual networks are best understood as hierarchies of increasingly abstract representations, given their resilience to layer manipulations. It proposes unrolled iterative estimation, derives both architectures from this view, and reports preliminary evidence that they perform similarly at equal size while gating can help when individual features must be replaced.
Problem
The representation view does not fully explain why Highway and Residual networks tolerate layer removal, dropout, and reshuffling despite assigning each layer a new representation.
Method
The paper interprets blocks within a stage as successive estimators that refine the same feature representation, then derives Residual and Highway networks from this perspective.
Results
Highway and Residual networks give very similar results for equal-sized networks, while non-gated identity skip-connections perform significantly worse in one example.
Takeaways & Limitations
Unrolled iterative estimation provides a unified framework for understanding both architectures and their resilience to lesioning, layer dropout, and mild reshuffling effects.
Takeaways & Limitations
The network may depend on a specific refinement order, so shuffling and lesioning can have moderate effects in some cases.
Abstract
from arXiv · showhide
The past year saw the introduction of new architectures such as Highway networks and Residual networks which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that these architectures defy a popular view of deep learning as a hierarchical computation of increasingly abstract features at each layer. In this report, we argue that this view is incomplete and does not adequately explain several recent findings. We propose an alternative viewpoint based on unrolled iterative estimation -- a group of successive layers iteratively refine their estimates of the same features instead of computing an entirely new representation. We demonstrate that this viewpoint directly leads to the construction of Highway and Residual networks. Finally we provide preliminary experiments to discuss the similarities and differences between the two architectures.
1 INTRODUCTION
Highway and Residual networks made very deep feedforward models trainable, but their behavior challenges the assumption that every layer computes a new, increasingly abstract representation. The paper proposes unrolled iterative estimation as an alternative interpretation of these architectures.
- The representation view treats deep networks as hierarchies in which each layer computes a new and increasingly abstract level of representation.
- Beyond a certain depth, performance can decline even on the training set, although deeper models are more powerful in principle.
- Highway and Residual networks made feedforward networks with hundreds of layers feasible and achieved broad success across vision, language modeling, and translation.
- Removing almost any layer from trained Highway or Residual networks reportedly has minimal effect, contradicting the expectation that each layer supplies an indispensable new representation.
- The paper proposes that blocks refine their input representations while preserving feature identity, with dimensionality-changing projections separating stages.
- This perspective is formalized to derive both Residual and Highway networks and to explain effects of lesioning and layer shuffling.
2 CHALLENGING THE REPRESENTATION VIEW
Several findings challenge a strict layer-by-layer representation view: features often remain stable across blocks, and networks tolerate layer removal, dropout, and reshuffling. These observations motivate viewing stages as iterative refinement rather than repeated construction of new representations.
- Residual blocks typically stay close to their input representation rather than replacing features with a higher-level representation.
- Highway transform gates are sparse per sample, so most units copy inputs while selectively replacing individual features.
- Iterative estimation refines the same representation across blocks, whereas classic layers compute a new representation at each layer.
- Removing layers from trained networks can have very different effects: VGG lesioning raises CIFAR-10 classification error to around 90%, while Highway and Residual networks show minimal overall impact.
- Layer dropout regularizes ResNets by training short subnetworks while using deep networks at test time, an effect described as an implicit ensemble of shallow networks.
- Reshuffling layers within a 110-layer ResNet produces a smooth error increase, with many reorderings causing only a small error increase.
3 UNROLLED ITERATIVE ESTIMATION VIEW
The paper augments the hierarchical representation view by treating blocks within a stage as iterative estimators that preserve feature identity while refining one representation. This perspective motivates Highway and Residual formulations through skip connections and estimation-based updates.
- Representation View: The one-to-one mapping between layers and abstraction levels is an implicit assumption that can be discarded without abandoning the representation view.A dimensionality change through projection separates stages and marks a transition to a new representation level.
- Unrolled Iterative Estimation: Blocks within a stage iteratively refine a single representation rather than compute entirely new representations.The first block provides a rough estimate, while subsequent blocks refine it without changing the representation level.
- Feature Identity: Feature identity is formalized by treating activations across layers as unbiased estimators of the same latent feature.The estimator and latent feature depend on the same data sample, so their difference remains inside the same expectation.
- Highway and Residual Networks: Identity skip connections improve error flow by allowing units to copy inputs unchanged, linking Highway and Residual networks to simplified unrolled LSTM-like computations.Highway layers additionally use transform and carry gates to control nonlinear transformation versus copying.
- Highway and Residual Networks: Residual networks express the desired transformation as the input plus a residual, while Highway networks combine a prior estimate with a new estimate using learned mixing.The residual form makes the identity case equivalent to driving the residual toward zero; the Highway formulation can be derived as a minimum-variance linear combination.
- Residual Derivation: A zero-mean residual implies zero expected change between consecutive blocks and therefore maintains feature identity.The paper notes that this assumption is reasonable, especially with batch normalization.
4 DISCUSSION
The discussion interprets Highway and Residual networks as stage-wise iterative estimators whose blocks preserve feature identity while progressively refining representations. Empirical errors, visualizations, and architectural behavior support this view, though refinement can depend moderately on layer order.
- Experimental corroboration: Residual blocks empirically have near-zero mean estimation error, while decreasing error variance across blocks indicates convergent refinement.This pattern appears in the first three stages of a 50-layer ResNet; stage four appears to underestimate representation values.
- Stage-wise estimation: Within a stage, identity shortcuts preserve the representation level, while projections between stages change dimensionality and abstraction.Stages therefore contain blocks operating on the same-dimensional representation, whereas transitions introduce a new representation level.
- Architectural implications: Iterative estimation explains why later blocks may benefit from greater depth, because complex features with interdependencies require more refinement than simple early representations.The perspective predicts shorter early stages and longer later stages.
- Visual evidence: Within-stage visualizations show a filter refining its response by incorporating surrounding context rather than changing feature identity.For one stage-1 filter, eight of nine later maximally activating patches are specular highlights, compared with six initially.
- Architectural implications: Staying close to the input motivates residual updates and sparse Highway transform-gate activity, which selectively changes features while preserving most of the representation.The discussion treats retaining the previous estimate as a common operation during iterative re-estimation.
- Lesioning and shuffling: Layer removal and shuffling should often have mild effects because within-stage blocks perform incremental, partly interchangeable refinements, although learned ordering can moderate this interchangeability.The stated limitation is that networks may depend on a specific refinement order.
5 COMPARATIVE CASE STUDIES
The case studies compare Highway and Residual architectures in image classification and language modeling. Results show similar ImageNet validation performance but task-dependent advantages for expressive gating in language modeling and for capacity under some normalization conditions.
- Image classification: In 50-layer ImageNet models, Residual networks achieve slightly lower top-5 error than Highway networks, while Highway-Full is closer to Residual performance.The reported errors are 7.17% for Residual, 7.53% for Highway, and 7.29% mean error for Highway-Full.
- Image classification: The image-classification comparison contradicts claims that Highway networks are difficult to train or require bias tuning, but suggests slight underfitting at matched parameter counts.The Highway network’s higher final training loss supports lower capacity in that setting.
- Image classification: Equally expressive transform and primary gates appear important for stable Highway learning, whereas using 1 × 1 transform gates was identified as a likely source of instability.The comparison emphasizes matching the gates’ receptive fields to the primary transformation H.
- Image classification: Without batch normalization, both networks achieve lower training error but worse validation error, indicating increased overfitting rather than failed optimization.Validation errors are 10.03% for Highway and 9.40% for Residual; Highway slightly exceeds Residual training capacity in this condition.
- Language modeling: In character-aware language modeling, Full, Coupled, and C-Only Highway variants outperform T-Only and Residual variants, making learned identity gating crucial for this task.The Residual variant performs close to a single plain layer despite using four Residual layers.
- Language modeling: Multiplicative input gates improve all Highway variants, suggesting that this task requires dynamically replacing some features while retaining others.The results imply a task-specific advantage for adaptive feature transformation and reuse.
6 CONCLUSION
The paper presents unrolled iterative estimation as a perspective on Highway and Residual networks, derives both architectures from it, and reports preliminary comparisons between them. The evidence is intended to guide further investigation rather than provide a final account.
- 6 CONCLUSION: Successive layers within a stage refine one representation while preserving feature identity, rather than each computing an entirely new representation.The first layer provides a rough estimate, and later layers iteratively improve it.
- 6 CONCLUSION: Both Residual and Highway networks can be directly derived from the unrolled iterative estimation perspective, providing a unified theory of the two architectures.
- 6 CONCLUSION: Preliminary experiments found similar results for equal-size Highway and Residual networks, challenging claims that Highway networks require more parameters or that gating impairs Residual networks.
- 6 CONCLUSION: Non-gated identity skip-connections performed significantly worse in one example, with gating proposed as beneficial when tasks require dynamically replacing individual features.
- 6 CONCLUSION: The report’s preliminary evidence is presented as a starting point for further investigation into understanding, improving, and possibly combining these techniques.
A.1 OPTIMAL LINEAR ESTIMATOR
This section formulates estimation of a latent variable from two noisy measurements as an unbiased minimum-variance linear-estimation problem. It imposes unbiasedness and reduces the variance objective to equations solved with Lagrangian multipliers.
- A.1 OPTIMAL LINEAR ESTIMATOR: A and B are modeled as noisy measurements of a latent variable C.
- A.1 OPTIMAL LINEAR ESTIMATOR: The estimator q(A, B) = q0 + q1A + q2B is required to be unbiased and to have minimum variance.
- A.1 OPTIMAL LINEAR ESTIMATOR: The unbiasedness condition is expanded by expressing A and B through their deviations from C and the residual coefficient q1 + q2 − 1.
- A.1 OPTIMAL LINEAR ESTIMATOR: The minimum-variance condition is reduced to an equation system and solved using Lagrangian multipliers.