Source-linked AI summary
Residual Networks Behave Like Ensembles of Relatively Shallow Networks
Andreas Veit, Michael Wilber, Serge Belongie
TL;DR
Residual networks raise the question of how very deep models remain trainable despite their depth and apparent layer redundancy. The paper rewrites them as collections of variable-length paths and tests their dependence through lesion studies. It finds ensemble-like path behavior and that short paths, rather than the deepest paths, provide most training gradient.
Problem
The paper examines why residual networks can remain effective at depths far beyond conventional architectures and whether their paths depend strongly on one another.
Method
The authors introduce an unraveled path representation and use lesion studies to analyze path dependence, redundancy, and the role of paths with different lengths.
Results
Residual-network paths show ensemble-like behavior, and most training gradient comes from short paths, with 10-34-layer paths dominating in a 110-layer network.
Takeaways & Limitations
The results support the conclusion that very deep residual networks are trained primarily through short effective paths rather than through their full depth.
Takeaways & Limitations
The analysis considers identity skip connections, although the framework is stated to generalize to projection skip connections for downsampling.
Abstract
from arXiv · showhide
In this work we propose a novel interpretation of residual networks showing that they can be seen as a collection of many paths of differing length. Moreover, residual networks seem to enable very deep networks by leveraging only the short paths during training. To support this observation, we rewrite residual networks as an explicit collection of paths. Unlike traditional models, paths through residual networks vary in length. Further, a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other. Finally, and most surprising, most paths are shorter than one might expect, and only the short paths are needed during training, as longer paths do not contribute any gradient. For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep. Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.
1 Introduction
Residual networks can be interpreted as collections of many paths rather than single ultra-deep networks. Their paths show ensemble-like redundancy, while training relies mainly on short paths that carry gradient through very deep networks.
- The findings contrast with traditional architectures, where removing a single layer can cause a dramatic performance loss.
- Residual networks can be viewed as collections of many paths instead of a single ultra-deep network.
- Lesion studies show that removing layers or corrupting paths has only a modest, smooth effect, indicating limited dependence and ensemble-like behavior.
- Most paths in a 110-layer network are about 55 layers deep, but most training gradient comes from paths only 10-34 layers deep.
- Residual networks appear to enable very deep training by shortening effective paths rather than preserving gradient flow across the full network depth.
2 Related Work
The paper situates residual networks within hierarchical feed-forward vision models while emphasizing their skip-connected, branching structure. It relates this structure to path ensembles, stochastic depth, and other architectures with parallel or gated paths.
- The sequential and hierarchical computer vision pipeline: Residual networks challenge the view that visual representations must be built through a strictly sequential hierarchy of increasingly complex features.
- Residual networks: Each residual block combines a residual module with an identity skip connection, allowing the input to bypass the module.
- Branching paths in neural networks: Inception networks likewise use parallel paths, but residual networks are selected here for their simpler design principle.
- Highway networks: Highway networks generalize residual connections with learned input-dependent gates, while residual networks correspond to equal flow through both paths when t_i(·) = 0.5.
- Residual networks: The study assumes identity skip connections, although the framework can extend to projection skip connections needed for downsampling.
- Residual networks: Expanding a three-block residual network reveals O(2^n) implicit input-output paths, with each added block doubling the path count.
- Ensembling: Residual networks resemble ensembles because performance changes smoothly with the number of valid paths, paralleling ensemble behavior.
3 The unraveled view of residual networks
The unraveled view rewrites a residual network as exponentially many paths formed by choosing which residual modules to enter or skip. Unlike ordinary feed-forward networks, these paths vary in length and layer composition, motivating questions about their dependence and effectiveness.
- 3 The unraveled view of residual networks: Residual networks contain 2^n paths, each defined by a binary configuration of entering or skipping n residual modules.The expression-tree representation exposes these paths while preserving shared module weights.
- 3 The unraveled view of residual networks: Each residual module receives mixtures of inputs generated by all configurations of preceding modules, rather than only the output of one previous layer.
- 3 The unraveled view of residual networks: Residual-network paths have varying lengths and traverse different subsets of layers, whereas paths in ordinary feed-forward networks have equal length.
- 3 The unraveled view of residual networks: The unraveled representation frames whether paths are dependent, ensemble-like, or differently influential according to their lengths as experimental questions.
4 Lesion study
Lesion studies show that residual networks tolerate deleting or rearranging modules far better than VGG, consistent with many relatively independent paths. Error changes smoothly with path loss or reordering, supporting ensemble-like behavior.
- 4.1 Experiment: Deleting individual layers from neural networks at test time: Deleting any VGG layer reduces CIFAR-10 performance to chance, whereas removing most residual-network blocks causes little change; downsampling blocks have greater impact.ImageNet results are comparable, with downsampling layers tending to matter more.
- 4.1 Experiment: Deleting individual layers from neural networks at test time: A residual layer deletion removes half the 2^n paths but leaves many valid paths, unlike VGG’s single viable input-to-output path.This supports limited dependence among jointly trained residual paths.
- 4.2 Experiment: Deleting many modules from residual networks at test-time: Deleting increasing numbers of residual modules increases error smoothly, matching the performance pattern expected when ensemble members are progressively removed.With k deletions from n modules, the valid-path count decreases to O(2^(n-k)); deleting 10 of 54 blocks leaves 2^44 paths and error around 0.2.
- 4.3 Experiment: Reordering modules in residual networks at test-time: Reordering compatible residual blocks also increases error smoothly as Kendall Tau corruption grows, despite introducing paths not seen during training.This suggests residual networks can be reconfigured to some extent at runtime.
5 The importance of short paths in residual networks
Residual networks contain paths of varying lengths, but gradient contributions decay with path length, making relatively shallow paths the effective paths during training.
- Distribution of path lengths: Paths in a 54-module residual network follow a Binomial length distribution, with more than 95% spanning 19–35 modules.
- Vanishing gradients in residual networks: Gradient magnitude decreases roughly exponentially as the number of modules traversed during backpropagation increases.The experiment isolates paths of length k by propagating through residual modules on selected blocks and skip connections elsewhere.
- The effective paths in residual networks are relatively shallow: 5–17-module paths supply almost all gradient updates despite comprising only 0.45% of all paths.These effective paths are shallow relative to the whole network.
- 5.96% error versus 6.10% for the full model shows that training only on effective paths performs comparably.The difference is not statistically significant, demonstrating that long paths are not needed during training.
6 Discussion
Deleting residual modules mainly removes long paths, while stochastic depth provides only a slight additional resilience benefit because plain residual networks already show limited dependence on individual layers.
- Removing 10 modules causes only a slight performance drop, whereas removing 20 causes a severe drop.After deleting 10 modules, many effective 5–17-module paths remain valid.
- Deleting residual modules mostly removes long paths, preserving many effective short paths even after substantial deletion.
- Connection to highway networks: Highway networks also tend to favor skip connections, reinforcing the observed importance of short paths.
- Effect of stochastic depth training procedure: Stochastic depth changes the training-path distribution toward short paths and encourages different paths to produce good results independently.
- Stochastic depth improves resilience to layer deletion only slightly, indicating that plain residual networks already do not depend strongly on individual layers.
7 Conclusion
The paper reframes residual networks as collections of jointly trained, relatively independent paths and argues that short paths, rather than full-depth gradient flow, explain their trainability.
- Residual networks can be viewed as collections of many paths rather than single ultra-deep networks, and the paths exhibit ensemble-like behavior.
- Only short paths contribute gradient during training; deep paths are not required because they contribute no gradient.
- Residual networks appear to enable very deep models by shortening effective paths rather than preserving gradient flow through the entire depth.The paper presents depth as an open research question.