Source-linked AI summary
Deep Networks with Stochastic Depth
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Weinberger
TL;DR
Very deep networks are expressive but difficult and slow to train because gradients and forward information can diminish. The paper proposes stochastic depth, which randomly bypasses layers during training while retaining full residual-network depth at testing. Experiments report substantially lower training time and test error, including improvements beyond 1000 layers on CIFAR-10, while the method also acts as a regularizer.
Problem
Very deep networks provide high expressiveness but face vanishing gradients, diminishing forward information flow, and long training times.
Method
Stochastic depth randomly removes layers from deep residual networks during training, creating a shorter expected network while retaining full depth at testing.
Results
Stochastic depth substantially reduces training time and test error, with significant CIFAR-10 test-error improvements when ResNets exceed 1000 layers.
Takeaways & Limitations
Stochastic depth can train residual networks at depths beyond 1000 layers while maintaining meaningful test-error improvements and acting as a regularizer with Batch Normalization.
Takeaways & Limitations
On the particular ImageNet architecture discussed, stochastic depth showed no immediate benefit; improvements may require larger models.
Abstract
from arXiv · showhide
Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10).
1 Introduction
Very deep CNNs offer greater expressiveness but suffer from vanishing gradients, diminishing forward information flow, and long training times. Stochastic depth addresses this tension by training shortened residual networks while retaining full depth at testing, reducing training time and test error.
- Very deep CNNs increase model expressiveness but introduce vanishing gradients, diminishing feature reuse, and long training times.
- Short networks train efficiently but lack sufficient expressiveness, whereas very deep networks are difficult and time-consuming to train.
- Stochastic depth randomly removes layers from deep residual networks during training, producing a small expected training depth and full testing depth.
- Stochastic depth substantially reduces training time and test error in extensive experiments.
- The method acts as a regularizer even when combined with Batch Normalization.
- Beyond 1000 layers, stochastic depth still produces significant improvements in CIFAR-10 test error.
2 Background
Prior approaches improve information flow and optimization in very deep networks through normalization or skip connections. Stochastic depth complements these methods by randomly shortening residual networks during training and implicitly ensembling networks of different depths.
- Batch Normalization standardizes hidden-layer means and variances within each mini-batch to reduce vanishing gradients.
- Highway Networks use parameterized skip connections to let earlier representations flow through multiple layers.
- ResNets simplify Highway Networks with mostly identity shortcuts, improving training efficiency and enabling more direct feature reuse.
- Residual propagation passes gradients and features between layers through identity transformations, with linear projections when dimensions differ.
- Dropout randomly removes hidden-node or connection activity and can be viewed as training an ensemble of smaller networks.
- Stochastic depth shortens networks rather than thinning them and implicitly trains networks with different depths.
- Stochastic depth is presented as complementary to prior methods and effective on ResNets with Batch Normalization.
3 Deep Networks with Stochastic Depth
Stochastic depth randomly bypasses residual blocks during training, shortening the expected network while retaining the full residual architecture at test time. The method uses survival probabilities, including a linearly decaying schedule, and implicitly ensembles subnetworks through shared weights.
- ResNet architecture: Residual blocks use convolutional transformations and identity shortcuts, with linear projections when output dimensions do not match.The construction is based on residual blocks, typically using Conv-BN-ReLU-Conv-BN sequences.
- Stochastic depth: Stochastic depth randomly drops entire ResBlocks during training and bypasses their transformations through identity skip connections.Each block is active according to a Bernoulli variable with survival probability pℓ.
- Survival probabilities: The linear survival schedule starts at p0 = 1 for the input and decays to pL for the last ResBlock.The authors compare uniform and decaying assignments and prefer the linear rule with pL = 0.5.
- Expected network depth: With pL = 0.5, the expected training depth is approximately 3L/4; a 54-block network therefore averages about 40 active ResBlocks.The full-depth network is recovered at test time despite the shorter expected training network.
- Training efficiency: Bypassing blocks reduces forward-backward computation and yields approximately 25% training-time savings under the linear decay rule with pL = 0.5.The reported practical timings are consistent with this calculation.
- Implicit model ensemble: Stochastic depth implicitly trains an ensemble of 2^L shared-weight ResNet subnetworks and combines them at test time by weighting layers by survival probability.All functions remain active during testing, with outputs recalibrated according to each function’s training survival probability.
4 Results
Across CIFAR-10, CIFAR-100, SVHN, and ImageNet, stochastic depth generally improves error while reducing training time, including gains for a 1202-layer ResNet. ImageNet is an important boundary: the reported benefit required additional training and was not immediate on the evaluated architecture.
- CIFAR-10: 18% relative improvement reduces CIFAR-10 test error from 6.41% to 5.25% with stochastic depth.The stochastic-depth result was lower than the previously reported 6.05% best single-model performance without massive data augmentation.
- CIFAR-100: CIFAR-100 test error falls from 27.22% with constant depth to 24.98% with stochastic depth.The result was reported as the best published single-model performance to the authors’ knowledge.
- CIFAR-10 and CIFAR-100: 15% improvements on both unaugmented CIFAR-10 and CIFAR-100 yield test errors of 11.66% and 37.8%, respectively.The corresponding constant-depth errors were 13.63% and 44.74%.
- SVHN: 1.75% SVHN error with stochastic depth improves on the 1.80% baseline and is reported as the second-best published result.The 152-layer baseline began overfitting during the second learning-rate phase, whereas stochastic depth improved the error.
- Training efficiency and ImageNet: 25% training-time savings are reported consistently across benchmark datasets with stochastic depth.The savings permit 120 ImageNet epochs in approximately the time used for 90 baseline epochs, reaching 21.98% error versus 23.06% at baseline epoch 90.
- 1202-layer ResNet: 4.91% CIFAR-10 test error results when stochastic depth scales the architecture from 110 to 1202 layers.The 1202-layer stochastic-depth model improves on the previous 5.25% record-low result without signs of overfitting; the constant-depth 1202-layer model reaches 6.67%.
5 Analytic Experiments
The analytic experiments examine whether stochastic depth strengthens gradients and how its survival-probability schedule affects accuracy, depth, and training time. They find larger gradients, robust performance under linear decay, and stronger benefits for deeper networks.
- Improved gradient strength: Stochastic depth produces larger gradients in the first convolutional layer, especially after scheduled learning-rate reductions.This supports reduced vanishing-gradient effects during training.
- Improved gradient strength: The method combines shortened training networks with deep test-time models, and its performance improves after the second learning-rate reduction.This observation links the gradient-strength findings to the train-short, test-deep design.
- Hyper-parameter sensitivity: Both survival-probability assignment rules outperform the baseline when pL is chosen properly, while linear decay consistently outperforms uniform assignment.The linear-decay rule remains competitive across pL values from 0.4 to 0.8.
- Hyper-parameter sensitivity: A linear-decay survival probability of pL = 0.2 still performs well while reducing training time by 40%.The result indicates substantial savings without compromising accuracy under this setting.
- Hyper-parameter sensitivity: Deeper networks perform best with pL = 0.5, and sufficiently deep models are needed for stochastic depth to significantly outperform the baseline.Shorter networks can still benefit from less aggressive skipping.
6 Conclusion
The paper introduces stochastic depth as a practical procedure for training very deep neural networks efficiently. It reduces expected depth during training while preserving full depth at test time, enabling networks beyond 1000 layers to reduce test error.
- 6 Conclusion: Stochastic depth reduces network depth during training in expectation while maintaining the full depth at testing time.The procedure is presented as a way to train very deep networks effectively and efficiently.
- 6 Conclusion: Networks exceeding 1000 layers can still achieve reduced test error when trained with stochastic depth.The conclusion frames this as evidence that the method can scale residual networks to previously unattainable depths.
- 6 Conclusion: The authors hope stochastic depth becomes a practical deep-learning tool for scaling models to greater depths and capabilities.This is stated as a prospective implication of the method’s simplicity and practicality.