Source-linked AI summary
Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks
Li Shen, Zhouchen Lin, Qingming Huang
TL;DR
Deep CNNs do not reliably benefit from simply adding layers because long backward paths degrade effective supervision. The paper proposes Relay Backpropagation, which bounds loss-specific gradient propagation through segmented networks, and reports first place in ILSVRC 2015 plus gains across architectures and datasets.
Problem
Simply increasing CNN depth can create optimization difficulties, while long backward propagation reduces information about the loss reaching lower layers.
Method
Relay Backpropagation segments the network, adds auxiliary output modules, and limits each loss gradient to at most N consecutive layers.
Results
Relay BP achieved first place in the ILSVRC 2015 Scene Classification Challenge and improved top-5 accuracy by 0.44% on ResNet-50 and 0.46% on Inception-v3.
Takeaways & Limitations
Experiments with four architectures on two large-scale datasets indicate that Relay BP is not restricted to a particular architecture or dataset.
Takeaways & Limitations
Auxiliary-branch placement is selected heuristically, and adding branches increases training computation; the experiments use one branch because the networks are not extremely deep.
Abstract
from arXiv · showhide
Learning deeper convolutional neural networks becomes a tendency in recent years. However, many empirical evidences suggest that performance improvement cannot be gained by simply stacking more layers. In this paper, we consider the issue from an information theoretical perspective, and propose a novel method Relay Backpropagation, that encourages the propagation of effective information through the network in training stage. By virtue of the method, we achieved the first place in ILSVRC 2015 Scene Classification Challenge. Extensive experiments on two challenging large scale datasets demonstrate the effectiveness of our method is not restricted to a specific dataset or network architecture. Our models will be available to the research community later.
1 Introduction
Deeper CNNs can improve accuracy, but simply adding layers creates optimization difficulties and may reduce performance. Relay Backpropagation addresses this by shortening backward information paths, and achieves first place in the ILSVRC 2015 Scene Classification Challenge.
- Motivation: Deeper architectures can improve classification accuracy, but increasing depth also enlarges parameter size and model complexity.Deeper networks face divergence, slower convergence, and overfitting risks.
- Motivation: Empirical results show that accuracy does not necessarily improve by simply adding layers, with deeper networks even declining on Places2.The cited comparison concerns ImageNet and preliminary Places2 experiments.
- Motivation: Information about target outputs diminishes during propagation even when gradients do not vanish, and the degradation amplifies with network depth.The paper argues that error information should not travel too many layers backward.
- Contribution: Relay Backpropagation divides the network into segments, adds interim output modules, and limits each loss gradient to at most N consecutive layers.The method shortens paths from outputs to lower layers while restraining less relevant information propagated through too many layers.
- Results: Relay BP achieved first place in the ILSVRC 2015 Scene Classification Challenge and was also evaluated on ImageNet with two other architectures.The Places2 challenge dataset involved 401 classes and more than 8 million training images.
2 Related Work
Prior work improves deep-network training through initialization, activations, optimizers, temporary classifier branches, and architectural shortcuts. Relay Backpropagation is presented as complementary because it addresses information reduction and long-term gradient propagation.
- Existing approaches: Prior approaches address deep-network optimization through pretraining, refined initialization, new activation functions, and improved optimizers.Examples include momentum-based stochastic gradient descent and RMSProp.
- Auxiliary supervision: Temporary classifier branches propagate supervision to lower layers, but standard multi-loss training does not address information reduction from long-term propagation.The paper also identifies adverse effects from less relevant information reaching lower layers.
- Architectural shortcuts: Inception and Residual structures use shortcut connections, yet long-term propagation remains when training deeper networks.The paper describes Relay Backpropagation as orthogonal to these architectural contributions.
- Positioning: Relay Backpropagation is proposed as complementary to prior structures and can further improve performance alongside them.The paper states that its method preserves relevant information while restraining less relevant information.
3 Standard BP and Information Reduction
Standard backpropagation sends error information through all preceding layers, creating long paths whose transmitted supervision can lose information. The paper frames this degradation using the Data Processing Inequality.
- Standard BP: Standard BP propagates gradients from the loss backward through intermediate layers to the first convolutional layer.Each iteration combines forward propagation with this full backward propagation process.
- Standard BP: As network depth grows, error information received by lower layers has flowed through more intermediate layers.The number of traversed layers increases with network depth.
- Information reduction: The gradient sequence y → x → u forms an information-flow Markov chain, so the Data Processing Inequality gives I(y; x) ≥ I(y; u).Here y is the starting supervision gradient, while x and u are gradients transmitted to successive layers.
- Information reduction: Repeated transformations tend to reduce loss-related information reaching lower layers, and this effect becomes stronger as propagation goes deeper.The paper therefore argues that effective parameter updates require limiting how many layers receive backward information.
4 Relay BackPropagation
Relay Backpropagation segments a deep network and assigns auxiliary and primary losses to bounded portions of layers. Overlapping loss flows relay information between segments while auxiliary branches are removed during testing.
- Network segmentation: The method divides a VGGNet-19-style network into five segments separated by max-pooling layers.Segments are numbered from lower to higher layers.
- Bounded propagation: Auxiliary output modules are attached to intermediate segments, and each loss propagates through at most N consecutive layers.Different losses therefore supervise different parts of the weight layers.
- Bounded propagation: In the example, auxiliary loss 1 reaches segment 1, auxiliary loss 2 reaches segment 3, and the primary loss reaches the lowest layer in segment 4.The flows are assigned distinct backward paths through the segmented network.
- Relay mechanism: Overlapping loss flows are fused at intermediate segments, allowing a segment to relay information between primary and auxiliary losses.The paper associates this mechanism with easier lower-layer optimization and better higher-layer discrimination.
- Method characteristics: Relay BP assigns different losses to different layer regions and overlaps their flows to reduce relevant-information degradation and long-term irrelevant propagation.These are the method’s two stated distinguishing characteristics.
- Training flow: During forward propagation, activations still move from input to output through adjacent layers as in standard BP with auxiliary branches.The modification concerns backward information flow rather than forward transmission.
- Testing: Auxiliary branches are ignored during testing, so Relay BP adds no parameter-size or time-expense cost at inference.Auxiliary supervision is introduced only during training.
- Design considerations: Branch placement is chosen heuristically, should avoid overly low layers, and depends on network depth and practical requirements.The paper notes that adding branches increases training computation and recommends using as few as possible.
5 Experiments
The experiments evaluate Relay BP against standard backpropagation and multi-loss standard BP on Places2 and ImageNet using multiple network architectures.
- Relay BP is evaluated on Places2 and ImageNet 2012 classification datasets across four different network architectures.
- The standard BP baseline uses traditional forward and backward propagation, while multi-loss standard BP adds an auxiliary output branch to intermediate layers.
- The training-stage architecture is identical for Relay BP and multi-loss standard BP; their difference is the backward information-propagation scheme.
- Places2 contains 401 scene categories, 8.1M training images, 20K validation images, and 381K testing images.
- The evaluated architectures include a VGGNet-19-based model and a model using 7 × 7 convolutions with a modified inception module.
Network Architectures.
The experiments address the learning challenges of large, imbalanced Places2 data and compare error rates for Relay BP under the reported network designs and sampling procedure.
- Places2 has more than 8M training images, with class sizes ranging from 4,000 to 30,000, creating large-scale and class-imbalance challenges.
- Table 3 reports single-crop error rates on the Places2 validation set, including improvements over the standard BP baseline.
- Table 4 reports single-model error rates on the Places2 validation set, also expressing improvements relative to standard BP.
- Class-aware sampling fills mini-batches as uniformly as possible across classes and avoids repeatedly presenting the same example and class order.
- The implementation uses Caffe, the Places2 training set without additional data, random 224×224 crops, horizontal flips, and color shifts.
Training and Testing.
Testing compares single-crop, single-model, and competition settings, showing Relay BP’s reported advantage over baselines and limited additional gain from broader evaluation configurations.
- Table 5 reports ILSVRC 2015 Scene Classification competition top-5 error rates on the Places2 test set as returned by the test server.
- Fully convolutional testing averages scores across image scales whose shorter sides are 224, 256, 320, 384, and 448 pixels.
- Relay BP consistently outperforms standard BP and multi-loss standard BP, by approximately 1.0% on model A and 0.8% on model B in top-5 measure.
- The single-model gain over center crop is about 1.5% top-5 error, decreasing from 17.83% single crop to 16.33%.
ILSVRC 2015 Scene Classification Challenge.
Relay BP achieved first place in the ILSVRC 2015 Scene Classification Challenge, while the reported examples and error analysis highlight both successful recognition and the complexity of scene labels.
- The WM team won first place, with five entries occupying the challenge’s top five places.
- The reported validation result is 15.74% top-5 error, while testing reaches 16.87%, roughly 1.1% worse.
- The method successfully classifies example Places2 validation images while listing each ground-truth label and five predictions in descending confidence order.
- Incorrect predictions can remain reasonable because scene images may have multiple labels and contain complex combinations of objects and places.
- The competition’s single-model results are reported separately for ImageNet 2012 classification in Table 6.
5.2 ImageNet 2012 Classification
The study evaluates Relay BP on ImageNet 2012, a 1000-class benchmark with separate training, validation, and testing sets. Performance is measured using top-1 and top-5 error rates without additional data.
- ImageNet 2012 contains 1,000 classes, with 1.2M training, 50K validation, and 100K testing images.
- Classification performance is measured using top-1 and top-5 error rates.
- The models are trained using the provided dataset without additional data.
Configurations.
Relay BP is evaluated with ResNet-50 and Inception-v3, while comparisons account for implementation differences between reported and reimplemented results. It improves both top-1 and top-5 performance over standard BP for both architectures.
- The evaluation uses the 50-layer residual network and Inception-v3 architectures, without scale jitter augmentation during training.
- Table 6 compares reported results with reimplementations, whose slight differences are attributed mainly to implementation details described in the configurations.
- Relay BP improves ResNet-50 performance by 0.44% on top-5 and 0.91% on top-1 measures over standard BP.
- Relay BP improves Inception-v3 performance by 0.46% on top-5 and 0.66% on top-1 measures over standard BP.
6 Conclusion
The conclusion presents Relay Backpropagation as a method for preserving relevant information during backward propagation while restraining less relevant information. Experiments across four architectures and two large-scale datasets support effectiveness beyond one architecture or dataset.
- Relay Backpropagation encourages effective-information transmission during backward propagation in deep convolutional neural networks.
- The method preserves relevant information and restrains the adverse effect of less relevant information.
- Experiments with four architectures on two challenging large-scale datasets demonstrate effectiveness beyond a specific architecture or dataset.
- The authors identify theoretical and mathematical support for Relay BP as a future research direction.