PaperlayerView exact revision on arXiv

Source-linked AI summary

Identity Mappings in Deep Residual Networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

arXiv:1603.05027v3cs.CVcs.LG

TL;DR

Deep residual networks lack a clear account of how information propagates through their depth and how skip-path design affects optimization. This paper analyzes identity mappings, proposes a pre-activation residual unit, and reports easier training with improved accuracy, including 20.7% error for a 200-layer ImageNet model, 1.1% below its baseline.

  • Problem

    The paper examines how residual-network connection and activation mappings affect direct forward and backward information propagation and training.

  • Method

    The paper derives propagation formulations for identity skip connections and after-addition activation, then evaluates a pre-activation residual unit through ablations and deep-network experiments.

  • Results

    The proposed identity-mapping design enables easily trained 1000-layer networks with improved accuracy, including 20.7% ImageNet error for ResNet-200, 1.1% below baseline.

  • Takeaways & Limitations

    Identity shortcut connections and identity after-addition activation are essential for smooth information propagation, while cleaner information paths ease optimization.

  • Takeaways & Limitations

    The propagation derivations do not strictly apply to the few units that increase dimensions or reduce feature-map sizes, where the skip mapping is not identity.

Abstract

from arXiv · show

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers

1 Introduction

The paper analyzes how identity mappings create direct forward and backward information paths across deep residual networks. It proposes a pre-activation residual unit that facilitates training and improves generalization, supported by ablation experiments and deep-network results.

  • 1 Introduction: ResNets learn additive residual functions F relative to identity skip mappings h(x_l) = x_l.Residual units stack many blocks, with F as the residual function and ReLU as the activation in the original formulation.
  • 1 Introduction: Identity mappings for both h(x_l) and f(y_l) enable direct signal propagation between any residual units during forward and backward passes.The analysis focuses on creating a direct path through the entire network, not only within individual residual units.
  • 1 Introduction: Scaling, gating, and 1×1-convolution skip connections produce higher training loss and error than identity skip connections.The experiments suggest that maintaining a clean information path eases optimization.
  • 1 Introduction: Reordering ReLU and batch normalization as pre-activation of the weight layers constructs an identity mapping f(y_l) = y_l.This design contrasts with conventional post-activation and yields the proposed residual unit.
  • 1 Introduction: A 1001-layer ResNet using the proposed unit achieves competitive CIFAR-10/100 results, is easier to train, and generalizes better than the original ResNet.The paper also reports improved results on ImageNet.

2 Analysis of Deep Residual Networks

The analysis shows that identity skip connections and after-addition activation enable direct forward and backward signal propagation between residual units. This residual formulation represents deep features as sums of residual outputs and supports gradient propagation even when weights are small.

  • Residual Unit formulation: The original Residual Unit stacks same-shape blocks, using an identity skip mapping h(x_l) = x_l and ReLU after addition.Each unit contains K = 2 or 3 layers in the original design, with F denoting the residual function.
  • Forward propagation: With identity after-addition activation, any deeper feature x_L equals a shallower feature x_l plus accumulated residual functions.The residual relationship holds recursively between any deeper unit L and shallower unit l.
  • Forward propagation: Deep features are sums of all preceding residual-function outputs plus x_0, unlike plain networks’ successive matrix-vector products.This gives the network a residual form between units rather than a purely compositional sequence of transformations.
  • Backward propagation: The gradient to a shallower unit decomposes into a direct term and a weight-mediated term, enabling information to propagate backward without traversing weight layers.The direct additive term makes cancellation unlikely across a mini-batch, so gradients do not vanish even when weights are arbitrarily small.
  • Role of identity mappings: Direct forward and backward propagation depends on two identity mappings: the skip connection h(x_l) = x_l and identity after-addition activation.The derivation does not strictly apply to units that increase dimensions or reduce feature-map sizes, which are few in the cited architectures.

3 On the Importance of Identity Skip Connections

Identity skip connections provide the most direct information paths in very deep residual networks, whereas scaling, gating, 1×1 convolutions, or dropout can impede propagation and create optimization problems. Experiments on ResNet-110 show that these alternatives generally worsen convergence or test error, while near-identity gating performs closer to the baseline.

  • Propagation analysis: In extremely deep networks, shortcut scaling can make the backpropagated signal exponentially large or small, blocking shortcut propagation and causing optimization difficulties.The multiplicative factor grows when λ_i > 1 and vanishes when λ_i < 1.
  • CIFAR-10 experiments: 12.35% test error results from constant scaling with λ = 0.5 and scaling F by 0.5, substantially exceeding the original ResNet-110 baseline.Leaving F unscaled does not converge well, while scaling it enables convergence but still produces higher training error.
  • CIFAR-10 experiments: 12.86% test error results from shortcut-only gating initialized with b_g = 0, whereas b_g = −6 yields 6.91%, much closer to the ResNet-110 baseline.A very negative bias makes 1−g(x) closer to 1, so the shortcut nearly becomes an identity mapping.
  • CIFAR-10 experiments: 12.22% test error results from 1×1 convolutional shortcuts in the 110-layer ResNet, despite their stronger representational ability and coverage of identity shortcuts.The higher training error indicates degradation from optimization issues rather than insufficient representational capacity.
  • CIFAR-10 experiments: Dropout shortcuts fail to converge to a good solution because their expected shortcut scale of 0.5 impedes signal propagation, similarly to constant scaling.The dropout ratio used is 0.5.
  • Propagation analysis: Multiplicative transformations such as scaling, gating, 1×1 convolutions, and dropout on shortcuts hamper information propagation and lead to optimization problems.These transformations alter the most direct paths available for signal propagation.

4 On the Usage of Activation Functions

The section investigates how rearranging ReLU and BN can make the after-addition activation an identity mapping. Full pre-activation improves optimization and regularization, yielding consistently better results than baseline counterparts across evaluated architectures and datasets.

  • Activation-function designs: Rearranging ReLU and BN targets an identity after-addition activation, replacing the original post-addition ReLU design.The original unit applies BN after each weight layer and places the final ReLU after element-wise addition.
  • Activation-function designs: 7.84% error results from moving ReLU before addition, worse than the 5.93% CIFAR-10 baseline.This design constrains the residual transform to nonnegative outputs and makes the forward signal monotonically increasing.
  • Pre-activation variants: Pre-activation makes the after-addition activation an identity mapping and is equivalent to using the asymmetric activation as the next unit’s pre-activation.This asymmetric formulation preserves a direct forward path and supports a corresponding backward formulation.
  • Pre-activation variants: Full pre-activation places both BN and ReLU before weight layers, whereas ReLU-only pre-activation performs similarly to the baseline.The ReLU-only design may not receive BN’s benefits because its ReLU is not paired with a BN layer.
  • Effects of pre-activation: Pre-activation improves optimization through identity mappings and improves regularization through BN used as a pre-activation.The models are reported as consistently better than baseline counterparts across ResNet-110, ResNet-164, ResNet-110(1layer), ResNet-1001, CIFAR-10, and CIFAR-100.
  • Effects of pre-activation: Pre-activation lowers test error despite slightly higher converged training loss, an effect attributed to BN’s regularization.This phenomenon is observed on ResNet-110, ResNet-110(1-layer), and ResNet-164 for both CIFAR-10 and CIFAR-100.

5 Results

The proposed pre-activation residual units achieve competitive CIFAR-10/100 results and improve ImageNet performance, especially for deeper ResNets. These gains come with computational cost that grows linearly with depth.

  • CIFAR-10/100 Results: Going deeper produces competitive CIFAR-10/100 results without specially tailoring network width, filter sizes, or using regularization techniques.The authors attribute these results to the simple but essential concept of increasing depth.
  • ImageNet Results: 24.6%/7.5% validation error for the “BN after addition” ResNet-101 exceeds the original ResNet-101’s 23.6%/7.1%.The errors are single-crop 224×224 validation results, consistent with CIFAR optimization findings.
  • ImageNet Results: 21.1% top-1 error for the pre-activation ResNet-152 improves on the original ResNet-152’s 21.3% on a 320×320 crop.The gain is small because the original ResNet-152 had not shown severe generalization difficulties.
  • ImageNet Results: 20.7% error for the pre-activation ResNet-200 is 1.1% lower than the baseline ResNet-200 and lower than both ResNet-152 versions.The original ResNet-200 had lower training error than ResNet-152, suggesting overfitting; with augmentation, the pre-activation model outperformed Inception v3.
  • Computational Cost: ∼10× computational complexity separates a 1001-layer network from a 100-layer network; ResNet-1001 trains about 27 hours on 2 GPUs.On ImageNet, ResNet-200 takes about 3 weeks to train on 8 GPUs, comparable to VGG networks.

6 Conclusions

The paper concludes that identity shortcut connections and identity after-addition activation enable smooth information propagation in deep residual networks. Ablations support these derivations, while 1000-layer networks can be trained easily and achieve improved accuracy.

  • 6 Conclusions: Identity shortcut connections and identity after-addition activation are essential for smooth information propagation.This conclusion follows from the paper’s propagation derivations.
  • 6 Conclusions: Ablation experiments demonstrate phenomena consistent with the propagation derivations.
  • 6 Conclusions: 1000-layer deep networks can be trained easily and achieve improved accuracy.

Appendix: Implementation Details

The appendix specifies ImageNet training settings and implementation details for pre-activation and bottleneck residual units. It also describes special handling of network endpoints and projection shortcuts when feature-map sizes change.

  • ImageNet training: ImageNet training uses a 0.1 learning rate without warmup, divides it by 10 at 30 and 60 epochs, and uses batches of 256 across 8 GPUs.Each GPU processes 32 examples; weight decay, momentum, and initialization match the preceding settings.
  • Pre-activation endpoints: Pre-activation networks apply an activation after conv1 before path splitting and add an extra activation after the final unit’s element-wise addition.These choices address the first and last residual units, which neighbor the standalone convolution and classifier, respectively.
  • Bottleneck units: CIFAR bottleneck units in ResNet-164/1001 follow the construction used in the cited baseline.The supplied passage introduces this construction but is truncated before giving its example.
  • Bottleneck shortcuts: Bottleneck ResNets use projection shortcuts to increase dimensions when reducing feature-map size, with pre-activation also applied to those shortcuts.The implementation compares ResNet-110 and ResNet-164 units with roughly the same parameter count.
Loading 1603.05027v3…