Deep Residual Learning for Image Recognition

1. Introduction

• Normalized initialization [37] [13] and intermediate normalization layers [16] largely addressed vanishing/exploding gradients, enabling networks with tens of layers to begin converging.This progress motivates asking whether simply stacking more layers makes learning better networks easier.Source: §1 · p. 1
• Adding layers can increase training error [11] [42] and degrade accuracy; this degradation is not caused by overfitting.A deeper model could preserve a shallower model’s training error by adding identity layers. Available solvers nevertheless fail to find comparably good solutions, or cannot do so in feasible time.Source: §1 · pp. 1–2
• Residual learning replaces direct fitting of H(x) with F(x) := H(x)−x, reconstructing the desired mapping as F(x)+x through identity shortcuts.The authors hypothesize that residual mappings are easier to optimize, particularly when the desired mapping is identity. Identity shortcuts add neither parameters nor computational complexity and permit end-to-end SGD training without solver modifications.Source: §1 · p. 2
• Over 100 layers were successfully trained in residual networks on CIFAR-10 [20], and models exceeding 1000 layers were explored.On ImageNet [36], residual networks are easy to optimize and gain accuracy with increased depth, while deeper plain counterparts exhibit higher training error. Similar phenomena appear on CIFAR-10.Source: §1 · p. 2
• 3.57% top-5 error on the ImageNet test set earned the residual-network ensemble first place in ILSVRC 2015 classification.The 152-layer residual network is reported as the deepest network presented on ImageNet, while having lower complexity than VGG nets [41].Source: §1 · p. 2
• First-place finishes also extended to ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation in the ILSVRC and COCO 2015 competitions.These results demonstrate that the extremely deep representations generalize beyond image classification.Source: §1 · p. 2

Residual Representations

• VLAD and Fisher Vector [30] are residual-based shallow representations used for image retrieval and classification.Fisher Vector [30] can be formulated as a probabilistic version of VLAD. For vector quantization, encoding residual vectors [17] is more effective than encoding original vectors.Source: §2 · p. 2
• Multigrid and hierarchical basis preconditioning exploit residuals between scales, converging faster than residual-unaware standard solvers and suggesting reformulation can simplify optimization.Multigrid assigns residual solutions to multiscale subproblems; hierarchical basis preconditioning uses variables representing residual vectors between two scales.Source: §2 · p. 2

Shortcut Connections

• Earlier shortcuts include input-to-output MLP links, auxiliary classifiers [24], response/gradient/error centering, and inception branches.Auxiliary classifiers [24] address vanishing/exploding gradients, while inception combines a shortcut branch with deeper branches.Source: §2 · p. 2
• Unlike highway networks [42] [43] with parameterized, data-dependent gates, this work uses parameter-free identity shortcuts that always transmit information and learn residual functions.Closed highway shortcuts instead produce non-residual functions; highway networks [42] [43] had not demonstrated accuracy gains with extremely increased depth, such as over 100 layers.Source: §2 · pp. 2–3

3.1. Residual Learning

• Residual learning makes stacked layers approximate F(x) = H(x) − x rather than directly approximate the target mapping H(x).When identity mappings are optimal, residual-layer weights can approach zero instead of requiring nonlinear layers to learn identity mappings directly. Experimentally small residual responses suggest that identity mappings provide reasonable preconditioning.Source: §3.1 · p. 3

3.2. Identity Mapping by Shortcuts

• Identity shortcuts add the input to a learned residual without extra parameters, enabling matched plain/residual comparisons apart from negligible element-wise addition.Residual branches use two or three layers; single-layer branches showed no advantages. For convolutional branches, feature maps are added channel by channel, with ReLU applied after addition.Source: §3.2 · p. 3

3.3. Network Architectures

• 3.6 billion FLOPs make the 34-layer plain baseline 18% as costly as VGG-19 [41], which requires 19.6 billion FLOPs.Inspired by VGG nets [41], the baseline mostly uses 3×3 filters, keeps filter counts fixed at each spatial resolution, and doubles them when stride-2 convolutions halve resolution. Global average pooling precedes a 1000-way fully connected softmax layer.Source: §3.3 · p. 3, Figure 3 · p. 4
• Residual architectures insert shortcuts into the plain baseline, using identity mappings for matching dimensions and zero-padding or projections when dimensions increase.Zero-padding adds no parameters, while projection shortcuts use 1×1 convolutions. Both options use stride 2 when crossing feature maps of different spatial sizes.Source: §3.3 · p. 4

3.4. Implementation

• 256-image SGD mini-batches train all ImageNet models from scratch, with batch normalization [16] after each convolution and before activation.Weights are initialized as in [13]; the learning rate starts at 0.1 and falls tenfold on plateaus, training lasts up to 60 × 10^4 iterations, weight decay is 0.0001, momentum is 0.9, and dropout [14] is omitted following [16]. Augmentation randomly samples the shorter side in [256, 480] [41], takes 224×224 crops with horizontal flipping and mean subtraction, and applies color augmentation.Source: §3.4 · p. 4
• 10-crop testing supports comparison studies; best results use fully convolutional evaluation [41] [13] with scores averaged across multiple scales.For multiscale evaluation, the image’s shorter side is resized to 224, 256, 384, 480, or 640.Source: §3.4 · p. 4

4.1. ImageNet Classification

• 3.5% top-1 error reduction distinguishes the 34-layer ResNet from its plain counterpart, demonstrating the benefit of residual learning without extra parameters.The 34-layer plain network has higher training error than its 18-layer counterpart, whereas ResNet-34 improves over ResNet-18 by 2.8% and lowers training error. At 18 layers, plain and residual networks have comparable accuracy, but ResNet converges faster.Source: Table 2 · p. 5, §4.1 · pp. 5–6, Figure 4 · p. 5
• 3.57% top-5 test error wins ILSVRC 2015 for a six-model ResNet ensemble; the 152-layer single model achieves 4.49% top-5 validation error.The single-model result outperforms all previous ensemble results.Source: Table 5 · p. 6, Table 4 · p. 6, §4.1 · p. 7
• Identity shortcuts address degradation without projections and keep bottleneck models efficient; replacing bottleneck identity shortcuts with projections doubles time complexity and model size.All shortcut options outperform plain networks: projections only for dimension increases slightly improve over zero-padding, while projecting every shortcut adds marginal gains and is discarded for efficiency. Bottlenecks use 1×1, 3×3, and 1×1 convolutions to reduce then restore dimensions for economy; non-bottleneck ResNets also benefit from depth, while plain bottleneck networks still degrade.Source: §4.1 · p. 6
• 11.3 billion FLOPs for ResNet-152 remains below VGG-16/19’s 15.3/19.6 billion FLOPs, while 50/101/152-layer ResNets improve accuracy over 34-layer models without degradation.The accuracy gains from increased depth hold across all reported evaluation metrics.Source: §4.1 · p. 7

4.2. CIFAR-10 and Analysis

• 6.43% CIFAR-10 test error places ResNet-110 among state-of-the-art results, with fewer parameters than FitNet [35] and Highway [42].Across 20–56 layers, deeper plain networks have higher training error, while residual networks gain accuracy with depth. ResNet-110 converges well after warming up at learning rate 0.01 until training error falls below 80%, then switching to 0.1.Source: Table 6 · p. 7, §4.2 · pp. 7–8
• 7.93% CIFAR-10 test error for ResNet-1202 is worse than ResNet-110 despite training error below 0.1% and no optimization difficulty.The authors attribute the gap to overfitting: the 19.4M-parameter network may be unnecessarily large for CIFAR-10. They use neither maxout nor dropout and leave stronger regularization for future study.Source: Table 6 · p. 7, §4.2 · p. 8
• Residual networks have smaller layer responses than plain networks, and deeper ResNets modify signals less per layer, supporting residual functions being closer to zero.Response magnitudes are measured using standard deviations after batch normalization and before ReLU/addition; comparisons include ResNet-20, ResNet-56, and ResNet-110.Source: §4.2 · p. 8

4.3. Object Detection on PASCAL and MS COCO

• 6.0% higher COCO [26] mAP@[.5, .95] (28% relative improvement) follows replacing VGG-16 [41] with ResNet-101 in otherwise identical Faster R-CNN [32].The detector comparison also covers PASCAL VOC 2007 and 2012. Residual nets won the 2015 ImageNet detection/localization and COCO detection/segmentation competition tracks.Source: Table 8 · p. 8, §4.3 · p. 8

A. Object Detection Baselines

• ResNet uses per-region conv5_x layers in place of VGG-16’s hidden fully connected layers, following “Networks on Conv feature maps” (NoC) [33].RoI pooling precedes conv5_1; the final classification layer is replaced by sibling classification and box-regression layers.Source: A. Object Detection Baselines · p. 10
• 16-pixel-stride feature maps from conv1 through conv4_x are shared by the region proposal network (RPN) [32] and Fast R-CNN detection network.These stages contain 91 convolutional layers in ResNet-101 and match VGG-16’s feature-map stride; the RPN generates 300 proposals [32].Source: A. Object Detection Baselines · p. 10
• Batch-normalization means and variances are computed on ImageNet after pretraining, then frozen during detection fine-tuning primarily to reduce memory consumption.The frozen layers act as linear activations with constant offsets and scales; fine-tuning does not update their statistics.Source: A. Object Detection Baselines · p. 10

• >3% mAP improvement over VGG-16 is achieved by ResNet-101, solely through its improved learned features.Faster R-CNN training uses the same hyper-parameters as [32].Source: PASCAL VOC · p. 10
• 5k VOC 2007 trainval images and 16k VOC 2012 trainval images form “07+12” training for VOC 2007 testing, following [32].Source: PASCAL VOC · p. 10
• 10k VOC 2007 trainval+test images and 16k VOC 2012 trainval images form “07++12” training for VOC 2012 testing, following [32].Source: PASCAL VOC · p. 10

MS COCO

• 80 object categories are evaluated on MS COCO [26], using 80k training images and 40k validation images.Evaluation uses the PASCAL VOC metric (mAP @ IoU = 0.5) and the standard COCO metric (mAP @ IoU = .5:.05:.95).Source: MS COCO · p. 10
• 8 GPUs train a COCO detection system similar to the PASCAL VOC system, with RPN and Fast R-CNN mini-batches of 8 and 16 images, respectively.Both stages train for 240k iterations at a learning rate of 0.001, followed by 80k iterations at 0.0001.Source: MS COCO · p. 10
• 6% absolute improvement in mAP@[.5, .95] over VGG-16, equivalent to 28% relative improvement, is achieved by ResNet-101 solely through better learned features.The absolute increase in mAP@[.5, .95] (6.0%) nearly matches that in mAP@.5 (6.9%), suggesting improvements in both recognition and localization.Source: MS COCO · p. 10

• The competition improvements are based on deep features.Source: B. Object Detection Improvements · p. 10
• Residual learning is expected to benefit these competition improvements.Source: B. Object Detection Improvements · p. 10
• The section reports improvements made for the competitions for completeness.Source: B. Object Detection Improvements · p. 10

Box refinement

• About 2 points of mAP improvement come from box refinement, which re-pools features from regressed boxes to obtain new classification scores and box regressions.The 300 new predictions join the original 300, followed by NMS at IoU 0.3 [8] and box voting [6].Source: MS COCO · p. 10, Table 9 · p. 11

Global context

• About 1 point of mAP@.5 improvement comes from global context, which concatenates a full-image feature with each region’s feature before classification and box regression.Single-level global Spatial Pyramid Pooling [12] pools the full-image feature, which passes through post-RoI layers. The combined structure is trained end-to-end.Source: MS COCO · p. 10, Table 9 · p. 11

Ensemble

• 59.0% and 37.4% mAP on test-dev earn the three-network ensemble first place in the COCO 2015 detection task.An ensemble generates region proposals, and an ensemble of per-region classifiers processes their union.Source: Table 9 · p. 11, MS COCO · p. 11

Multi-scale testing

• Over 2 points of mAP improvement come from multi-scale testing, which merges features from two adjacent pyramid scales by maxout following [33].The pyramid uses image shorter sides of 200, 400, 600, 800, and 1000 pixels. Multi-scale testing applies only to Fast R-CNN, not RPN; training remains single-scale at 600 pixels.Source: MS COCO · pp. 10–11, Table 9 · p. 11

Using validation data

• 55.7% mAP@.5 and 34.9% mAP@[.5, .95] are achieved by the single model trained on the 80k+40k trainval set.Evaluation uses the 20k test-dev set through its evaluation server; ground truth is not publicly available.Source: MS COCO · p. 11, Table 9 · p. 11

PASCAL VOC

• 55.7% mAP@.5 on COCO characterizes the single model fine-tuned to obtain these PASCAL VOC results.The approach also adopts box refinement, context, and multi-scale testing.Source: PASCAL VOC · p. 11
• 83.8% mAP on PASCAL VOC 2012 exceeds the previous state-of-the-art result [6] by 10 points.Source: PASCAL VOC · p. 11
• 85.6% mAP is achieved on PASCAL VOC 2007.Source: PASCAL VOC · p. 11

ImageNet Detection

• ImageNet DET evaluates 200 object categories using mAP@.5, with networks pretrained on 1000-class ImageNet classification and fine-tuned on DET data.The detector uses the same algorithm as MS COCO in Table 9; following [8], fine-tuning uses the DET training set and val1, while val2 provides validation. No other ILSVRC 2015 data are used.Source: ImageNet Detection · p. 11
• 62.1% mAP on the DET test set earned the three-model ensemble first place in ILSVRC 2015 detection, exceeding second place by 8.5 absolute points.Source: ImageNet Detection · p. 12
• 58.8% mAP on the DET test set was achieved by a single ResNet-101 model.Source: ImageNet Detection · pp. 11–12

C. ImageNet Localization

• The system fine-tunes ImageNet-pretrained networks for per-class regression [40] [41], adapting the RPN framework [32] to class-specific object classification and bounding-box regression.Image-level classifiers predict labels before localization predicts boxes for those classes, using the 1000-class ImageNet training set. Sibling 1×1 convolutional heads produce 1000 binary classification outputs and 1000×4 box-regression outputs relative to multiple anchors.Source: C. ImageNet Localization · p. 12
• 9.0% test-set top-5 localization error from the ensemble represents a 64% relative reduction over ILSVRC 14 results and wins ILSVRC 2015 localization.The refined single-model system achieves 10.6% top-5 localization error on the validation set.Source: Table 14 · p. 12, C. ImageNet Localization · p. 12
• Using ground-truth classes, the RPN method with ResNet-101 substantially reduces center-crop error from a reported baseline of 33.1% to 13.3% when evaluated under the same setting.Dense (fully convolutional), multi-scale testing with ResNet-101 achieves 11.7% error using ground-truth classes. Using ResNet-101 to predict classes, with 4.6% top-5 classification error, yields 14.4% top-5 localization error.Source: C. ImageNet Localization · p. 12
• RoI-centric R-CNN [8] replaces Fast R-CNN because heavily overlapping proposals produce similar pooled features and limited sample variation for image-centric stochastic training.The authors attribute this overlap to images usually containing a single dominant object and suggest that low sample variation may be undesirable for stochastic training.Source: C. ImageNet Localization · p. 12
• 200 highest-scoring per-class RPN proposals supply refinement training samples; at test time, the refinement network updates scores and box positions for each predicted class’s 200 proposals.Training proposals correspond to the ground-truth class; their regions are cropped, warped to 224×224 pixels, and processed by per-class classification and regression heads.Source: C. ImageNet Localization · p. 12