PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Residual Learning for Image Recognition

arXiv:1512.03385v1

TL;DR

The paper asks whether simply stacking more layers makes networks better despite persistent degradation and optimization difficulties. It introduces residual learning and shows that extremely deep residual networks achieve strong ImageNet and cross-task recognition results, including 3.57% top-5 test error.

  • Problem

    The paper asks whether deeper networks improve performance despite degradation and optimization difficulties that persist beyond vanishing or exploding gradients.

  • Method

    The paper reformulates learning H(x) as learning a residual F(x)=H(x)−x, then reconstructs the mapping as F(x)+x.

  • Results

    3.57% top-5 error on the ImageNet test set was achieved by an ensemble, alongside first-place results across multiple recognition tasks.

  • Takeaways & Limitations

    Residual learning supports successfully trained networks exceeding 100 layers and shows evidence of a generic principle across vision recognition tasks.

  • Takeaways & Limitations

    On small CIFAR-10, the 1202-layer network may be unnecessarily large, and stronger regularization may improve results.

Abstract

from arXiv · show

1. Introduction

Increasing depth can cause optimization degradation even when deeper models contain a constructed identity-mapping solution, motivating residual learning, which enables very deep networks and achieves strong ImageNet and transfer-task results. The framework trains stacked layers to learn F(x) = H(x)−x, with identity shortcuts adding no parameters or computational complexity.

  • Problem: As depth increases, plain networks can suffer rapidly worsening training and test error after accuracy saturation, a degradation not caused by overfitting.Figure 1 compares 20-layer and 56-layer plain networks on CIFAR-10 and shows higher error for the deeper model.
  • Problem: A deeper model has a constructed solution matching a shallower model by setting added layers to identity mappings, yet optimization may fail to reach a comparably good solution.This explains why increasing depth can produce higher training error despite the existence of a no-worse solution by construction.
  • Method: Residual learning recasts H(x) as F(x)+x, letting stacked nonlinear layers fit the residual F(x) := H(x)−x instead of the full underlying mapping.The framework is implemented with identity shortcut connections that add the shortcut output to the stacked-layer output.
  • Results: Residual networks remained effective across datasets, with successfully trained CIFAR-10 models exceeding 100 layers and explored models exceeding 1000 layers.The results suggest the optimization difficulties and residual-learning effects are not specific to one dataset.

2. Related Work

The related work connects residual representations to image recognition and multiscale solvers, and situates identity shortcuts among earlier shortcut and gated-network approaches [18, 30, 3, 42, 43].

  • Residual Representations: VLAD and Fisher Vector [30] encode residuals for effective shallow image-retrieval and classification representations.For vector quantization, residual-vector encoding is more effective than encoding original vectors [17].
  • Residual Representations: Multigrid and hierarchical basis preconditioning solve multiscale problems using residual vectors between coarser and finer scales.The passage describes Multigrid as reformulating PDE systems into subproblems at multiple scales.
  • Shortcut Connections: Earlier shortcut approaches connected network inputs to outputs, attached auxiliary classifiers to intermediate layers [44] [24], or centered responses and gradients [47].These practices and theories address optimization issues including vanishing or exploding gradients.
  • Shortcut Connections: Highway networks [42] [43] use parameterized, data-dependent gates, whereas identity shortcuts are parameter-free, never closed, and always learn residual functions.Highway networks had not demonstrated accuracy gains with depth exceeding 100 layers.

3. Deep Residual Learning

Deep residual learning reformulates the target as a residual function relative to an identity shortcut, easing optimization when identity mappings are difficult for deep nonlinear stacks. Residual blocks add the learned mapping to the input with negligible computational overhead and support dimension changes through projections.

  • Residual learning: Residual learning fits H(x)−x instead of H(x), helping optimization because identity mappings are difficult for multiple nonlinear layers to approximate.If the optimal function is closer to identity than zero, learning perturbations around identity is easier; experiments find learned residual responses are generally small.
  • Building block: A residual block computes F(x,{W_i})+x, with F typically using two or three layers and the post-addition nonlinearity applied afterward.For two layers, F=W_2σ(W_1x); the shortcut performs element-wise addition before the second nonlinearity.
  • Building block: Shortcut connections add no parameters or meaningful computation, enabling fair plain-versus-residual comparisons with matched parameters, depth, width, and computational cost.The only exception is negligible element-wise addition.
  • Shortcut connections: When input and output dimensions differ, a linear projection W_s matches them; otherwise, identity shortcuts are sufficient and are used for economy.Identity shortcuts can also pad extra zero entries when dimensions increase in the ImageNet architectures.
  • Architectural scope: The residual formulation extends from fully connected to convolutional layers, where F comprises convolutional layers and addition operates channel by channel on feature maps.A single-layer F reduces to y=W_1x+x, for which the authors observed no advantages.

4. Experiments

Experiments show that residual learning removes the degradation problem, enabling deeper networks to achieve better accuracy without prohibitive complexity. The method also generalizes across ImageNet, CIFAR-10, object detection, localization, and segmentation tasks.

  • Optimization and ablations: Plain networks showed higher training error as depth increased, whereas residual networks achieved lower training error and validation generalization, addressing the degradation problem.The deeper plain network underperformed despite containing the shallower network’s solution space; at 18 layers, ResNet mainly improved convergence speed.
  • Optimization and ablations: Identity shortcuts were more efficient than projection shortcuts for bottleneck architectures because replacing them doubled time complexity and model size.Projection shortcuts were not essential for solving degradation, and the selected bottleneck design was primarily motivated by practical efficiency.
  • ImageNet results: The 50-, 101-, and 152-layer ResNets were considerably more accurate than 34-layer models, with no observed degradation and gains across all evaluation metrics.Residual learning reversed the plain-network trend: the 34-layer ResNet outperformed the 18-layer ResNet by 2.8%, while reducing top-1 error by 3.5%.
  • ImageNet results: 3.57% top-5 test error was achieved by a six-model ResNet ensemble, while the 152-layer single model reached 4.49% top-5 validation error and surpassed previous ensemble results.The 152-layer model also used fewer FLOPs than VGG-16/19 despite its greater depth.
  • CIFAR-10 experiments: On CIFAR-10, a 110-layer ResNet converged well with learning-rate warmup, used fewer parameters than FitNet [35] and Highway [42], and achieved 6.43% error.An over-1000-layer model still avoided optimization difficulty and reached training error below 0.1% with 7.93% test error, although its size led to overfitting.
  • Analysis and other tasks: Residual networks produced generally smaller layer responses than plain networks, supporting the motivation that residual functions are closer to zero.Using the same detection implementation, replacing VGG-16 [41] with ResNet-101 improved recognition-task baselines on PASCAL VOC and COCO; the method also won first places in several ILSVRC and COCO 2015 tracks.

A. Object Detection Baselines

The detection method builds on Faster R-CNN [32], using ImageNet-pretrained ResNet-50/101 models fine-tuned for object detection. It adapts ResNet’s convolutional features through NoC [33] and fixes batch-normalization statistics during fine-tuning to reduce memory use.

  • A. Object Detection Baselines: The method uses ImageNet-pretrained ResNet-50/101 models initialized for Faster R-CNN [32] and fine-tuned on object-detection data.These models were explored during the ILSVRC and COCO 2015 detection competitions.
  • A. Object Detection Baselines: NoC [33] replaces ResNet’s absent hidden fully connected layers by computing shared full-image convolutional feature maps through conv4_x, whose strides are at most 16 pixels.For ResNet-101, this comprises conv1 through conv4_x, totaling 91 convolutional layers, analogous to VGG-16’s 13 convolutional layers.
  • A. Object Detection Baselines: Batch-normalization layers use ImageNet-training means and variances fixed during detection fine-tuning, becoming linear activations with constant offsets and scales.The statistics are not updated during fine-tuning, primarily reducing Faster R-CNN training memory consumption.

PASCAL VOC

On PASCAL VOC, ResNet-101 improves mAP by more than 3% over VGG-16, with the gain attributed solely to improved features. The evaluation uses standard VOC training splits and Faster R-CNN hyperparameters from [32].

  • PASCAL VOC: ResNet-101 improves mAP by >3% over VGG-16 on PASCAL VOC, and Table 7 reports the result.The passage attributes this gain solely to improved features.
  • PASCAL VOC: For VOC 2007, training uses 5k VOC 2007 and 16k VOC 2012 trainval images (“07+12”), following [32].
  • PASCAL VOC: For VOC 2012, training uses 10k VOC 2007 trainval+test and 16k VOC 2012 trainval images (“07++12”), with Faster R-CNN hyperparameters matching [32].

MS COCO

On MS COCO, ResNet-101 substantially outperformed VGG-16, improving both recognition and localization under standard and PASCAL VOC metrics. The evaluation used 80k training images and 40k validation images across 80 object categories.

  • MS COCO: The MS COCO benchmark contains 80 object categories, with 80k train images and 40k validation images evaluated using mAP@.5 and mAP@[.5, .05:.95].The passage labels the standard COCO metric as mAP @ IoU = .5:.05:.95.
  • MS COCO: Both RPN and Fast R-CNN stages were trained for 240k iterations at 0.001 followed by 80k iterations at 0.0001.The 8-GPU implementation used RPN and Fast R-CNN mini-batches of 8 and 16 images, respectively.
  • MS COCO: 6.0% absolute mAP@[.5, .95] improvement over VGG-16 (28% relative) shows ResNet-101 improves both recognition and localization.The corresponding mAP@.5 increase was 6.9%.

B. Object Detection Improvements

The competition system combined box refinement, global context, and multi-scale testing with Faster R-CNN and ResNet-101, achieving first place in COCO 2015 detection. On test-dev, the single model reached 55.7% mAP@.5 and 34.9% mAP@[.5, .95], while a three-network ensemble achieved 59.0% and 37.4%.

  • MS COCO Box refinement: Box refinement pooled features from each regressed box to produce new classification and regression predictions, which were combined with the original predictions before NMS at IoU 0.3.The method generated 300 new predictions and combined them with the original 300 predictions.
  • Global context: Global context concatenated a full-image spatial-pyramid-pooled feature with each per-region feature before sibling classification and box regression.The global feature was obtained by treating the entire image bounding box as an RoI.
  • Ensemble: A three-network Faster R-CNN ensemble achieved 59.0% mAP@.5 and 37.4% mAP@[.5, .95] on COCO test-dev, winning first place in the COCO 2015 detection task.The ensemble combines networks for region proposal and per-region classification.
  • Multi-scale testing: Multi-scale testing pooled RoI features from two adjacent pyramid scales and merged them with maxout, improving mAP by over 2 points.Multi-scale testing was used without multi-scale training because of limited time.
  • Using validation data: The single-model system trained on the 80k+40k trainval set achieved 55.7% mAP@.5 and 34.9% mAP@[.5, .95] on the 20k test-dev set.The test-dev ground truth was unavailable publicly, so evaluation was performed by the evaluation server.

PASCAL VOC

Fine-tuning the COCO-trained model with box refinement, context, and multi-scale testing achieves strong PASCAL VOC detection performance. It reaches 85.6% mAP on VOC 2007 and 83.8% on VOC 2012, 10 points above the previous state of the art on VOC 2012 [6].

  • PASCAL VOC: The model is fine-tuned from a single COCO model using box refinement, context, and multi-scale testing improvements.The COCO model achieves 55.7% mAP@.5 before fine-tuning on PASCAL VOC.
  • PASCAL VOC: 85.6% mAP on PASCAL VOC 2007 and 83.8% mAP on PASCAL VOC 2012 demonstrate strong detection performance.These results are reported in Tables 10 and 11.
  • PASCAL VOC: 10 points higher than the previous state-of-the-art result [6], the VOC 2012 result establishes a substantial improvement.

ImageNet Detection

On ImageNet DET, the proposed detector achieved 62.1% mAP with a three-model ensemble, winning ILSVRC 2015 by 8.5 absolute points over second place. The task covers 200 categories and evaluates detection accuracy using mAP@.5.

  • Evaluation: ImageNet DET evaluates detection over 200 object categories using mAP@.5.
  • Evaluation protocol: The detection models use ImageNet classification pretraining, are fine-tuned on DET training data and val1, and use val2 for validation.The detection algorithm is the same as the one used for MS COCO in Table 9.
  • Results: 62.1% mAP from an ensemble of 3 models won the ImageNet detection task at ILSVRC 2015, surpassing second place by 8.5 absolute points.A single model achieved 58.8% mAP on the DET test set.

C. ImageNet Localization

The per-class RPN/R-CNN localization framework achieves a 9.0% top-5 localization error with an ensemble, winning first place in ILSVRC 2015 and outperforming ILSVRC 2014 by 64% relatively. Its ResNet-101 single-model variant reaches 10.6% error, while oracle-class testing gives 11.7% with dense multi-scale evaluation.

  • Method: The method uses a per-class RPN with separate binary classification and box-regression outputs, followed by class-dependent proposals for an R-CNN classifier.The RPN is based on Faster R-CNN [32], while the R-CNN classifier follows R-CNN [8].
  • Results: 11.7% localization error is obtained by ResNet-101 with ground-truth classes under dense fully convolutional and multi-scale testing.Using ground-truth classes, the same framework reduces center-crop error to 13.3% from VGG’s 33.1% under the comparable setting [41].
  • Results: 9.0% top-5 localization error on the test set with an ensemble significantly outperforms ILSVRC 2014, yielding a 64% relative error reduction and first place in ILSVRC 2015.The comparison is reported against the ILSVRC 2014 results.
  • Results: 10.6% top-5 localization error is achieved by the single-model method on the validation set.This is the reported single-model result before ensembling classification and localization networks.
Loading 1512.03385v1…