PaperlayerView exact revision on arXiv

Source-linked AI summary

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Zifeng Wu, Chunhua Shen, Anton van den Hengel

arXiv:1611.10080v1cs.CV

TL;DR

Increasing depth may not be the best route to stronger visual recognition, especially for residual networks. The paper reinterprets ResNets to derive shallower architectures that outperform much deeper models across ImageNet classification and semantic segmentation.

  • Problem

    It remains unclear whether increasing residual-network depth is preferable to width, given evidence that very deep ResNets behave like ensembles of shallower subnetworks.

  • Method

    The paper analyzes ResNet ensembles and effective depth, then derives spatially efficient, relatively shallow residual architectures for classification and segmentation.

  • Results

    The proposed networks outperform very deep residual networks on ImageNet and semantic segmentation, achieving 82.5% mean IoU on PASCAL VOC.

  • Takeaways & Limitations

    The architecture delivers better feature extraction than current state-of-the-art models while using a more spatially efficient design.

  • Takeaways & Limitations

    Very deep ResNet-152 models can exceed available GPU memory, preventing tuning on mainstream GPUs with 12GB memory.

Abstract

from arXiv · show

The trend towards increasingly deep neural networks has been driven by a general observation that increasing depth increases the performance of a network. Recently, however, evidence has been amassing that simply increasing depth may not be the best way to increase performance, particularly given other limitations. Investigations into deep residual networks have also suggested that they may not in fact be operating as a single deep network, but rather as an ensemble of many relatively shallow networks. We examine these issues, and in doing so arrive at a new interpretation of the unravelled view of deep residual networks which explains some of the behaviours that have been observed experimentally. As a result, we are able to derive a new, shallower, architecture of residual networks which significantly outperforms much deeper models such as ResNet-200 on the ImageNet classification dataset. We also show that this performance is transferable to other problem domains by developing a semantic segmentation approach which outperforms the state-of-the-art by a remarkable margin on datasets including PASCAL VOC, PASCAL Context, and Cityscapes. The architecture that we propose thus outperforms its comparators, including very deep ResNets, and yet is more efficient in memory use and sometimes also in training time. The code and models are available at https://github.com/itijyou/ademxapp

1. Introduction

This section motivates revisiting increasingly deep convolutional networks by examining how ResNets operate and identifying a gap between their unravelled view and true ensembles of shallow subnetworks. It introduces relatively shallow architectures based on this understanding, reporting state-of-the-art ImageNet results and best results on three semantic-segmentation datasets.

  • Motivation: Convolutional networks have grown deeper since AlexNet, culminating in a 1,202-layer ResNet trained on CIFAR-10’s tiny images.Small images produce relatively small feature maps, which is critical for training extremely deep networks.
  • ResNet interpretation: ResNets have been described as exponential ensembles of shallow networks, but their unravelled view differs from a real ensemble because residual units are nonlinear.The paper examines this gap to clarify ResNet mechanisms, effective depth, and subnetworks’ actual depths.
  • ResNet interpretation: Most gradients in a 110-layer ResNet traverse at most seventeen residual units, supporting the existence of a network-specific effective depth.This empirical finding motivates distinguishing effective depth from the network’s nominal depth.
  • Contributions: The paper introduces a further developed intuitive view of ResNets to explain their behaviour and identify directions for improvement.This understanding informs the design of relatively shallow convolutional networks.
  • Contributions: The proposed relatively shallow networks achieve state-of-the-art ImageNet classification results and boost semantic-segmentation algorithms, achieving the best results on PASCAL VOC, PASCAL Context, and Cityscapes.The segmentation evaluation treats the networks as pre-trained features and measures their impact on existing algorithms.

2. Related work

This section situates the work in ResNet-based image classification and fully convolutional semantic segmentation. It highlights ResNets’ broad success and unresolved mechanism, alongside FCNs’ role as an efficient mainstream approach to dense prediction.

  • Scope: The work is closely related to residual-network image classification and fully convolutional semantic image segmentation.These are the two research topics framing the related-work discussion.
  • ResNet-based image classification: ResNets were proposed to combat vanishing gradients in very deep convolutional networks and have outperformed earlier models in object detection and semantic segmentation.They are gradually replacing VGGNets as standard computer-vision feature extractors, although their effectiveness remains incompletely understood.
  • ResNet-based image classification: The unravelled view attributes a linearly increasing number of independent classifiers to nonlinear interactions among residual units.Because f2(a + b) ≠ f2(a) + f2(b), f2 does not operate independently of f1; analyzing effective-depth interactions also clarifies gradient paths.
  • Semantic image segmentation: Fully convolutional networks predict categories for every image pixel and became mainstream for dense prediction because of their efficiency.Prior empirical results also indicate that stronger pretrained features can further improve FCN performance.

3. Residual networks revisited

The paper revisits pre-activation ResNets through an unravelled view, arguing that they form linearly growing ensembles of subnetworks whose actual depths are constrained by effective depth. This explains over-deepening and motivates reducing residual units while preserving sufficient depth for feature learning.

  • Model formulation: The analysis assumes full pre-activation ResNets with identity shortcut mappings and omits input, classifier, and initial stem layers for clarity.Residual stages consist of batch normalization, ReLU, and convolution components.
  • Unravelled view: A ResNet is more accurately a linearly growing ensemble of subnetworks, not an exponentially growing ensemble, because residual mappings are nonlinear.For a two-unit ResNet, three subnetworks correspond to the input, the first residual mapping, and the composed second-unit path.
  • Effective depth: Each subnetwork M_i has actual depth min(i, l), so paths exceeding effective depth l are not fully end-to-end trained.When effective depth is one, gradients through the second block vanish and the first block is updated only through a shallower subnetwork.
  • Very deep ResNets: Very deep ResNets avoid vanishing gradients by reshaping into multiple shallower subnetworks, although the adequacy of their resulting training remains unresolved.Gradients vanish exponentially as path length increases, and the paper notes that the effect of nonlinear later blocks makes equivalence to fully end-to-end training non-trivial.
  • Width–depth trade-off: The width–depth dilemma implies that going deeper eventually adds features that are not completely end-to-end trained, whereas going wider preserves the end-to-end characteristic.This motivates removing residual units from over-deepened models, while retaining appropriate depth because an eight-unit, twenty-layer network can outperform ResNet-152 but sufficient depth remains vital for good features.

4. Approach to image classification

The proposed image-classification networks comprise three input-size variants built from residual units with inner shortcuts. Classification uses pooling for down-sampling, 4,096-dimensional averaged final features, and a linear learning-rate schedule in the reported experiments.

  • Network architectures: The approach defines three architectures with different input sizes, using convolution stages composed of batch normalization, ReLU, and convolution layers.Closely stacked two or three stages form residual units B1–B7 with inner shortcut connections; each unit type corresponds to a level whose units share characteristics.
  • Classification features: The networks use pooling layers for down-sampling and average the top-most feature maps into 4,096-dimensional final features.The feature dimensionality matches the cases of AlexNet and VGGNets.
  • Implementation details: Experiments use MXNet on four devices on a single node and apply a linear learning-rate schedule from 0.1 to 10−6 within 450k iterations.The implementation follows the re-implementation settings of ResNets by Gross and Wilber as much as possible.

5. Approach to semantic image segmentation

The approach reshapes a pre-trained network for semantic segmentation through resolution, classifier, and dropout modifications. It follows an FCN/early-DeepLab setup while avoiding most multi-scale, supervision, context, testing, averaging, and CRF enhancements.

  • Design choices: The method follows FCN and first-version DeepLab, without multi-scale structures, deep supervision, global context features, multi-scale testing, model averaging, or CRF post-processing except on ADE20K.These omissions reduce entangled factors when evaluating the proposed approach.
  • Resolution: The first reshaping step produces 1/8-resolution score maps by removing down-sampling operations and increasing dilation rates in selected convolution layers.For 224×224 inputs, stride changes and dilation increases are applied in B5 and B6.
  • Resolution: Because spatial pooling caused harmful over-invariance in preliminary FCN experiments, the method replaces several top-most pooling operations and fine-tunes the modified network.The example removes the top-most three pooling layers before B4, B5, and B6.
  • Classifier: The classifier step removes global pooling and the top linear classifier, then adds either one linear layer or a two-convolution nonlinear stage with 3×3 kernels.The two alternatives are called ‘1 convolution’ and ‘2 convolutions’.
  • Dropout: Dropout rates are 0.3 for 2,048-channel residual units and 0.5 for 4,096-channel units to alleviate over-fitting.The rates apply to very wide residual units, including specified upper units in ResNets and the proposed networks.

6. Experimental results

Experiments show that shallower, properly designed models can match or outperform deeper ResNets while improving efficiency, and that the proposed features transfer strongly across semantic segmentation datasets. Results also reveal that classification performance alone does not reliably predict segmentation quality, while task complexity and scene pretraining affect model rankings.

  • ImageNet classification: Model A performs best among the evaluated ImageNet networks, while shallower Model B achieves error rates similar to ResNet-152 and runs slightly faster.The authors attribute this advantage to large model capacity in relatively shallow networks.
  • ImageNet classification: Smaller input sizes trade performance for efficiency: Model D is slightly worse than ResNet-152 but almost two times faster.The authors note that down-sampling should be adjusted when reducing input size to avoid degraded performance.
  • ImageNet classification: Models C, D, and E perform comparably despite Model C having greater depth and more parameters, highlighting the importance of network design.The models place too many layers at the 7×7 low-resolution level.
  • PASCAL VOC: 3.4%: Model A outperforms ResNet-152 on PASCAL VOC, while one hidden layer provides a further 2.1% improvement.ResNet-152 does not yield a statistically significant improvement over ResNet-101, and Model B underperforms ResNet-101 despite stronger ImageNet classification.
  • PASCAL VOC: 82.5%: Using only augmented PASCAL VOC training data, the method achieves this mean IoU and exceeds the previous best by 3.4%.It wins for seventeen of the twenty object categories.
  • Cityscapes and ADE20K: 4.2%: Model A1 outperforms ResNet-101 on Cityscapes mean IoU, while the method also exceeds prior work by 6.6% in class-level IoU and 6.4% in instance-weighted IoU.On ADE20K, Model A2 outperforms ResNet-152 by 4.0% in mean IoU, and Model C ranks second despite weaker ImageNet classification.
  • PASCAL Context: 2.4%: On PASCAL Context, the method improves the IoU score over the previous best method, while Model A2 performs best among methods using one pre-trained model.The authors apply multi-scale testing, model averaging, and CRF post-processing for the broader comparison.

7. Conclusion

The paper analyzes ResNet through its ensemble classifiers and effective residual-unit depths, then derives a more spatially efficient architecture for fully end-to-end training of large networks. Shallow networks built with this architecture outperform previous very deep residual networks on ImageNet.

  • The analysis examines ResNet’s ensemble classifiers and the effective depths of its residual units.
  • The resulting architecture is more spatially efficient, better performing, and enables fully end-to-end training for large networks.
  • Networks designed with the new architecture are correspondingly shallow and outperform previous very deep residual networks on ImageNet.

A. Appendix · A.2. Gradients in residual networks

The appendix evaluates gradient propagation along sampled paths in trained residual networks. For paths of length k, it measures the input-gradient norm after propagating gradients only through selected trainable mappings, excluding shortcut connections.

  • A.2. Gradients in residual networks: The experiment tests gradients in various trained residual networks with n units.It follows the gradient experiment proposed by Veit et al. [31].
  • A.2. Gradients in residual networks: For each path-length setting, the procedure samples individual paths containing k units.The sampled path length k is the variable used to analyze gradient magnitude.
  • A.2. Gradients in residual networks: Each trial first feeds a batch forward through the whole network.The subsequent gradient measurement occurs during the backward pass.
  • A.2. Gradients in residual networks: During backpropagation, the method randomly samples k units from the network.Only the selected units participate in the restricted gradient propagation.
  • A.2. Gradients in residual networks: For sampled units, gradients propagate only through their trainable mapping functions.This isolates the contribution of the selected mappings along the sampled path.
  • A.2. Gradients in residual networks: The restricted backward pass omits shortcut connections when propagating gradients.The passage contrasts propagation through trainable mappings with propagation through shortcuts.
  • A.2. Gradients in residual networks: Figure 4 reports the gradient magnitude at the input for different path lengths k across residual networks.The figure caption identifies gradient magnitude at input as the plotted quantity.

A.3. Qualitative results

Qualitative semantic-segmentation results are shown for PASCAL VOC, Cityscapes, ADE20K, and PASCAL Context, alongside failure cases for PASCAL VOC and Cityscapes. Difference maps distinguish correctly and incorrectly labelled pixels, and no CRF post-processing is applied.

  • A.3. Qualitative results: Qualitative segmentation results are presented for PASCAL VOC, Cityscapes, ADE20K, and PASCAL Context validation sets.Each example includes the original image, ground-truth, predicted label, and difference map.
  • A.3. Qualitative results: Failure cases are shown for the PASCAL VOC 2012 and Cityscapes validation sets.The failure-case figures use the same four-panel layout as the qualitative results.
  • A.3. Qualitative results: Difference maps mark correctly labelled pixels in grey, wrongly labelled pixels in black, and officially ignored evaluation pixels in white.These conventions apply to the qualitative segmentation examples and failure cases.
  • A.3. Qualitative results: The segmentation outputs use no CRF post-processing because, although it can smooth outputs, it is too slow in practice.The passage explicitly notes that CRFs are not applied.
Loading 1611.10080v1…