PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Residual Networks for Image and Video Recognition

Ionut Cosmin Duta, Li Liu, Fan Zhu, Ling Shao

arXiv:2004.04989v1cs.CV

TL;DR

Deep ResNets face optimization and degradation difficulties as depth increases. The paper improves information flow, projection shortcuts, and residual building blocks, reporting consistent gains without increased complexity and successful training at extreme depths.

  • Problem

    Increasing CNN depth can produce optimization difficulties, and existing ResNets do not completely solve degradation in very deep networks.

  • Method

    The paper introduces coordinated improvements to staged information flow, projection shortcuts, and residual building blocks, including greater spatial-channel capacity.

  • Results

    The approach consistently improves the baseline without increasing model complexity and trains 404-layer ImageNet and 3002-layer CIFAR networks that the baseline cannot converge on.

  • Takeaways & Limitations

    Improved residual-network design supports accuracy, convergence, and training of extremely deep visual-recognition CNNs across the reported datasets.

Abstract

from arXiv · show

Residual networks (ResNets) represent a powerful type of convolutional neural network (CNN) architecture, widely adopted and used in various tasks. In this work we propose an improved version of ResNets. Our proposed improvements address all three main components of a ResNet: the flow of information through the network layers, the residual building block, and the projection shortcut. We are able to show consistent improvements in accuracy and learning convergence over the baseline. For instance, on ImageNet dataset, using the ResNet with 50 layers, for top-1 accuracy we can report a 1.19% improvement over the baseline in one setting and around 2% boost in another. Importantly, these improvements are obtained without increasing the model complexity. Our proposed approach allows us to train extremely deep networks, while the baseline shows severe optimization issues. We report results on three tasks over six datasets: image classification (ImageNet, CIFAR-10 and CIFAR-100), object detection (COCO) and video action recognition (Kinetics-400 and Something-Something-v2). In the deep learning era, we establish a new milestone for the depth of a CNN. We successfully train a 404-layer deep CNN on the ImageNet dataset and a 3002-layer network on CIFAR-10 and CIFAR-100, while the baseline is not able to converge at such extreme depths. Code is available at: https://github.com/iduta/iresnet

1 Introduction

Increasing CNN depth improves representation power but creates optimization difficulties, and deeper ResNets can suffer degradation despite residual shortcuts. This work proposes coordinated improvements to information flow, projection shortcuts, and building blocks, yielding consistent gains without added complexity.

  • Deeper CNNs can develop optimization difficulties, so adding layers does not guarantee better results.
  • Residual shortcuts facilitate identity mapping and address the degradation problem, but very deep ResNets can still worsen in accuracy and training error.
  • The proposed architecture improves information propagation, projection shortcuts, and spatial-channel allocation while controlling parameters and computational cost.
  • The approach provides consistent baseline improvements without increasing model complexity across image and video datasets.
  • 404-layer ImageNet and 3002-layer CIFAR networks were trained successfully, whereas the baseline did not converge at such depths.

2 Related Work

Prior work improves ResNets through pre-activation ordering, grouped convolutions, additional modules, or collections of refinements. The proposed work differs by introducing a distinct building block and complementary architectural directions, while also overlapping on projection shortcuts.

  • Pre-activation ResNets change building-block element ordering to improve signal propagation.
  • Grouped-convolution methods have targeted distributed computation or improved recognition performance in ResNet architectures.
  • The proposed building block uses a different shape and twice as many spatial filters as the cited grouped-convolution approach, showing improved performance.
  • Squeeze-and-excitation and non-local blocks improve ResNets but increase model and computational complexity because they are additional inserted blocks.
  • The proposed directions are mostly complementary to the cited bag of tricks, with projection shortcuts as an area of overlap.

3 Improved residual networks

The paper improves ResNets by redesigning information flow, projection shortcuts, and residual building blocks while maintaining similar model complexity. These changes address signal propagation, information loss, and limited spatial-pattern capacity.

  • Improved information flow through the network: ResStage splits a ResNet into stages with Start, Middle, and End ResBlocks organized around changes in spatial size or channel count.In ResNet-50, four main stages contain 3, 4, 6, and 3 ResBlocks, respectively.
  • Improved information flow through the network: ResStage fixes the number of ReLUs on the main propagation path and uses normalization at stage boundaries to stabilize signal flow.The main stages use four ReLUs independent of network depth; End and Start ResBlocks prepare and normalize signals between stages.
  • Improved residual networks: The proposed stages, shortcut, and building block provide performance gains over baselines without increasing model complexity.The paper reports similar component and parameter counts for ResNet-50 variants, with improvements attributed to rearrangement and architectural design.
  • Improved projection shortcut: The proposed projection shortcut separates spatial and channel projection using 3×3 max pooling followed by stride-1 1×1 convolution and batch normalization.Max pooling selects high-activation spatial information before channel projection, addressing noise and information loss in the original shortcut.
  • Grouped building block: The ResGroup block places the largest channel count on the 3×3 convolution and uses grouped convolution to control parameters and FLOPs.The 3×3 convolution is the only component responsible for learning spatial patterns, so widening it increases spatial-pattern capacity.

4 Experiments

Experiments across image classification, video recognition, and object detection show that iResNet variants improve accuracy and convergence over ResNet baselines without increasing model complexity. The improvements persist at extreme depths, where baseline networks degrade or fail to converge.

  • ImageNet: iResNet outperforms ResNet on ImageNet at 50, 101, 152, and 200 layers, with a 1.19% reduction in top-1 error at 50 layers.At 200 layers, iResNet outperforms ResNet by 1.93% top-1 error, while the baseline shows severe optimization issues.
  • ImageNet: iResNet speeds ImageNet training convergence, requiring fewer epochs to outperform the baseline and maintaining optimization as depth increases.At 50 layers, it surpasses the best ResNet results during the first 30 epochs in fewer than 8 epochs; these gains occur nearly without added model complexity.
  • Extreme depth: At 302 and 404 layers on ImageNet, iResNet enables extreme-depth experiments while the baseline exhibits increasing optimization difficulty.Table 3 reports error rates for these depths and identifies P as the parameter count.
  • Video recognition: On video recognition, iResNet outperforms ResNet on Kinetics-400 and Something-Something-v2, including a 3.1% top-1 error improvement on Kinetics-400.The comparison uses 50-layer networks; the experiments report video error rates and parameter counts.
  • Extreme depth: On CIFAR-10 and CIFAR-100, iResNet improves over the baseline through 3002 layers, while the baseline fails to converge at 2000 and 3002 layers.Validation accuracy begins to decrease for iResNet at 3002 layers, probably because of overfitting, although training loss shows no optimization issues.
  • Object detection: On COCO object detection, iResNet backbones improve all reported metrics while maintaining efficiency; 50-layer iResGroup nearly matches the 101-layer ResNet baseline.The evaluation uses SSD with 300×300 input images.

5 Conclusion

The improved residual network addresses information propagation, projection shortcuts, and building blocks, yielding consistent baseline improvements without added complexity and enabling extreme depths.

  • The proposed improvements target information propagation, the projection shortcut, and the residual building block.
  • 1.19% to 2.33% improvements in top-1 error are reported for 50-layer ResNets across different settings.
  • The improvements are obtained without increasing model complexity.
  • The approach facilitates learning networks over 400 layers on ImageNet and over 3000 layers on CIFAR-10/100 without optimization issues.

A Appendix

The appendix provides experimental setup details and states that the models follow common training and testing procedures across six datasets.

  • The experiments follow common training and testing procedures established by the research community across all six datasets.
  • All models are trained from scratch except for object detection, which uses ImageNet pre-training.
  • Convolutional layers use the weight initialization of.

A.1 iResNet on ImageNet

The ImageNet appendix examines training behavior and validation performance across multiple ResNet depths, emphasizing iResNet’s faster convergence and improved results under the reported evaluation settings.

  • Original ResNets converge very slowly at the beginning of ImageNet training, with increasing depth making convergence more difficult.Training and validation curves show degraded training performance as the number of layers increases.
  • iResNets show consistent and significant validation improvements over baseline ResNets.
  • iResNet training-set convergence shows greater benefits over baseline ResNets, indicating improved optimization efficiency.
  • Fig. 7 presents the beginnings of ImageNet training and validation curves for ResNet and iResNet at 50, 101, 152, and 200 layers.
  • Table 10 reports ImageNet error rates using a 320×320 validation crop size.The models are trained with 224×224 inputs, while validation can use the larger crop because ResNets are fully convolutional.

A.2 iResNet on video recognition

The video-recognition appendix evaluates iResNet3D on Kinetics-400 and Something-Something-v2 using specified clip sampling and validation procedures, with training curves indicating facilitated learning.

  • iResNet3D facilitates learning on Kinetics-400 and Something-Something-v2 according to the training and validation curves.The curves compute top-1 error independently for each video clip.
  • The video experiments use 16-frame input clips with dataset-specific frame skipping: four frames for Kinetics-400 and one for Something-Something-v2.
  • Fig. 8 visualizes training curves for Kinetics-400 and Something-Something-v2.The left panel represents Kinetics-400 and the right panel Something-Something-v2.
  • The appendix includes tables describing the ResNet3D video architecture and the ResNet architecture for CIFAR-10/100.
  • Final validation samples up to 10 clips per video and applies 3 spatial crops, producing a maximum of 30 clips per video.

A.3 iResNet on CIFAR-10/100

On CIFAR-10/100, iResNet optimization remains effective as depth increases to 3002 layers, unlike the original ResNet. The evaluated networks use three main stages and depths from 164 to 3002 layers.

  • 3002-layer iResNets continue improving training loss, whereas original ResNets face critical optimization issues and fail to start converging at depths 2000 and 3002.
  • 164- to 3002-layer iResNets show continuously improving training loss, with smaller training fluctuations than baseline ResNets.
  • Training follows the original ResNet procedure for 164 epochs, while using mini-batches of 64 and one GPU per model.
  • CIFAR-10/100 networks contain three main stages because the image resolution is 32×32.
  • The experiments consider 164-, 1001-, 2000-, and 3002-layer networks, with building-block counts specified separately for each stage and depth.

A.4 Comparison on the projection shortcut

The proposed projection shortcut differs from a prior refinement in pooling type and kernel size, and the resulting ResMax network outperforms that refinement on ImageNet.

  • Max pooling replaces average pooling to improve translation invariance and select the highest local activation.
  • A 3x3 pooling kernel matches the 3×3 convolution in the Start Block, aligning the element-wise addition inputs.
  • On a 50-layer ImageNet network, the proposed projection shortcut outperforms by a large margin; this variant is called ResMax.

A.5 Discussion

The discussion frames extreme network depth as impractical today but potentially ordinary in the future. It connects this prospect to the brain’s scale, speed, and energy efficiency.

  • 404- and 3002-layer networks are presented as currently unrealistic depths whose future usefulness may increase as capabilities expand.
  • The human brain is described as having far more neurons, connections, and layers than artificial learning systems while processing information quickly and using low energy.
  • A proposed future scenario is building networks closer to the brain’s capacity without using their full capacity simultaneously.
Loading 2004.04989v1…