PaperlayerView exact revision on arXiv

Source-linked AI summary

Wide Residual Networks

Sergey Zagoruyko, Nikos Komodakis

arXiv:1605.07146v4cs.CVcs.LGcs.NE

TL;DR

Very deep residual networks suffer from diminishing feature reuse and slow training. This paper experimentally studies ResNet architectures and proposes wider, shallower blocks, finding that 16-layer networks can outperform 1000-layer networks on CIFAR while training several times faster.

  • Problem

    Residual-network research had focused mainly on activation order and depth, while diminishing feature reuse made very deep networks potentially inefficient to train.

  • Method

    The paper experimentally studies ResNet block architectures and proposes widening residual blocks while reducing network depth.

  • Results

    16-layer wide networks significantly outperform 1000-layer networks on CIFAR, while 50-layer networks outperform 152-layer networks on ImageNet.

  • Takeaways & Limitations

    The findings indicate that residual blocks contribute more to performance than extreme depth, while wide residual networks are several times faster to train.

  • Takeaways & Limitations

    The study did not train larger bottleneck networks because doing so required 8-GPU machines.

Abstract

from arXiv · show

Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at https://github.com/szagoruyko/wide-residual-networks

1 Introduction

The paper experimentally studies ResNet block architectures beyond activation order and depth, arguing that widening residual blocks can improve performance more effectively than increasing depth. It proposes widened blocks and dropout use, reporting improved performance, regularization, accuracy, and speed across datasets.

  • Architecture study: The study examines a richer set of ResNet block architectures and evaluates how aspects beyond activation order affect performance.This extends prior work focused mainly on activation order and network depth.
  • Motivation: Residual identity mappings may let gradients bypass block weights, causing few blocks to learn useful representations or many blocks to contribute little.The authors identify this as a weakness of residual networks that can limit feature reuse and training effectiveness.
  • Width versus depth: Properly widening ResNet blocks provides a more effective way to improve performance than increasing depth.The work builds on prior research to study how wide deep residual networks should be and to address training difficulties.
  • Contributions: The proposed architecture widens ResNet blocks and enables residual networks with significantly improved performance.The paper presents this widened architecture as a central contribution.
  • Regularization and results: The paper introduces dropout within deep residual networks to regularize them properly and prevent overfitting during training.It also reports state-of-the-art results on several datasets with improved residual-network accuracy and speed.

2 Wide residual networks

This section defines wide residual networks by reducing reliance on depth and scaling residual-block width, then describes the block structures and controlled experiments used to study these design choices. It also motivates widening as computationally efficient and introduces dropout-based regularization for the resulting increase in parameters.

  • Architecture: Residual-block representational power can be increased by adding layers, widening convolutional layers, or increasing filter sizes, but the study restricts filters to at most 3×3.The paper focuses on deepening and widening while retaining small filters shown to be effective in prior work.
  • Architecture: WRNs introduce a widening factor k that multiplies feature planes in residual layers, while the baseline basic block uses l = 2 and k = 1.The network keeps the initial convolution fixed and scales the widths of residual groups conv2–conv4.
  • Block design: The study compares multiple residual-block layouts, including the original B(3,3) block and variants containing 1×1 convolutions, while keeping feature-plane dimensionality constant across each block.The examined variants include B(3,1,3), B(1,3,1), B(1,3), B(3,1), and B(3,1,1).
  • Experimental design: Deepening experiments compare networks at roughly constant parameter counts, requiring the total number of blocks d to decrease when the number of convolutions per block l increases.This controls network complexity while isolating the effect of block depth.
  • Efficiency: Although widening makes parameters and computational complexity quadratic in k, large tensors are more efficient on GPUs than thousands of small kernels.The authors therefore seek an optimal balance between the number of blocks d and widening factor k.
  • Regularization: To regularize wider networks, dropout is inserted between convolutions in each residual block and after ReLU to perturb subsequent batch normalization and reduce overfitting.This supplements the regularization already provided by batch normalization while reducing dependence on heavy data augmentation.

3 Experimental results

Experiments across CIFAR, SVHN, ImageNet, and COCO show that widening residual networks generally improves accuracy and efficiency, while very deep networks provide limited benefit. The study identifies effective block designs and regularization strategies, achieving strong benchmark results with substantially fewer layers.

  • Experimental setup: Experiments used CIFAR-10, CIFAR-100, SVHN, and ImageNet, with horizontal flips and reflected-padding random crops rather than heavy augmentation.CIFAR-10 and CIFAR-100 contain 32 × 32 color images from 10 and 100 classes, with 50,000 training and 10,000 test images.
  • Block design: Comparable-parameter blocks produced similar results, so subsequent experiments restricted WRNs to 3×3 convolutions.The block comparison used test-accuracy medians over 5 runs and training time per epoch.
  • Block design: B(3,3) was optimal, while B(3) and deeper B(3,3,3) or B(3,3,3,3) blocks performed worse under matched parameter and layer counts.The deeper blocks were speculated to be harder to optimize because they contain fewer residual connections.
  • Width and depth: Widening consistently improved performance across depths, while jointly increasing depth and width helped until parameter counts became too high and stronger regularization was needed.The experiments varied widening factor k from 2 to 12 and depth from 16 to 40.
  • Width and depth: WRN-28-10 outperformed ResNet-1001 by 0.92% on CIFAR-10 and 3.46% on CIFAR-100 while using 36 times fewer layers.WRN-40-4 also achieved better accuracy on both datasets, had comparable parameters, and was 8 times faster to train.
  • Regularization: Dropout reduced WRN-28-10 test error by 0.11% on CIFAR-10 and 0.4% on CIFAR-100, while WRN-16-8 achieved 1.54% on SVHN.Cross-validated dropout probabilities were 0.3 on CIFAR and 0.4 on SVHN, without increasing training epochs; dropout also mitigated post-learning-rate-drop instability.
  • ImageNet and COCO: On ImageNet, width improved accuracy at comparable parameter counts, while WRN-50-2-bottleneck was almost 2× faster than pre-activation ResNet-200; WRN-34-2 achieved state-of-the-art COCO single-model performance.The COCO model outperformed ResNet-152 and Inception-v4-based models, and the authors judged more than 50 residual layers unnecessary for computational reasons.

4 Conclusions

The paper proposes wide residual networks based on studying residual-network width and dropout, achieving state-of-the-art results on several benchmarks and significant ImageNet improvements. Its 16-layer networks significantly outperform 1000-layer networks on CIFAR.

  • 4 Conclusions: The study examines residual-network width and dropout, leading to a proposed wide residual network architecture.The architecture is based on the paper’s experimental study of width and dropout in residual architectures.
  • 4 Conclusions: Wide residual networks provide state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, and COCO, with significant improvements on ImageNet.These results span several commonly used benchmark datasets and ImageNet.
  • 4 Conclusions: 16-layer wide networks significantly outperform 1000-layer deep networks on CIFAR.The comparison is stated directly for CIFAR and contrasts network depth.
Loading 1605.07146v4…