PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Networks of Residual Networks: Multilevel Residual Networks

Ke Zhang, Miao Sun, Tony X. Han, Xingfang Yuan, Liru Guo, Tao Liu

arXiv:1608.02908v2cs.CV

TL;DR

Very deep residual networks face optimization limits associated with vanishing gradients and overfitting. RoR adds level-wise shortcut connections and applies them across ResNets, Pre-ResNets, and WRN; experiments report state-of-the-art results across several image-classification benchmarks.

  • Problem

    Very deep residual networks can suffer vanishing gradients, overfitting, and limited optimization ability when residual blocks are simply stacked.

  • Method

    RoR adds identity shortcut connections level by level to original residual networks and extends the architecture to ResNets, Pre-ResNets, and WRN.

  • Results

    RoR achieves state-of-the-art image-classification results on CIFAR-10, CIFAR-100, SVHN, and ImageNet, including 3.77% test error on CIFAR-10 with RoR-3-WRN58-4+SD.

  • Takeaways & Limitations

    RoR provides an effective complement to the residual-networks family and improves multiple residual-network structures with a few additional shortcuts.

Abstract

from arXiv · show

A residual-networks family with hundreds or even thousands of layers dominates major image recognition tasks, but building a network by simply stacking residual blocks inevitably limits its optimization ability. This paper proposes a novel residual-network architecture, Residual networks of Residual networks (RoR), to dig the optimization ability of residual networks. RoR substitutes optimizing residual mapping of residual mapping for optimizing original residual mapping. In particular, RoR adds level-wise shortcut connections upon original residual networks to promote the learning capability of residual networks. More importantly, RoR can be applied to various kinds of residual networks (ResNets, Pre-ResNets and WRN) and significantly boost their performance. Our experiments demonstrate the effectiveness and versatility of RoR, where it achieves the best performance in all residual-network-like structures. Our RoR-3-WRN58-4+SD models achieve new state-of-the-art results on CIFAR-10, CIFAR-100 and SVHN, with test errors 3.77%, 19.73% and 1.59%, respectively. RoR-3 models also achieve state-of-the-art results compared to ResNets on ImageNet data set.

I. INTRODUCTION

Very deep residual networks can still face vanishing gradients, overfitting, and optimization limits. RoR addresses these issues by adding level-wise shortcuts, extending the approach across residual-network variants and achieving strong benchmark results.

  • Very deep ResNets can suffer vanishing gradients and overfitting, with thousand-layer ResNets performing worse than hundred-layer ResNets.
  • RoR adds extra identity shortcuts level by level to original residual networks, forming a multilevel architecture.The paper analyzes shortcut levels, shortcut types, and training epochs.
  • RoR also applies to Pre-ResNets and WRN, while stochastic depth provides an additional drop-path strategy for alleviating overfitting.
  • The paper evaluates RoR across CIFAR-10, CIFAR-100, SVHN, and ImageNet, reporting state-of-the-art results on these data sets.
  • RoR improves ResNet optimization ability and outperforms same-depth ResNets on different data sets using only a few identity shortcuts.

II. RELATED WORK

Related work addresses deep-network degradation through residual shortcuts, direct information paths, stochastic depth, and wider architectures. These approaches target vanishing gradients, overfitting, or training difficulty through different structural and stochastic modifications.

  • ResNets use shortcut connections around nonlinear layers because residual mappings are considered easier to optimize, improving training efficiency and reducing degradation.
  • The 1202-layer ResNet performs worse than the 110-layer ResNet on CIFAR-10, showing that extreme depth does not guarantee better results.
  • Pre-ResNets create a direct information path by using identity mappings and a BN-ReLU-conv order, reducing training difficulty and vanishing gradients.
  • Stochastic Depth randomly drops layers and bypasses them with identity mappings, addressing overfitting and reducing the vanishing problem.
  • Wide residual variants such as WRN and CRMN increase width or add memory mechanisms, with wider and shallower networks reported to train easily and perform well.

III. RESIDUAL NETWORKS OF RESIDUAL NETWORKS

RoR extends residual networks with root-, middle-, and final-level shortcuts, reframing optimization as learning residual mappings of residual mappings. The architecture supports ResNets, Pre-ResNets, and WRN variants while creating additional direct information paths.

  • III. RESIDUAL NETWORKS OF RESIDUAL NETWORKS: RoR adds shortcuts above original residual blocks level by level, constructing a multilevel residual network.The paper describes root, middle, and final shortcut levels when m=3.
  • III. RESIDUAL NETWORKS OF RESIDUAL NETWORKS: With m=3, RoR uses root-level, middle-level, and final-level shortcuts across three residual block groups.The original residual blocks are treated as final-level shortcuts, while additional shortcuts are added above groups and subdivisions.
  • III. RESIDUAL NETWORKS OF RESIDUAL NETWORKS: RoR is constructed on ResNets, Pre-ResNets, and WRN, with block ordering and identity mappings adjusted for each residual-network family.ResNet-based RoR uses conv-BN-ReLU blocks, while Pre-RoR and RoR-WRN use BN-ReLU-conv with identity mappings for g, h, and f.
  • III. RESIDUAL NETWORKS OF RESIDUAL NETWORKS: RoR transfers the learning problem from the original mapping to the residual mapping of residual mapping.The paper motivates this by treating the output of the original residual network as another residual mapping.
  • III. RESIDUAL NETWORKS OF RESIDUAL NETWORKS: RoR creates direct paths between different original residual blocks, allowing information to propagate from upper blocks to lower blocks.These paths are formed by extra shortcuts and are intended to alleviate vanishing gradients.

IV. OPTIMIZATION OF ROR

RoR optimization requires selecting architectural and training parameters that govern shortcut structure, training duration, and regularization.

  • IV. OPTIMIZATION OF ROR: RoR optimization requires choosing the shortcut level number, identity mapping type, maximum epoch number, and use of drop-path.These parameters and principles are identified as important optimization decisions.

A. Shortcut level number of RoR

RoR performance depends on selecting an appropriate number and type of shortcut levels. The best choices vary with dataset and balance optimization benefits against overfitting and added parameters.

  • A. Shortcut level number of RoR: RoR with m=3 achieved the best performance, while m=4 or 5 performed worse because additional levels increase branches, parameters, and overfitting.Too few levels provide smaller improvements, so the level count must balance optimization and overfitting.
  • B. Identity Mapping Types of RoR: Type C shortcuts were not selected for final-level shortcuts because their marginal performance benefit over Type B comes with too many extra parameters.The paper therefore used Type A or Type B for final-level shortcuts.
  • B. Identity Mapping Types of RoR: Type B shortcuts performed better than Type A on CIFAR-10, whereas Type A performed better than Type B on CIFAR-100.The paper attributes this dataset dependence to CIFAR-100's greater overfitting risk and fewer training examples per class.
  • B. Identity Mapping Types of RoR: RoR uses Type B projection shortcuts in Levels 1 and 2, while the final level uses Type A on CIFAR-100 or Type B on CIFAR-10.Using Type B in other levels with the dataset-specific final-level choice outperformed using Type A throughout for m=2 and m=3.

C. Maximum Epoch Number of RoR

The authors found that 164 training epochs were inadequate for optimizing ResNets and RoR. Increasing training to 500 epochs improved performance on both CIFAR-10 and CIFAR-100.

  • C. Maximum Epoch Number of RoR: 500 epochs achieved better performance than 164 epochs for 110-layer ResNets and RoR-3 on CIFAR-10.This comparison is shown in Fig. 8.
  • C. Maximum Epoch Number of RoR: 500 epochs achieved better performance than 164 epochs for 110-layer ResNets and RoR-3 on CIFAR-100.This comparison is shown in Fig. 9.

D. Drop Path by Stochastic Depth

The drop-path method, implemented through Stochastic Depth (SD), was evaluated as an overfitting countermeasure for RoR. Its value differed between CIFAR-10 and CIFAR-100.

  • D. Drop Path by Stochastic Depth: CIFAR-100 overfitting motivated adding SD to RoR experiments, while CIFAR-10 overfitting was considered non-critical.The experimental setup compared RoR and ResNet baselines across CIFAR benchmarks using the described architectures and training procedures.
  • D. Drop Path by Stochastic Depth: On CIFAR-10, 110-layer ResNets without SD achieved a 5.43% test error, while RoR-3 without SD achieved 5.08%.The 110-layer RoR-3 improvement over the 110-layer ResNet baseline was 6.4%.
  • D. Drop Path by Stochastic Depth: SD had little effect on CIFAR-10 performance, although it reduced training time without affecting performance.The authors therefore included SD in subsequent CIFAR-10 experiments.

C. CIFAR-100 Classification by RoR

On CIFAR-100, RoR-3 without SD was not ideal, but adding SD substantially improved both 110-layer and 164-layer RoR-3 models. The results support SD as an overfitting countermeasure in this setting.

  • C. CIFAR-100 Classification by RoR: 23.48% and 22.47% test errors were obtained by 110-layer and 164-layer RoR-3+SD on CIFAR-100.These models outperformed the corresponding ResNet and ResNet+SD baselines by the reported relative percentages.
  • C. CIFAR-100 Classification by RoR: 110-layer and 164-layer ResNets without SD achieved 26.80% and 25.85% test errors on CIFAR-100.The corresponding RoR-3 models without SD were reported as not ideal.
  • C. CIFAR-100 Classification by RoR: Adding extra branches and convolutional layers to RoR can escalate overfitting on CIFAR-100.The paper links this observation to the poor performance of RoR-3 without SD.
  • C. CIFAR-100 Classification by RoR: The two-convolution residual block B(3,3) outperformed B(3,3,3) when RoR was fully trained for 500 epochs.At 164 epochs, B(3,3,3) instead achieved better performance.

E. Versatility of RoR for other residual networks

RoR extends residual architectures with additional level-wise shortcuts and can be adapted to Pre-ResNets and WRN. Its benefits depend on controlling depth, width, and overfitting, with stochastic depth improving WRN-based variants.

  • Architecture adaptations: RoR adapts residual networks by adding level-wise shortcuts, while Pre-RoR uses two-level shortcuts to accommodate the BN-ReLU-conv ordering of Pre-ResNets.WRN-based RoR replaces the original filter counts with width-scaled filters.
  • Pre-RoR results: 14.1% and 2.4%: 164-layer Pre-RoR-3+SD outperformed 164-layer Pre-ResNets and Pre-ResNets+SD on CIFAR-100.The corresponding errors were 4.51% on CIFAR-10 and 21.94% on CIFAR-100.
  • RoR-WRN results: 4.59% and 22.48%: RoR-3-WRN40-2+SD achieved these errors on CIFAR-10 and CIFAR-100, respectively.Without stochastic depth, RoR-3-WRN could overfit more readily because WRN adds feature planes and parameters.
  • Depth limits: Very deep RoR remained vulnerable to vanishing gradients: performance decreased as depth increased from 164 to 182 and 218 layers.Pre-RoR behaved differently, with accuracy increasing through 1202 layers and reaching 4.49% CIFAR-10 error and 20.64% CIFAR-100 error.
  • Depth and width: RoR-WRN performance improved with depth and width, but widening increased parameters exponentially and required stochastic depth to reduce overfitting.The paper recommends increasing depth until performance saturates, then increasing width until the best results are reached.

G. Training time comparison on CIFAR-10/100

RoR-3-110 required similar training time to ResNets-110 on CIFAR-10/100, while stochastic depth provided an approximately 25% speedup.

  • Training time: Similar training times: ResNets-110 and RoR-3-110 required comparable training time on CIFAR-10/100.The comparison indicates that adding RoR shortcuts did not add extra training time relative to the original residual network.
  • Stochastic depth: About 25% speedup: stochastic depth consistently reduced training time in the comparison.This conclusion matches the speedup reported for stochastic depth in prior work.

H. SVHN Classification by RoR

RoR-3-WRN58-4+SD achieved strong SVHN classification results under the paper’s standard training setup. The model reached 1.59% test error and outperformed the WRN baselines reported here.

  • SVHN setup and result: 1.59% test error: RoR-3-WRN58-4+SD achieved this result on SVHN without data augmentation.The experiment used all training samples, preprocessing by mean subtraction and standard-deviation scaling, and batch size 128.
  • Baseline comparison: 5.9% and 4.2%: RoR-3-WRN58-4+SD outperformed WRN58-4 and WRN58-4+SD on SVHN by these reported amounts.The comparison was made using test error on the SVHN classification task.
  • Training behavior: After 700×200 iterations, WRN58-4 and RoR-3-WRN58-4 began to overfit, whereas the reported RoR-3-WRN58-4+SD result remained the strongest.Figure 14 plots test error across training epochs for these models.
  • Cross-dataset comparison: RoR’s broader comparisons included CIFAR-10, CIFAR-100, and SVHN, where the paper reported state-of-the-art results using a simple residual-network modification.The paper attributes the approach to optimizing residual mappings of residual mappings with added shortcut levels.
  • Parameter scale: 13.3M parameters: the best RoR-3-WRN58-4+SD model achieved the paper’s reported new state-of-the-art performance with this model size.The paper contrasts this result with larger competing models and reports errors of 3.77%, 19.73%, and 1.59%.

J. ImageNet Classification

The paper extends RoR evaluation from small image datasets to ImageNet classification and reports state-of-the-art performance there. ImageNet training was slower for RoR, so the authors pretrained from ResNet models.

  • Dataset and scope: ImageNet 2012 contains 1.28 million high-resolution training images, 50,000 validation images, and 1,000 object categories.The dataset evaluation broadens the paper beyond the preceding small-scale, low-resolution image benchmarks.
  • Training procedure: RoR training on ImageNet was slower than ResNet training, so the authors initialized RoR using pretrained ResNet models.The passage describes this as a practical training choice for the ImageNet experiments.
  • ImageNet result: RoR-3 models achieved state-of-the-art results compared with ResNets on ImageNet classification.The paper’s conclusion identifies ImageNet among the image-classification datasets where RoR obtained new state-of-the-art performance.
Loading 1608.02908v2…