PaperlayerView exact revision on arXiv

Source-linked AI summary

ShakeDrop Regularization for Deep Residual Learning

Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, Koichi Kise

arXiv:1802.02375v3cs.CV

TL;DR

Deep residual networks remain vulnerable to overfitting, and Shake-Shake is limited to ResNeXt while its effectiveness is not fully explained. This paper proposes ShakeDrop, combines perturbation with a stabilizing mechanism, and reports broad gains across residual architectures and tasks. Experiments identify conditions under which the method works successfully.

  • Problem

    Deep residual architectures still exhibit substantial generalization error, while Shake-Shake is limited to ResNeXt and lacks an identified explanation for its effectiveness.

  • Method

    ShakeDrop is a stochastic regularization method that uses Bernoulli-controlled branch behavior and random coefficients, with a stabilizing mechanism for difficult-to-train networks.

  • Results

    ShakeDrop generally outperformed Shake-Shake and RandomDrop across CIFAR and ImageNet experiments, and outperformed RandomDrop and vanilla networks on Faster R-CNN and Mask R-CNN.

  • Takeaways & Limitations

    ShakeDrop can, in principle, be applied across ResNeXt, ResNet, Wide ResNet, and PyramidNet, with experiments demonstrating conditions for successful use.

Abstract

from arXiv · show

Overfitting is a crucial problem in deep neural networks, even in the latest network architectures. In this paper, to relieve the overfitting effect of ResNet and its improvements (i.e., Wide ResNet, PyramidNet, and ResNeXt), we propose a new regularization method called ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake, which is an effective regularization method, but can be applied to ResNeXt only. ShakeDrop is more effective than Shake-Shake and can be applied not only to ResNeXt but also ResNet, Wide ResNet, and PyramidNet. An important key is to achieve stability of training. Because effective regularization often causes unstable training, we introduce a training stabilizer, which is an unusual use of an existing regularizer. Through experiments under various conditions, we demonstrate the conditions under which ShakeDrop works well.

I. INTRODUCTION

Deep residual architectures still overfit despite advances and established regularizers. The paper motivates ShakeDrop by extending Shake-Shake-style perturbation beyond ResNeXt while addressing unstable training.

  • I. INTRODUCTION: Deep residual architectures can still have large generalization errors because improved base architectures do not by themselves prevent overfitting.Regularization introduces additional information to improve test errors.
  • I. INTRODUCTION: Shake-Shake perturbs forward and backward calculations with different random variables and achieved the lowest error rate on CIFAR-10/100 with ResNeXt.Its limitations are that it applies only to ResNeXt and its effectiveness was not yet explained.
  • I. INTRODUCTION: ShakeDrop is proposed as a more effective regularizer with potential applicability to ResNeXt, ResNet, Wide ResNet, and PyramidNet.The method targets both three-branch and two-branch residual architectures.
  • II. REGULARIZATION METHODS FOR THE RESNET FAMILY: Shake-Shake uses independent random coefficients α and β for the forward and backward passes, respectively, with values drawn per image or batch.The paper suggests training six times as long as usual for Shake-Shake.
  • II. REGULARIZATION METHODS FOR THE RESNET FAMILY: RandomDrop regularizes ResNet-family networks by stochastically dropping building blocks, making the network appear shallow during learning.Its layer-level dropping differs from dropout, which drops elements.
  • II. REGULARIZATION METHODS FOR THE RESNET FAMILY: RandomDrop uses a Bernoulli variable bl to choose whether a residual branch is retained during training and uses its expected value during testing.The paper recommends a linear decay rule for the retention probability, with initial parameter pL = 0.5.

A. SHAKEDROP REGULARIZATION

ShakeDrop combines stochastic branch behavior with perturbations controlled by random coefficients. When the Bernoulli variable is active it matches the original network; otherwise it perturbs the residual branch.

  • A. SHAKEDROP REGULARIZATION: ShakeDrop is a proposed ResNet-family regularization method whose behavior is controlled by a Bernoulli variable bl and random coefficients α and β.The coefficients are independent uniform random variables in each element, while bl follows a layer-wise linear decay probability.
  • A. SHAKEDROP REGULARIZATION: When bl = 1, ShakeDrop becomes equivalent to the original residual network in both forward and backward training passes.This branch behavior preserves the unperturbed residual computation for the active case.
  • A. SHAKEDROP REGULARIZATION: When bl = 0, ShakeDrop perturbs the residual-branch calculation using α and β.The intermediate Single-branch Shake method illustrates this perturbation-based behavior.

B. DERIVATION OF SHAKEDROP

ShakeDrop extends Shake-Shake’s feature-space perturbation from multi-branch networks to two-branch architectures by perturbing a single residual branch. The resulting Single-branch Shake is effective in principle but can fail badly in practice.

  • Shake-Shake interpolates two residual-branch outputs with random weight α, which can be interpreted as synthesizing augmented feature-space data.The paper motivates this interpretation through prior work on feature-space interpolation.
  • Because Shake-Shake relies on multiple residual branches, it cannot directly regularize two-branch ResNet, Wide ResNet, and PyramidNet architectures.
  • Single-branch Shake instead applies random perturbation to one residual branch and uses its expected coefficient at test time.
  • A 110-layer PyramidNet with Single-branch Shake reached a 77.99% CIFAR-100 error rate in preliminary experiments, indicating severely poor practical performance.

3) Stabilization of training

Single-branch Shake fails because strong perturbations can destabilize learning without Shake-Shake’s two-branch fail-safe. ShakeDrop stabilizes this regime by probabilistically mixing the original and strongly perturbed networks.

  • Single-branch Shake lacks this stabilizing mechanism, so strong perturbation can prevent convergence rather than merely improving regularization.
  • Shake-Shake’s two residual branches provide a fail-safe: when one branch receives an excessively large gradient multiplier, the other remains trainable.
  • ShakeDrop uses RandomDrop’s mechanism in an unusual role, mixing the original network with a strongly perturbed network instead of simply making the network shallower.
  • The probabilistic mixture is intended to promote correct learning when the original network is selected and disturb learning when the perturbed network is selected.
  • The balance between the two networks is controlled by pL, which the paper identifies as important for achieving good performance.
  • ShakeDrop differs from RandomDrop and dropout because it explicitly generates new data and updates parameters using noisy gradients.

IV. PRELIMINARY EXPERIMENTS

Preliminary experiments show that ShakeDrop’s best α and β settings depend on the base architecture. PyramidNet favors case O, while ResNet favors case G, and the EraseReLU variant benefits from both.

  • pL was fixed at 0.5, while α and β ranges and their update rules were experimentally searched on CIFAR-100.
  • The experiments tested ShakeDrop on ResNet, ResNet with EraseReLU, and PyramidNet.
  • On all three architectures, RandomDrop case B outperformed the vanilla network case A.
  • PyramidNet: PyramidNet had the lowest error rates among the three architectures; only cases N and O exceeded case B, with case O best.
  • ResNet: ResNet showed a different pattern: case O did not converge, and only case G outperformed case B.
  • ResNet (EraseReLU version): Both cases O and G outperformed case B on ResNet with EraseReLU, with case G best; O favored EraseReLU architectures, whereas G favored non-EraseReLU architectures.

B. UPDATE RULE OF α AND β

ShakeDrop’s update rule explores how α and β should be sampled and paired across residual blocks. Pixel-level updates and architecture-dependent parameter ranges were effective, while fixed combinations showed that random sampling itself was not the primary source of improved error rates.

  • Update levels: ShakeDrop can sample α and β at batch, image, channel, or pixel level, while block dropping remains distinct from how the coefficients are chosen.Unlike Dropout and RandomDrop, a dropped block retains freedom in coefficient selection.
  • Update levels: Pixel-level updates produced the best result for both ResNet and PyramidNet.The tested ranges were α = 0, β ∈[0, 1] for ResNet and α ∈[−1, 1], β ∈[0, 1] for PyramidNet.
  • Fixed combinations: The combinations (1, 1), (1, 0), (−1, 1), and (−1, 0) were formed from the endpoints of PyramidNet’s parameter ranges.When bl = 0, one pair was selected uniformly from the pool instead of drawing α and β from continuous ranges.
  • Fixed combinations: pL strongly affected error rates because cases with a normal-state pair effectively altered the probability of selecting that state.For example, pL = 0.5 in case i was equivalent to pL = 0.75 in case l.

V. EXPERIMENTS

Experiments compare ShakeDrop with RandomDrop, Shake-Shake, and unregularized networks across several residual architectures on CIFAR datasets. The results support broad applicability and generally stronger performance, with a noted exception for shallow Wide ResNet with BN.

  • Experimental setup: ShakeDrop was evaluated against RandomDrop, Shake-Shake, and vanilla networks on ResNet, Wide ResNet, ResNeXt, and PyramidNet.The experiments used CIFAR datasets and covered both two-branch and three-branch architectures.
  • Experimental conditions: PyramidNet + ShakeDrop parameter combinations were evaluated using final-epoch top-1 error on CIFAR-100 under the batch-level update rule.Table 4 marks the α and β combinations used in each case.
  • Experimental conditions: The CIFAR experiments used α = 0, β ∈[0, 1] for original networks and α ∈[−1, 1], β ∈[0, 1] for modified networks whose residual branches end with BN.The latter group includes EraseReLU versions; ResNet and two-branch ResNeXt were examined in both original and EraseReLU forms.
  • Applicability: ShakeDrop applied to ResNet, Wide ResNet, and PyramidNet as well as three-branch ResNeXt architectures.This extends evaluation beyond the three-branch setting associated with Shake-Shake.
  • Results: ShakeDrop outperformed RandomDrop and Shake-Shake in most tested cases, while improving over the vanilla network in the reported Wide ResNet BN setting.For 28-layer Wide ResNet with BN, ShakeDrop did not outperform RandomDrop.

B. COMPARISON ON THE IMAGENET DATASET

On ImageNet, ShakeDrop improved performance across ResNet, ResNeXt, and PyramidNet, outperforming vanilla networks and, in the reported comparisons, RandomDrop. The experiments used 152-layer architectures and compared original with modified residual-block variants.

  • ShakeDrop was evaluated on 152-layer ResNet, ResNeXt, and PyramidNet architectures using ImageNet classification.The best parameters from CIFAR were used except for pL, which was selected as 0.9.
  • 0.84% and 0.15% gains over the vanilla network were reported for original ResNet and ResNeXt, respectively.
  • 0.60% and 0.29% gains were reported for PyramidNet over the vanilla network and RandomDrop, respectively.
  • Across ResNet, ResNeXt, and PyramidNet, ShakeDrop clearly outperformed RandomDrop and the vanilla network.The same passage reports that modified EraseReLU versions were worse than the original networks in contrast to the CIFAR cases.

D. SIMULTANEOUS USE OF SHAKEDROP WITH MIXUP

The paper also tested ShakeDrop together with mixup and found that the combination usually further reduced the base networks’ error rates. This supports using ShakeDrop alongside, rather than instead of, mixup.

  • In most cases, ShakeDrop further improved the error rates of networks already using mixup.
  • The authors characterize ShakeDrop as a collaborator rather than a rival to other regularization methods such as mixup.
  • The paper reports successful combined use of ShakeDrop and mixup, while also examining how pL and network depth relate to error rates.The depth experiments found that error rates varied with both pL and network depth.

APPENDIX A EXPERIMENTAL CONDITIONS

The appendix describes shared implementation practices and dataset-specific training conditions for CIFAR, ImageNet, and COCO experiments. It also specifies the evaluation setup used for feature extraction on COCO.

  • All experiments used back-propagation with SGD, Nesterov accelerated gradients, momentum, and multi-GPU training.Four GPUs were used for CIFAR and eight GPUs for ImageNet, producing separate stochastic observations across GPUs.
  • The implementations were based on publicly available code for the evaluated architectures and detection models, with learning conditions made as common as possible on CIFAR.The modified code was made available through the paper’s repository.
  • CIFAR training used normalization, 50% horizontal flips, padding, and random crops, with dataset- and architecture-specific initial learning rates.
  • ImageNet training used random aspect-ratio distortion, 224 × 224 random crops, 50% horizontal flips, color noise, scheduled learning-rate decay, and specified batch sizes.
  • COCO models used ResNet-152 backbones with FPN, and feature extraction used E(bl + α −blα) instead of ShakeDrop regularization.The COCO setup trained on the union of the 80k training set and 35k validation subset and evaluated on the remaining 5k validation images.
Loading 1802.02375v3…