ShakeDrop Regularization for Deep Residual Learning

I. INTRODUCTION

• Deep residual architectures can still have large generalization errors because improved base architectures do not by themselves prevent overfitting.Regularization introduces additional information to improve test errors.Source: §I · p. 1
• Shake-Shake perturbs forward and backward calculations with different random variables and achieved the lowest error rate on CIFAR-10/100 with ResNeXt.Its limitations are that it applies only to ResNeXt and its effectiveness was not yet explained.Source: §I · p. 1
• ShakeDrop is proposed as a more effective regularizer with potential applicability to ResNeXt, ResNet, Wide ResNet, and PyramidNet.The method targets both three-branch and two-branch residual architectures.Source: §I · pp. 1–2

II. REGULARIZATION METHODS FOR THE RESNET FAMILY

• Shake-Shake uses independent random coefficients α and β for the forward and backward passes, respectively, with values drawn per image or batch.The paper suggests training six times as long as usual for Shake-Shake.Source: §II · p. 2
• RandomDrop regularizes ResNet-family networks by stochastically dropping building blocks, making the network appear shallow during learning.Its layer-level dropping differs from dropout, which drops elements.Source: §II · p. 2
• RandomDrop uses a Bernoulli variable bl to choose whether a residual branch is retained during training and uses its expected value during testing.The paper recommends a linear decay rule for the retention probability, with initial parameter pL = 0.5.Source: §II · p. 2

A. SHAKEDROP REGULARIZATION

• ShakeDrop is a proposed ResNet-family regularization method whose behavior is controlled by a Bernoulli variable bl and random coefficients α and β.The coefficients are independent uniform random variables in each element, while bl follows a layer-wise linear decay probability.Source: A. SHAKEDROP REGULARIZATION · p. 2
• When bl = 1, ShakeDrop becomes equivalent to the original residual network in both forward and backward training passes.This branch behavior preserves the unperturbed residual computation for the active case.Source: A. SHAKEDROP REGULARIZATION · p. 2
• When bl = 0, ShakeDrop perturbs the residual-branch calculation using α and β.The intermediate Single-branch Shake method illustrates this perturbation-based behavior.Source: A. SHAKEDROP REGULARIZATION · p. 2, Figure 1 · p. 3

• Shake-Shake interpolates two residual-branch outputs with random weight α, which can be interpreted as synthesizing augmented feature-space data.The paper motivates this interpretation through prior work on feature-space interpolation.Source: 1) Interpretation of Shake-Shake… · p. 3
• Because Shake-Shake relies on multiple residual branches, it cannot directly regularize two-branch ResNet, Wide ResNet, and PyramidNet architectures.Source: 2) Single-branch Shake Regularization · p. 3
• Single-branch Shake instead applies random perturbation to one residual branch and uses its expected coefficient at test time.Source: 2) Single-branch Shake Regularization · p. 3
• A 110-layer PyramidNet with Single-branch Shake reached a 77.99% CIFAR-100 error rate in preliminary experiments, indicating severely poor practical performance.Source: 2) Single-branch Shake Regularization · p. 3

• Single-branch Shake lacks this stabilizing mechanism, so strong perturbation can prevent convergence rather than merely improving regularization.Source: 3) Stabilization of training · pp. 3–4
• Shake-Shake’s two residual branches provide a fail-safe: when one branch receives an excessively large gradient multiplier, the other remains trainable.Source: 3) Stabilization of training · p. 4
• ShakeDrop uses RandomDrop’s mechanism in an unusual role, mixing the original network with a strongly perturbed network instead of simply making the network shallower.Source: 3) Stabilization of training · p. 4
• The probabilistic mixture is intended to promote correct learning when the original network is selected and disturb learning when the perturbed network is selected.Source: 3) Stabilization of training · p. 4, Figure 2 · p. 4
• The balance between the two networks is controlled by pL, which the paper identifies as important for achieving good performance.Source: 3) Stabilization of training · p. 4
• ShakeDrop differs from RandomDrop and dropout because it explicitly generates new data and updates parameters using noisy gradients.Source: C. RELATIONSHIP WITH EXISTING… · p. 4

• pL was fixed at 0.5, while α and β ranges and their update rules were experimentally searched on CIFAR-100.Source: §IV · p. 5
• The experiments tested ShakeDrop on ResNet, ResNet with EraseReLU, and PyramidNet.Source: A. RANGES OF α AND β · p. 5
• On all three architectures, RandomDrop case B outperformed the vanilla network case A.Source: A. RANGES OF α AND β · p. 5

PyramidNet

• PyramidNet had the lowest error rates among the three architectures; only cases N and O exceeded case B, with case O best.Source: A. RANGES OF α AND β · p. 5

ResNet

• ResNet showed a different pattern: case O did not converge, and only case G outperformed case B.Source: A. RANGES OF α AND β · p. 5

ResNet (EraseReLU version)

• Both cases O and G outperformed case B on ResNet with EraseReLU, with case G best; O favored EraseReLU architectures, whereas G favored non-EraseReLU architectures.Source: A. RANGES OF α AND β · p. 5

Update levels

• ShakeDrop can sample α and β at batch, image, channel, or pixel level, while block dropping remains distinct from how the coefficients are chosen.Unlike Dropout and RandomDrop, a dropped block retains freedom in coefficient selection.Source: B. UPDATE RULE OF α AND β · p. 5
• Pixel-level updates produced the best result for both ResNet and PyramidNet.The tested ranges were α = 0, β ∈[0, 1] for ResNet and α ∈[−1, 1], β ∈[0, 1] for PyramidNet.Source: B. UPDATE RULE OF α AND β · p. 5

Fixed combinations

• The combinations (1, 1), (1, 0), (−1, 1), and (−1, 0) were formed from the endpoints of PyramidNet’s parameter ranges.When bl = 0, one pair was selected uniformly from the pool instead of drawing α and β from continuous ranges.Source: C. COMBINATIONS OF (α, β) FOR… · p. 5
• pL strongly affected error rates because cases with a normal-state pair effectively altered the probability of selecting that state.For example, pL = 0.5 in case i was equivalent to pL = 0.75 in case l.Source: C. COMBINATIONS OF (α, β) FOR… · pp. 5–6

Experimental setup

• ShakeDrop was evaluated against RandomDrop, Shake-Shake, and vanilla networks on ResNet, Wide ResNet, ResNeXt, and PyramidNet.The experiments used CIFAR datasets and covered both two-branch and three-branch architectures.Source: A. COMPARISON ON CIFAR DATASETS · pp. 6–7

Experimental conditions

• PyramidNet + ShakeDrop parameter combinations were evaluated using final-epoch top-1 error on CIFAR-100 under the batch-level update rule.Table 4 marks the α and β combinations used in each case.Source: Table 4 · p. 7
• The CIFAR experiments used α = 0, β ∈[0, 1] for original networks and α ∈[−1, 1], β ∈[0, 1] for modified networks whose residual branches end with BN.The latter group includes EraseReLU versions; ResNet and two-branch ResNeXt were examined in both original and EraseReLU forms.Source: A. COMPARISON ON CIFAR DATASETS · p. 7

Applicability

• ShakeDrop applied to ResNet, Wide ResNet, and PyramidNet as well as three-branch ResNeXt architectures.This extends evaluation beyond the three-branch setting associated with Shake-Shake.Source: A. COMPARISON ON CIFAR DATASETS · p. 7

Results

• ShakeDrop outperformed RandomDrop and Shake-Shake in most tested cases, while improving over the vanilla network in the reported Wide ResNet BN setting.For 28-layer Wide ResNet with BN, ShakeDrop did not outperform RandomDrop.Source: A. COMPARISON ON CIFAR DATASETS · p. 7

• ShakeDrop was evaluated on 152-layer ResNet, ResNeXt, and PyramidNet architectures using ImageNet classification.The best parameters from CIFAR were used except for pL, which was selected as 0.9.Source: B. COMPARISON ON THE IMAGENET DATASET · p. 7
• 0.84% and 0.15% gains over the vanilla network were reported for original ResNet and ResNeXt, respectively.Source: B. COMPARISON ON THE IMAGENET DATASET · p. 7
• 0.60% and 0.29% gains were reported for PyramidNet over the vanilla network and RandomDrop, respectively.Source: B. COMPARISON ON THE IMAGENET DATASET · p. 7
• Across ResNet, ResNeXt, and PyramidNet, ShakeDrop clearly outperformed RandomDrop and the vanilla network.The same passage reports that modified EraseReLU versions were worse than the original networks in contrast to the CIFAR cases.Source: B. COMPARISON ON THE IMAGENET DATASET · p. 7

• In most cases, ShakeDrop further improved the error rates of networks already using mixup.Source: D. SIMULTANEOUS USE OF SHAKEDROP WITH… · p. 9
• The authors characterize ShakeDrop as a collaborator rather than a rival to other regularization methods such as mixup.Source: D. SIMULTANEOUS USE OF SHAKEDROP WITH… · p. 9
• The paper reports successful combined use of ShakeDrop and mixup, while also examining how pL and network depth relate to error rates.The depth experiments found that error rates varied with both pL and network depth.Source: D. SIMULTANEOUS USE OF SHAKEDROP WITH… · p. 9

• All experiments used back-propagation with SGD, Nesterov accelerated gradients, momentum, and multi-GPU training.Four GPUs were used for CIFAR and eight GPUs for ImageNet, producing separate stochastic observations across GPUs.Source: Appendix A · p. 9
• The implementations were based on publicly available code for the evaluated architectures and detection models, with learning conditions made as common as possible on CIFAR.The modified code was made available through the paper’s repository.Source: Appendix A · p. 9
• CIFAR training used normalization, 50% horizontal flips, padding, and random crops, with dataset- and architecture-specific initial learning rates.Source: Appendix A · p. 9
• ImageNet training used random aspect-ratio distortion, 224 × 224 random crops, 50% horizontal flips, color noise, scheduled learning-rate decay, and specified batch sizes.Source: Appendix A · pp. 9–10
• COCO models used ResNet-152 backbones with FPN, and feature extraction used E(bl + α −blα) instead of ShakeDrop regularization.The COCO setup trained on the union of the 80k training set and 35k validation subset and evaluated on the remaining 5k validation images.Source: Appendix A · p. 10