Source-linked AI summary
Shake-Shake regularization
Xavier Gastaldi
TL;DR
Residual networks can overfit small datasets, motivating a regularizer for multi-branch architectures. The paper replaces branch summation with stochastic affine blending, reports state-of-the-art CIFAR results, and finds encouraging evidence beyond residual networks and Batch Normalization. Remaining questions concern the method’s exact dynamics and broader scope.
Problem
Residual networks still overfit on small datasets, while existing regularization approaches do not fully address this problem.
Method
Shake-Shake regularization replaces standard summation of parallel branches with stochastic affine combinations during training.
Results
The method reaches test errors of 2.86% on CIFAR-10 and 15.85% on CIFAR-100, while experiments without skip connections or Batch Normalization are encouraging.
Takeaways & Limitations
The results support using branch decorrelation to combat overfitting and suggest potential applicability to architectures without ResNets or Batch Normalization.
Takeaways & Limitations
The method’s exact dynamics remain unresolved, and fewer CIFAR-100 tests were performed because of the larger model and long training time.
Abstract
from arXiv · showhide
The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem. The idea is to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination. Applied to 3-branch residual networks, shake-shake regularization improves on the best single shot published results on CIFAR-10 and CIFAR-100 by reaching test errors of 2.86% and 15.85%. Experiments on architectures without skip connections or Batch Normalization show encouraging results and open the door to a large set of applications. Code is available at https://github.com/xgastaldi/shake-shake
1 Introduction
The paper addresses overfitting in multi-branch networks by stochastically blending parallel branches during training, extending regularization beyond standard dropout and drop-path approaches.
- Motivation: Residual networks remain prone to overfitting on small datasets despite techniques including weight decay, early stopping, dropout, Batch Normalization, and noisy-gradient SGD.The passage frames overfitting as a continuing problem for powerful residual models.
- Contribution: Shake-Shake regularization replaces the standard summation of parallel branches with a stochastic affine combination.The method targets the generalization ability of multi-branch networks.
- Mechanism: The method stochastically blends viable intermediate tensors by multiplying the whole tensor with a single scalar coefficient rather than applying element-wise noise.This distinguishes Shake-Shake from dropout variants such as shakeout and whiteout.
- Relation to prior methods: The approach relates to drop-path and gradient-noise methods, but scales residual branches continuously instead of completely dropping them or adding element-wise noise.The comparison places Shake-Shake among stochastic regularization methods for multi-branch networks.
- Training procedure: During training, scaling coefficients are replaced with new random numbers before each forward and backward pass, producing stochastic forward and backward flows.At test time, coefficients are set to their expected value of 0.5.
2 Improving on the best single shot published results on CIFAR
The experiments evaluate Shake-Shake variants across CIFAR-10 and CIFAR-100, including image-level coefficient updates and a ResNeXt-based CIFAR-100 model. Shake-Shake achieves strong error rates while its training curves and image-level application support improved regularization.
- Implementation: Image-level Shake-Shake multiplies each image slice in a mini-batch by a scalar αi.j or 1 − αi.j.For a 128-image batch, each slice has dimensions 64x16x16 inside the second stage of a 26 2x32d model.
- CIFAR-10: Image-level coefficient application improves regularization, while Shake-Shake training curves become more advantageous as width increases to 64d.Shake-Keep has little effect, Even-Shake works only at image level, and Shake-Even and Shake-Shake are strong at 32d.
- CIFAR-100: The evaluated CIFAR-100 network is a ResNeXt-29 2x4x64d model without pre-activation, using four grouped convolutions per branch.Its larger 34.4M-parameter size and long training time led to fewer tests than on CIFAR-10.
- CIFAR-100: Reducing the CIFAR-100 batch size from 128 to 32 with 2 GPUs is required for the E-E-B network to produce competitive results.The smaller batch size increases regularization and makes S-E-I a slightly better choice.
- CIFAR-10: 2.86% test error is reached by the 26 2x96d Shake-Shake-Image ResNet on CIFAR-10.The result averages five runs, with a median of 2.87%, minimum of 2.72%, and maximum of 2.95%.
- CIFAR-100: 15.85% test error is obtained on CIFAR-100 after adding shake-even regularization to a modified ResNeXt-29 8x64d baseline.The baseline reaches 16.34%; the shake-even result averages three runs, with median 15.85%, minimum 15.66%, and maximum 16.04%.
3 Correlation between residual branches
The experiments examine whether shake-shake regularization decorrelates residual branches and whether summation aligns corresponding layers. Results indicate reduced output correlation and higher same-layer correlation across branches.
- Correlation measurement: The experiment forwards each test image through both residual branches and stores their output tensors for comparison.Branch outputs are flattened before covariance and variance calculations.
- Correlation measurement: Correlation is computed from corresponding-vector covariance and variances after all test images have been processed.The resulting covariance and variances are used to calculate correlation.
- Output correlation: Correlation between the two residual branches seems reduced by regularization, supporting the assumption that the branches learn different representations.The test used three E-E-B and three S-S-I CIFAR-10 models.
- Layer alignment: The analysis treats layer alignment as a concern because residual-block summation may align layers across the left and right branches.Layer-wise correlations are calculated for configurations involving the first three layers of each block.
- Layer alignment: Same-layer pairings such as L1R1 and L2R2 have higher correlation than other pairings, consistent with summation-induced alignment.Figure 4 reports layer-wise correlations between the first three layers of each residual block.
4 Regularization strength
This section tests how changing backward-pass coefficients relative to forward-pass coefficients affects regularization strength. The effect appears stronger as the coefficients diverge, with a notable change when the backward coefficient crosses 0.5.
- Coefficient relationship: The backward pass can assign a large weight to a branch that received a small forward-pass weight, and vice versa.α_i.j denotes the forward coefficient and β_i.j the backward coefficient for the same image and residual-block position.
- Tested methods: Method 1 sets β_i.j = 1 - α_i.j, producing a drastic effect that keeps training error high.The comparison used CIFAR-10 26 2x32d image-level models against a Shake-Keep-Image model.
- Observed effects: The regularization effect seems linked to the relative position of β_i.j compared with α_i.j.Methods M2 to M5 were designed to investigate Method 1's strong effect.
- Observed effects: The further β_i.j moves from α_i.j, the stronger the regularization effect appears to become.The observations are intended to help control regularization strength more precisely.
- Observed effects: A jump in regularization strength seems to occur when β_i.j crosses 0.5.This is reported as an observed pattern rather than a definitive threshold.
5 Removing skip connections / Removing Batch Normalization
Experiments test shake-shake regularization without skip connections and without Batch Normalization. The method works in some altered architectures, but its strength and stability depend strongly on architecture, capacity, and coefficient range.
- Removing skip connections: Shake-shake regularization works without a skip connection in architecture A.Architecture A uses two branches with two convolutional layers per branch and was tested on a 26 2x32d model.
- Removing skip connections: On architecture A, S-S-I is too strong and underfits, while the softer S-E-I effect works better.The relative behavior could change with increased capacity such as 64d or 96d.
- Removing skip connections: Architecture B shows no effective regularization, suggesting interaction between the two convolutions in each branch may matter.Architecture B has one convolutional layer per branch and twice as many blocks.
- Removing skip connections: In architecture B, S-E-I and E-E-B have identical training and test curves, while S-S-I has a different training curve but nearly identical test behavior.The S-S-I test curve has smaller variance.
- Removing Batch Normalization: Without Batch Normalization, a softened S-E-I model with α_i.j restricted to [0.4,0.6] works fairly well but can easily diverge.Architecture C removes both skip connections and Batch Normalization; the depth and learning rate were also adjusted.
6 Conclusion
Experiments indicate that shake-shake regularization can combat overfitting by decorrelating branches and may extend to architectures without ResNets or Batch Normalization. The dynamics remain incompletely understood.
- Conclusion: Experiments indicate that shake-shake regularization combats overfitting by decorrelating branches in multi-branch networks.The conclusion describes the evidence as indicative rather than definitive.
- Conclusion: The method achieves state-of-the-art results on CIFAR datasets and could potentially improve architectures without ResNets or Batch Normalization.The conclusion characterizes these results as encouraging.
- Conclusion: The exact dynamics remain unresolved, and understanding them could expand the method's application to more complex architectures.This is presented as an open direction rather than an established outcome.