Source-linked AI summary
Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift
Xiang Li, Shuo Chen, Xiaolin Hu, Jian Yang
TL;DR
The paper asks why Dropout and Batch Normalization often perform worse together and analyzes their train/test variance mismatch theoretically and statistically. It identifies variance shift, validates it across modern architectures, and evaluates ways to avoid the risk through placement and Dropout modification.
Problem
The paper addresses why Dropout and BN often conflict in common architectures despite each improving neural-network performance.
Method
The paper combines theoretical analysis of neural variance with statistical experiments across PreResNet, ResNeXt, DenseNet, and Wide ResNet on CIFAR10 and CIFAR100, then tests two mitigation strategies.
Results
Variance shift explains the performance drop, and experiments verify the explanation across four modern networks; placing Dropout after BN or modifying its formula generally works well.
Takeaways & Limitations
Avoiding variance shift through Dropout placement or a less variance-sensitive formula can provide extra improvements at nearly no additional cost.
Takeaways & Limitations
When BN directly follows Dropout, alleviating variance shift requires increasing the retain ratio p toward 1, effectively eliminating Dropout.
Abstract
from arXiv · showhide
This paper first answers the question "why do the two most powerful techniques Dropout and Batch Normalization (BN) often lead to a worse performance when they are combined together?" in both theoretical and statistical aspects. Theoretically, we find that Dropout would shift the variance of a specific neural unit when we transfer the state of that network from train to test. However, BN would maintain its statistical variance, which is accumulated from the entire learning procedure, in the test phase. The inconsistency of that variance (we name this scheme as "variance shift") causes the unstable numerical behavior in inference that leads to more erroneous predictions finally, when applying Dropout before BN. Thorough experiments on DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to the uncovered mechanism, we next explore several strategies that modifies Dropout and try to overcome the limitations of their combination by avoiding the variance shift risks.
1. Introduction
Dropout and Batch Normalization are individually effective but often fail to provide complementary gains when combined, with performance depending on architecture and placement. The paper attributes this disharmony to variance shift and evaluates strategies designed to avoid it.
- Motivation: Dropout and BN often produce worse performance together despite their individual effectiveness as regularization techniques.BN was reported to eliminate the need for Dropout in some cases, while both methods are widely used independently.
- Motivation: Modern ResNet, ResNeXt, and DenseNet results favor BN without Dropout, whereas Wide ResNet can benefit from including Dropout.These contrasting observations motivate the paper’s investigation of when the two techniques conflict or cooperate.
- Proposed explanation: Variance shift occurs because Dropout changes neural variance when switching from training to testing, while BN retains a training-accumulated moving variance.The mismatch can create numerical instability that amplifies through deeper layers and degrades final predictions.
- Proposed explanation: Without Dropout, inference variances remain close to BN’s accumulated moving variances and are associated with higher test accuracy.Figure 1 contrasts the shifted and closely matched variance behaviors.
- Evidence: Theoretical deductions under two general conditions and experiments on four modern networks across CIFAR10 and CIFAR100 verify the variance-shift explanation.The evaluated architectures are PreResNet, ResNeXt, DenseNet, and Wide ResNet.
- Strategies: Applying Dropout after all BN layers or modifying Dropout to reduce variance sensitivity generally avoids variance-shift risks and yields extra improvements.The strategies target the identified mechanism rather than simply removing one of the techniques.
2. Related Work and Preliminaries
The preliminaries describe Dropout as stochastic activation masking with train/test scaling and BN as deterministic normalization supported by moving statistics during inference. These distinct policies establish the setting for analyzing variance shift.
- Dropout: Dropout regularizes hidden units by multiplying activations by independent Bernoulli variables, retaining each unit with probability p.The retain ratio is p and the drop ratio is 1 − p.
- Train/test policies: Dropout’s test policy approximates an equally weighted geometric mean of predictions from exponentially many learned models sharing parameters.This approximation leaves the variance behavior of hidden units as the subject of the paper’s analysis.
- Dropout: During training, Dropout scales retained activations by 1/p, while inference uses unmasked activations after equivalent weight scaling.The paper adopts activation scaling during training and no activation modification at test time.
- Batch Normalization: BN normalizes each neuron toward zero mean and unit variance using mini-batch statistics during training.The normalization depends on the current mini-batch and participates in backpropagation.
- Batch Normalization: During inference, BN replaces mini-batch statistics with moving averages of neural means and variances accumulated during learning.These moving statistics are fixed for the inference linear transform.
3. Theoretical Analyses
The analysis identifies variance shift as the mismatch between Dropout’s train-to-test variance behavior and BN’s fixed statistical variance, then derives conditions governing that mismatch. It also relates the theory to channel width and observed statistics across modern architectures.
- Case (b): weighted responses: For case (b), the variance shift depends on p, channel width d, correlations, and the geometric term d(cos θ)^2 in the derived expression.The analysis treats convolutional and fully connected responses as weighted sums over a feature vector and assumes late-training weights remain effectively constant.
- Variance-shift mechanism: Dropout before BN changes a neural response’s variance during the train-to-test transition, while BN retains its moving variance from training.This variance mismatch is the paper’s central mechanism for variance shift.
- Case (a): Dropout before BN: In case (a), where BN directly follows Dropout, the shift coefficient satisfies △(p) ≤1 and approaches 1 only as the retain ratio p approaches 1.Thus, eliminating Dropout is the only way to remove the shift in this arrangement.
- Propagation through depth: The resulting numerical deviations can amplify through deeper BN layers, altering later variances and means and dislocating final predictions.With network weights fixed, changing only the network state from train to test changes response distributions and can produce wrong classifications.
- Architectural statistics: Approximately 0.01–0.10: (cos θ)^2 remains in this range across network structures and datasets, while d(cos θ)^2 increases with channel width.The WRN value is significantly larger than those of the other networks, mainly because WRN has larger channel width d.
- Conditions for reducing shift: The theoretical conditions for △(p,d) →1 are p →1 or d →∞, corresponding respectively to eliminating Dropout or greatly increasing channel width.These conditions explain why wide architectures can reduce the relative variance-shift risk.
4. Statistical Experiments
Experiments across four modern architectures and two CIFAR datasets measure variance shift and its relationship to train–test prediction instability. The results connect larger variance mismatch with inconsistent inference behavior and performance degradation, while WRN is less affected at moderate Dropout ratios.
- Experimental setup: Experiments evaluate DenseNet, PreResNet, ResNeXt, and Wide ResNet on CIFAR10 and CIFAR100 using several Dropout placements and ratios.The statistical procedure compares BN moving variances with real variances measured after switching fixed networks to evaluation mode.
- Variance shift and performance: When the Dropout ratio is 0.1, variance-shift curves remain near no-Dropout baselines and performance stays close to those baselines.The experiments report that reducing the drop ratio alleviates variance-shift risks.
- Variance shift and performance: WRN Dropout-(b) models with ratios 0.1, 0.3, and 0.5 approach the no-Dropout variance curve and outperform their baselines.The paper attributes this pattern to WRN’s larger channel dimension, which limits variance explosion at these ratios.
- Train–test instability: A Dropout ratio of 0.5 produces significant accuracy shifts when fixed networks move from train mode to evaluation mode on the training data.The comparison keeps model parameters fixed while applying the test policies of Dropout and BN.
- Train–test instability: Mode transfer changes final-response distributions and can cause wrong classifications, confirming unstable numerical behavior as a reason for performance drop.This effect occurs even on training examples and with all network weights fixed.
- Statistical correction: Adjusting BN moving means and variances with training data under evaluation mode improves all tested Dropout-(a)/(b) 0.5 models.DenseNet gains are approximately 2 points on CIFAR10 and 4.5 points on CIFAR100 in the reported examples.
5. Strategies to Combine Them Together
The paper tests two ways to combine Dropout with BN while avoiding variance shift: placing Dropout after BN layers or making its variance effect more stable. Both strategies are evaluated as routes to recovering or improving performance.
- Strategy design: The proposed strategies either apply Dropout after all BN layers or modify Dropout to reduce sensitivity to variance.The first strategy avoids placing Dropout before a subsequent BN layer; the second targets the test-phase weight scaling that disturbs variance.
- Empirical outcomes: Table 5 reports consistent improvements when Dropout is applied after all BN layers on representative state-of-the-art ImageNet models.The table values are averaged over five parallel runs with different random seeds.
- Strategy design: Placing Dropout after all BN layers adds one Dropout layer immediately before the softmax layer in the four architectures.This placement ensures subsequent layers do not include BN.
- Empirical outcomes: With drop ratio 0.1, Uout scales variance by approximately 0.9966777 rather than vanilla Dropout’s 0.9 in the illustrated case.The paper presents this as a variance shift rate much closer to 1.0.
- Empirical outcomes: Except for ResNeXt, nearly all architectures using Uout gain up to 0.2–0.3 accuracy on both CIFAR10 and CIFAR100.Uout is also reported as less affected by insufficient training data on CIFAR100 than last-layer Dropout.
6. Conclusion
When Dropout precedes Batch Normalization, their differing test-time policies shift neural variance during inference and can reduce performance. Two low-cost strategies avoid this risk and work well in practice.
- Dropout before Batch Normalization shifts neural variance during inference because the layers use distinct test policies.This variance shift makes neural variance improper as information flows through the network.
- The resulting variance mismatch leads to unexpected final predictions and reduced performance.
- Applying Dropout after all Batch Normalization layers avoids the variance-shift risk.
- Modifying Dropout’s formula makes it less sensitive to variance and helps avoid the same risk.
- Both strategies work well in practice, with nearly free extra cost and easy implementation.