Revisiting Batch Normalization for Improving Corruption Robustness

1. Introduction

• Common corruptions such as Gaussian noise can substantially reduce the performance of models trained on clean images.The paper frames this as a practical concern for corruption robustness.Source: §1 · p. 1
• The paper treats clean and corrupted images as source and target domains, respectively, and proposes adapting BN statistics using a few corruption-domain representation samples.It motivates this approach by noting that a short inference period may have a stable corruption variant.Source: §1 · pp. 1–2
• Training with one corruption type can improve robustness to that type while reducing robustness to others, such as contrast and fog.The introduction gives Gaussian-noise augmentation as an example.Source: §1 · p. 1
• Adapting BN statistics improves corruption robustness across several benchmarks and further raises DeepAugment + AugMix accuracy from 58.1% to 63.3%.The approach is presented as orthogonal to existing state-of-the-art robustness methods.Source: §1 · p. 2

2.1. Batch Normalization

• BN is widely used in modern networks, but its training benefits have been linked to several mechanisms, and its effect on adversarial robustness involves trade-offs.Prior work associates BN with smoother optimization, decoupled length and direction, and regularization from mini-batch stochasticity; its effects on adversarial robustness are also discussed.Source: §2.1 · p. 2

2.2. From domain shift to corruption robustness

• Common corruptions can be viewed as a shift between clean and corruption domains, while augmentation may improve robustness to one corruption and reduce it for others.The related-work discussion also describes ImageNet-C and other corruption benchmarks.Source: §2.2 · p. 2

2.3. Aligning or Adapting feature statistics

• Adapting feature-normalization statistics with target-domain images is prior work that this paper tests using only a few corruption samples.The passage identifies AdaBN as adapting target-domain statistics and contrasts this direction with methods requiring special training data or augmentation.Source: §2.3 · p. 3

3.1. Revisiting classical batch normalization

• During training, BN normalizes mini-batch features; at inference, it uses population statistics estimated over the training data to make predictions deterministic for each input.For 2D feature layers, the mean and variance are computed over spatial positions as well as the batch.Source: §3.1 · p. 3

3.2. BN statistics: Style instead of content

• BN statistics vary more between clean and corrupted batches than between batches from the same distribution, supporting their interpretation as style rather than content.Figure 2 reports absolute mean and variance distances averaged over 100 measurements for a randomly selected layer of ImageNet-pretrained ResNet50.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Motivation for rectifying batch normalization

• Corruption severity lowers clean–corrupted feature similarity, while rectifying BN statistics raises similarity across severity levels.Figure 3 compares the unrectified model with the rectified model on corrupted inputs; the paper uses this observation to motivate rectification with a small number of representation samples.Source: §3.3 · p. 4, Figure 3 · p. 4

4. Experimental Setup

• ImageNet-C preserves ImageNet validation content across 1,000 classes and 50 images per class, with 15 test corruptions evaluated at five severities.It also includes four hold-out corruptions; evaluation uses the 15 test corruptions.Source: §4 · p. 4
• For each corruption severity, rectification uses a random batch of 32 corruption samples to estimate batch statistics and update population statistics.The batch mean and variance are calculated according to Eq. 2.Source: §4 · p. 4
• Evaluation covers state-of-the-art models trained on their corresponding clean datasets and uses corruption error as a metric.Source: §4 · p. 4
• Mean CE averages error across corruption types, while Acc averages top-1 accuracy; after adaptation, the metrics are marked with an asterisk.Higher Acc is better and lower mCE is better.Source: §4 · p. 4

5.1. Evaluation on CIFAR10-C and CIFAR100-C

• CIFAR10-C accuracy rises around 10% points for most models, with ResNet-20 gaining around 5% points.ResNet-20's mCE changes from 106.4% to 90.7% after rectification.Source: §5.1 · p. 5, Table 1 · p. 5
• On CIFAR100-C, WRN-28-10, ResNeXt-29, and DenseNet gain more than 12% points, while smaller models gain around 5% points.The passage reports that higher-capacity models also have higher initial accuracy.Source: §5.1 · p. 5, Table 2 · p. 5

5.2. Evaluation on ImageNet-C

• ImageNet-C results show significant gains of up to 10% points, with relatively smaller models improving slightly more than higher-capacity models.Higher-capacity models have initial accuracy higher than 46%.Source: §5.2 · p. 5, Table 3 · p. 5

5.3. Evaluation on state-of-the-art models

• 58.1% to 63.3%: adapting BN statistics improves the accuracy of DeepAugment + AugMix on ImageNet-C.The method also raises AugMix accuracy from 48.3% to 56.7%.Source: §5.3 · p. 5, Table 4 · p. 5

5.4. Evaluation on adversarially trained models

• Rectification adds around 3% points of corruption accuracy for adversarially trained CIFAR10 models and about 15% for adversarially trained ImageNet models.Before rectification, adversarially trained ImageNet ResNet-50 models have lower corruption accuracy than normal models.Source: §5.4 · p. 6, Table 5 · p. 6

6.1. Number of representation samples

• Above 32 representation samples, accuracy gains are not significant; 8 samples already yield performance above 70%, while 2 give a noticeable increase over no rectification.With only 1–4 samples, accuracy is relatively low and mCE relatively high, consistent with statistics that are not representative of the corruption dataset.Source: Figure 4 · p. 6, §6.1 · p. 6

6.2. Impact of mean and variance

• For CIFAR, rectifying only variance raises ResNet-20 Acc* from 73.0% to Acc*σ2 of 79.3%; ImageNet-C requires adapting both mean and variance.On CIFAR-C, mean-only rectification improves accuracy by no more than 1%, while variance-only adaptation can outperform adapting both parameters.Source: Table 6 · p. 6, §6.2 · p. 6
• Across corruption types, BN rectification helps most cases but slightly lowers accuracy for elastic, jpeg, snow, fog, and brightness corruptions.For those cases, rectifying either parameter alone can outperform rectifying both, while remaining around the original corruption robustness.Source: §6.2 · pp. 6–7, Figure 5 · p. 7

6.3. Location of rectified parameters

• Rectifying the first network third tends to achieve the highest accuracy, while rectifying only the final third performs about as well as no rectification.ResNet-50 and VGG-19 are exceptions, with marginally better performance from rectifying only the first third than from adapting all statistics.Source: Table 7 · p. 7, §6.3 · p. 7

6.4. Effectiveness of BN rectification

• For Gaussian noise, shot, impulse, glass, and motion corruptions, the gap between AccUB and Acc* is 25% ± 0.5; brightness is already about 1% from AccUB without rectification.Rectification does not improve every corruption, including defocus, snow, brightness, and jpeg.Source: §6.4 · p. 7, Table 8 · p. 8

6.5. Comparison with other normalization techniques

• On CIFAR10-C, GN and IN outperform non-rectified BN models, and ResNet-50 with IN reaches 83.2% versus 83.1% with rectified BN; on ImageNet-C, GN beats non-rectified BN while IN performs worst.For ImageNet-C, GN reaches 35.1% for ResNet-18 and 43.6% for ResNet-50, compared with 33.7% and 40.2% for non-rectified BN, respectively.Source: §6.5 · pp. 7–8, Table 9 · p. 8, Table 10 · p. 8

6.6. t-SNE analysis

• After BN rectification, ResNet-56 feature vectors for 1,000 severity-3 Gaussian-noise images are more clustered in the t-SNE visualization.Source: Figure 6 · p. 8, §6.6 · p. 8

6.7. Rectifying BN for adversarial perturbation

• Rectifying BN statistics with representative adversarial examples reduces rather than improves adversarial robustness.The authors suggest this indicates a difference between natural, average-case corruptions and worst-case adversarial corruption.Source: §6.7 · p. 8

• Rectifying BN statistics significantly improves corruption robustness across several benchmark datasets and models, with an accuracy boost of around 10%.The conclusion reports that variance adaptation suffices for CIFAR-C, while ImageNet-C requires adapting both mean and variance; gains also increase with representation samples until saturating near 32.Source: §7 · p. 8