PaperlayerView exact revision on arXiv

Source-linked AI summary

Removing Batch Normalization Boosts Adversarial Training

Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, Zhangyang Wang

arXiv:2207.01156v1cs.LGcs.CV

TL;DR

Adversarial training can reduce clean accuracy, which remains important in real-world applications, while batch normalization struggles with the different statistics of clean and adversarial samples. NoFrost removes batch normalization from adversarial training. On ImageNet, it reports higher clean accuracy and PGD robustness than BN-based adversarial training, and additional augmentations extend the approach to out-of-distribution benchmarks.

  • Problem

    Adversarial training seeks robustness but can reduce clean accuracy, and batch normalization struggles to capture the different statistics of clean and adversarial training samples.

    Source: §1 · p. 1

  • Method

    NoFrost uses normalizer-free networks for adversarial training by removing all batch normalization layers.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    74.06% clean accuracy and 23.56% adversarial robustness against PGD attack for NoFrost, versus 59.28% clean accuracy and 13.57% adversarial robustness for BN-based AT.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NoFrost is reported to have better model smoothness and larger decision margins, alongside its improved accuracy-robustness trade-off.

    Source: §1 · p. 2

    NoFrost cannot be directly combined with existing test-time adaptation methods that rely on batch normalization.

    Source: §5 · p. 8

Abstract

from arXiv · show

Adversarial training (AT) defends deep neural networks against adversarial attacks. One challenge that limits its practical application is the performance degradation on clean samples. A major bottleneck identified by previous works is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT. Although the dominant approach is to extend BN to capture this mixture of distribution, we propose to completely eliminate this bottleneck by removing all BN layers in AT. Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT for its unexplored advantage on handling the mixture distribution challenge. We show that NoFrost achieves adversarial robustness with only a minor sacrifice on clean sample accuracy. On ImageNet with ResNet50, NoFrost achieves $74.06\%$ clean accuracy, which drops merely $2.00\%$ from standard training. In contrast, BN-based AT obtains $59.28\%$ clean accuracy, suffering a significant $16.78\%$ drop from standard training. In addition, NoFrost achieves a $23.56\%$ adversarial robustness against PGD attack, which improves the $13.57\%$ robustness in BN-based AT. We observe better model smoothness and larger decision margins from NoFrost, which make the models less sensitive to input perturbations and thus more robust. Moreover, when incorporating more data augmentations into NoFrost, it achieves comprehensive robustness against multiple distribution shifts. Code and pre-trained models are public at https://github.com/amazon-research/normalizer-free-robust-training.

1. Introduction

NoFrost removes batch normalization from adversarial training to address clean/adversarial distribution differences, aiming to preserve clean accuracy while improving robustness. The introduction reports stronger ImageNet results and extension to out-of-distribution robustness with additional augmentations.

1. Introduction

  • Because clean and adversarial images come from different distributions, BN struggles to capture both statistics, while mixture BN requires choosing a path without an inference-time oracle.Source: §1 · p. 1
  • NoFrost reports 11.96% higher robustness against APGD-CE and 0.42% higher clean accuracy than MBNAT, and 12.15% higher clean accuracy and 7.25% higher APGD-CE robustness than TRADES-FAT.Source: §1 · p. 2
  • Combining adversarial samples with additional augmentations extends NoFrost to adversarial and out-of-distribution robustness, with better or comparable performance on ImageNet-C, ImageNet-R, and ImageNet-Sketch.The extension uses DeepAugment and texture-debiased augmentation.Source: §1 · p. 2
  • NoFrost removes all BN layers and uses normalizer-free networks to address the mixture-distribution challenge in adversarial training.The paper presents normalizer-free models’ handling of mixture distributions as an unexplored advantage.Source: §1 · p. 2
  • 74.06% clean accuracy and 23.56% adversarial robustness against PGD attack for NoFrost, versus 59.28% clean accuracy and 13.57% adversarial robustness for BN-based AT.NoFrost’s clean accuracy drops 2.00% from standard training.Source: §1 · p. 2

2. Preliminary

The preliminary section frames robustness across adversarial and natural distribution shifts, then reviews the BN mixture-distribution challenge and normalizer-free networks.

2.1. Model Robustness

  • AT trains on clean and adversarial images, while OOD benchmarks probe corruptions, sketches, naturally occurring images, and renditions.The section also describes data augmentation as a popular approach to improving OOD robustness.Source: §2.1 · pp. 2–3

2.2. The Mixture Distribution Challenge

  • Clean and adversarial samples have different BN statistics, making it difficult for BN to estimate normalization statistics for their mixture.Figure 1 plots channel-wise running means and variances for the 15th ResNet26 layer under standard and adversarial training.Source: Figure 1 · p. 3, §2.2 · p. 3
  • MBN routes clean and adversarial images through separate BN paths, but test-time path selection lacks an oracle and trades clean accuracy against robustness.The clean path favors clean accuracy, while the adversarial path favors robustness.Source: §2.2 · p. 3

2.3. Normalizer-Free Networks

  • Normalizer-free networks aim to remove BN for hardware efficiency, and scaled weight standardization and adaptive gradient clipping support competitive accuracy and large-batch training.Scaled weight standardization constrains layer weights during the forward pass; adaptive gradient clipping supports strong data augmentations.Source: §2.3 · pp. 3–4

3. Method

NoFrost removes BN from adversarial training using normalizer-free networks; NoFrost* adds complementary augmentations to target multiple distribution shifts.

3.1. NoFrost for Adversarial Training

  • NoFrost removes all BN layers, uses normalizer-free networks with PGD-generated adversarial samples, and sets λ = 0.5.Without BN, it avoids the inference-time oracle path-selection limitation of MBNAT.Source: §3.1 · p. 4

3.2. NoFrost∗for Comprehensive Robustness

  • Adversarial training and DeepAugment excel at different distribution shifts, while combining augmentations may intensify the mixture-distribution challenge.The passage contrasts adversarial robustness with natural shifts and DeepAugment's rendition-shift strength with its limited adversarial benefit.Source: §3.2 · p. 4
  • NoFrost* adds DeepAugment and TDA to adversarial training, sampling either augmentation with half probability in its objective.The augmented image comes from DeepAugment or TDA, while the adversarial image is generated from the clean input.Source: §3.2 · p. 4

4. Experiments

Across ImageNet experiments, NoFrost improves the accuracy–robustness trade-off and extends to multiple distribution shifts. It also yields stronger measured robustness-related model properties than SAT.

4.1. Experimental Settings

  • Experiments use ImageNet with ResNet26 and ResNet50, measuring clean accuracy, adversarial accuracy, and OOD robustness on ImageNet-C, ImageNet-R, and ImageNet-Sketch.Adversarial evaluations include white-box and black-box attacks, including AutoAttack.Source: §4.1 · pp. 4–5

4.2. Adversarial Training Results

  • On ResNet26, NoFrost achieves 70.13% accuracy and 6.34% APGD-CE robustness, while MBNAT must trade sharply between the two as γ increases.At comparable accuracy MBNAT has 0% APGD-CE robustness; at comparable robustness it has 57.08% accuracy and 6.27% APGD-CE robustness.Source: Figure 2 · p. 5, §4.2 · p. 6
  • On ImageNet, NoFrost achieves significantly higher clean-image accuracy and better or comparable robustness than the compared adversarial training methods; on ResNet26, it exceeds TRADES-FAT by 14.48% clean and 0.55% against APGD-CE.Results cover ResNet26 and ResNet50 under perturbation magnitude ϵ = 8.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • Compared with SAT w/ IN, NoFrost achieves 13.35% higher accuracy and 1.18% higher PGD robustness.Both SAT w/ IN and NoFrost outperform the naive SAT counterpart on clean accuracy and PGD robustness.Source: §4.2 · pp. 6–7, Table 3 · p. 6
  • Across three random seeds, NoFrost has small standard deviations in clean accuracy and adversarial robustness.The evaluation uses ResNet26 and reports mean and standard deviation across the three runs.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. NoFrost Leads to More Robust Model Properties

  • Compared with SAT, NoFrost improves decision margins, boundary thickness, and model smoothness; MBNAT improves smoothness while leaving the other two properties almost unchanged.The paper describes all three properties as beneficial for model robustness.Source: §4.3 · p. 7, Table 5 · p. 7

4.4. Comprehensive Robustness

  • On ResNet26, NoFrost* achieves the best robustness across evaluated OOD datasets and attacks; on ResNet50, it leads on all but ImageNet-C, where DeepAugment is 3.06% better.On ResNet50, NoFrost* exceeds DeepAugment by 16.30% on PGD, 8.20% on Square, and 2.05% on ImageNet-R.Source: §4.4 · p. 8, Figure 3 · p. 8

5. Discussions

The discussion places NoFrost alongside approaches that modify BN for robustness and distinguishes its normalizer-free strategy. It notes that BN-dependent test-time adaptation methods are not directly applicable to NoFrost.

Related work

Limitations and future work

  • BN-dependent test-time adaptation methods cannot be directly combined with NoFrost; enabling test-time adaptation for normalizer-free networks remains future work.The authors also note that NoFrost may benefit from future improvements in normalizer-free networks and adversarial training.Source: §5 · p. 8

6. Conclusion

NoFrost removes all batch-normalization layers to improve adversarial training’s clean-accuracy and robustness trade-off. For MBNAT, test-time logit interpolation provides the best trade-off among the compared interpolation strategies.

6. Conclusion

  • NoFrost removes all BNs in adversarial training and achieves a more favorable clean-accuracy and adversarial-robustness trade-off than previous BN-based methods.The conclusion describes this as decent adversarial robustness with only minor clean-accuracy degradation.Source: §6 · p. 8

A. How to Interpolate between Two BN Branches in MBNAT

  • MBNAT (logits) averages the clean- and adversarial-branch logits at test time, while alternatives mix branch outputs at all or selected layers.The alternatives are named MBNAT (all) and MBNAT (random p%).Source: A. How to Interpolate between Two BN… · p. 12
  • MBNAT (logits) achieves the best robustness-accuracy trade-off curve among the compared interpolation strategies and is used as the default.Figure 5 presents the trade-off between robustness and accuracy for interpolation strategies on MBNAT with ResNet26.Source: A. How to Interpolate between Two BN… · p. 12

B.1. Histograms of Decision Margin, Boundary Thickness, and Model Smoothness

Histograms visualize decision margins, boundary thickness, and model smoothness for adversarially trained models, complementing average metrics reported in Table 5.

B.1. Histograms of Decision Margin, Boundary Thickness, and Model Smoothness

  • The section compares average decision margins, boundary thickness, and model smoothness for SAT, MBNAT, and NoFrost.These averages were numerically compared in Table 5.Source: §B.1 · p. 12
  • Histograms show the distributions of decision margin, boundary thickness, and model smoothness across the compared models.The histogram view gives more detailed information than the averages in Table 5.Source: §B.1 · p. 12
  • The histogram visualization presents another view of the Table 5 metric results rather than a separate numerical comparison.Source: §B.1 · p. 12

B.2. Normalizer-free Networks with Standard Training is Not Robust to Adversarial Attacks

Standard training on ResNet50 and NF-ResNet50 yields almost no adversarial robustness in the reported comparison. The authors attribute NoFrost’s robustness to combining adversarial training with a normalizer-free network structure.

B.2. Normalizer-free Networks with Standard Training is Not Robust to Adversarial Attacks

  • Table 6 covers standard training on clean images and methods using ImageNet with ResNet50 or NF-ResNet50.Source: Table 6 · p. 12
  • Standard training on both ResNet50 and NF-ResNet50 has almost zero adversarial robustness in the reported comparison.Table 6 compares standard training on these networks with adversarial training on NF-ResNet50.Source: §B.2 · p. 12, Table 6 · p. 12
  • The authors state that NoFrost’s robustness comes from combining the AT algorithm with an AT-friendly normalizer-free network structure, not from network structure alone.Source: §B.2 · p. 12

B.3. Robustness under Different Perturbation Magnitudes

NoFrost’s robustness advantage holds across the tested perturbation magnitudes. Figure 7 compares adversarial robustness for ImageNet-trained ResNet26 methods under targeted PGD attacks.

B.3. Robustness under Different Perturbation Magnitudes

  • NoFrost’s robustness advantage holds across targeted PGD perturbation magnitudes from 8 to 16.The compared models were trained with perturbation magnitude ϵ = 8.Source: §B.3 · p. 12, Figure 7 · p. 13
  • The evaluation compares models from Table 1, trained with ϵ = 8, under targeted PGD attacks spanning ϵ values from 8 to 16.Source: §B.3 · p. 12
  • Figure 7 presents adversarial robustness under different perturbation magnitudes for methods trained on ImageNet with ResNet26.Source: Figure 7 · p. 13

B.4. Adversarial Training Results with Small Perturbation Magnitudes

Under the small perturbation magnitudes ε = 2 and 4, NoFrost largely outperforms FastAT and FreeAT in adversarial robustness. The comparison evaluates ResNet50 using PGD-10 and PGD-50.

B.4. Adversarial Training Results with Small Perturbation Magnitudes

  • NoFrost largely outperforms FastAT and FreeAT in adversarial robustness under the small ε setting.The comparison covers perturbation magnitudes ε = 2 and 4.Source: §B.4 · p. 13, Table 7 · p. 13
  • Robustness is evaluated with PGD-10 and PGD-50, following Shafahi et al. (2019).The paper contrasts this small-ε setting with its main-text setting of ε = 8 for training and evaluation.Source: §B.4 · p. 13
  • Table 7 reports clean-image accuracy and accuracy under adversarial attacks at ε = 2 and 4.Source: Table 7 · p. 13

Related papers

Loading 2207.01156v1…