PaperlayerView exact revision on arXiv

Source-linked AI summary

On the effectiveness of adversarial training against common corruptions

Klim Kireev, Maksym Andriushchenko, Nicolas Flammarion

arXiv:2103.02325v2cs.LGcs.AIcs.CVstat.ML

TL;DR

The paper addresses uncertainty about whether adversarial training improves performance on common corruptions, despite their large ℓp distance from clean images. It evaluates carefully selected ℓp radii and an efficient LPIPS-based relaxation, reporting improved accuracy and calibration and state-of-the-art corruption performance when combined with existing augmentations.

  • Problem

    Prior work offered no consensus on whether ℓp adversarial training improves common-corruption performance, whose high ℓp distance from clean samples makes its usefulness unclear.

    Source: §1 · p. 2

  • Method

    The paper evaluates ℓp adversarial training with selected radii and introduces an efficient LPIPS relaxation based on layerwise perturbations.

    Source: §1 · p. 2

  • Results

    The approach improves the ℓp adversarial-training baseline and combines cumulatively with AugMix, DeepAugment, ANT, and SIN for state-of-the-art common-corruption performance.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Carefully selected ℓp adversarial training can improve corrupted-image accuracy and calibration, while the relaxed LPIPS approach can be combined with existing augmentations.

    Source: §1 · p. 2, §7 · p. 12

    Direct optimization with LPIPS is challenging because dLPIPS is defined by a neural network.

    Source: §5 · p. 8

Abstract

from arXiv · show

The literature on robustness towards common corruptions shows no consensus on whether adversarial training can improve the performance in this setting. First, we show that, when used with an appropriately selected perturbation radius, $\ell_p$ adversarial training can serve as a strong baseline against common corruptions improving both accuracy and calibration. Then we explain why adversarial training performs better than data augmentation with simple Gaussian noise which has been observed to be a meaningful baseline on common corruptions. Related to this, we identify the $\sigma$-overfitting phenomenon when Gaussian augmentation overfits to a particular standard deviation used for training which has a significant detrimental effect on common corruption accuracy. We discuss how to alleviate this problem and then how to further enhance $\ell_p$ adversarial training by introducing an efficient relaxation of adversarial training with learned perceptual image patch similarity as the distance metric. Through experiments on CIFAR-10 and ImageNet-100, we show that our approach does not only improve the $\ell_p$ adversarial training baseline but also has cumulative gains with data augmentation methods such as AugMix, DeepAugment, ANT, and SIN, leading to state-of-the-art performance on common corruptions. The code of our experiments is publicly available at https://github.com/tml-epfl/adv-training-corruptions.

1 Introduction

The paper asks whether adversarial training can improve robustness to common corruptions, where its utility is unclear because such corruptions are far from clean samples in ℓp distance. It argues that carefully chosen ℓp radii and an efficient perceptual-distance relaxation offer effective approaches, including alongside existing augmentations.

  • On CIFAR-10-C, ε = 1/255 yields significantly higher accuracy than the standardly used ε = 8/255 for adversarially trained ResNet-18 models.Source: Figure 1 · p. 1
  • Common corruptions have high ℓp distance from clean samples, leaving the usefulness of ℓp adversarial training for them unclear.Source: §1 · p. 2
  • ℓp adversarial training with an appropriately selected radius improves accuracy and calibration on corrupted images.Source: §1 · p. 2
  • Gaussian augmentation can overfit to its training perturbation size, whereas adversarial training does not show this overfitting.Source: §1 · p. 2
  • The paper introduces an efficient LPIPS adversarial-training relaxation based on layerwise perturbations, reported as at least as effective as prior approaches and significantly faster to train.Source: §1 · p. 2
  • The proposed relaxation combines cumulatively with AugMix, DeepAugment, ANT, and SIN, achieving state-of-the-art results on CIFAR-10-C and ImageNet-100-C.Source: §1 · p. 2

2 Related work

Related work covers common-corruption evaluation, data augmentation, and ℓp and non-ℓp adversarial robustness. Prior findings include mixed transfer across corruption types and differing conclusions about adversarial training on common corruptions.

Common image corruptions

  • Findings on transfer across corruption types differ: augmentation on one type often does not help others, while Gaussian augmentation has been reported to help broadly.Source: §2 · p. 2
  • CIFAR-10-C and ImageNet-C standardize evaluation across 15 corruptions in four categories, and common corruptions can degrade both accuracy and calibration.Source: §2 · p. 2
  • Test-time batch-normalization adaptation and large-scale contrastive pretraining have also been reported to improve robustness to common corruptions or distribution shifts.Source: §2 · pp. 2–3

Data augmentations

  • AugMix combines selected augmentations with Jensen-Shannon-divergence regularization and is among the strongest methods for common-corruption accuracy and calibration.Source: §2 · p. 3

ℓp adversarial robustness

  • Prior work reports mixed ℓp adversarial-training results on common corruptions: some models do not help on ImageNet-C, while CIFAR-10 models and AdvProp report improvements.Source: §2 · p. 3

Non-ℓp adversarial robustness

  • Non-ℓp approaches include input- or latent-space adversarial training and LPIPS-based perceptual adversarial training; latent-space success depends on generative-model quality and did not scale beyond simple image recognition datasets.Source: §2 · p. 3

3 ℓp adversarial training improves the performance on common corruptions

With a carefully selected perturbation radius, ℓp adversarial training improves common-corruption accuracy and calibration. Small perturbations are important: the most robust model is not necessarily the best on corruptions.

3 ℓp adversarial training improves the performance on common corruptions

  • Experiments evaluate accuracy on CIFAR-10-C and ImageNet-C using PreAct ResNet-18 unless noted otherwise, reporting accuracy averaged over all five severity levels; the datasets contain 15 synthetic corruptions.ImageNet-100 is used because grid-searching perturbation radii would be too expensive on full ImageNet.Source: §3 · p. 4
  • 82.7% accuracy is achieved with adversarial training at ε = 1/255, versus 74.5% with standard training and 73.8% at ε∞ = 8/255.The results show that selecting the most robust ℓp model does not yield the best common-corruption performance.Source: §3 · p. 5
  • For small ε, the study uses FGM/FGSM for ℓ2/ℓ∞ inner maximization, respectively, with 2× computational overhead.PGD is used with 10 steps at ε ∈ {8/255, 10/255} to prevent catastrophic overfitting and enable comparison with prior work.Source: §3 · p. 5
  • 10.8% ECE versus 16.6% for the standard model, and 6.7% versus 11.3% after temperature rescaling, for the most accurate ε∞ = 1/255 model.The passage reports that ECE decreases as ℓ∞ training radius increases; the stated comparisons are for the most accurate model.Source: §3 · pp. 5–6

4 Understanding the effect of adversarial training on image corruptions

ℓ2 adversarial training performs best among the tested natural baselines and improves accuracy across corruption types. The paper relates this advantage to worst-case training and contrasts it with Gaussian augmentation’s sensitivity to training noise magnitude.

Comparing natural baselines across corruption types

  • ℓ2 adversarial training is the best-performing tested method and improves accuracy for every corruption type, while Gaussian augmentation degrades digital and weather performance.Gaussian augmentation substantially improves noise-corruption performance; fog and contrast accuracy decline for all methods.Source: §4 · p. 6, Figure 3 · p. 6

Worst-case vs average-case behavior

  • Gaussian augmentation minimizes an average loss over perturbations near a sphere, whereas adversarial training minimizes worst-case loss within a ball.The paper notes that minimizing expected loss does not guarantee behavior inside the sphere.Source: §4 · p. 7
  • 100% Gaussian augmentation has its lowest loss near its training σ = 0.05, while 50% augmentation improves common-corruption accuracy to 85.0% from 80.5%.The 50% scheme alleviates σ-overfitting and improves clean accuracy to 93.2% from 92.5%; adversarial training does not show this overfitting and its 50% and 100% schemes perform nearly equally well.Source: §4 · p. 7

Local vs global ℓp behavior

  • Common corruptions have ℓ2 norms an order of magnitude larger than the ε = 0.1 used for ℓ2 adversarial training.The authors suggest that improving minimum distance to the decision boundary may also improve average distance, potentially explaining generalization to larger average-case perturbations.Source: §4 · pp. 7–8

5 Improving adversarial training by relaxing a perceptual distance

Because ℓ2 adversarial training appears more suitable than ℓ∞ training for common corruptions, the authors explore perceptual distances and introduce a relaxed LPIPS adversarial-training method. The relaxation permits efficient layerwise perturbations without requiring a corresponding input-space perturbation.

From ℓp distances to LPIPS

  • ℓ2 adversarial training appears more suitable than ℓ∞ adversarial training on both datasets, motivating exploration of perceptual distances for corruption robustness.The authors suggest perceptual distances may further improve corruption robustness.Source: §5 · p. 8
  • LPIPS is a learned perceptual distance based on neural-network activations and learned coefficients, and is reported to correlate better with network error than ℓ2 on CIFAR-10-C.Its error increase across severity levels is also more monotonic than with ℓ2.Source: §5 · p. 8

LPIPS adversarial training

  • The LPIPS projection needed for iterative adversarial training has no closed-form expression, making the projection computationally expensive and limiting scalability.Prior approximate LPIPS attacks also require tuning a Lagrange multiplier or performing approximate projection, and use 10-step attacks.Source: §5 · pp. 8–9

Relaxed LPIPS adversarial training

  • RLAT relaxes the requirement that layerwise perturbations correspond to an input-space perturbation, enabling a single-iteration attack with cheap per-layer ℓ2 projections.The method applies layerwise bounds at the layers used by LPIPS.Source: §5 · p. 9
  • The authors present layerwise adversarial training as a novel efficient relaxation of LPIPS adversarial training and apply it to general robustness, including common corruptions.Related layerwise adversarial training methods had been proposed previously.Source: §5 · p. 9

6 Experimental evaluation of RLAT

The experiments find that RLAT improves corruption accuracy and calibration on CIFAR-10-C and ImageNet-100-C, including when combined with established augmentations. They also report stronger LPIPS adversarial robustness and substantially lower training time than Fast PAT.

LPIPS robustness of RLAT

  • RLAT substantially improves LPIPS adversarial accuracy over ℓ2 adversarial training and Fast PAT on CIFAR-10, and the authors report no catastrophic overfitting for one-step RLAT or ℓ2 training.The evaluation uses a Lagrangian Perceptual Attack across LPIPS radii.Source: §6 · p. 10, Figure 6 · p. 10

Main experimental setup

  • The comparison covers adversarial-training, Gaussian-augmentation, AdvProp, Fast PAT, and four data-augmentation baselines, with main hyperparameters selected by grid search.The study avoids overlapping training augmentations because the benchmark targets unseen corruptions, while Gaussian augmentation is included as an illustrative exception.Source: §6 · pp. 10–11

Main experimental results

  • 84.1% versus 74.6% corruption accuracy on CIFAR-10-C and 48.8% versus 47.5% on ImageNet-100-C; RLAT also lowers ECE from 16.6% to 9.9% and from 10.0% to 9.1%, respectively.These comparisons are against standard models.Source: Table 2 · p. 11, §6 · p. 12
  • 88.5% versus 86.6% CIFAR-10-C accuracy and 54.8% versus 52.3% ImageNet-100-C accuracy when RLAT is combined with AugMix; RLAT also consistently improves ECE across settings.The reported combinations with DeepAugment, SIN, and ANT3x3 also improve corruption accuracy.Source: §6 · p. 12

Runtime of RLAT

  • RLAT takes 1.8 hours versus 9.4 hours for Fast PAT on CIFAR-10, with 1.8 hours versus 1.3 hours for ℓ2/ℓ∞ adversarial training.On ImageNet-100, RLAT takes 6.2 hours on one V100 GPU versus 120 hours for Fast PAT on four RTX 2080 Ti GPUs; Fast PAT uses a larger network.Source: §6 · p. 12

7 Conclusions and future work

The conclusion is that carefully tuned ℓp adversarial training can improve accuracy and calibration on common corruptions, while efficient relaxed LPIPS adversarial training further improves LPIPS robustness and can be combined with existing augmentations. The appendix collects additional implementation, derivation, and experimental material.

7 Conclusions and future work

  • The authors conclude that carefully chosen ℓp perturbation radii provide a strong corruption-robustness baseline, while efficient single-step RLAT improves LPIPS robustness and combines with existing augmentations.They identify possible future application to other domains, such as robustness to typos in natural language processing.Source: §7 · p. 12

Organization of the Appendix

  • The appendix covers setup and runtime, calibration and ℓ2 results, AdvProp and σ-overfitting analyses, LPIPS suitability and derivations, distribution shifts, and detailed corruption results.It also includes breakdowns by corruption and severity, larger architectures, and multiple random seeds.Source: Appendix Organization of the Appendix · p. 16

A Experimental details

The experiments use CIFAR-10 and ImageNet-100, with common-corruption results reported as average accuracy on both datasets. The appendix details the training, evaluation, hyperparameter-search, augmentation, and runtime procedures.

Datasets and evaluation metrics

  • Experiments use 32 × 32 CIFAR-10 images and 224 × 224 ImageNet-100 images, selecting every tenth ImageNet class because grid searches on full ImageNet exceed available resources.Corruption datasets include 15 main corruption types at five severity levels.Source: A Experimental details · p. 16
  • Average accuracy is reported on both corruption datasets because CIFAR-10 lacks a standard AlexNet reference for defining mCE and relative mCE.Relative mCE does not account for absolute accuracy; a constant model can score zero.Source: A Experimental details · p. 18

Training details

  • PreAct ResNet-18 models are trained with SGD and momentum 0.9 for 150 CIFAR-10 epochs or 100 ImageNet-100 epochs, using a scheduled learning-rate decay and the final model.No early stopping is used.Source: A Experimental details · p. 18

Augmentation and hyperparameter choices

  • The study avoids training augmentations that overlap with test corruptions, and searches main hyperparameters such as Gaussian σ and adversarial radius ε using corruption accuracy.For 100% Gaussian augmentation on ImageNet-100, the best tested σ was 0.001, yet it still harmed overall performance, which the authors attribute to σ-overfitting.Source: A Experimental details · p. 19

Runtime and augmentation details

  • RLAT is reported to be significantly faster than Fast PAT, with only a slight runtime overhead over ℓ2/ℓ∞ adversarial training.The appendix provides the full runtime comparison and implementation details for DeepAugment, SIN, and ANT.Source: A Experimental details · p. 19, Table 6 · p. 19

B Additional details and results on calibration

The section describes the calibration procedure and reports that ℓ2 adversarial training improves calibration on corrupted images, including after temperature rescaling.

Calibration details

  • ECE uses 15 equally sized bins, and temperature rescaling minimizes ECE by grid search over t ∈ [0.001, 1.0] and 1/t.Temperatures are optimized only on in-distribution test samples so corrupted out-distribution samples remain unseen.Source: B Additional details and results on… · p. 20

Calibration of ℓ2 adversarially trained models

  • 10.5% ECE vs 16.6% ECE for the standard model, and 5.8% vs 11.3% after temperature rescaling, for the most accurate ℓ2 model on corruptions.Across ℓ2 and ℓ∞ adversarial training, calibration improves before and after temperature rescaling; ℓ2 has lower ECE than ℓ∞ in both cases.Source: B Additional details and results on… · p. 20
  • Figure 7 presents expected calibration error for CIFAR-10-C models trained with different ℓ2 perturbation radii.The reported ECE trend decreases as the perturbation radius increases.Source: Figure 7 · p. 20, B Additional details and results on… · p. 20

C Ablation study for AdvProp and adversarial training

The ablation finds that mixing clean and adversarial examples does not benefit standard adversarial training, while AdvProp yields only a small gain and costs additional training time.

AdvProp baseline

  • AdvProp combines clean and adversarial examples using separate BatchNorm layers, then uses only the clean-example BatchNorm layers at test time.The method reports 94.7% standard accuracy and 82.9% common-corruption accuracy.Source: C Ablation study for AdvProp and… · p. 20

AdvProp comparison

Ablation study

  • AdvProp beats 100% standard adversarial training by only 0.2%, while 100% standard training performs comparably to 75% and better than 50% and 25% adversarial-example mixtures.AdvProp may require up to 50% more training time for the same number of adversarial examples.Source: C Ablation study for AdvProp and… · p. 21, Figure 8 · p. 21

D Details on σ-overfitting

On ImageNet-100, Gaussian augmentation can overfit to its training noise magnitude, while ℓ2 adversarial training avoids that pattern; the section also evaluates LPIPS as a corruption-distance metric and finds it generally better aligned with error rates.

D Details on σ-overfitting

  • 100% Gaussian augmentation severely overfits to its training noise magnitude; 50% augmentation mitigates but does not eliminate the effect, with a local loss maximum near σ = 0.2.At σ = 0.2, the loss is about 1.6× the loss at σ = 0.45; standard and ℓ2 adversarially trained models show no σ-overfitting trend over σ ∈ [0, 0.1].Source: D Details on σ-overfitting · pp. 21–22, Figure 9 · p. 22
  • The authors suggest σ-overfitting may explain why 50% adversarial training does not outperform 100%, and hypothesize that Gaussian augmentation's larger perturbation norms contribute to the difference.On CIFAR-10-C, the optimal ℓ2 radius is 0.1, compared with approximate Gaussian-vector norms of 2.77 for 100% augmentation and 5.54 for 50%.Source: D Details on σ-overfitting · p. 22

E Suitability of LPIPS for common corruptions.

  • LPIPS distances track corruption severity more plausibly than ℓ2 for several corruptions, including elastic transformations, JPEG, and frost, though LPIPS rises rapidly for noise corruptions.Distances are computed using a standardly trained VGG network, and full images are used rather than smaller patches.Source: E Suitability of LPIPS for common… · p. 23
  • LPIPS distance correlates more with error rates than ℓ2 across corruption types except noise, with the largest difference coming from digital corruptions.The reported correlation is averaged over corruption types and calculated across each type’s five severity levels.Source: E Suitability of LPIPS for common… · p. 23
  • LPIPS distances are less often non-monotonic across corruption severity than ℓ2 distances, while ℓ2 magnitude does not always reflect accuracy degradation.Fog and brightness have similar large ℓ2 magnitudes but differ substantially in their accuracy effects.Source: E Suitability of LPIPS for common… · p. 24, Table 8 · p. 24

F Details on the relaxed LPIPS adversarial training

The appendix details RLAT’s relaxed, layerwise LPIPS perturbations and computational efficiency, then reports its behavior on other distribution shifts, where gains are modest and no method wins across all shifts.

F Details on the relaxed LPIPS adversarial training

  • RLAT relaxes layerwise LPIPS perturbation constraints into ℓ2-bounded perturbations at selected layers, with bounds scaled by layer dimension and depth.The selected layers include the input, first convolution, and outputs of residual blocks; the scaling is described as empirically more effective than other simple schemes.Source: F Details on the relaxed LPIPS… · p. 26
  • RLAT requires only 2× training overhead because it can use single-step adversarial training.The appendix contrasts this with the more computationally expensive perceptual attacks and notes that standard PGD requires T forward and backward passes.Source: F Details on the relaxed LPIPS… · pp. 24–26
  • All three adversarial-training methods substantially improve ℓ2 robustness over the standard model on CIFAR-10.The evaluation uses APGD-CE with 100 iterations and five random restarts across ℓ2 radii 0.05, 0.1, 0.25, and 0.5.Source: Figure 10 · p. 27, F Details on the relaxed LPIPS… · p. 27

G Performance under various distribution shifts

  • RLAT moderately improves ImageNet-100-A and Stylized ImageNet-100 accuracy by 0.4% and 1.1%, respectively, but changes ImageNet-100-R accuracy by −0.1%.The models were selected to maximize ImageNet-100-C performance; target-dataset model selection could improve results, and no method performs best on every shift.Source: G Performance under various… · p. 28, Table 9 · p. 28
  • RLAT adds gains when combined with some augmentations, including SIN on ImageNet-100-R, where accuracy increases from 35.1% to 37.0%.The authors conclude that improvements are relatively small unless a target distribution shift is used for training.Source: G Performance under various… · p. 28

H Supplementary figures and tables

Supplementary experiments find that RLAT generally outperforms adversarial-training baselines, while corruption-specific performance depends on the perturbation radius and severity. They also report calibration, validation-selection, and stability results that qualify these comparisons.

Performance across individual corruptions

  • ℓ2 adversarial training improves average accuracy across corruption types, but performs worse than the standard model on contrast, fog, and slightly on brightness.Its gains are particularly noted for JPEG compression, elastic transform, pixelate, and zoom blur.Source: H Supplementary figures and tables · p. 29, Figure 12 · p. 30

Effect of perturbation radius

Performance by corruption severity

  • Combining RLAT with AugMix improves accuracy on all severity levels; on ImageNet-100-C severity 4, ℓ2 adversarial training slightly exceeds RLAT, 37.0% versus 36.8%.Source: H Supplementary figures and tables · p. 30

AugMix with adversarial training

  • With AugMix and the JSD term, RLAT reaches 89.4% CIFAR-10-C accuracy, exceeding the best AugMix-paper model despite using ResNet-18 rather than ResNeXt-29.RLAT also outperforms ℓ2 and ℓ∞ adversarial training when combined with AugMix, with or without JSD.Source: H Supplementary figures and tables · p. 30, Table 14 · p. 33

Performance across individual corruptions

  • RLAT consistently improves on ℓ2 adversarial training and has better average performance than other baselines, excluding 50% Gaussian augmentation because it uses CIFAR-10-C corruptions.The comparison is reported for CIFAR-10-C using ResNet-18.Source: H Supplementary figures and tables · p. 29, Table 10 · p. 31

Validation-based model selection

  • Validation-corruption selection chose the same ε as test-corruption selection for adversarially trained models except ImageNet-100 ℓ∞, whose test accuracy was unchanged.The authors attribute the limited risk of overfitting to validation corruptions being from the same category and to searching a small one-dimensional grid.Source: H Supplementary figures and tables · pp. 31–32, Table 15 · p. 33

Architecture and stability

Calibration

Related papers

Loading 2103.02325v2…