PaperlayerView exact revision on arXiv

Source-linked AI summary

On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness

Eric Mintun, Alexander Kirillov, Saining Xie

arXiv:2102.11273v2cs.CVcs.LG

TL;DR

The relationship between training augmentations and test-time corruptions remains poorly understood, despite their importance for corruption robustness. The paper constructs a feature space for image transforms and uses Minimal Sample Distance to measure augmentation-corruption similarity. Similarity predicts corruption error, while robustness can degrade when test corruptions are perceptually dissimilar from the benchmark.

  • Problem

    The relationship between training augmentations and test-time corruptions is not well understood, although models trained on clean inputs often perform dramatically worse under image corruptions.

    Source: §1 · pp. 1–2

  • Method

    The authors construct a feature space for image transforms and use Minimal Sample Distance (MSD) to quantify distances between augmentation schemes and corruption classes.

    Source: §1 · p. 2

  • Results

    Perceptual similarity strongly predicts corruption error, while augmentation schemes designed to improve robustness show significantly degraded performance on corruptions sampled to be dissimilar from ImageNet-C.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training on perceptually similar augmentations can improve corruption robustness, and only a small fraction of sampled augmentations need to be similar to a given corruption.

    Source: §1 · p. 3

    Improvements on a fixed finite set of test corruptions may generalize poorly to perceptually dissimilar corruptions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed that significantly improve performance on ImageNet-C, a benchmark of such corruptions. However, there is still a lack of basic understanding on the relationship between data augmentations and test-time corruptions. To this end, we develop a feature space for image transforms, and then use a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate a strong correlation between similarity and performance. We then investigate recent data augmentations and observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C in this feature space. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may not generalize well beyond the existing benchmark. We hope our results and tools will allow for more robust progress towards improving robustness to image corruptions. We provide code at https://github.com/facebookresearch/augmentation-corruption.

1 Introduction

The paper asks how similarity between training augmentations and test-time corruptions affects robustness, especially when the corruptions are unknown. It introduces a feature-space measure and finds that perceptually similar transforms predict performance, while robustness may generalize poorly to dissimilar corruptions.

1 Introduction

  • Low-level image corruptions affect colors, textures, or geometry, and data augmentations using random low-level transformations have improved ImageNet-C performance.Models trained on clean, undistorted inputs typically perform dramatically worse under image corruptions [8] [13].Source: §1 · p. 1
  • The study addresses how similarity between training augmentations and test corruptions affects error, and how well augmentations generalize beyond a benchmark.Test corruptions are expected to be unknown during training, and excluding their exact transformations may not prevent visually similar augmentations from improving benchmark results.Source: §1 · pp. 1–2
  • The authors develop a feature space for image transforms and use Minimal Sample Distance (MSD) to quantify distance between augmentation schemes and corruption classes.They report that only a small fraction of sampled augmentations need to be similar to a corruption for generalization to emerge among perceptually similar transforms.Source: §1 · pp. 2–3
  • Augmentation-corruption perceptual similarity is a strong predictor of corruption error, but robustness degrades on corruptions sampled to be perceptually dissimilar from ImageNet-C.Some augmentation schemes still improve over baseline on the dissimilar corruptions, with differing degrees of generalization.Source: §1 · p. 2

2 Related Work

Related work covers corruption-robustness benchmarks and analyses, as well as training and test-time approaches intended to improve robustness. The paper distinguishes its focus on perceptual similarity and generalization to new corruptions from several adjacent research questions.

Corruption robustness benchmarks and analysis

  • ImageNet-C [13] is a corruption dataset often used as a benchmark in robustness studies; other corruption datasets [15] [29] collect real-world corrupted images with semantic distribution shifts and perceptual transforms.Corruption robustness differs from adversarial robustness [33], which seeks invariance to small, worst-case distortions; improving corruption robustness often slightly improves regular test error instead of harming it.Source: §2 · p. 3

Improving corruption robustness

  • Robustness-training approaches include AugMix [14], Patch Gaussian [20], ANT [27], and methods such as DeepAugment [15] and Stylized-ImageNet [9].Noisy Student [39] and Assemble-ResNet [18] combine augmentation with new models and training procedures.Source: §2 · p. 3
  • Other work adapts to unseen corruptions at test time using self-supervised tasks or entropy minimization [37]; this paper does not directly study that setting.The paper also does not directly address how synthetic-corruption performance transfers to real-world corruption datasets [15] [35].Source: §2 · p. 3

3 Perceptual similarity for augmentations and corruptions

The section defines a feature space for perceptual image transforms and introduces Minimal Sample Distance to compare augmentation and corruption distributions. It motivates MSD as a measure suited to robustness across unknown corruptions and reports that it correlates with corruption error.

Measuring similarity between perceptual transforms

  • The feature space encodes each transform by averaging feature changes over sampled images, and transform similarity is the L2 distance between these representations.The image feature extractor uses the network’s last hidden layer and default task augmentation so it is independent of the transforms being studied.Source: §3 · pp. 3–4

Minimal sample distance

  • For robustness to unknown corruption distributions, a good augmentation scheme should not be equivalent to any one corruption distribution.The paper contrasts this goal with optimizing error on a known corruption distribution, where distributional equivalence is appropriate.Source: §3 · p. 4
  • In a toy mixture of impulse noise and motion blur, MMD grows linearly as impulse-noise mixing decreases, while impulse-noise error stays low until the mix is nearly zero.This illustrates why distributional similarity can poorly predict corruption error.Source: §3 · p. 4, Figure 2 · p. 4
  • MSD measures the perceptual similarity between an average corruption and its closest augmentation among finite samples, accommodating rare augmentation samples near a corruption.Unlike MMD, MSD can remain small for a broad augmentation distribution that occasionally produces samples near multiple corruption distributions.Source: §3 · pp. 4–5, Figure 2 · p. 4

Measuring similarity and corruption error

  • In CIFAR-10-C, MSD correlates well with corruption error across all four corruption categories, with ρ denoting Spearman rank correlation; brightness is shown as a negative example with poor correlation.Source: Figure 3 · p. 5

4 Perceptual similarity is predictive of corruption error

This section tests whether feature-space perceptual similarity between augmentations and corruptions predicts corruption error, using CIFAR-10-C and sampled augmentation schemes. It reports broad correlations, interpretable matches for real augmentation methods, and concerns about generalization beyond a benchmark.

4.1 Experimental setup

  • The setup averages error and distance over five severities for each of CIFAR-10-C’s 15 corruptions and measures 512 schemes built from subsets of nine base augmentations.MSD uses a WideResNet-40-2 feature extractor and averages over 100 images, 100 corruptions, and a minimization over 100k augmentations.Source: §4.1 · pp. 5–6

4.2 Analysis

  • 12 of 15 CIFAR-10-C corruptions have Spearman rank correlation above 0.6 between MSD and corruption error, with strong correlations across all four corruption categories.Brightness is a low-correlation exception, possibly because it is a single image statistic that varies substantially across images and may be poorly represented by the feature extractor.Source: §4.2 · p. 6
  • Adding solarize lowers MSD to impulse noise, and adding x translation lowers MSD to motion blur, while dissimilar augmentation-corruption pairs do not show decreased MSD.The examples link visually similar transforms to lower distance without a corresponding decrease for dissimilar pairs.Source: §4.2 · p. 6, Figure 4 · p. 6
  • Patch Gaussian has low MSD to noise, while AutoAugment has low MSD to contrast and defocus blur; MSD does not predict AutoAugment error on fog.Glass blur behaves more like noise than blur for these schemes, likely because it randomly permutes pixels.Source: §4.2 · p. 6, Figure 5 · p. 7
  • Perceptual similarity between non-identical augmentations and corruptions also predicts error, suggesting such pairs warrant care when assessing whether gains generalize beyond a benchmark.Explicitly identical transforms, such as AutoAugment contrast and ImageNet-C contrast, are typically removed from augmentation schemes during testing.Source: §4.2 · pp. 6–7

5 ImageNet-C: benchmarking with dissimilar corruptions

The authors construct corruption benchmarks perceptually dissimilar from existing ones and find that augmentation methods lose performance by differing amounts. Their results also show that transfer depends on the corruption and augmentation considered, limiting what aggregate benchmark scores reveal about robustness to unknown corruptions.

5 ImageNet-C: benchmarking with dissimilar corruptions

  • The authors select human-interpretable corruptions dissimilar to ImageNet-C in their feature space, fixing the selection procedure before evaluating augmentation methods.They build 30 candidate corruptions, then select 10 while matching average error to the existing benchmark for default augmentation.Source: §5 · pp. 7–8
  • +0.7% for AutoAugment to +7.3% for PatchGaussian: every tested augmentation performs worse on the new benchmark, with degradation varying enough to change corruption-error rankings.The table caption identifies test error on CIFAR-10-C and ImageNet-10-C and notes that the increase differs across methods.Source: §5 · p. 8, Table 1 · p. 9
  • Performance varies by corruption: excluding blue sample noise and checkerboard leaves AutoAugment at 57.3% error and Patch Gaussian at 57.2%, near the 57.4% baseline.DeepAugment+AugMix improves single frequency noise by 31% but inverse sparkles by only 2.3%.Source: §5 · p. 8
  • The measure may misrepresent similarity for impulse noise versus blue noise sample, and may not cover methods such as Stylized-ImageNet, whose degradation may have other causes.Stylized-ImageNet primarily augments texture and appears to help more with higher-frequency corruptions.Source: §5 · pp. 8–9
  • Robust models not primarily based on data augmentation show no consistent degradation on the new benchmark, suggesting it is not simply harder overall.The comparison covers WSL, EfficientNet-B0, Noisy Student, ViT-S, and ResNeSt models.Source: §5 · p. 8, Table 2 · p. 9
  • Broader augmentation schemes generalize better, plausibly because they are perceptually similar to more corruption types; AugMix and DeepAugment still improve over baseline.The authors frame this as a reason for optimism about unknown corruptions, provided evaluation is careful.Source: §5 · p. 9

6 Discussion

The discussion treats corruption robustness as generalization across a small set of transforms and argues for held-out corruption evaluation. It also notes that transfer can occur among perceptually similar transforms, while the paper does not evaluate real-world corruptions.

6 Discussion

  • The authors note that dissimilar-corruption generation could be used to attack vision systems, while also potentially testing whether claimed robustness extends beyond common benchmarks.They cite possible use in content moderation or self-driving systems and warn that robust models could also be used detrimentally.Source: §6 · pp. 9–10
  • Because corruption benchmarks and augmentation schemes involve relatively few transforms, the authors argue that evaluation should hold out corruption transforms for validation and testing.This proposed split concerns transforms applied to the ImageNet validation set, not a test set for the underlying classification task.Source: §6 · p. 10
  • The paper does not study real-world corruptions; it proposes perceptual similarity as a possible explanation for conflicting prior findings about transfer to them.The authors suggest seeking transforms with maximal perceptual similarity to real-world corruptions.Source: §6 · p. 10
  • Generalization occurs among perceptually similar transforms, and dissimilar augmentations often add little performance loss when a similar augmentation is also present.The authors suggest broad schemes may therefore confer robustness to a larger class of unknown corruptions, with careful evaluation.Source: §6 · p. 10

A Sampling similar augmentations more frequently gives minor performance improvements

The experiments test how the size and perceptual similarity of an AugMix transform subset affect CIFAR-10-C error. Random subsets need only be small to match AugMix, while the closest transforms perform better and the farthest perform worse.

A Sampling similar augmentations more frequently gives minor performance improvements

  • The experiment probes whether a broad augmentation scheme can retain robustness when only a small fraction of sampled transforms are similar to each corruption.The authors expect rare similar augmentations may suffice, while many sampled augmentations may be dissimilar without significant loss in corruption error.Source: A Sampling similar augmentations more… · p. 14
  • About 100 closest transforms exceed regular AugMix performance, while using the farthest transforms causes around a 5% drop in accuracy.The gain from sampling the closest transforms more frequently is only around 1%.Source: A Sampling similar augmentations more… · p. 14
  • Randomly sampled subsets of around 1,000 transforms match AugMix despite its usual 5 million uniquely sampled transforms.The experiment trains on a fixed-size subset selected randomly, by closeness, or by distance from CIFAR-10-C corruptions.Source: A Sampling similar augmentations more… · p. 14
  • MSD correlates well with corruption error across four CIFAR-10-C categories, whereas MMD is typically not predictive.The authors suggest that reducing the frequency of similar transforms to include transforms similar to another corruption set may trade a small loss on the original set for a likely gain on the new set.Source: A Sampling similar augmentations more… · p. 14, Figure 8 · p. 15

B Additional MSD and MMD experiments

Additional experiments compare MSD with MMD, examine MMD patterns across augmentations, and test MSD against batch-norm-adapted error. MSD correlates with corruption error where MMD generally does not, and its relationship with error remains present after batch-norm adaptation, though weaker than with training on augmentations.

B.1 Comparison of MSD and MMD

  • MSD correlates strongly with corruption error in all four corruption categories, while MMD typically correlates poorly.The comparison calculates both distances for each augmentation in the augmentation powerset against CIFAR-10-C corruptions.Source: §B.1 · p. 15

B.2 Analyzing generalization with MMD

  • Patch Gaussian has low MMD to noise corruptions and high MMD to others, suggesting its perceptual similarity is distributionally concentrated on noise.AutoAugment and AugMix show no strong MMD-error correlation; the authors suggest Patch Gaussian may generalize poorly to corruptions unlike noise, including ImageNet-C.Source: §B.2 · p. 15, Figure 9 · p. 16

B.3 MSD vs Batch-Norm Adaptation

  • MSD still correlates with error after adapting batch-norm statistics to the augmentation distribution, but less well than when training on augmentations.The result supports the authors’ claim that batch-norm statistics are an important aspect of augmentation choice.Source: §B.3 · pp. 15–16

C.1 Architecture choice

The authors test whether MSD’s correlation with corruption error depends on feature-extractor architecture and on the number of sampled images and corruptions. The qualitative correlation pattern is stable across the tested architectures, and the chosen sampling counts yield distance variability smaller than the plotted feature scale.

C.1 Architecture choice

  • The qualitative relationship between MSD and corruption error is unchanged when VGG-19-BN replaces WideResNet-40-2 as the feature extractor.WideResNet has slightly better rank correlations overall, but the relative behavior across corruptions remains the same.Source: §C.1 · p. 16, Table 3 · p. 18

C.2 Parameter dependencies

  • Batch-norm-adapted test error still correlates strongly with MSD, though less than error from models trained from scratch on augmentations..Source: Figure 10 · p. 17
  • Using VGG-19-BN for feature extraction leaves the qualitative MSD-error correlation structure intact.Figure 11 uses VGG-19-BN for feature extraction and WideResNet-40-2 to calculate corruption error.Source: Figure 11 · p. 17
  • With 100 images and 100 corruptions, distance standard deviation is around 5% of the mean, smaller than the feature size in Figure 8.The variability estimate averages over 100 augmentation-corruption pairs and 100 resamplings.Source: §C.2 · pp. 16, 18, Figure 12 · p. 18

D.1 Dataset construction details

The construction procedure creates new corruption datasets intended to be perceptually distant from CIFAR/ImageNet-C while keeping baseline error comparable. It samples candidate corruption-severity groups, ranks their distance contributions, and selects the most distant corruptions.

D.1 Dataset construction details

  • Thirty new corruptions are adapted from common image filters and noise distributions, with 10 severities chosen to remain human-interpretable and roughly match ImageNet-C baseline error.Severity groups contain five severities, centered on one severity from 3 through 8, and are designed to roughly match ImageNet-C’s average error spread across severities.Source: §D.1 · p. 18
  • Each candidate dataset contains 10 groups of severities from distinct corruptions, with baseline average error within 1% of ImageNet-C.Matching baseline error normalizes relative error decreases against a fixed baseline.Source: §D.1 · p. 18
  • The candidate pool is sampled as 100,000 datasets because exhaustive enumeration was computationally infeasible.For each choice of five corruptions, a severity choice is sampled when it meets the baseline-error constraint, then disjoint pairs of five-corruption sets form candidate datasets.Source: §D.1 · p. 19
  • Candidate distance uses the minimum MMD to ImageNet-C corruptions so the new corruptions are far from all benchmark corruptions.For ImageNet, the procedure jointly samples 10,000 images and corruptions; for CIFAR-10 it uses the parameters from Section 4.Source: §D.1 · p. 19
  • The new dataset selects the top 10 corruption types ranked by average contribution to distance from ImageNet-C, averaged across repeated runs.The procedure is repeated 10 times for CIFAR and 5 times for ImageNet; the selected severity configuration is chosen for error closest to ImageNet-C.Source: §D.1 · p. 19, Figure 13 · p. 19

D.2 Complete results

The appendix supplies corruption-level and average comparisons, reports the MSD–error relationship on CIFAR-10-C, and documents experiment costs and the transforms used.

D.2 Complete results

  • Table 4 summarizes average ImageNet/CIFAR-10-C performance, while Table 5 breaks results down by individual corruption.The reported standard deviations span 10 CIFAR-10 runs and 5 ImageNet runs; ANT and DeepAugment variants use provided pretrained models and only one run.Source: §D.2 · p. 19, Table 4 · p. 20, Table 5 · p. 21

IN-C IN-C

  • Stylized-ImageNet is jointly trained with ImageNet for half the epochs; results average five ImageNet runs and ten CIFAR-10 runs.Each Stylized-ImageNet run uses a newly generated dataset with a different random seed.Source: IN-C IN-C · p. 20

D.3 MSD for CIFAR-10-C

  • The CIFAR-10-C MSD–error correlation remains strong for many corruptions, but is weak for some, including inverse sparkles.Figure 14 plots the correlation; ρ denotes Spearman rank correlation.Source: §D.3 · p. 20, Figure 14 · p. 20

E Resource usage

  • ImageNet training takes approximately 20 hours on 8 V100 GPUs, while CIFAR-10 training takes about 45 minutes to an hour on one V100 GPU.Corruption evaluation can take up to approximately 6 hours per corruption and severity on 80 CPUs for ImageNet, or 8 minutes on 40 CPUs for CIFAR-10.Source: E Resource usage · pp. 20–21

F Glossary of transforms

  • The appendix illustrates 30 new human-interpretable corruptions, nine base augmentation transforms, and ImageNet-C examples grouped into four corruption categories.The new corruptions span noise, obscuring, warping, and color shifts; the base transforms comprise five geometric and four color transforms.Source: F Glossary of transforms · p. 21, Figure 15 · p. 22, Figure 16 · p. 23
Loading 2102.11273v2…