Source-linked AI summary
Improving robustness against common corruptions by covariate shift adaptation
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, Matthias Bethge
TL;DR
Machine vision models are sensitive to common image corruptions, while standard benchmarks often assume only ad hoc corrupted samples despite applications providing repeated unlabeled observations. The paper adapts batch-normalization statistics to corrupted images and finds substantial robustness gains, including a 10–15% points mCE improvement typically and a reduction from 53.6% to 45.4% mCE for DeepAugment+AugMix.
Problem
Machine vision models are often sensitive to distribution shifts from common image corruptions, and corruption benchmarks emphasize ad hoc scenarios that may not reflect applications with continuously observed corruptions.
Method
The paper treats corruption robustness as covariate-shift adaptation by estimating batch-normalization statistics on unlabeled corrupted images across ad hoc, partial, and full adaptation settings.
Results
10–15% points mCE improvements are typical on ImageNet-C, while DeepAugment+AugMix improves from 53.6% mCE to 45.4% mCE after adaptation.
Takeaways & Limitations
Single-sample adaptation improves robustness, with most further gains obtained from 32 to 64 samples, so adapted-statistics metrics can better reflect robustness in applications with repeated unlabeled inputs.
Takeaways & Limitations
Robustness to common corruptions does not ensure reliability under slightly different perturbations such as surface reflections, and the conditions for trustworthy decisions remain open.
Abstract
from arXiv · showhide
Today's state-of-the-art machine vision models are vulnerable to image corruptions like blurring or compression artefacts, limiting their performance in many real-world applications. We here argue that popular benchmarks to measure model robustness against common corruptions (like ImageNet-C) underestimate model robustness in many (but not all) application scenarios. The key insight is that in many scenarios, multiple unlabeled examples of the corruptions are available and can be used for unsupervised online adaptation. Replacing the activation statistics estimated by batch normalization on the training set with the statistics of the corrupted images consistently improves the robustness across 25 different popular computer vision models. Using the corrected statistics, ResNet-50 reaches 62.2% mCE on ImageNet-C compared to 76.7% without adaptation. With the more robust DeepAugment+AugMix model, we improve the state of the art achieved by a ResNet50 model up to date from 53.6% mCE to 45.4% mCE. Even adapting to a single sample improves robustness for the ResNet-50 and AugMix models, and 32 samples are sufficient to improve the current state of the art for a ResNet-50 architecture. We argue that results with adapted statistics should be included whenever reporting scores in corruption benchmarks and other out-of-distribution generalization settings.
1 Introduction
Machine vision models are sensitive to corruptions that arise when deployment data differ from training data. The paper argues that repeated unlabeled corrupted samples enable domain-adaptation methods, especially batch-normalization statistic adaptation, to improve robustness.
- Real-world shifts such as changed acquisition systems, lighting, dirt, sandstorms, and hail can expose machine vision models to image corruptions.
- ImageNet-C evaluations often assume models receive isolated corrupted samples, although applications may provide repeated samples sharing the same corruption.
- Domain adaptation studies adapting models from labeled clean source data to unlabeled corrupted target data, but this connection had not been reported on popular robustness benchmarks.
- Batch normalization estimates training activation statistics for normalizing intermediate representations, which may not match corrupted test-distribution statistics.
- 62.2% mCE is reached by a ResNet-50 after adaptation, compared with 76.7% mCE without adaptation.
2 Measuring robustness against common corruptions
ImageNet-C measures corruption robustness under a non-adaptive single-sample setting. The paper proposes evaluating models with partial or full access to unlabeled batches sharing corruption type and severity.
- ImageNet-C applies 15 test and four hold-out corruptions at five severity levels to 50,000 ImageNet validation images.
- The benchmark’s main metric, mean corruption error (mCE), normalizes top-1 errors against AlexNet across 15 corruptions and five severities.
- The proposed adaptive benchmark lets models condition responses on complete unlabeled batches of corrupted validation samples.
- The ad hoc scenario sets n = 1, whereas partial adaptation uses n = 8 and full adaptation uses n = 50 000 same-corruption, same-severity samples.
3 Correcting Batch Normalization statistics as a strong baseline for reducing covariate shift induced by common corruptions
The method treats corruption-induced shifts as changes in internal activation moments and corrects them by estimating batch-normalization statistics from unlabeled target samples, optionally blended with source statistics.
- Batch normalization normalizes feature-map activations using first- and second-order statistics estimated across spatial dimensions and training samples.
- Using training-time statistics under distribution shift can break the zero-mean, unit-variance assumption on which downstream layers depend.
- Under the paper’s covariate-shift definition, source and target distributions share p(y|x) but differ in p(x).
- The adaptation estimates target statistics from unlabeled test samples and combines them with source statistics when the available batch is small.
- The paper limits its internal-covariate-shift claim to dataset-induced shifts whose effects appear in first- and second-order activation moments.
- The pseudo-sample size N controls the tradeoff between target-statistic estimation and reliance on source statistics, with N ∈ [8, 128] suggested for small n < 32.
4 Experimental Setup
The experiments evaluate many pretrained ImageNet models and several out-of-distribution datasets, including corrupted, naturally difficult, distribution-shifted, object-centric, and artistic-image benchmarks.
- The study evaluates 25 torchvision model architectures spanning DenseNet, GoogLeNet, Inception, MNASNet, MobileNet, ResNet, ResNeXt, ShuffleNet, VGG, and WRN families.
- The models are trained on the 1.2-million-image, 1,000-class ILSVRC2012 ImageNet training set.
- The evaluation covers ImageNet-C, ImageNet-A, ImageNet-V2, ObjectNet, and ImageNet-R.
5 Results
Batch-normalization adaptation improves corruption robustness across model families and robust models, with gains evident under both partial and extensive adaptation. The strongest reported result reaches 45.4% mCE for DeepAugment+AugMix on ImageNet-C.
- ResNet-50 adaptation: 62.2% mCE is reached after around 16 to 32 adaptation samples, improving substantially over the non-adapted ResNet-50 baseline of 76.7% mCE.With a training-statistics prior, even one adapted sample can increase robustness.
- Model families: Adaptation consistently improves mCE across 25 ImageNet-trained models, typically by about 10 percentage points across architecture families.DenseNets also show higher corruption robustness than ResNets in both evaluation modes.
- Robust models: 45.4% mCE establishes a new ImageNet-C state of the art for a ResNet-50 DeepAugment+AugMix model after batch-normalization adaptation.All evaluated models improve when their batch-normalization statistics are adapted.
- Robust models: Eight adaptation samples suffice for AugMix to outperform AssembleNet, the previous state-of-the-art ResNet-50 model without adaptation.The comparison uses the number of samples available for adaptation.
6 Analysis and Ablation Studies
The analysis links covariate-shift measurements to performance, examines adaptation across datasets and normalization schemes, and evaluates how sample size and model pretraining affect its benefits.
- Covariate-shift analysis: Wasserstein distance between source and target statistics correlates with top-1 error for both non-adapted and adapted models on ImageNet-C.The metric remains predictive of residual source–target mismatch after adaptation.
- Covariate-shift analysis: 1–2% absolute mean deviation estimates hold-out corruption accuracy within a corruption, increasing to 5–15% across corruption categories.A systematic offset typically remains in the category-level estimates.
- Pretraining and normalization: 45.7% mCE is achieved by ResNeXt models pretrained on 3.5 × 10^9 Instagram images, while adaptation improvements vanish after full large-scale pretraining.The result holds for both the small and largest evaluated models, although the small IN-pretrained model still improves.
- Pretraining and normalization: 72.0% mCE for Fixup ResNet-50 is better than 76.7% for vanilla batch normalization but worse than 62.2% for adapted batch normalization.GroupNorm and Fixup models likewise outperform non-adapted BN models but underperform adapted BN models.
- Dataset-dependent effects: BN adaptation fails on IN-A, does not improve IN or IN-V2, helps ObjectNet only slightly with more than 64 samples, and improves IN-R beyond 32 samples.Shuffling IN-C corruptions removes the adaptation benefit, consistent with adaptation helping systematic shifts across multiple inputs.
7 Related Work
Related work connects corruption robustness to unsupervised domain adaptation, adaptive normalization, and training-based robustness methods.
- Robustness methods: ImageNet-C robustness research includes data augmentation, consistency-based augmentation mixtures, stylized-image training, and adversarial approaches.These methods generally involve specialized training protocols and additional time or resources.
- Domain adaptation: Unsupervised domain adaptation adapts models trained on a source domain to an unlabeled target domain using additional test-dataset information.Prior work includes feature-statistics adaptation, BN-parameter adaptation, single-example self-supervision, and pseudo-labeling.
- Normalization-based adaptation: Separate normalization layers for clean and adversarial images can improve adversarial-training robustness, while adapting affine BN parameters has shown expressive power in multitask and domain-adaptation settings.Concurrent work also reports increased ImageNet-C performance from BN adaptation.
8 Discussion and Conclusion
Adapting to corruption-specific statistics substantially improves robustness and suggests that current benchmarks understate performance when unlabeled repeated corruptions are available. The results also motivate broader evaluation of adaptation-aware robustness.
- 10–15% points mCE improvements are typical when reducing corruption-induced covariate shift in models with batch-normalization layers.
- Most further gains arrive with 32 to 64 adaptation samples, after which additional improvements are minor.
- Correcting feature-wise first- and second-order moments suggests corrupted images retain features used for prediction, unlike IN-A or ObjectNet.
- Batch Norm is not itself the source of corruption susceptibility; adapted BN models remain more robust than Group Normalization and Fixup alternatives.
- Large-scale pre-training can at least partially remove first- and second-order covariate shift between clean and corrupted images.
- Because many applications encounter similar corruptions continuously, corruption benchmarks should account for adaptation using available unlabeled samples.
Broader Impact
More robust machine-vision systems may improve reliability and safety across several applications, but robustness does not guarantee reliable decisions under all perturbations. The paper therefore highlights both societal implications and unresolved reliability boundaries.
- More robust machine-vision systems can potentially enhance reliability and safety in autonomous driving, manufacturing, surveillance, and health care.
- These applications also carry societal risks including autonomous weapons, job losses, waste, surveillance, and misinformation.
- Models robust to common corruptions may still fail quickly on slightly different perturbations such as surface reflections.
Supplementary Material
The supplementary analysis estimates domain shifts by comparing source and corruption-adapted batch-normalization statistics across layers, using several distance and divergence measures. It also documents implementation choices, practical adaptation variants, and limitations of the analysis.
- Domain-shift analysis: The procedure adapts an ImageNet-trained model separately to all 95 ImageNet-C conditions, producing source and target statistics across network layers.The model is split into stages, BN statistics are updated on corruption inputs, and the adapted model is returned.
- Limitations: The considered covariance matrices are diagonal, and the authors expect full covariance matrices could improve domain-shift estimates.This is an explicit simplification in the supplementary analysis.
- Distances and divergences: Three measures quantify source-target shifts: Wasserstein distance, normalized Wasserstein distance, and Jeffrey divergence.The measures compare statistics associated with clean source images and corrupted target images.
- Layer-wise findings: Unnormalized Wasserstein distance is magnitude-sensitive, whereas normalized Wasserstein distance and Jeffrey divergence provide qualitatively different layer-wise analyses.Across metrics, ResNet-50 downsampling layers show the most notable source-target differences, and corruption severity generally increases the measured shifts.
- Implementation: Practical variants include target-only statistics, exponential moving averages, and source statistics as a prior.Target-only adaptation is recommended when sufficiently large batches are available; exponential moving averages support sequential or hardware-limited settings.
- Implementation: The implementation uses BN training mode for adaptation while keeping the model in evaluation mode, with dataset shuffling recommended during test-time statistic computation.The supplementary material states that the implementation adds minimal overhead and provides reference code.
B.4 Notes on datasets
The supplementary material documents datasets, evaluation details, model metadata, and additional results. It reports that BN adaptation helps some converted SimCLRv2 models but can hurt another, while larger models generally show lower ImageNet-C mCE.
- Datasets: ImageNet-C evaluation uses JPEG-compressed images from the authors’ recommended repository for reproducibility.The supplementary material notes that JPEG compression artefacts can reduce performance relative to applying corruptions directly in memory.
- Datasets: ObjectNet evaluation discards nine classes with ambiguous ImageNet mappings and tests models on objects from new viewpoints and backgrounds.ImageNet-trained models experience a performance drop of roughly 40–45% on ObjectNet.
- Datasets: The supplementary material describes three ImageNet-V2 test sets distinguished by the selection frequencies of the crowdsourcing workers.The listed variants are MatchedFrequency, Threshold0.7, and TopImages.
- Models and metrics: The evaluated models include parameter-count and batch-normalization metadata, with BN parameters comprising at most 1% of trainable parameters.Additional tables provide AlexNet errors used for mCE conversion and model-specific evaluation information.
- Additional results: Figure 8 reports adaptation improvements across all 25 torchvision model architectures, often on the order of 10 percentage points.The figure compares adapted and baseline mCE values.
- Additional results: Adaptation improves ResNet50 and ResNet101 performance after TensorFlow-to-PyTorch checkpoint conversion but hurts ResNet152 performance.The supplementary results explicitly report this model-dependent outcome.
- Additional results: Larger model parameter counts generally correlate with smaller ImageNet-C mCE values.This relationship is reported for the evaluated models, without establishing a causal explanation.
C.3 Per-corruption results on IN-C
The supplementary analysis examines per-corruption behavior, corruption-category similarity, batch-size choices, and a statistical model for source-target statistic mixing. It finds consistent per-corruption improvements while identifying estimation trade-offs and analytical assumptions.
- Per-corruption results: Full BN adaptation consistently improves error rates on every ImageNet-C corruption for both vanilla and AugMix models.The result is shown for models adapted on the full test sets.
- Corruption similarity: Corruption categories mostly induce similar shifts in BN parameters, except that the digital category is grouped differently in the t-SNE visualization.The visualization embeds Wasserstein distances between adapted and non-adapted models.
- Error estimation: Wasserstein distance and top-1 error have an approximately linear relationship over the considered ImageNet-C range, with similar corruption types showing similar slopes.This relationship is used to estimate error rates for unseen corruptions within corruption classes.
- Analytical model: The analytical model qualitatively explains the characteristic performance curves observed when varying n and N.Its derivation assumes a linear relationship between Wasserstein distance and error under domain shift.
- Batch-size analysis: The preferred pseudo batch size N depends on the input batch size n, and larger n generally makes performance less sensitive to N.Figures report mCE averaged across 15 test corruptions and select optimal N using holdout corruptions.
- Analytical model: Mixing source and estimated target statistics creates a trade-off between source-target mismatch and unreliable target estimates from small batches.The analysis studies this trade-off using convex combinations of source and target moments.
- Proof sketch: The expected Wasserstein-distance expression cannot be evaluated in closed form because it contains the square root of a convex combination of target and source variance.The proof therefore bounds the square-root term using Jensen’s inequality and Hölder’s defect formula.