Source-linked AI summary
MNIST-C: A Robustness Benchmark for Computer Vision
Norman Mu, Justin Gilmer
TL;DR
Computer vision models can perform superhumanly on in-distribution tests yet remain less robust on out-of-distribution corruptions. MNIST-C introduces a 15-corruption MNIST benchmark, showing substantial failures across models and limitations of adversarial defenses and naive augmentation.
Problem
Computer vision models still lag behind humans on out-of-distribution data, while MNIST lacks a standardized corrupted variant for robustness evaluation.
Method
MNIST-C constructs and evaluates a diverse suite of 15 MNIST corruptions selected for non-triviality, semantic invariance, realism, and breadth.
Results
MNIST-C exposes model failures that adversarial-robustness metrics do not detect, with large performance degradations across adversarially trained CNNs and alternative architectures.
Takeaways & Limitations
MNIST-C provides a broader and more reproducible benchmark for measuring robustness progress in computer vision.
Takeaways & Limitations
Semantic preservation is verified through thorough visual inspection and a random sample of images misclassified by a simple CNN.
Abstract
from arXiv · showhide
We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions applied to the MNIST test set, for benchmarking out-of-distribution robustness in computer vision. Through several experiments and visualizations we demonstrate that our corruptions significantly degrade performance of state-of-the-art computer vision models while preserving the semantic content of the test images. In contrast to the popular notion of adversarial robustness, our model-agnostic corruptions do not seek worst-case performance but are instead designed to be broad and diverse, capturing multiple failure modes of modern models. In fact, we find that several previously published adversarial defenses significantly degrade robustness as measured by MNIST-C. We hope that our benchmark serves as a useful tool for future work in designing systems that are able to learn robust feature representations that capture the underlying semantics of the input.
1. Introduction
MNIST-C addresses the lack of a standardized corrupted MNIST benchmark for evaluating out-of-distribution robustness. Its 15 corruptions are designed to preserve semantic content while exposing substantial model failures.
- MNIST lacks a standardized corrupted variant, despite its continued use in robustness research and the availability of corruption benchmarks for CIFAR-10 and ImageNet.
- MNIST-C introduces 15 image corruptions tailored to low-resolution, black-and-white MNIST images for measuring out-of-distribution robustness.
- The benchmark is intended to expose broad, realistic failure modes while preserving the semantic content of the underlying images.
- Relative to clean data, MNIST-C increases convolutional-neural-network error rates by a factor of 10 while preserving image semantics.
- Four prior adversarial defense methods significantly degrade performance on MNIST-C, and training on other corruptions yields minimal improvement on a held-out corruption.
2. MNIST-C
MNIST-C is constructed from a broad corruption suite and selected to test non-trivial, semantically invariant, realistic, and diverse model failures. Its corruptions cover multiple perturbation mechanisms while remaining suitable for reproducible evaluation.
- MNIST-C begins with 31 corruptions spanning additive noise, blurring, digital effects, geometric transformations, and superimposed patterns.
- The benchmark selects 15 corruptions by tuning severity to degrade model performance while preserving semantic content and avoiding redundant failure patterns.
- The four design principles are non-triviality, semantic invariance, realism, and breadth.
- Severity is tuned to expose convolutional-network blind spots, with CNN error rates increasing by up to 1000% relative to clean MNIST error rates.
- The corruptions model plausible real-world perturbations from camera setup, environmental factors, and physical modification.
- The authors release source code for all 31 corruptions and a static pre-computed dataset, because the generation algorithms are unsuitable for direct training use.
3. Experimental Results
Experiments show that MNIST-C substantially degrades diverse models, including adversarially trained networks, while exposing errors that standard adversarial robustness measures may miss. Direct training on the benchmark corruptions undermines its out-of-distribution purpose.
- Conv1 achieves 91.21% mean accuracy on MNIST-C, corresponding to a 1100% increased error rate relative to 99.22% clean-test accuracy.
- The evaluation compares clean-trained, adversarially trained, alternative-architecture, capsule, and generative models on the corruption benchmark.
- A majority of MNIST-C model errors are images that humans can classify correctly.
- Relative mean corruption error shows large testing-performance degradations across adversarially trained CNNs and alternative architectures.
- Because MNIST-C corruptions are not constrained to a small l_p-ball, the benchmark can detect solutions that overfit to l_p robustness.
4. Conclusion
MNIST-C is presented as a robustness benchmark that exposes model failures beyond adversarial-robustness metrics and supports more reproducible evaluation of progress.
- MNIST-C exposes model failures that adversarial-robustness metrics cannot detect.
- The benchmark measures a more comprehensive notion of robustness and reduces the difficulty of reproducibly evaluating robustness.
- MNIST-C is intended to make robustness evaluation more reliable for measuring scientific progress in computer vision.
A. Additional Corruptions
The paper releases additional corruption implementations, including random, blur, environmental, compression, and geometric transformations drawn from prior work.
- Speckle noise is described as a random imaging-process corruption, while pessimal noise uses an adversarially trained covariance matrix as a noninteractive worst-case proxy.
- The additional corruption suite includes speckle noise, Gaussian blur, defocus blur, zoom blur, frost, snow, contrast, saturation, JPEG compression, pixelation, and elastic transformation.
- Several additional corruptions, including speckle noise and multiple blur types, are taken from prior work on visual corruptions.
B. Additional Results
Additional results report accuracy across all six benchmarked models and examine how direct and indirect augmentation affect Conv1 on MNIST-C.
- Test accuracy is reported for all 6 benchmarked models on MNIST-C.
- The augmentation experiment compares methods that do and do not directly access the corruption used for testing Conv1.
C. Corruption Error Examples
The paper presents sampled Conv1 errors across corruption families and figures that summarize model accuracy and augmentation effects on MNIST-C.
- Randomly sampled Conv1 test errors are shown across brightness, Canny edges, dotted line, fog, impulse noise, stripe, motion blur, and glass blur.
- Additional sampled Conv1 errors cover translate, rotate, scale, shear, shot noise, spatter, and zigzag.
- Figure 3 summarizes test accuracy of various models on MNIST-C, shading each row's highest, lowest, and average accuracy green, red, and white.
- Figure 4 summarizes Conv1 accuracy under clean, single-corruption, all-but-one, and all-corruption training and testing conditions.