Source-linked AI summary
Natural Adversarial Examples
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, Dawn Song
TL;DR
The paper addresses limited evidence about model robustness when natural images shift away from standard ImageNet conditions and when anomalies come from semantically related concepts. It constructs adversarially filtered ImageNet-A and ImageNet-O datasets, finding that clean examples transfer across unseen models and reveal substantial shared weaknesses, while architectural changes offer a promising path toward robustness.
Problem
Standard ImageNet performance does not adequately measure robustness to shifted inputs, spurious-cue-limited examples, or out-of-distribution concepts.
Method
The paper uses adversarial filtration to construct ImageNet-A from hard in-class images and ImageNet-O from outside-ImageNet-1K concepts that induce high-confidence predictions.
Results
The datasets transfer across tested architectures, including vision Transformers, while data augmentation and additional public data provide limited or difficult improvements and architecture changes are promising.
Takeaways & Limitations
ImageNet-A and ImageNet-O provide measures of performance under distribution shift for evaluating robustness as models are deployed in real-world environments.
Abstract
from arXiv · showhide
We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets' real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called ImageNet-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called ImageNet-O, which is the first out-of-distribution detection dataset created for ImageNet models. On ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on ImageNet-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.
1. Introduction
ImageNet performance can appear strong while failing on shifted, anomalous, or difficult natural images. The paper addresses this with two adversarially filtered ImageNet test sets that expose shared model weaknesses and challenge robustness improvements.
- ImageNet classifiers can perform well on standard benchmarks yet fail when test-time conditions include weather, occlusion, or anomalous objects.
- Adversarial filtration removes simple-to-classify examples and limits spurious cues, creating hard test sets with clean natural images that are simple for humans but difficult for models.
- IMAGENET-A contains in-class images from a distribution unlike ImageNet training, producing consistent mistakes across models through scene complications and classifier blind spots.
- Adversarially filtered examples transfer to unseen architectures, indicating shared weaknesses across models rather than failures limited to the model used for filtering.
- IMAGENET-O contains concepts outside ImageNet-1K that induce high-confidence in-distribution predictions despite being out of distribution, enabling tests of detection under label shifts.
- Data augmentation can decrease performance, other public training data yields less than a 10% accuracy increase from a 10× data increase, and architecture improvements are promising but insufficient.
2. Related Work
Prior work studies artificial adversarial perturbations, shifted distributions, spurious cues, and OOD detection using datasets that may not resemble ordinary ImageNet examples. This paper positions its natural adversarial datasets as a broader-transfer and more ImageNet-like evaluation setting.
- Out-of-Distribution Detection: IMAGENET-O uses OOD examples closer to ImageNet than textures, scenes, or synthetic blobs, including visually similar but still out-of-distribution concepts.
- Adversarial Examples: Artificial ℓp adversarial examples use nearly worst-case small distortions, whereas the paper's adversarially filtered natural images move beyond that threat model.
- Out-of-Distribution Detection: Prior OOD detection work often uses visually dissimilar datasets as anomaly stand-ins, producing anomalies that are far from the in-distribution data.
- Spurious Cues and Unintended Shortcuts: Adversarial filtration has been used to remove examples solvable through simple spurious cues, which can otherwise produce optimistic and inaccurate performance estimates.
- Robustness to Shifted Input Distributions: ImageNetV2 was estimated to reduce accuracy by less than 3.6%, while IMAGENET-A can reduce model accuracy by over 50%.
3. IMAGENET-A and IMAGENET-O
IMAGENET-A and IMAGENET-O are adversarially filtered datasets designed to expose failures in classification and out-of-distribution detection. The section describes their construction, substantial distribution shift, and representative classifier failure modes.
- IMAGENET-A: IMAGENET-A is built by filtering images that fixed ResNet-50 classifiers predict correctly, then manually selecting remaining examples.The process uses weakly labeled images and 10-crop classification with ResNet-50 filters.
- IMAGENET-O: IMAGENET-O contains non-ImageNet-1K images selected for high-confidence ResNet-50 ImageNet predictions, followed by manual quality selection.Candidates come from ImageNet-22K after ImageNet-1K classes are removed; the dataset contains 2,000 examples.
- Distribution shift: 50.40 FID separates IMAGENET-A from the ImageNet validation set, while estimated human accuracy remains approximately 90%.The FID against the ImageNet test set is approximately 50.25, compared with approximately 0.99 between ImageNet validation and test sets.
- Failure modes: IMAGENET-A examples transfer straightforwardly to unseen models and expose failure modes including visual overgeneralization, reliance on color or texture, and erroneous background cues.Observed errors include confusing semantically distinct classes and associating objects with recurring scene elements.
4. Experiments
The experiments test transfer, robustness metrics, augmentation, additional data, and architecture changes on IMAGENET-A and IMAGENET-O. Architectural changes generally provide the strongest improvements, while augmentation and more data offer limited or mixed gains.
- Experimental setup: Adversarially filtered examples collected against fixed ResNet-50 models transfer reliably to unseen models, indicating shared convolutional-network weaknesses.The experiments evaluate data augmentation and training data with fixed architectures, then compare different architectures.
- Data augmentation: 2.17% IMAGENET-A accuracy is obtained by a scratch-trained ResNet-50 baseline, while adversarial training decreases accuracy to 1.68%.The section reports that some other augmentation methods help only by a few percent.
- More labeled data: 11.41% IMAGENET-A accuracy and 21.86% IMAGENET-O AUPR are achieved after ImageNet-21K pretraining, improving accuracy by 9.24% over the stated baseline.ImageNet-21K contains approximately 21,000 classes and 14 million images; the OOD gain is described as less significant because of image overlap.
- Architectural changes: 6.1% top-1 accuracy and 18.0% AUPR are achieved by ResNet-152, compared with 2.17% and 16.2% for ResNet-50.The passage attributes these gains to increasing network capacity.
- Architectural changes: 14.59% accuracy and 19.5% AUPR are achieved by Res2Net-50, while Res2Net-152 reaches 22.4% accuracy and 23.9% AUPR.These results come from exchanging ResNet residual blocks for Res2Net v1b blocks.
- Vision Transformers: 19.0% IMAGENET-A accuracy is obtained by DeiT-small, compared with 14.6% for a similarly sized Res2Net-50.The transfer result extends beyond convolutional architectures to a vision Transformer trained with aggressive augmentation.
5. Conclusion
The paper presents IMAGENET-A and IMAGENET-O as challenging measures of performance under distribution shift. Improvements are possible through augmentation, pretraining data, and architectural changes, but the examples transfer broadly, making robustness difficult.
- Conclusion: Performance on IMAGENET-A and IMAGENET-O can improve through data augmentation, pretraining data, and architectural changes.The paper describes these datasets as measures of performance under distribution shift.
7. Expanded Results
Expanded results examine OOD detection, calibration, response-rate accuracy, and architecture comparisons. The supplied passages define the uncertainty measures and report that several additional OOD detectors provide little improvement.
- OOD detection: 14.3% AUPR is obtained by the auxiliary confidence branch, about 2% below the 16.1% MSP baseline with DenseNet-121.With ResNeXt-101 (32 × 8d), Max Logit increases AUPR from 20.5% to 20.6%.
- OOD detection: 16.2% AUPR is obtained both with rotation prediction and MSP for ResNet-50.The rotation-prediction method requires network training and does not work out-of-the-box.
- Calibration: ℓ2 Calibration Error measures whether confidence forecasts match observed accuracy, such as 60% confidence corresponding to 60% correctness.The supplied passage introduces this metric for IMAGENET-A calibration results.
- Response-rate analysis: Figure 10 compares Response Rate Accuracy for ResNeXt-101 (32×4d) with and without Squeeze-and-Excitation.Response Rate is the percent classified; accuracy at n% response is measured on the n% most-confident examples.
- Expanded architecture results: Figure 8 reports expanded architecture results by model size, while its caption notes that IMAGENET-O performance improves more slowly.The figure distinguishes normal, large, and XLarge model sizes.
- Calibration: AURRA summarizes the Response Rate Accuracy curve, which measures accuracy on the highest-confidence fraction of examples as response rates vary.The paper computes it using maximum softmax probability and selects the examples with highest confidence.
8. IMAGENET-A Classes
This section lists the 200 ImageNet classes selected for IMAGENET-A and provides associated class names and WordNet identifiers. The supplied passages are primarily enumerations rather than analytical results.
- Class selection: IMAGENET-A uses a selected set of 200 ImageNet classes, including animals, objects, foods, and scenes.The passages enumerate class names such as goldfish, ambulance, banana, pizza, and volcano.
- Identifiers: The selected classes are also represented by WordNet IDs and alternate lexical labels.The passages include identifiers such as n01498041 and labels such as “African chameleon, Chamaeleo chamaeleon.”
9. IMAGENET-O Classes
IMAGENET-O comprises 200 selected ImageNet classes, listed by class name and associated WordNet ID.
- IMAGENET-O contains 200 selected ImageNet classes.
- The class list includes diverse object, animal, and artifact categories such as goldfish, triceratops, centipede, harmonica, and lawn mower.
- Each selected class is associated with a WordNet ID.