Source-linked AI summary
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
Dan Hendrycks, Kevin Gimpel
TL;DR
Detecting misclassified and out-of-distribution examples is important because neural networks can fail silently with high-confidence predictions. The paper evaluates a softmax-probability baseline across vision, language, and speech tasks, finding it consistently useful while sometimes surpassed by an auxiliary-decoder method.
Problem
The paper asks whether trained classifiers’ errors and out-of-distribution inputs can be detected on individual held-out examples.
Method
The paper uses statistics from softmax prediction probabilities as a baseline for detecting misclassified and out-of-distribution examples across multiple tasks.
Results
The baseline is consistently useful across computer vision, natural language processing, and speech recognition, but an auxiliary-decoder abnormality module outperforms it on some tested cases.
Takeaways & Limitations
Simple softmax-distribution statistics provide a strong baseline for detecting errors and out-of-distribution examples, while the results leave room for improved methods.
Takeaways & Limitations
The paper does not evaluate correctness detection for speech recognition because recognition errors vary along a range of edit distances rather than forming a binary outcome.
Abstract
from arXiv · showhide
We consider the two related problems of detecting if an example is misclassified or out-of-distribution. We present a simple baseline that utilizes probabilities from softmax distributions. Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection. We assess performance by defining several tasks in computer vision, natural language processing, and automatic speech recognition, showing the effectiveness of this baseline across all. We then show the baseline can sometimes be surpassed, demonstrating the room for future research on these underexplored detection tasks.
1 INTRODUCTION
The paper introduces a softmax-probability baseline for detecting misclassified and out-of-distribution examples, motivated by classifiers’ tendency to produce confidently incorrect predictions. It evaluates this approach across vision, language, and speech tasks, and identifies reconstruction-based detection and standardized evaluation as directions beyond the baseline.
- Motivation: Classifiers can fail silently with high-confidence, incorrect predictions when training and test distributions differ, limiting adoption and risking serious accidents.The introduction motivates detecting errors and distribution shifts because confident failures can be consequential in real-world applications.
- Motivation: Softmax probabilities can be misleading because the exponential transformation makes small logit changes produce substantial output changes and rarely yields uniform outputs for out-of-distribution inputs.The passage attributes overconfidence partly to softmax’s fast-growing exponential function and its smooth approximation of an indicator function.
- Baseline: Prediction probabilities for incorrect and out-of-distribution examples tend to be lower than those for correct examples, making statistics from correct or in-sample examples sufficient for detection.The authors distinguish aggregate probability statistics from interpreting an individual prediction probability in isolation.
- Baseline: The proposed prediction-probability baseline is effective across computer vision, natural language processing, and automatic speech recognition tasks, though it is not uniformly optimal.Its inconsistent superiority motivates further research into detection methods.
- Beyond the baseline: A reconstruction-based method assesses neural-network input reconstruction quality to detect abnormal examples and outperforms the baseline on some, but not all, tasks.The method is presented as one contribution beyond the softmax-probability baseline.
- Evaluation framework: The work designates standard tasks and evaluation metrics for automatic detection of errors and out-of-distribution examples across three research areas and well-studied neural-network architectures.For out-of-distribution detection, the authors also provide ways to supply such examples at test time.
2 PROBLEM FORMULATION AND EVALUATION
The paper formulates error/success prediction and in-/out-of-distribution detection as two related binary detection problems. It evaluates detectors with AUROC and AUPR because accuracy and fixed-threshold comparisons can be misleading, then introduces a maximum-softmax-probability baseline before an auxiliary reconstruction model.
- Problem formulation: The evaluation addresses whether classifiers correctly classify held-out examples and whether test examples belong to the training distribution.These correspond to error and success prediction, and in- and out-of-distribution detection.
- Evaluation metrics: Accuracy can mislead when negative examples greatly outnumber positive examples, motivating explicit score thresholds and alternative detector metrics.A detector outputs scores for both positive and negative classes, so always predicting the more likely negative class may yield high accuracy.
- Evaluation metrics: AUROC provides threshold-independent evaluation and measures the probability that a positive example receives a higher detector score than a negative example.For success and error scores, AUROC = P(S > E) = P(−E > −S).
- Evaluation metrics: AUPR is used as the second metric because it adjusts for differing positive and negative base rates, whereas AUROC is not ideal under large class imbalance.AUPR plots precision and recall, while AUROC plots true positive rate against false positive rate.
- Methods overview: The experiments first use maximum softmax probability as a simple baseline and then introduce an auxiliary model component trained to reconstruct the input.The baseline uses the maximum probability from the softmax label distribution in neural network classifiers.
3 SOFTMAX PREDICTION PROBABILITY AS A BASELINE
The baseline uses the predicted class’s maximum softmax probability to detect misclassified and out-of-distribution examples. Correct and incorrect, as well as in- and out-of-distribution, examples show sufficiently different probabilities for reliable discrimination, although these probabilities should not be treated as direct confidence.
- 3 SOFTMAX PREDICTION PROBABILITY AS A BASELINE: The detector scores each example by its maximum softmax probability, evaluating separation with areas under precision-recall and ROC curves.For in-distribution detection, correctly classified examples are positive and the probability is the score; for out-of-distribution detection, out-of-distribution examples are positive and the score is negated.
- Experimental evaluation: Across the experiments, AUROCs differ from random baselines with high statistical significance according to the Wilcoxon rank-sum test.The paper reports results across vision, natural language processing, and automatic speech recognition tasks.
- Computer vision: Correctly and incorrectly classified examples are sufficiently distinct for reliable discrimination, while curve areas degrade as image-recognizer test error increases.The computer-vision experiments use MNIST, CIFAR-10, and CIFAR-100.
- Computer vision: Out-of-distribution image examples sufficiently differ from in-distribution examples in prediction probabilities, enabling their detection.Mean predicted or maximum class probabilities exceed 75%, illustrating that high probabilities do not necessarily represent calibrated confidence.
- Natural language processing: The baseline also detects correct and incorrect sentiment classifications reliably, with softmax distributions differing between the two groups.This result is reported for binary sentiment classification on IMDB using a linear classifier over averaged 50-dimensional word vectors.
4 ABNORMALITY DETECTION WITH AUXILIARY DECODERS
This section introduces auxiliary decoders and abnormality modules that use learned internal representations to detect abnormal examples beyond softmax statistics. Experiments on speech and MNIST show improved or generalized detection, including for novel distortions and examples that do not severely reduce accuracy.
- Auxiliary decoders: Auxiliary decoders reconstruct inputs from learned neural-network representations, providing information that can sometimes be more useful for abnormality detection than softmax probabilities.A normal classifier is augmented with an auxiliary decoder and scorer, which are trained together.
- Speech detection: The TIMIT abnormality module generalizes from simple test noise to realistically distorted audio and detects out-of-distribution examples even when accuracy is not severely degraded.The module is not trained on the same noise type used for testing; experiments also include Aurora-2 distortions and THCHS-30 Chinese speech.
- MNIST detection: On MNIST, adding an auxiliary decoder and abnormality module consistently improves out-of-sample detection over softmax prediction probabilities for blurred, rotated, and Gaussian-noised images.The abnormal examples are generated from training images using these three transformations.
5 DISCUSSION AND FUTURE WORK
The baseline can be surpassed by exploiting network representations, motivating research into new detection methods and evaluation practices across diverse tasks, architectures, and datasets. Future evaluations should report AUPR, AUROC, and classifier accuracy to make detection performance interpretable.
- Future detection methods: The abnormality module can sometimes beat the baseline by exploiting network representations, suggesting numerous research directions.Proposed avenues include using intra-class variance or feeding layer-activation summaries into an RNN.
- Evaluation scope: Future detection methods should be tested across varied tasks, architectures, and datasets because vision-only demonstrations may not transfer broadly.Suggested datasets include MNIST, CIFAR, IMDB, and tweets.
- Evaluation metrics: Reporting AUPR, AUROC, and the underlying classifier’s accuracy is important for interpreting detection results.An always-wrong classifier can achieve maximum AUPR for error detection when error is the positive class.
6 CONCLUSION
The paper demonstrates a softmax prediction probability baseline for error and out-of-distribution detection across architectures and datasets. An abnormality module achieves superior discrimination on tested cases, showing the baseline can be surpassed and motivating future research.
- 6 CONCLUSION: The softmax prediction probability baseline detects errors and out-of-distribution examples across several architectures and numerous datasets.The study evaluates the baseline across multiple model architectures and datasets.
- 6 CONCLUSION: The abnormality module provides superior scores for discriminating between normal and abnormal examples on tested cases.Its results exceed those of the baseline in the reported tested cases.
- 6 CONCLUSION: The baseline can be beaten in some cases, indicating room for future research on these detection tasks.The authors hope other researchers investigate improved architectures.
A ABNORMALITY MODULE EXAMPLE
The example depicts a neural network classifying a diamond image with an auxiliary decoder and an abnormality module. It combines softmax probabilities with reconstruction-based processing, while distinguishing layers trained on in-distribution data from those trained on both data types.
- A ABNORMALITY MODULE EXAMPLE: The network classifies a diamond image using an auxiliary decoder and an abnormality module.The diagram represents neurons with either GELU or sigmoid activations.
- A ABNORMALITY MODULE EXAMPLE: A blurred diamond reconstruction is subtracted from the input and then elementwise squared.
- A ABNORMALITY MODULE EXAMPLE: The probability vector is the softmax probability vector.
- A ABNORMALITY MODULE EXAMPLE: Blue layers train on in-distribution data, whereas red layers train on both in- and out-of-distribution examples.