Source-linked AI summary
COMBOOD: A Semiparametric Approach for Detecting Out-of-distribution Data for Image Classification
Magesh Rajasekaran, Md Saiful Islam Sajol, Frej Berglind, Supratik Mukhopadhyay, Kamalika Das
TL;DR
OOD data can violate the shared-distribution assumption behind image classification, making inference-time detection important for automated applications. COMBOOD combines nearest-neighbor and Mahalanobis distance signals into an unsupervised semiparametric confidence score, and experiments show it outperforms state-of-the-art methods across far- and near-OOD benchmarks and document images.
Problem
Image classifiers need an inference-time confidence score to identify inputs that fall outside the training distribution.
Method
COMBOOD combines nonparametric nearest-neighbor and parametric regularized Mahalanobis distance signals using their log distributions to produce an OOD confidence score.
Results
COMBOOD outperforms state-of-the-art OOD detection methods across OpenOOD far- and near-OOD benchmarks and the documents dataset, with statistically significant accuracy gains on a majority of benchmark datasets.
Takeaways & Limitations
The semiparametric combination addresses the complementary strengths of Mahalanobis detection for far-OOD and nearest-neighbor detection for OOD detection more broadly.
Takeaways & Limitations
The current combination of the two approaches' priors is not data-driven, and future work will investigate data-driven combinations using ID and OOD distribution kurtosis.
Abstract
from arXiv · showhide
Identifying out-of-distribution (OOD) data at inference time is crucial for many machine learning applications, especially for automation. We present a novel unsupervised semi-parametric framework COMBOOD for OOD detection with respect to image recognition. Our framework combines signals from two distance metrics, nearest-neighbor and Mahalanobis, to derive a confidence score for an inference point to be out-of-distribution. The former provides a non-parametric approach to OOD detection. The latter provides a parametric, simple, yet effective method for detecting OOD data points, especially, in the far OOD scenario, where the inference point is far apart from the training data set in the embedding space. However, its performance is not satisfactory in the near OOD scenarios that arise in practical situations. Our COMBOOD framework combines the two signals in a semi-parametric setting to provide a confidence score that is accurate both for the near-OOD and far-OOD scenarios. We show experimental results with the COMBOOD framework for different types of feature extraction strategies. We demonstrate experimentally that COMBOOD outperforms state-of-the-art OOD detection methods on the OpenOOD (both version 1 and most recent version 1.5) benchmark datasets (for both far-OOD and near-OOD) as well as on the documents dataset in terms of accuracy. On a majority of the benchmark datasets, the improvements in accuracy resulting from the COMBOOD framework are statistically significant. COMBOOD scales linearly with the size of the embedding space, making it ideal for many real-life applications.
1 Background and Motivation
OOD detection is needed because neural networks can misclassify inputs outside their training distribution, while existing distance-based methods have complementary strengths. COMBOOD combines nearest-neighbor and Mahalanobis signals to target both near- and far-OOD detection, with strong benchmark results and linear embedding-space scaling.
- Motivation: OOD inputs violate the shared training–test distribution assumption and can be assigned an in-distribution class by image classifiers.This creates risks for applications including automatic text extraction, medical diagnosis, and autonomous systems.
- Motivation: Softmax class probabilities are only weakly correlated with confidence, motivating dedicated inference-time OOD detection methods.Existing methods vary in accuracy and computational overhead, while scalable convolutional-network algorithms should ideally remain linear-complexity.
- COMBOOD: Mahalanobis distance is effective for far-OOD detection but performs unsatisfactorily near the training distribution, where nearest-neighbor methods provide a complementary signal.COMBOOD combines these parametric and non-parametric distance measures into one confidence score.
- COMBOOD: COMBOOD combines nearest-neighbor and Mahalanobis evidence in a semi-parametric framework intended to provide accurate confidence scores for both near-OOD and far-OOD cases.The framework combines the two signals using their distance distributions.
- Results: COMBOOD outperforms state-of-the-art OOD methods on OpenOOD far- and near-OOD benchmarks and the documents dataset, with statistically significant gains on most benchmark datasets.The experiments include different feature-extraction strategies.
- Results: COMBOOD scales linearly with embedding-space size, supporting its use in real-life applications.This matches the practical need for scalable inference-time OOD detection in convolutional networks.
2 Methods and Technical Solutions
The paper formulates OOD detection as assigning an inference-time confidence score alongside an image-classification label. It explores global-extrema and normalized penultimate-layer features, whose strengths differ across Mahalanobis and nearest-neighbor detection.
- Problem formulation: The problem is to learn a function that indicates whether a test image is out of distribution while the classifier returns its class label.The classifier maps images to one of K classes, but OOD inputs require an additional confidence-like score.
- Feature extraction: Two feature-extraction strategies are evaluated: global extrema of input features and embeddings from a pretrained network’s penultimate layer.The global-extrema strategy uses layerwise maximum and minimum values, while penultimate embeddings represent images at a high abstraction level.
- Global extrema: Global-extrema features enable accurate Mahalanobis-based results on many far-OOD tasks with improved computational efficiency, but near-OOD performance remains unsatisfactory.The extrema are transformed and standardized without using class labels or image topology.
- Penultimate embeddings: Penultimate-layer embeddings improve nearest-neighbor OOD detection across far- and near-OOD scenarios, whereas their Mahalanobis combination significantly slows computation without improving performance.The embeddings are normalized by their L2 norms.
3 Algorithms for OOD Detection
The paper develops two unsupervised OOD detection algorithms: regularized Mahalanobis distance and nearest-neighbor density estimation, using feature statistics or pretrained-network embeddings.
- The regularized Mahalanobis method begins by extracting features and estimating their training-set mean and covariance.
- It adds C·I to the covariance matrix, with standardized features making the regularizer's effect consistent across instances.C is a nonnegative regularization constant and I is the identity matrix.
- The resulting Mahalanobis distance measures how far a test feature vector lies from the training distribution.For large C, the distance approaches an L2-distance-based model.
- The method operates without prior OOD knowledge and is completely unsupervised.
- With global-extrema features, regularized Mahalanobis detection has minimal computational overhead and performs well when training and OOD extreme-value distributions are compact and separated.Its near-OOD performance remains unsatisfactory, while penultimate-layer embeddings increase computation without improving performance.
- Nearest-neighbor detection estimates density by computing each test image's k-nearest neighbors among training images in embedding space.A threshold is typically selected so that in-distribution data are classified correctly.
4 COMBOOD: A Semiparametric Framework
COMBOOD combines nearest-neighbor and regularized Mahalanobis evidence through log-distance distributions to support OOD decisions across near- and far-OOD settings. Experiments indicate that pairing distinct feature-extraction strategies performs best.
- COMBOOD combines parametric regularized Mahalanobis and non-parametric nearest-neighbor distances in a semiparametric OOD detector.
- COMBOOD is designed to provide accurate confidence scores for both near-OOD and far-OOD scenarios.
- The framework converts both distances into confidence scores from their log distributions and adds them without user-selected weights.The scores are derived from nearest-neighbor and regularized Mahalanobis distances.
- COMBOOD performs best when nearest-neighbor uses penultimate-layer embeddings while Mahalanobis uses global extrema of features.
- The feature-strategy difference reflects complementary information: penultimate embeddings capture abstraction, whereas global extrema retain information across layers and feature magnitudes.
5 Empirical Evaluation
COMBOOD was evaluated against distance-based and established OOD detectors across image-classification benchmarks, document datasets, and efficiency measures. It generally achieved stronger detection performance, including statistically significant gains over KNN on most benchmark datasets.
- Evaluation measures: The experiments measured detection quality with AUROC and AUPR and measured efficiency using average per-image inference time.AUROC ranges from 50% for a random detector to 100% for an ideal detector.
- Benchmark comparisons: COMBOOD outperformed ODIN and Mahalanobis in all evaluated near-OOD and far-OOD scenarios, and outperformed Gram despite Gram's higher computational cost.It outperformed KNN in most near-OOD and far-OOD scenarios.
- Efficiency: COMBOOD combines its parametric and nonparametric components in parallel, with inference time determined by the slower component.The resulting priors are combined to produce the final confidence score.
- Document case study: On document datasets, COMBOOD increased AUROC to 1.5 and reduced misclassified test images from 79 with KNN to 39.The corresponding error rate decreased from 15.8% to 7.8% on a test set of 500 images.
- Statistical significance: McNemar tests found COMBOOD's AUROC improvements over KNN statistically significant for most benchmark datasets.Among the post-hoc methods evaluated by OpenOOD, KNN performed best before this comparison.
6 Related Work
Prior OOD research includes many methods with inconsistent performance across datasets and substantial computational overhead. COMBOOD is positioned as a post-hoc alternative that does not require OOD data during training and improves on the strongest post-hoc baseline in most evaluated scenarios.
- OOD detection landscape: Prior standardized evaluations found inconsistent performance across OOD algorithms and no single algorithm that could be considered state of the art.The comparison covered three benchmark in-distribution datasets and seven benchmark OOD datasets.
- Post-hoc methods: COMBOOD does not require access to OOD data during training and outperforms KNN in most near-OOD and far-OOD scenarios.KNN was reported as the best-performing post-hoc algorithm in the OpenOOD evaluation.
- Computational efficiency: Earlier methods involving matrix multiplication have at least quadratic complexity, making them potentially prohibitively expensive for neural networks with many features.The cited comparison reports overhead above 100% for algorithms that cannot train on OOD data.
- Feature extraction: The paper also uses a feature-extraction strategy that computes global extrema of input features, distinct from prior extreme-value-theory work on time-series outliers.
7 Significance and Impact
COMBOOD is presented as an unsupervised semi-parametric approach that combines nearest-neighbor and Mahalanobis signals for OOD detection. Its reported benefits span accuracy and efficiency, while the current prior-combination rule remains nondatadriven and motivates future work.
- Contribution: COMBOOD combines nearest-neighbor and Mahalanobis distance signals to produce an OOD confidence score for image classification.The framework is unsupervised and semi-parametric.
- Reported impact: Experiments report improvements over state-of-the-art OOD methods on OpenOOD far-OOD and near-OOD benchmarks and on a document dataset in accuracy and efficiency.
- Limitation and future work: The current method combines the two approaches' priors using a nondatadriven rule, with data-driven combinations identified as future work.The proposed future direction includes considering the kurtosis of in-distribution and OOD distributions.
- Future work: Future work also extends OOD detection across near-OOD and far-OOD settings using layer and class applicability and toward generative models.