Source-linked AI summary
OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, Hai Li
TL;DR
OOD detection supports reliable operation of open-world intelligent systems, but inconsistent evaluation and limited benchmark scalability and scope impede standardized assessment. OpenOOD v1.5 addresses these gaps with larger-scale, broader evaluation and comprehensive analysis, finding challenges across current approaches and no consistently best method across datasets.
Problem
OOD detection is needed for reliable open-world intelligent systems, while existing evaluation definitions and practices remain inconsistent and limited in scope.
Method
OpenOOD v1.5 constructs more rigorous benchmarks and protocols, expands evaluation to broader settings including full-spectrum detection, and provides comprehensive results and analysis.
Results
Full-spectrum detection poses a significant challenge for all current approaches, and no single method consistently outperforms others across multiple datasets.
Takeaways & Limitations
OpenOOD v1.5 provides insights for assessing the current state of OOD detection and future research, with implications for applications involving unexpected anomalies and unknown inputs.
Takeaways & Limitations
OpenOOD currently focuses on discriminative classifiers and does not include generative OOD detection methods.
Abstract
from arXiv · showhide
Out-of-Distribution (OOD) detection is critical for the reliable operation of open-world intelligent systems. Despite the emergence of an increasing number of OOD detection methods, the evaluation inconsistencies present challenges for tracking the progress in this field. OpenOOD v1 initiated the unification of the OOD detection evaluation but faced limitations in scalability and scope. In response, this paper presents OpenOOD v1.5, a significant improvement from its predecessor that ensures accurate and standardized evaluation of OOD detection methodologies at large scale. Notably, OpenOOD v1.5 extends its evaluation capabilities to large-scale data sets (ImageNet) and foundation models (e.g., CLIP and DINOv2), and expands its scope to investigate full-spectrum OOD detection which considers semantic and covariate distribution shifts at the same time. This work also contributes in-depth analysis and insights derived from comprehensive experimental results, thereby enriching the knowledge pool of OOD detection methodologies. With these enhancements, OpenOOD v1.5 aims to drive advancements and offer a more robust and comprehensive evaluation benchmark for OOD detection research.
1 Introduction
OpenOOD v1.5 addresses inconsistent OOD detection evaluation by expanding OpenOOD to larger-scale data, foundation models, and full-spectrum distribution shifts. Its comprehensive results provide observations about method performance and augmentation across settings.
- OOD detection is important for reliable open-world intelligent systems, but progress tracking is hindered by confusing terminology, inconsistent data sets, and erroneous practices.
- OpenOOD v1.5 extends its predecessor with fair and accurate evaluation at larger scale and broader scope.
- Large-scale experiments and results: Nearly 40 methods and combinations are evaluated on ImageNet-1K, alongside an ImageNet-200 benchmark designed for more affordable large-scale research.
- Large-scale experiments and results: CLIP and DINOv2 are evaluated to provide an initial inspection of foundation-model performance on OOD detection tasks.
- Investigation on full-spectrum detection: Full-spectrum OOD detection is evaluated for the first time in OpenOOD v1.5, extending beyond the standard setting to include OOD generalization.
- New insights: There is no single winner across multiple data sets, while data augmentations help OOD detection in both standard and full-spectrum settings.
2 Problem Statement
The paper formalizes OOD detection for general tasks and specifies its image-classification setting, including standard and full-spectrum shifts and established evaluation metrics.
- The framework begins with a mathematical OOD definition applicable to classification and regression, then specializes the objective and metrics to image classification.
- Existing OOD work lacks a definition of what is out-of-distribution relative to a reference distribution, motivating the paper’s proposed definition.
- The proposed definition requires regions with significant OOD probability mass to have arbitrarily small in-distribution probability mass.
- The formal detection objective minimizes open space risk, while practical evaluation also considers the base task and commonly uses AUROC.
- Standard OOD detection: In standard image classification, an OOD detector identifies whether an input belongs to YID or YOOD, while the classifier assigns ID inputs to known categories.
- Full-spectrum OOD detection: Full-spectrum detection jointly considers semantic-shifted OOD images and covariate-shifted ID images, which remain in-distribution because their labels stay within YID.
- Evaluation metrics: Evaluation uses AUROC, AUPR, and FPR@95, with the first two threshold-independent and the last reflecting performance at a specific threshold.
3 Evaluation Protocol
OpenOOD v1.5 defines a rigorous evaluation protocol for standard and full-spectrum OOD detection. It separates validation, training, and test data to reduce unrealistic or overoptimistic evaluation, while distinguishing near-OOD, far-OOD, and covariate-shifted ID samples.
- Standard OOD detection: The protocol evaluates standard OOD detection by testing ID samples against multiple OOD datasets grouped as near-OOD or far-OOD.The grouping uses image semantics or empirical difficulty to provide a finer-grained assessment.
- Full-spectrum OOD detection: Full-spectrum detection adds covariate-shifted ID samples to the ID test data while retaining the same OOD datasets for direct comparison.The covariate shifts include corruptions, style changes, and resampling bias in the broader protocol description.
- Data partitioning: Validation ID and OOD samples are introduced for hyperparameter tuning and checkpoint selection instead of using test samples.This design aims to ensure realistic evaluation and avoid overoptimistic results.
- Data partitioning: OOD training samples are distinguished from OOD test samples, with category-disjoint construction intended to prevent trivial evaluation.The protocol explicitly separates Dtrain_OOD from Dtest_OOD and requires non-overlapping training and test OOD categories.
4 Supported Benchmarks and Methods
OpenOOD v1.5 broadens its benchmark suite from smaller standard settings to large-scale ImageNet and full-spectrum evaluation. It also curates disjoint OOD training and test data and supports a range of post-hoc and training-based methods.
- Benchmark scope: OpenOOD v1.5 includes four standard benchmarks and two large-scale full-spectrum benchmarks centered on ImageNet-200 and ImageNet-1K.The full-spectrum benchmarks add covariate-shifted samples to the ID test set while keeping OOD datasets unchanged.
- ImageNet benchmarks: ImageNet-1K uses 45,000 validation images for ID testing and 5,000 for ID validation without modifying the original 1.2M-image training set.Its near-OOD group includes SSB-hard and NINCO, while its far-OOD group includes iNaturalist, Textures, and OpenImage-O.
- ImageNet benchmarks: ImageNet-200 reuses the ImageNet-1K OOD datasets with 200 ID categories, enabling lower-compute experiments and direct scalability comparisons.The 200 categories match those of ImageNet-R.
- Supported methods: Supported methods include post-hoc inference methods and other methods prioritized for public implementations and reliable adaptation into the framework.Post-hoc methods apply scoring functions to a base classifier at inference time.
5 Experiment Setup
The experiments evaluate a broad set of OOD detection methods across the supported benchmarks using standardized training, pretrained-model, reproducibility, and metric procedures. The setup also documents missing results and excludes some training configurations when data overlap or implementation constraints prevent meaningful evaluation.
- Training: CIFAR-10/100 and ImageNet-200 models are trained with ResNet-18, while ImageNet-1K post-hoc evaluation uses pretrained ResNet-50, ViT, and Swin Transformer models.Training methods on ImageNet-1K focus on ResNet-50 and use official checkpoints when possible.
- Reproducibility: All training runs are reproducible through OpenOOD configuration files and documented training scripts.Except for ImageNet-1K experiments, each method uses three independent training runs.
- Evaluation metrics: The evaluation reports AUROC, AUPR, and FPR@95, emphasizing near-OOD and far-OOD AUROC averaged across datasets in each group.AUROC is interpreted as the probability of correctly separating ID and OOD samples, with a 50% random-guessing baseline.
- Scope boundaries: Some methods lack reported results because they are infeasible on ImageNet-1K, lack exact ID-accuracy evaluation code, or rely on OOD training data excluded from the benchmark.The benchmark also notes difficulty finding non-overlapping OOD training samples for ImageNet-1K.
6 Analysis
OpenOOD v1.5 finds that OOD detection performance varies substantially across datasets, scales, architectures, and distribution-shift settings. Data augmentation and post-processing are complementary, while full-spectrum detection remains difficult and introduces practical trade-offs.
- No single method consistently outperforms others across benchmarks, with rankings changing substantially between datasets.
- Data augmentations improve OOD detection and amplify gains when combined with powerful post-processors.
- 82.16% AUROC is the best reported ImageNet-1K near-OOD score, achieved by AugMix combined with ASH.
- Near-OOD AUROC improves roughly with far-OOD AUROC, but progresses more slowly because near-OOD detection is more difficult.
- Training-time methods outperform post-hoc methods on CIFAR-10/100, but generally not on ImageNet-200/1K.On CIFAR-10, RotPred and LogitNorm improve near- and far-OOD AUROC by approximately 2% and 3%, respectively, over the best post-processors.
- Several methods impose substantial practical costs: some increase inference time by orders of magnitude, while others require storing many ID feature vectors.
- Full-spectrum detection causes many detectors to lose more than 10% near-OOD AUROC, indicating sensitivity to covariate shift and possible false OOD flags for shifted ID samples.MOS has the smallest reported drop, from 72.85% to 66.17% AUROC.
- AugMix combined with SHE improves full-spectrum near-OOD AUROC by 8.45% over the cross-entropy baseline, although augmentation benefits are unclear for full-spectrum far-OOD.
7 Conclusion and Discussion
OpenOOD v1.5 strengthens OOD detection evaluation through rigorous benchmarks, large-scale ImageNet analysis, and full-spectrum detection. Its results expose persistent gaps, including the absence of a universally strong detector and the exclusion of generative methods.
- OpenOOD v1.5 constructs more rigorous evaluation protocols and benchmarks, extends experiments to ImageNet, and investigates full-spectrum detection.
- No single OOD detector performs competitively across multiple benchmarks.
- Data augmentations combined with strong post-processors are particularly effective in the reported results.
- Full-spectrum detection remains challenging and may require insights from both OOD generalization and detection research.
- The benchmark currently assumes a discriminative ID classifier and excludes generative OOD methods, whose integration requires additional effort.
- OpenOOD is intended to support machine-learning safety and research on unknown inputs in applications including remote sensing and anomaly detection.
Appendix A. Evaluation Pitfalls of OOD Detection
OOD detection research suffers from inconsistent terminology, datasets, and evaluation practices. These problems hinder direct comparison and can produce overoptimistic results.
- Confusing terminologies: OOD detection and OSR pursue essentially the same goal but are often developed and compared separately under different benchmarks.
- Inconsistent data sets: Evaluation datasets vary greatly across papers, making straightforward comparison between methods difficult.
- Erroneous practices: Training or tuning on test OOD data violates basic machine-learning principles and leads to overoptimistic results.
Appendix B. New Features and Updates of OpenOOD v1.5
OpenOOD v1.5 adds an online leaderboard and lightweight evaluator to make standardized OOD evaluation easier to run and track.
- The new online leaderboard tracks state-of-the-art results across methods, while the lightweight evaluator enables evaluation with a few lines of code.
Appendix C. Supported Methods
OpenOOD v1.5 supports post-hoc scoring methods, training methods with and without outlier data, and data augmentations. Together, these categories cover inference-time scoring, training-time regularization, auxiliary-OOD supervision, and augmentation-based approaches.
- Post-Hoc Inference Methods: Post-hoc methods assign OOD scores from classifier outputs and threshold them to produce binary predictions.
- Post-Hoc Inference Methods: Examples of post-hoc methods include MSP, OpenMax, temperature scaling, ODIN, MDS, MDSEns, and RMDS.
- Training methods without outlier data: Training methods without outlier data enhance OOD detection through training-time regularization, including uncertainty branches, reciprocal points, hierarchical classifiers, and feature-space regularization.
- Training methods with outlier data: Training methods with outlier data use auxiliary OOD samples, including supervised OOD learning, prediction disagreement, clustering, and pixel-level mixing.
- Data augmentations: The benchmark also evaluates data augmentations such as StyleAugment, RandAugment, and AugMix to improve image-classifier generalization.
Appendix D. Generalizing OpenOOD to Other Tasks
OpenOOD can be applied beyond classification when a task’s model output has a similar structure. An age-estimation experiment demonstrates this applicability, although detection performance is not uniformly strong.
- OpenOOD investigates whether its framework applies to tasks beyond classification, using image-based age regression as an example.
- Three simple post-hoc OOD detectors are applied to face-age estimation, with Textures and Places providing OOD images.
- The age predictor computes continuous age from a softmax-weighted sum over discrete ages, making its output structurally similar to classification.
- 10% over the random-guessing baseline is achieved by some method on each OOD dataset, while overall performance is not extremely satisfying.Table 5 identifies AUROC as the evaluation metric and sets the random-guessing baseline at 50%.
Appendix E. Related Work
The related work spans studies of method inconsistency, benchmark construction, evaluation libraries, and ImageNet OOD datasets. OpenOOD distinguishes itself through standardized benchmarks and broader method coverage.
- OpenOOD v1.5 is presented as comprehensively benchmarking OOD methods across multiple ID-OOD pairs.
- Earlier experiments found OOD methods inconsistent across datasets, while OpenOOD reports a similar “no single winner” conclusion using four datasets and nearly 40 methods.
- Galil et al.’s benchmark groups ImageNet-21K OOD images by scores from specific classifier–post-processor pairs, making comparisons dependent on those choices.
- OpenOOD’s benchmarks are standardized and agnostic to classifiers and post-processors, while covering methods beyond a few post-hoc approaches.
- OpenOOD supports 40 approaches, compared with PyTorch-OOD’s 19 methods as of May 2023, including approaches published in 2023.
- OpenOOD is complementary to NINCO because it evaluates varied method types rather than restricting evaluation to post-hoc methods.