Source-linked AI summary
OpenOOD: Benchmarking Generalized Out-of-Distribution Detection
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, Ziwei Liu
TL;DR
OOD detection lacks a unified benchmark despite its importance for reliable machine learning and its close ties to anomaly detection, open set recognition, and uncertainty. OpenOOD addresses this gap with a generalized evaluation framework covering 9 benchmarks and 35 methods, finding strong potential in preprocessing and post-hoc methods. The benchmark results are limited by single-run evaluations and exclusion of large-scale ImageNet training methods.
Problem
OOD detection lacks a unified, comprehensive benchmark, and related fields such as anomaly detection, open set recognition, and model uncertainty have developed in isolation despite overlapping applicability.
Method
OpenOOD provides 9 carefully designed benchmarks and integrates 35 methods from OOD detection-related fields under a unified generalized OOD detection framework.
Results
Across the benchmarks, simple data augmentation preprocessing methods are generally most effective, while post-hoc methods make significant progress and are generally no worse than methods requiring training.
Takeaways & Limitations
OpenOOD enables fairer cross-task comparison and indicates that simple preprocessing and post-hoc methods are strong candidates for OOD detection.
Takeaways & Limitations
Each method is evaluated only once without multiple random-seed runs, and training methods on large-scale ImageNet are excluded because of limited computational resources.
Abstract
from arXiv · showhide
Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often results in unfair comparisons and inconclusive results. From the problem setting perspective, OOD detection is closely related to neighboring fields including anomaly detection (AD), open set recognition (OSR), and model uncertainty, since methods developed for one domain are often applicable to each other. To help the community to improve the evaluation and advance, we build a unified, well-structured codebase called OpenOOD, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework. With a comprehensive comparison of these methods, we are gratified that the field has progressed significantly over the past few years, where both preprocessing methods and the orthogonal post-hoc methods show strong potential.
1 Introduction
OpenOOD addresses fragmented and unfair OOD evaluation by unifying related tasks, benchmarks, and methods in a structured codebase. Its comparisons indicate strong potential for simple preprocessing and post-hoc approaches.
- OOD detection matters for model safety because deployed models inevitably encounter unseen samples that deviate from training distributions.
- The field lacks a uniform, comprehensive benchmark, while dataset-dependent reporting, saturated scores, and inconsistent preprocessing hinder fair comparisons.
- Methods from anomaly detection, open set recognition, model uncertainty, and data augmentation can apply to OOD detection under a generalized framework.
- OpenOOD provides 9 benchmarks and integrates 35 representative methods from related fields for unified evaluation and comparison.
- Comprehensive comparisons find that simple preprocessing methods can achieve the best benchmark scores, while post-hoc methods generally match methods requiring training.
2 Supported Tasks, Benchmarks, and Metrics
OpenOOD organizes evaluation across anomaly detection, open set recognition, and OOD detection, using task-specific benchmarks and metrics. Its OOD benchmarks distinguish near- and far-OOD while cleaning semantic overlaps from test sets.
- OpenOOD covers 9 benchmarks spanning AD, OSR, and OOD detection, with metrics including FPR@95 and AUROC.
- Anomaly Detection: Anomaly detection identifies sensory or semantic anomalies, and the MVTec-AD benchmark trains on defect-free images before testing normal and defective samples.
- Open Set Recognition: OSR partitions dataset classes into closed and open sets, requiring models trained on closed classes to separate unknown open-set samples during testing.
- OOD Detection: OOD detection targets samples with label distributions different from training, using intact ID datasets and disconnected datasets as OOD sources.
- OOD Detection: Near-OOD datasets share greater similarity with ID data, whereas far-OOD datasets have obviously different styles, enabling detailed detector analysis.
- OOD Benchmark Construction: OpenOOD removes semantically overlapping images, including 1,207 TinyImageNet images from the CIFAR-10 benchmark and 2,502 from CIFAR-100.
3 Supported Methodologies
OpenOOD organizes 35 supported methods by originating field and methodological type, covering AD, OSR, OOD detection, uncertainty, and augmentation. The codebase includes both training-free and training-based approaches.
- OpenOOD prioritizes open-source methods and organizes 35 approaches chronologically by field and methodological type.
- Anomaly Detection: Its AD coverage includes distance-, augmentation-, memory-bank-, and reconstruction-based methods such as Deep-SVDD, CutPaste, PatchCore, and DRÆM.
- Open Set Recognition: Its OSR coverage includes OpenMax, ARPL, and OpenGAN, which model unknowns through Weibull-based probabilities, reciprocal points, or generated negative features.
- Out-of-Distribution Detection: OpenOOD integrates 22 OOD detection methods spanning post-hoc scoring, training-time regularization, and outlier exposure.
- Post-hoc Methods: Post-hoc methods operate on pretrained models, offering plug-and-play integration and compatibility with varied architectures and training procedures.
- Model Uncertainty and Augmentation: The broader collection also includes Bayesian uncertainty methods, temperature scaling, and augmentation-based uncertainty estimation.
4 Experiments
OpenOOD evaluates 35 methods under unified generalized OOD benchmarks, revealing patterns across augmentation, post-hoc, extra-data, dataset, anomaly-detection, and open-set settings.
- Experimental setup: OpenOOD compares 35 methods using unified architectures, hyperparameters, training procedures, and validation-based tuning to support fair evaluation.The benchmark uses LeNet for MNIST, ResNet-18 for CIFAR and TinyImageNet, and ResNet50 for ImageNet.
- Main results: 93.1% AUROC: PixMix achieves the best reported Near-OOD result on CIFAR-10, while Ensemble and Mixup also perform strongly.Data augmentation and other model-uncertainty methods are generally the most effective category in the benchmark.
- Main results: Extra data provides limited advantage: UDG outperforms KNN mainly on CIFAR-10 Near-OOD, while the training outliers are not purely OOD.The authors note that the choice of training outliers can greatly affect detector performance.
- Main results: Inference-only post-hoc methods generally do not underperform methods requiring training, although trained models can potentially be combined with post-hoc methods.Post-hoc methods also have minimal computational cost, whereas data augmentation achieves strong results at somewhat higher cost.
- Main results: ImageNet generally yields higher OOD detection scores than CIFAR-10 and CIFAR-100, despite its more complex data and apparent difficulty.The paper notes that strong performance on higher-resolution datasets is more practical for real-world applications.
- Cross-task findings: Recent post-hoc methods improve performance on realistic datasets, AD methods perform well on Far-OOD, and OSR results align substantially with Near-OOD detection.KNN maintains good performance across MNIST, CIFAR-10, CIFAR-100, and ImageNet, while DRAEM and CutPaste perform strongly on Far-OOD.
- Cross-task findings: Most OSR/OOD methods do not affect ID classification performance on the CIFAR-100 benchmark.ID accuracy is reported alongside OOD results to assess whether detection methods downgrade classification capability.
5 Outlook and Conclusion
OpenOOD’s comparison highlights strong potential for simple preprocessing and post-hoc methods, while documenting benchmark limitations and broader safety relevance. The authors position the codebase as a practical resource for fairer evaluation and future OOD research.
- Simple preprocessing methods can achieve the best score among the benchmark, while extra data seems unnecessary or requires further exploration.
- Post-hoc methods make significant progress and generally outperform methods that require training.
- Each benchmark method runs only once without multiple random-seed runs because of limited computational resources.
- OOD detection can identify unexpected anomalies, emergent phenomena, unknown unknowns, Black Swans, malicious use, and network intruders, reducing exposure to hazards and improving safety.
- Future work includes extending the benchmark to robust detectors and object-level OOD detection for safety-critical applications.
- OpenOOD’s unified evaluation helps the community select suitable methods, while its open-source codebase reduces redundant work.