PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Out-of-Distribution Detection for Real-World Settings

Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, Dawn Song

arXiv:1911.11132v4cs.CVcs.LG

TL;DR

OOD detection research has largely focused on small-scale settings, despite its importance for safe and reliable machine learning. The paper introduces large-scale multiclass, multilabel, and anomaly-segmentation benchmarks, including Species and CAOS, and evaluates maximum-logit detection. MaxLogit outperforms prior methods across the large-scale tasks, providing a baseline for future work.

  • Problem

    Existing OOD research mainly uses small-scale datasets, while realistic safety-critical applications involve high-resolution images, many classes, complex scenes, and localized anomalies.

  • Method

    The paper creates Species, a multi-label evaluation setup, and CAOS, then evaluates maximum-logit OOD detection across multiclass, multilabel, and anomaly-segmentation settings.

  • Results

    MaxLogit outperforms prior baselines across the large-scale multiclass, multilabel, and anomaly-segmentation tasks.

  • Takeaways & Limitations

    Species and CAOS, together with the MaxLogit baseline, support future research on large-scale OOD detection.

Abstract

from arXiv · show

Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes. To make future work in real-world settings possible, we create new benchmarks for three large-scale settings. To test ImageNet multiclass anomaly detectors, we introduce the Species dataset containing over 700,000 images and over a thousand anomalous species. We leverage ImageNet-21K to evaluate PASCAL VOC and COCO multilabel anomaly detectors. Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks, establishing a simple new baseline for future work.

1. Introduction

The paper extends OOD detection beyond small-scale benchmarks to large-scale multiclass, multilabel, and anomaly-segmentation settings, introducing benchmarks and finding that maximum-logit detection consistently outperforms prior methods.

  • OOD detection supports safer ML systems by enabling conservative fallback policies or deferring decisions to human judgment.
  • Existing OOD research primarily uses small-scale datasets with low-resolution images and few classes, motivating stronger large-scale benchmarks.
  • The paper introduces Species for controlled multiclass evaluation, a multi-label setup using ImageNet-21K, and CAOS for anomaly segmentation in driving scenes.
  • StreetHazards uses simulated driving scenes to integrate diverse foreign objects with correct lighting and orientation, avoiding artificial copy-paste cues.
  • MaxLogit outperforms all baselines on Species, the multi-class benchmark, and CAOS, establishing a simple baseline for large-scale OOD detection.

2. Related Work

Prior OOD research relies heavily on small-scale or mismatched benchmarks, motivating large-scale datasets that avoid class overlap and artificial anomaly cues.

  • Recent OOD methods primarily evaluate small-scale datasets with low-resolution images and few classes.
  • ImageNet-21K pretraining can overlap with earlier OOD datasets, causing erroneous evaluations of supposedly unseen categories.
  • Species contains hundreds of anomalous species disjoint from ImageNet-21K classes, enabling controlled evaluation of large-scale detectors.
  • Existing anomaly-segmentation benchmarks often treat whole images as anomalous or use cut-and-paste objects with unnatural contextual cues.
  • StreetHazards inserts foreign 3D objects into simulated driving scenes with integrated lighting and orientation, avoiding artificial overlay cues.

3. Multi-Class Prediction for OOD Detection

Large-scale multiclass classification disperses probability across visually similar classes, weakening MSP; the paper proposes MaxLogit and evaluates it with the Species benchmark.

  • Problem with existing baselines: Large-scale classifiers can disperse probability mass among visually similar classes, making low confidence misclassify difficult in-distribution images as anomalies.
  • MaxLogit detector: MaxLogit uses the negative maximum unnormalized logit, whose value is not affected by the number of classes.
  • Species dataset: Over 700,000 images and over a thousand anomalous species make Species a large-scale, class-disjoint OOD test dataset.
  • Experimental setup: The experiments evaluate MSP and MaxLogit with ImageNet-21K models, alongside ImageNet-1K and Places365 settings and standard AUROC, FPR95, and AUPR metrics.
  • Results: MaxLogit outperforms MSP on every OOD test dataset for all three ImageNet-21K-trained models.

4. Multi-Label Prediction for OOD Detection

The paper extends OOD detection to multi-label data with a dedicated setup and finds MaxLogit to be the strongest baseline across PASCAL VOC and MS-COCO evaluations.

  • Multi-label OOD detection is increasingly relevant, but prior work provides little dedicated evaluation infrastructure.
  • Setup: The setup uses PASCAL VOC and MS-COCO as in-distribution data with 20 ImageNet-21K classes absent from the training domains.
  • Methods: Models use an ImageNet-1K-pretrained ResNet-101 with sigmoid multi-label prediction and frozen batch-normalization parameters.
  • Methods: Compared detectors include iForest, LOF, Dropout, logit average, MSP, and MaxLogit, with MaxLogit scoring labels by −max_i f(x)_i.
  • Results: MaxLogit achieves the highest performance in every evaluated multi-label case and is naturally applicable to multi-label problems.

5. The CAOS Benchmark

The CAOS benchmark evaluates anomaly segmentation on diverse, realistic driving-scene anomalies using simulated and real-image datasets. Experiments show MaxLogit performs best, although class imbalance remains a major challenge.

  • Benchmark construction: CAOS combines StreetHazards’ simulated anomalies with BDD-Anomaly’s real driving images to test segmentation across diverse, realistic conditions.Both datasets use 720 × 1280 images.
  • Benchmark construction: StreetHazards inserts foreign objects into rendered CARLA scenes with contextually correct lighting and orientation, avoiding common cut-and-paste artifacts.The dataset uses Unreal Engine and CARLA to integrate anomalies into driving scenes.
  • Results: MaxLogit obtains the best performance on CAOS, while all methods substantially outperform chance despite low AUPR from severe class imbalance.The evaluation averages per-image metrics because pixel-wise predictions are too numerous to fit in memory directly.
  • Results: MaxLogit outperforms all other methods by a substantial margin, whereas the background-class detector performs poorly and the autoencoder struggles in the complex CAOS domain.The results suggest the background class may not align with rare visual features.
  • Results: MaxLogit and MSP produce false positives near semantic boundaries, but the problem is less severe for MaxLogit.The authors suggest maximum logits can remain stable during class hand-offs even when prediction confidence falls.

6. Conclusion

The paper scales OOD detection to thousands of classes and high-resolution images through new datasets, experimental settings, and the CAOS benchmark. MaxLogit consistently improves over MSP in multi-label and anomaly-segmentation settings, establishing a baseline for large-scale OOD research.

  • The paper scales OOD detection to thousands of classes and high-resolution images.
  • The authors introduce Species, a multi-label experimental setup, and CAOS to support large-scale multiclass, multi-label, and anomaly-segmentation research.
  • MaxLogit provides consistent and significant gains over MSP in multi-label and anomaly-segmentation settings.
  • CAOS baselines substantially improve over random guessing but remain limited, indicating room for future work.

A. Full Multiclass OOD Detection Results

The multiclass experiments evaluate OOD detection across synthetic and realistic datasets and examine methods designed for large-scale settings. MaxLogit consistently outperforms MSP and other baselines, while several alternative approaches do not scale straightforwardly.

  • Datasets: The evaluation uses ImageNet-1K and Places365 as in-distribution datasets, with synthetic and realistic unseen OOD examples.OOD datasets include Gaussian noise, Rademacher noise, blobs, textures, LSUN, and Places69.
  • Methods: The KL Matching method compares each test posterior with class-specific validation-derived posterior templates and uses the minimum KL divergence as its anomaly score.It does not require validation labels but requires more data than MaxLogit.
  • Results: MaxLogit outperforms MSP on all three metrics for both ImageNet and Places365, with a Places365 AUROC improvement over 10%.
  • Results: On CIFAR-10, MSP reaches 90.08% average AUROC versus 90.22% for MaxLogit, a minor 0.14% difference that contrasts with larger-scale gains.The paper reports that differences can exceed 10% on individual large-scale OOD datasets.
  • Alternative methods: Auxiliary rotation prediction yields 59.1% AUROC with MSP and 73.6% with MaxLogit, but the network performs better without the auxiliary task.Mahalanobis scaling to 1000 classes consistently produced NaN errors from high condition numbers.
  • Alternative methods: Combining MaxLogit with ODIN produces ImageNet FPR95 33.6, AUROC 88.8, and AUPR 51.3, while Places365 reaches FPR95 35.3, AUROC 86.5, and AUPR 24.2.The results indicate compatibility between MaxLogit and techniques using different assumptions.

B. OOD Segmentation

This section adapts anomaly-detection methods to semantic segmentation and evaluates them on realistic driving-scene benchmarks. It describes confidence estimation, lighting-aware StreetHazards examples, and dataset limitations.

  • Segmentation setup: Semantic segmentation models use f(x)i,j to produce per-pixel prediction distributions P for image locations i,j.The network input is an image x, and yi,j denotes the prediction for pixel i,j.
  • Confidence estimation: Confidence estimation adds a confidence branch that assigns every pixel a confidence value alongside the network predictions.The method modifies the training objective with an auxiliary confidence-related term.
  • Confidence estimation: A beta-distributed training hint regularizes confidence learning so the network does not rely exclusively on true labels.The final loss includes an additional term, with its weighting adaptively adjusted during training.
  • Dataset limitations: 3.9% of BDD-Anomaly validation images are estimated to contain label noise, with a small reported impact on results.The estimate comes from manually checking training and validation examples in BDD100K.
  • StreetHazards: StreetHazards inserts diverse foreign objects into simulated driving scenes and re-renders them with correct lighting and orientation.This construction avoids giveaway cues associated with pasted anomalies, such as edge effects and mismatched lighting.

C. Species

The Species dataset is cleaned using automated and human quality checks, then provided in two configurations for ImageNet-based and ecological OOD research.

  • Dataset construction: Images scraped from iNaturalist undergo blind quality assessment followed by Mechanical Turk review for low-quality or irrelevant examples.Images below the automated quality cutoff are discarded before manual flagging.
  • Usage: The dataset offers separate configurations for off-the-shelf ImageNet classifiers and researchers simulating ecological OOD detection.The first uses ImageNet as in-distribution data and Species examples as OOD; the second divides images into individual species classes.
Loading 1911.11132v4…