Source-linked AI summary
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, Zsolt Kira
TL;DR
OoD detection matters because neural networks can fail when deployment data differs from training data, while many methods depend on difficult-to-specify OoD data for tuning. This paper adapts ODIN with decomposed confidence and modified preprocessing that use no OoD tuning, and reports improved detection with differing difficulty across semantic and non-semantic shifts.
Problem
Many OoD detection methods train or tune with OoD data, whose prior selection is difficult and can introduce selection bias.
Method
The paper decomposes confidence scoring and develops an input-preprocessing strategy whose perturbation magnitude is tuned using only in-distribution data.
Results
Across eight OoD benchmarks, the combined DeConf-C* strategy outperforms ODIN* and Mahalanobis* trained without OoD data, while often surpassing their OoD-tuned originals.
Takeaways & Limitations
Learning OoD detection without OoD data is feasible, but DomainNet shows that semantic-only shifts are harder to detect.
Takeaways & Limitations
Some DeConf variants can suffer from overfitting, and extra regularization affects variants differently.
Abstract
from arXiv · showhide
Deep neural networks have attained remarkable performance when applied to data that comes from the same distribution as that of the training set, but can significantly degrade otherwise. Therefore, detecting whether an example is out-of-distribution (OoD) is crucial to enable a system that can reject such samples or alert users. Recent works have made significant progress on OoD benchmarks consisting of small image datasets. However, many recent methods based on neural networks rely on training or tuning with both in-distribution and out-of-distribution data. The latter is generally hard to define a-priori, and its selection can easily bias the learning. We base our work on a popular method ODIN, proposing two strategies for freeing it from the needs of tuning with OoD data, while improving its OoD detection performance. We specifically propose to decompose confidence scoring as well as a modified input pre-processing method. We show that both of these significantly help in detection performance. Our further analysis on a larger scale image dataset shows that the two types of distribution shifts, specifically semantic shift and non-semantic shift, present a significant difference in the difficulty of the problem, providing an analysis of when ODIN-like strategies do or do not work.
1. Introduction
Out-of-distribution detection is needed because closed-world neural networks can fail silently when distributions shift or new concepts appear. The paper proposes confidence decomposition and OoD-free input preprocessing to improve ODIN-style detection.
- Closed-world neural networks can fail catastrophically and silently when test distributions shift or unseen concepts appear.
- OoD detection asks whether an input comes from the training distribution or a different distribution, enabling rejection or user alerts.
- ODIN improves softmax-based OoD scoring through temperature scaling and input preprocessing but tunes both strategies with OoD data.
- OoD-tuned hyperparameters may not generalize across datasets, while broad OoD spaces can create selection bias.
- Prior methods learning without OoD data reported performance similar to ODIN, making the setting challenging.
- The paper introduces decomposed confidence and modified input preprocessing, both designed to avoid tuning with OoD data.
- The proposed strategies are evaluated on standard benchmarks and DomainNet to analyze when OoD detection strategies work.
2. Background
OoD detection distinguishes in-distribution data from inputs drawn from potentially intractable alternative distributions, including semantic and non-semantic shifts. The paper frames scoring as estimating in-distribution domain confidence and removes OoD tuning from established baselines.
- The classifier is trained on in-distribution samples Din generated from pin(x, y), then predicts class posterior probabilities p(y|x).
- Open-world deployment may encounter pout ≠ pin, and enumerating all possible OoD distributions is generally intractable for high-dimensional images.
- Non-semantic shift preserves object classes but changes presentation, whereas semantic shift involves different classes or concepts.
- An explicit domain variable d distinguishes din from dout, representing whether an input belongs to the training distribution.
- OoD detection uses a score S(x) whose threshold separates inputs according to their estimated in-distribution domain probability.
- ODIN and Mahalanobis begin with a classifier and add scoring parameters originally tuned separately for each OoD dataset.
- For OoD-free baselines, the experiments fix ODIN's temperature at T = 1000 and use uniform Mahalanobis weighting.
3. Approach
The approach removes out-of-distribution data from ODIN-style tuning by decomposing confidence and modifying input preprocessing. It uses a dividend/divisor classifier structure and selects perturbations using only in-distribution validation data.
- 3.1. The Decomposed Confidence: The method rewrites class confidence using an explicit in-distribution domain variable, separating joint class-domain probability from domain probability.This decomposition provides a probabilistic view of softmax overconfidence and motivates learning without out-of-distribution supervision.
- 3.1. The Decomposed Confidence: The dividend/divisor classifier defines each class logit fi(x) as hi(x) divided by g(x), then applies softmax to produce class probabilities.The network uses two branches after the penultimate layer and trains the resulting logits with cross-entropy loss.
- 3.1. The Decomposed Confidence: The cross-entropy objective can increase hi(x) or decrease g(x), encouraging different behavior in low- versus high-density in-distribution regions.This behavior motivates using hi(x) or g(x) as OoD detection scores, while g(x) may estimate p(din|x) beyond simple temperature scaling.
- 3.1.1 Design Choices: The decomposition is implemented with inner-product, negative Euclidean distance, and cosine-similarity choices for hi(x), producing DeConf-I, DeConf-E, and DeConf-C variants.The dividend/divisor structure encourages the desired confidence decomposition but does not guarantee it; the characteristics of hi(x) and g(x) matter.
- 3.2. A Modified Input Preprocessing Strategy: The modified input preprocessing selects one perturbation magnitude ϵ* by maximizing the score on an in-distribution validation set rather than tuning separately with OoD data.The search avoids the benchmark performance metric and uses a coarser six-value grid, making it faster while allowing possible overshooting.
4. Experiments
Experiments evaluate OoD detection without OoD data across benchmark datasets and DomainNet, testing the proposed scoring and input-preprocessing strategies, their robustness, and distribution-shift difficulty.
- Experimental Settings: All methods in Table 1 are evaluated without OoD data during training and validation across eight benchmark datasets.The base network is DenseNet trained on CIFAR-10/100, with values averaged over three runs.
- Results and Discussion: DeConf-C* significantly outperforms ODIN* and Mahalanobis* in the no-OoD-data benchmark comparison.The authors report this as evidence that OoD detection without OoD data is feasible and that the combined proposed methods are effective.
- Results and Discussion: The original ODIN and Mahalanobis methods generally perform better than their no-OoD-data variants, while DeConf-C* surpasses them in many cases.The original methods were tuned on each OoD dataset, giving them an advantage in this cross-setting comparison.
- Robustness and Distribution Shift: DeConf-C* is most robust in high-data regimes and across class counts, while semantic shift is harder to detect than non-semantic or combined shifts.DomainNet results also report DeConf-C* as best in the more realistic distribution-shift setting, despite substantial room for improvement.
- Results and Discussion: Extra regularization improves DeConf-I and all g(x) variants, reducing the CIFAR-100 classifier-accuracy drop to 1%.The authors associate the unregularized weakness with overfitting and report dropout rate 0.7 as the regularization setting.
5. Conclusion
The paper proposes decomposed confidence and modified input preprocessing to adapt ODIN without OoD tuning. These strategies are effective, sometimes outperform OoD-tuned methods, while semantic shifts remain harder to detect.
- The decomposed confidence and modified input preprocessing strategies eliminate the need for OoD data when tuning the method.
- The proposed strategies are effective and outperform methods tuned separately for each OoD dataset in several cases.
- Larger-scale image analysis shows that data with only semantic shift are harder to detect, identifying a challenge for future work.
Supplementary
The supplementary material visualizes score distributions and documents benchmark performance and experimental settings across multiple classifier backbones and regularization conditions.
- Figure 9 visualizes score distributions with t-SNE features from penultimate neural-network layers for DeConf-I with ResNet-34.It contrasts CIFAR-10 in-distribution data with Imagenet-resized OoD data and uses scores from h, g, or logits.
- Table 5 reports six OoD detection methods on eight benchmark datasets using DenseNet backbones and the h(x) branch for all DeConf results.Parenthesized values are standard deviations.
- Table 6 reports the corresponding eight-dataset experiment with ResNet-34 backbones and the h(x) branch for all DeConf results.Parenthesized values are standard deviations.
- Table 7 lists AUROC values for individual settings in Figures 3 and 4 without input preprocessing, averaged over three runs.A plus sign denotes extra regularization with dropout rate 0.7; parenthesized values are standard deviations.
- Table 8 summarizes the classifiers analyzed experimentally and reports their in-domain classification accuracy.A plus sign denotes classifiers trained with extra regularization using dropout rate 0.7.