PaperlayerView exact revision on arXiv

Source-linked AI summary

POEM: Out-of-Distribution Detection with Posterior Sampling

Yifei Ming, Ying Fan, Yixuan Li

arXiv:2206.13687v1cs.LGcs.AIcs.CV

TL;DR

OOD detection must use auxiliary outlier data efficiently because the potential OOD sample space is prohibitively large and many outliers are uninformative. POEM uses posterior sampling to mine near-boundary outliers, achieving state-of-the-art performance on CIFAR-10 and CIFAR-100, including a 4.82% FPR95 improvement over NTOM on CIFAR-100.

  • Problem

    Auxiliary outlier datasets improve OOD detection, but their prohibitively large sample spaces make inefficient outlier use a central challenge.

  • Method

    POEM treats outlier mining as sequential decision making, using a boundary score and posterior sampling to select informative near-boundary outliers while balancing exploration and exploitation.

  • Results

    POEM establishes state-of-the-art OOD detection performance on CIFAR-10 and CIFAR-100 and outperforms NTOM by 4.82% in FPR95 on CIFAR-100.

  • Takeaways & Limitations

    Selecting near-boundary outliers can improve OOD detection and use auxiliary outlier data more effectively, while maintaining comparable in-distribution classification accuracy.

  • Takeaways & Limitations

    The theoretical analysis assumes Gaussian feature distributions and a large signal/noise ratio.

Abstract

from arXiv · show

Out-of-distribution (OOD) detection is indispensable for machine learning models deployed in the open world. Recently, the use of an auxiliary outlier dataset during training (also known as outlier exposure) has shown promising performance. As the sample space for potential OOD data can be prohibitively large, sampling informative outliers is essential. In this work, we propose a novel posterior sampling-based outlier mining framework, POEM, which facilitates efficient use of outlier data and promotes learning a compact decision boundary between ID and OOD data for improved detection. We show that POEM establishes state-of-the-art performance on common benchmarks. Compared to the current best method that uses a greedy sampling strategy, POEM improves the relative performance by 42.0% and 24.2% (FPR95) on CIFAR-10 and CIFAR-100, respectively. We further provide theoretical insights on the effectiveness of POEM for OOD detection.

1. Introduction

POEM addresses the inefficient use of auxiliary outliers in OOD detection by mining informative samples with posterior sampling. It targets a compact ID–OOD decision boundary while balancing exploration and exploitation, achieving strong benchmark performance and tractable training.

  • Motivation: Auxiliary outlier methods can waste informative potential because the space of candidate OOD samples is prohibitively large.Random selection often yields outliers that do not meaningfully improve the estimated ID–OOD decision boundary.
  • POEM framework: POEM selects informative outliers from a large auxiliary pool to help estimate a compact decision boundary between ID and OOD data.Its reward, called boundary score, is higher for outliers close to the ID–OOD boundary.
  • POEM framework: POEM formulates outlier mining as sequential decision making and uses Thompson sampling to balance exploration and exploitation during selection.Actions correspond to outlier selection, while rewards depend on closeness to the ID–OOD boundary.
  • POEM framework: POEM maintains and periodically updates a model posterior, approximated with Bayesian linear regression over neural-network feature representations.The posterior guides selection of samples with high boundary scores as training changes the estimated decision boundary.
  • POEM framework: POEM is computationally tractable and can be trained efficiently end-to-end with modern deep neural networks.The framework is designed to improve posterior-guided outlier selection without making training impractical.
  • Results: 42.0% and 24.2% relative FPR95 improvements are reported over greedy sampling on CIFAR-10 and CIFAR-100, respectively.The evaluation spans six diverse OOD test datasets, and the paper reports state-of-the-art benchmark performance.

2. Preliminaries

OOD detection asks models to distinguish known in-distribution inputs from unknown inputs at deployment. Auxiliary outlier data regularizes models during training, but selecting useful outliers efficiently remains an open challenge.

  • OOD detection: Reliable deployment requires both accurate classification of known ID samples and identification of OOD inputs as unknown.These are described as dual objectives for deployed classifiers.
  • OOD detection: OOD detection can be formulated as binary classification between the in-distribution marginal Pin and unknown inputs outside it.At test time, the model decides whether x comes from Pin or is OOD.
  • OOD detection: A score threshold classifies samples with higher S(x) as ID and lower scores as OOD.The threshold is denoted γ.
  • Auxiliary outliers: Auxiliary outlier data are unlabeled training examples drawn from Paux and used to regularize the model toward lower confidence on outliers.These training-time auxiliary outliers are distinguished from unknown OOD inputs encountered at test time.
  • Auxiliary outliers: Previous auxiliary-outlier methods improve OOD detection but can use outliers inefficiently because the potential OOD sample space is prohibitively large.This motivates sample-efficient utilization of auxiliary outlier data.

3. Method

POEM formalizes outlier mining as sequential decision making and uses posterior sampling to select informative near-boundary outliers. Its tractable Bayesian approximation interleaves posterior and feature updates while training a shared classifier.

  • 3.1. Outlier Mining: A Thompson Sampling View: POEM formalizes outlier selection as sequential decision making, using boundary closeness as the reward for identifying informative outliers.
  • 3.1. Outlier Mining: A Thompson Sampling View: Thompson sampling balances exploration and exploitation by sampling models from a maintained posterior before selecting auxiliary outliers.
  • 3.1. Outlier Mining: A Thompson Sampling View: The boundary score G(x) = −|foutlier(x; w∗)| is higher for outliers whose logits are closer to zero and therefore nearer the ID–OOD boundary.
  • 3.2. Approximate Posterior with Neural Networks: POEM approximates the intractable posterior with Bayesian linear regression on neural-network features and periodically updates it during training.
  • 3.3. Training Objective: The two branches share features: posterior-based mining selects outliers, while classification and energy regularization train the network on ID and selected outlier data.
  • 3.2. Approximate Posterior with Neural Networks: A fixed-size feature queue reduces posterior-update computation, with queue size M much smaller than the auxiliary dataset size.

4. Experiments

Experiments evaluate POEM on CIFAR-10 and CIFAR-100 using ImageNet-RC outliers and six diverse OOD test datasets. POEM achieves strong OOD detection, improves over greedy and energy-regularized baselines, and maintains comparable ID classification accuracy.

  • 4.1. Experimental Setup: Experiments compare POEM with pre-trained, randomly sampled-outlier, and greedily sampled-outlier OOD detection methods.
  • 4.1. Experimental Setup: CIFAR-10 and CIFAR-100 serve as ID datasets, ImageNet-RC supplies auxiliary outliers, and six image datasets provide OOD tests.
  • 4.1. Experimental Setup: 400,000 ImageNet-RC samples form the outlier pool, while 50,000 are selected for training using boundary scores.
  • 4.2. Results and Discussion: 4.1% lower FPR95 on CIFAR-100, a relative 21.4% error reduction, distinguishes POEM from the best energy-regularized baseline.
  • 4.2. Results and Discussion: 4.82% (FPR95) separates POEM from NTOM on CIFAR-100 under matched configurations, while POEM avoids NTOM’s confidence-hyperparameter sensitivity.
  • 4.2. Results and Discussion: POEM reaches lower FPR95 with fewer training epochs on both CIFAR-10 and CIFAR-100, indicating more effective use of auxiliary outliers.
  • 4.2. Results and Discussion: 26.59% test error on CIFAR-100 is comparable to NTOM’s 26.14% and the pre-trained model’s 25.95%, alongside improved OOD detection.

5. Further Discussion on Computation

POEM trades additional Bayesian computation for stronger and more efficient OOD detection, with early stopping offering a practical runtime reduction. Across the reported comparisons, selectively mined outliers outperform simply increasing the number of randomly sampled outliers.

  • Computation-performance trade-off: POEM achieves strong detection performance with moderate computational cost, despite Bayesian uncertainty tracking adding computation.The training time of POEM is shorter than the 14-hour SOFL baseline, while maintaining state-of-the-art performance.
  • Computation-performance trade-off: 19.25% to 15.14%: POEM reduces average FPR95 on CIFAR-100 while using comparable computations to NTOM.NTOM reports 19.96% under comparable computations.
  • Early stopping: 2.4 hours: early stopping reduces average training time, with AUROC 97.33% on CIFAR-100 after a marginal performance decrease.POEM reaches state-of-the-art performance around epoch 80, motivating the early-stopping strategy.
  • Random sampling comparison: 19.25% to 19.19%: tripling randomly sampled auxiliary data only marginally improves CIFAR-100 FPR95, while training time rises from 5h to 8.9h.AUROC changes from 96.68% to 97.18% in the same comparison.

6. Theoretical Insights: Sample Complexity with High Boundary Scores

The theoretical analysis studies high-boundary-score outlier selection in a Gaussian-mixture feature model. It shows that selecting such outliers improves sample efficiency by decreasing bounds on both false-negative and false-positive rates.

  • Theoretical setting: The analysis shows that selecting outliers with high boundary scores benefits sample complexity under a simple Gaussian mixture model for binary classification.The paper notes that more general OOD models are discussed in Appendix F.
  • Theoretical setting: The feature-space model assumes P_in = N(µ, σ^2I) and P_aux = N(−µ, σ^2I), with equal class priors.The hypothesis class consists of linear classifiers sign(θ^⊤x).
  • Error-rate analysis: The analysis defines false-negative and false-positive rates for the Gaussian-mixture data model and studies their bounds under high-boundary-score sampling.The displayed formulation expresses both rates through the classifier and the feature-distribution parameters.
  • Theoretical result: The lower bounds on FNR and FPR decrease when outliers with high boundary scores are selected, demonstrating a sample-efficiency benefit from outlier mining.The theorem assumes a large signal-to-noise ratio and auxiliary points satisfying an average high-boundary-score constraint.

7. Related Works

Related work spans OOD detection without auxiliary data, auxiliary-data training, posterior sampling, and applications beyond vision. POEM is positioned against greedy auxiliary-outlier mining by using posterior sampling to balance exploration and exploitation.

  • Posterior sampling: Posterior sampling, also called Thompson sampling, balances exploration and exploitation in bandit problems and has been applied across recommendation, marketing, and web optimization.Prior work also combines Thompson sampling with neural networks.
  • Posterior sampling: Gaussian-process and neural-network extensions study posterior sampling for optimizing unknown reward functions and solving bandit problems.Linear Bayesian models built on neural representations are presented as an extension of this line of work.
  • OOD detection without auxiliary data: OOD detection without auxiliary data includes OpenMax, maximum softmax probability, feature-space scores, KNN-based scores, and flow-based density estimation.The cited approaches include both parametric and non-parametric feature-space methods.
  • Auxiliary-data OOD detection: Auxiliary-data OOD methods use natural or synthesized outliers during training, while Chen et al. introduce greedy confidence-based outlier mining that falls short on exploration.These methods differ from approaches that detect OOD without auxiliary data.
  • OOD detection in NLP: NLP OOD detection addresses shifts in topics, domains, and unexpected utterances using methods including ensembling, data augmentation, and contrastive learning.The paper states that POEM does not depend on assumptions about the task domain and therefore has potential for NLP applications.

8. Conclusion and Outlook

The paper proposes POEM to identify near-boundary outliers through posterior sampling and learn a more compact ID–OOD decision boundary. Experiments report state-of-the-art performance, while theory explains why high-boundary-score selection improves OOD detection.

  • Conclusion: POEM uses posterior sampling to find near-boundary outlier training examples for regularization and compact ID–OOD decision-boundary learning.The framework is presented as a posterior sampling-based learning method for OOD detection.
  • Conclusion: POEM establishes state-of-the-art performance among competitive OOD detection methods in extensive experiments.The paper also provides theoretical insights into why selecting outliers with high boundary scores improves detection.

A. Discussion on the Choice of Auxiliary Dataset and Pool Size

POEM remains competitive across auxiliary-dataset and pool-size settings, while its outlier-mining strategy improves performance over random and greedy alternatives. The discussion also identifies practical boundaries for auxiliary outlier data and training duration.

  • Scope and training duration: When the auxiliary dataset is limited or very different from ID data, outlier-exposure approaches may not be the most suitable choice.The authors also report diminishing OOD-detection gains from training longer after the performance gap plateaus around epoch 85.
  • Choice of auxiliary outlier dataset: POEM achieves an average FPR95 of 1.56% on CIFAR-10 when TinyImages is used as the auxiliary outlier dataset.This result is averaged across six OOD test datasets and five independent training runs.
  • Effect of pool size: Halving the auxiliary pool size changes average AUROC from 99.40 to 99.29 and FPR95 from 2.54 to 4.23 on CIFAR-10.The authors report that POEM remains competitive with other baseline methods despite this change.
  • K + 1 training scheme: On CIFAR-100, POEM reduces average FPR95 by 1.97% relative to cross-entropy with K + 1 classes under the alternative training scheme.The ablation replaces confidence-based mining with POEM while using the same training objective as NTOM.
  • Outlier selection: POEM targets informative auxiliary samples near the estimated decision boundary rather than relying on fixed target values for outlier probabilities.Thompson sampling selects auxiliary samples with the highest estimated boundary score at each iteration.

D. Results on Individual Datasets

The supplied passages identify the per-dataset evaluation as being based on publicly available checkpoints, with results reported for POEM on each OOD dataset. The table caption specifies a DenseNet-101 basis but provides no individual values here.

  • Evaluation source: The per-dataset reference results are based on publicly available checkpoints.
  • Per-dataset results: Table 6 reports POEM’s OOD-detection results separately for each OOD dataset using a DenseNet-101 backbone.The supplied passages do not include the table’s individual metric values.

E. Theoretical Insights on Sample Complexity with High Boundary Scores

The theoretical analysis models ID and auxiliary data with a Gaussian mixture and studies how selecting outliers with high boundary scores affects sample complexity. Its results support a benefit from constraining selected samples near the decision boundary.

  • Definitions and model: The analysis assumes extracted features approximately follow an equal-prior Gaussian mixture model, with ID data from N(µ, σ^2I) and auxiliary data from N(−µ, σ^2I).The classifier classifies samples using sign(θ^⊤x), and the boundary score is defined as G(x) = −|foutlier(x)|.
  • High-boundary-score sampling: High boundary scores correspond to auxiliary points constrained near two hyperplanes around the decision boundary.For fixed n, the selected samples are viewed as draws from Paux subject to this constraint.
  • Sample complexity analysis: The sample-complexity analysis uses the boundary-score constraint to control the auxiliary sample distribution and derive concentration bounds.The analysis assumes a large signal-to-noise ratio and compares the constrained auxiliary component with the unconstrained case.
  • Interpretation: The theoretical interpretation states that decreasing ϵ improves the lower bound associated with high-boundary-score sampling.The authors identify this as showing the benefit of outlier mining with high boundary scores.
  • Interpretation: The boundary-score analysis links the selected-sample constraint to the benefit of outlier mining in the theoretical model.

F. Extension: Towards a More General Data Model for OOD Detection

The extension broadens the theoretical setting to mixture-generated ID and auxiliary data and a wider family of test OOD distributions. Under this model, the paper retains a qualitative boundary-score argument but no longer has a clean closed-form boundary score.

  • Generalized data model: The generalized model uses auxiliary and ID data generated from mixtures of Gaussians and allows test OOD distributions Qv beyond Paux.The test distribution is modeled as N(−µ + v, σ^2I) for bounded v.
  • Analytical limitation: The generalized setting does not provide a clean solution for G(x) because the class densities require integrating Gaussian densities over the mixture variables.This differs from the simpler model analyzed in Appendix E.
  • Qualitative extension: Even without a closed-form G(x), high boundary scores can generally bound selected auxiliary points near the decision hyperplane.The paper therefore expects a revised bound and similar sample-complexity behavior under the generalized model.
  • Implementation: The experiments were run on an NVIDIA GeForce RTX-2080Ti under Ubuntu Linux 20.04 with Python 3.8.
Loading 2206.13687v1…