PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges

Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, Mohammad Sabokrou

arXiv:2110.14051v5cs.CVcs.LG

TL;DR

Models can assign unseen samples misleading confidence, undermining classifier reliability in safety-critical open-world settings. This survey unifies research on unfamiliar-sample detection across multiple domains, reviews methods and evaluations, and identifies limitations and future directions. It also reports comprehensive baseline testing and aims to support a broader cross-domain perspective.

  • Problem

    Unseen samples can receive misleading in-class predictions, while related detection fields remain fragmented and lack a comprehensive, up-to-date OOD survey.

  • Method

    The paper comprehensively reviews techniques, datasets, evaluation criteria, limitations, and future challenges across anomaly, novelty, one-class, open-set, and OOD detection.

  • Results

    The survey identifies commonalities across detection domains, reports comprehensive tests of existing baselines, and points to promising research directions.

  • Takeaways & Limitations

    A unified cross-domain perspective can help researchers develop future methodologies synergistically and bring these research fields closer together.

  • Takeaways & Limitations

    Vision Transformer OOD evaluations can lose train-test integrity when pretraining datasets overlap with the evaluation data.

Abstract

from arXiv · show

Machine learning models often encounter samples that are diverged from the training distribution. Failure to recognize an out-of-distribution (OOD) sample, and consequently assign that sample to an in-class label significantly compromises the reliability of a model. The problem has gained significant attention due to its importance for safety deploying models in open-world settings. Detecting OOD samples is challenging due to the intractability of modeling all possible unknown distributions. To date, several research domains tackle the problem of detecting unfamiliar samples, including anomaly detection, novelty detection, one-class learning, open set recognition, and out-of-distribution detection. Despite having similar and shared concepts, out-of-distribution, open-set, and anomaly detection have been investigated independently. Accordingly, these research avenues have not cross-pollinated, creating research barriers. While some surveys intend to provide an overview of these approaches, they seem to only focus on a specific domain without examining the relationship between different domains. This survey aims to provide a cross-domain and comprehensive review of numerous eminent works in respective areas while identifying their commonalities. Researchers can benefit from the overview of research advances in different fields and develop future methodology synergistically. Furthermore, to the best of our knowledge, while there are surveys in anomaly detection or one-class learning, there is no comprehensive or up-to-date survey on out-of-distribution detection, which our survey covers extensively. Finally, having a unified cross-domain perspective, we discuss and shed light on future lines of research, intending to bring these fields closer together.

1 Introduction

Unfamiliar-sample detection is important for reliable, safety-critical machine learning, but related fields have largely been studied separately. This survey connects these domains through a comprehensive methodological review, experiments, and future research directions.

  • Closed-set models can assign misleading confidence to unseen inputs, raising reliability concerns for safety-critical applications.
  • Anomaly detection, novelty detection, one-class classification, open-set recognition, and OOD detection address related unknown-sample problems but differ in connections and testing protocols.
  • The survey identifies relationships among research areas that have been examined separately, enabling cross-domain methodological connections.
  • The paper provides comprehensive methodological analysis, theoretical and visual explanations, and tests of existing baselines.
  • Future research directions address fairness, adversarial robustness, privacy, data efficiency, and explainability.

2 A Unified Categorization

The unified categorization organizes unfamiliar-sample detection by supervision and task formulation across anomaly detection, open-set recognition, and OOD detection. It also relates generative and discriminative modeling approaches while documenting important assumptions and limitations.

  • The hierarchy unifies approaches across fields and shows that unfamiliar-sample detection can use different supervision levels.
  • Discriminative modeling benefits from labeled training data, whereas unlabeled one-class settings make conditional-probability modeling challenging.
  • Autoencoders assume precise reconstruction of normal samples and poorer reconstruction of abnormal samples, but these assumptions are not always true.
  • GAN-based methods can compare generated inputs with test samples, but they suffer from mode collapse, unstable training, and irreproducible results.
  • Auto-regressive and flow-based likelihood methods can assign higher likelihood to abnormal samples, weakening detection performance across several unfamiliar-sample tasks.

3 Anomaly and Novelty Detection

Anomaly and novelty detection address samples that deviate from learned normal patterns, but differ in training-data assumptions and detection settings. The survey reviews traditional density, isolation, boundary, reconstruction, adversarial, and contrastive approaches across these domains.

  • Concepts and assumptions: Anomaly detection may include abnormal training samples, whereas novelty detection assumes training data is entirely normal.Anomaly methods exploit the rarity of abnormal events and may filter samples during training.
  • Traditional methods: Isolation Forest identifies anomalies through shorter tree branches because unusual points require fewer random partitions to isolate.The figure contrasts four partitions for anomaly xo with twelve for normal point xi.
  • Traditional methods: DBSCAN detects outliers as points in low-density regions after clustering core points and assigning reachable non-core points.Core points require at least the minimum neighborhood count; non-core points not assigned to a cluster become noise.
  • Traditional methods: OC-SVM and SVDD learn boundaries around normal data, with SVDD using a compact hypersphere to capture richer shared normal features.The survey notes that half-space boundaries can be insufficiently compact and produce false negatives.
  • Deep methods: Autoencoder methods use reconstruction-based scores, but blurry reconstructions can inflate errors for normal samples and motivate adversarial or distributional alternatives.ALOCC uses a discriminator for higher-quality outputs and semantic similarity, while AMA minimizes Wasserstein distance between joint distributions.

4 Open-set Recognition

Open-set recognition studies classification with known classes while detecting unknown classes at test time, connecting this setting to novelty detection and OOD detection. Methods span EVT-based rejection, generated unknown samples, counter-factual examples, reconstruction, probabilistic modeling, and architectural analysis.

  • OSR trains on K of N classes, then classifies the K known classes while detecting the N−K unseen classes at test time.
  • OpenMax: OpenMax models extreme activation values with EVT and rejects inputs when the unknown-class probability dominates or top-k confidence is insufficient.
  • G-OpenMax: G-OpenMax augments OpenMax by generating unknown-unknown samples with a conditional GAN, training a classifier to assign them to class 0, and removing the validation-dataset requirement.
  • Counter-factual examples: Counter-factual generation produces samples near unknown-class boundaries by keeping them close to training examples while lowering confidence for known classes.
  • Representation learning: CROSR separates reconstruction and classification representations because information loss in the penultimate layer can hinder known–unknown discrimination.
  • Evaluation and architecture: Closed-set accuracy is highly correlated with OSR performance across loss objectives and architectures, while the Semantic Shift Benchmark distinguishes Easy and Hard semantic-shift splits.

5 Out-of-distribution Detection

OOD detection identifies test samples that are semantically different from training categories and should not receive known-class predictions. The surveyed methods include softmax-based baselines, post-hoc score modification, feature-space generative modeling, uncertainty decomposition, and synthetic OOD generation.

  • OOD detection identifies semantically different test samples while classifying in-distribution samples into known categories; in multi-class settings, it is canonical to OSR.
  • Softmax-based detection: Maximum softmax probability provides a general-purpose OOD baseline: large maxk p(y = k | x) indicates an in-distribution sample.
  • Post-hoc methods: ODIN combines temperature scaling with input perturbation to amplify ID/OOD separability, although its score is designed for separation rather than calibrated predictive confidence.
  • Feature-space methods: Mahalanobis-based detection fits class-conditional Gaussians to penultimate-layer features, ensembles multiple layers, and adds controlled input noise for OOD detection.
  • Uncertainty modeling: Uncertainty methods distinguish data, distributional, and model uncertainty, representing class predictions as categorical distributions and approximating intractable parameter integrals.
  • Uncertainty modeling: Dirichlet Prior Networks encourage flat simplex distributions for OOD samples and sharp sparse distributions for in-class samples using KL-divergence objectives.
  • Synthetic OOD generation: GAN-based approaches generate OOD samples and train classifiers to make their confidence distributions uniform, with test-time detection based on maximum-softmax thresholding.

5.6 Deep Anomaly Detection with Outlier Exposure (OE) (58):

The reviewed methods improve unfamiliar-sample detection through auxiliary outlier data, self-supervised objectives, classifier discrepancy, and generative-model analysis. They also expose confidence and likelihood pathologies that motivate alternative training objectives and scores.

  • Outlier Exposure: Outlier Exposure trains classifiers to produce uniform softmax outputs on outliers, improving separation between in- and out-of-distribution examples.Outlier data may come from existing datasets unrelated to the task, whose diverse variations can improve performance.
  • Self-Supervised Learning: Auxiliary self-supervised tasks improve robustness to corruptions and improve maximum softmax probability detection, despite not improving classification accuracy.The evaluated corruptions include Gaussian noise, shot noise, blurring, zooming, and fogging.
  • Classifier Discrepancy: Maximum classifier discrepancy trains two classifiers to agree on in-class samples while maximizing confidence discrepancy on unlabeled outliers.The discrepancy is defined using the difference between the classifiers’ final-layer entropies.
  • Confidence Pathology: ReLU networks can produce infinitely many high-confidence predictions away from training data because their outputs are piece-wise affine on input polytopes.Confidence-enhancing data augmentation and adversarial confidence-enhancing training are introduced to address this behavior.
  • Generative Models: Generative models may assign higher likelihoods to outliers, while their likelihood objective can encourage high sensitivity to small input changes.Constant-volume transformations can alleviate, but not entirely solve, the likelihood problem; a curvature and data-second-moment relationship is also reported.

5.11 Likelihood Ratios for Out-of-Distribution Detection (132):

Likelihood-ratio and multi-head approaches separate in-distribution, background, and domain information to improve OOD detection. The section also describes probabilistic and re-sampling interpretations of outlier exposure and related detection architectures.

  • Likelihood Ratios: Likelihood-ratio detection models foreground and background information separately using autoregressive models trained on original and noisy inputs.The likelihood ratio is thresholded at test time.
  • Generalized ODIN: Generalized ODIN decomposes class prediction and inlier-domain probability into separate heads of a shared feature extractor.The explicit domain variable distinguishes inlier from outlier inputs and addresses overconfident conditional class scores.
  • Background Resampling: Background-data resampling selects an influential subset of auxiliary outliers, emphasizing on-boundary samples during optimization.The method provides a probabilistic interpretation of outlier exposure and alternates optimization over classifier parameters and sample weights.

5.14 Input Complexity and Out-of-Distribution Detection With Likelihood-Based Generative Models (152):

Likelihood-based generative OOD detection is affected by input complexity, with simpler samples receiving higher likelihoods. The reviewed approaches correct or reinterpret these scores using compression, energy, and likelihood-regret methods.

  • Input Complexity: Simpler inputs can receive higher likelihood values from generative models, linking OOD likelihoods to input complexity.Complexity is measured using normalized compressed size in bits per dimension.
  • Input Complexity: A universal-compressor likelihood correction adjusts learned-model likelihoods to account for input complexity when defining an OOD score.The correction is intended to reduce inflated likelihoods for simple OOD samples.
  • Energy-Based Detection: Energy-based OOD detection maps logits to a scalar energy and detects OOD samples using a threshold on −E(x; f).The reviewed work reports stronger performance than OE and describes energy as hyperparameter-free and easy to compute.
  • Energy-Based Detection: Softmax scoring is biased because training pushes energy down for in-distribution samples while shifting it by the maximum logit.Energy-based regularization instead directly optimizes an ID–OOD energy gap.
  • VAE Detection: VAE OOD detection is difficult because VAEs can reconstruct OOD samples perfectly and exhibit a narrower range of average test likelihoods than PixelCNN or Glow.Likelihood regret optimizes latent statistics and thresholds the resulting regret values at test time.

5.17 Understanding Anomaly Detection With Deep Invertible Networks Through Hierarchies of Distributions and Features (146):

Flow-based generative models can let local smoothness dominate likelihood, causing smoother datasets to score higher independently of the training distribution. The section reviews local-feature analysis, likelihood ratios, auxiliary outliers, self-supervised features, and groupwise OOD scoring.

  • Flow-Based Models: Local features such as smooth patches can dominate flow-model likelihood, making smoother datasets like SVHN score higher than CIFAR-10.Pseudo-likelihoods from local 3 × 3 pixel differences strongly correlate with exact likelihoods.
  • Likelihood Ratios: Likelihood-ratio detection trains one generative network on a general image distribution and another on the in-distribution, then compares their likelihoods.An outlier loss using OOD samples can be added to the maximum-likelihood objective.
  • Self-Supervised Detection: Self-supervised contrastive learning uses unlabeled data to learn semantically meaningful features for flexible novelty, open-set, and OOD detection.The framework can incorporate a few OOD samples and extend to supervised contrastive learning when in-class labels are available.
  • MOS: MOS addresses degradation with many in-distribution classes by grouping related categories and adding an “others” category to each group.Its Minimum Others Score detects OOD inputs using the minimum others-class score across groups.

5.21 Can Multi-Label Classification Networks Know What They Don’t Know? (171):

This section presents OOD detection methods for classification networks, including JointEnergy for multi-label settings, gradient-based scoring, activation rectification, virtual outlier synthesis, and object-level uncertainty modeling.

  • JointEnergy: JointEnergy sums label-wise scores to increase separation between in-distribution and OOD inputs in multi-label networks.The method uses the summed score as an OOD indicator rather than only the maximum-valued label score.
  • JointEnergy: JointEnergy combines joint likelihood across labels with underlying data density, producing stronger ID–OOD separability.
  • Gradient-based detection: GradNorm uses the norm of gradients backpropagated from KL divergence with the uniform distribution to distinguish ID from OOD data.GradNorm is generally higher for ID data, and using last-layer parameters reportedly performs best among the tested parameter choices.
  • Activation-based detection: ReAct improves ID–OOD separation by rectifying penultimate-layer activations before computing OOD scores.
  • Virtual outlier synthesis: VOS synthesizes virtual outliers in feature space and uses them to regularize the decision boundary for OOD detection.It models class-conditional feature distributions and samples low-likelihood features, avoiding image-space synthesis in high-dimensional pixel space.
  • Object-level detection: Object-level OOD detection applies energy and uncertainty modeling to predicted objects, enabling detection when scenes contain both ID and OOD categories.Testing uses the logistic-regression uncertainty branch and a threshold γ selected to retain a high fraction of correctly classified ID data.

5.25 Out-of-distribution Detection with Deep Nearest Neighbors (164):

This section describes a deep nearest-neighbor method that detects OOD samples from distances in normalized penultimate-layer embedding space without requiring unknown data.

  • Method: The method uses non-parametric nearest-neighbor distances instead of parametric distributional assumptions about the feature space.
  • Embedding-space detection: Test inputs are mapped to normalized penultimate features and compared by Euclidean distance with the training embedding set.Training embeddings are reordered by increasing distance from the test embedding.
  • Decision rule: OOD decisions use distances to the nearest neighbors and an indicator-based decision function.The distance to the k-th nearest neighbor is denoted by r_k(z∗).
  • Properties: The KNN detector is OOD-agnostic and model-agnostic because its threshold is estimated from ID data and it applies across losses and architectures.The passage also reports millisecond-scale execution with modern implementations such as Faiss, even for large databases.

6 A Summary of the Shared Core Ideas

The survey identifies shared strategies across anomaly, novelty, open-set, and OOD detection, including generative modeling, pretrained knowledge, self-supervision, compact representations, gradients, outliers, and prototypes.

  • Compressed normal representations: Compact feature representations force normal samples together, helping separate abnormal inputs from shared normal features.Examples include compact hyperspheres in DSVDD, DSAD, and GOAD, with related labeled variants in open-set recognition.
  • Normal gradients: Gradient-based methods exploit the observation that normal samples can alter gradients more substantially than abnormal samples.In OOD detection, gradient norms from KL divergence to a uniform distribution are reported as larger for normal samples.
  • Outliers: Outlier exposure and synthetic-outlier methods use abnormal or generated samples to improve normal boundaries and detection performance.
  • Prototypes: Prototype-based methods learn a small set of prototypes covering the normal feature space and score inputs by their relation to those prototypes.
  • Generative models: Generative methods distinguish normal from abnormal inputs using reconstruction error or discriminator outputs because abnormal inputs are assumed harder to generate or reconstruct.
  • Pretrained knowledge: Pretrained models provide transferable feature knowledge that can support downstream anomaly and open-set recognition tasks.
  • Self-supervised learning: Self-supervised learning is used across all four detection areas because better representations are associated with improved open-set recognition performance.

7 Dataset

The survey organizes datasets for semantic anomalies, pixel-level anomalies, and robustness evaluation, spanning toy, hard, fine-grained, industrial, medical, and corruption benchmarks.

  • Semantic anomalies: Semantic-anomaly datasets range from toy datasets such as MNIST and SVHN to harder datasets such as CIFAR-10, CIFAR-100, LSUN, and TinyImageNet.Flowers and Birds provide fine-grained semantic settings with substantial within-class variation or few training images per species.
  • Semantic datasets: The listed toy and image datasets vary in scale, resolution, category count, scene content, and pose or illumination variation.Examples include MNIST’s 60k training images, CIFAR-10’s 50k training images, and COIL-100’s 7,200 images across 100 objects.
  • Pixel-level anomalies: Pixel-level anomaly datasets preserve the original semantic meaning while introducing defects or abnormal regions.
  • Industrial and medical datasets: Industrial and medical benchmarks include MVTec AD, PCB, LaceAD, Retinal-OCT, CAMELYON16, and Chest X-Ray datasets.Their challenges include manufacturing defects, subtle tissue differences, metastasis detection, multiple disease labels, and variable imaging conditions.
  • OOD and robustness: OOD and robustness datasets include Species, ImageNet-O, MNIST-C, ImageNet-C, and ImageNet-P.These datasets test semantic novelty, adversarially filtered examples, common corruptions, or subtle perturbations while often preserving semantic content.
  • Dataset overview: Figure 43 provides an overall visualization of the datasets discussed in this section.

8 Evaluation Protocols

Evaluation protocols differ across anomaly detection, OOD detection, and open-set recognition in their class construction, sample proportions, and treatment of unknowns. Metrics such as AUC-ROC, FPR@TPRx, normalized accuracy, and modified F-measures capture different aspects of detection and classification performance.

  • Metrics: AUC-ROC measures the threshold-independent relation between false positive rate and true positive rate, with 1 as the highest value and 0.5 as random guessing.
  • Evaluation protocols: Anomaly and novelty detection use one-vs-all evaluation, OOD detection contrasts an in-distribution dataset with another dataset, and OSR tests K known classes against unseen classes.
  • Metrics: FPR@TPRx reports the false positive rate at a selected true positive rate, commonly x = 0.95, to represent an application-specific operating point.
  • Evaluation protocols: Open-world evaluation extends closed-set assessment by requiring correct classification of unknown-unknown classes, including their correctly predicted count in the metric.
  • Metrics: Normalized accuracy weights accuracy for known-known classes and unknown-unknown classes to address accuracy's sensitivity to imbalanced sample counts.
  • Metrics: Macro-F-measure and micro-F-measure modify precision and recall calculations for known-known classes because correctly classified unknowns are not represented during training.

9 Core Challenges

The survey identifies challenges involving augmentation alignment, data construction, limited supervision, and generalization across detection settings. It highlights the need for task-specific self-supervision, richer data, and methods that operate under few-shot and zero-shot conditions.

  • Self-supervision: Self-supervised augmentations are not always beneficial for novelty detection, OSR, or OOD detection, and their success depends on alignment with real abnormalities.Misaligned self-supervision can worsen detection performance.
  • Self-supervision: Carefully selected augmentations can let the simple MSP baseline on Vision Transformers outperform many recent OSR methods.
  • Self-supervision: The survey calls for self-supervised tasks specifically designed for outlier detection rather than relying on generic augmentation objectives.
  • Data construction: Generating unseen abnormal or synthetic outlier samples can enrich training data and convert one-class learning into a two-class classification problem.SDE-based generation has been used to create synthetic training outliers for OE-based novelty detection.
  • Small-sample learning: Few-shot detection remains underexplored for generalizing from a few known classes to unknown classes and for achieving convergence in one-class meta-learning.
  • Small-sample learning: Pre-trained vision-language and vision-transformer models have been explored for zero-shot and few-shot anomaly and OOD detection.CLIP with textual queries can define the normal class without visual inputs.

10 Future Challenges

Future work must test unfamiliar-sample detection under realistic distribution shifts, difficult data conditions, and operational constraints while addressing fairness, robustness, multimodality, and explainability.

  • OOD Evaluation: OOD methods often fail to detect both near and far OOD samples; auxiliary-data Outlier Exposure is the notable exception.Mahalanobis distance can improve far-OOD detection while degrading near-OOD performance and remains sensitive to low-level statistics.
  • Evaluation Scope: Semantic OSR benchmarks can overstate performance because subtle class variations, such as emerging diseases in chest X-rays, are substantially harder.Existing discriminative and generative methods model widely separated semantic classes more effectively than closely related classes.
  • Fairness and Bias: Spurious correlations can severely worsen OOD detection and produce high-confidence predictions for semantically different inputs sharing the same background.Fairness-aware anomaly detection uses protected-variable labels as additional supervision, while biased auxiliary anomalies can favor particular anomaly types.
  • Explainability: Explainability remains underexplored for one-class and novelty-detection models because only one training label is available.Receptive field upsampling offers one proposed approach, but explainable OCC models still require further investigation.
  • Reliability in Practice: Realistic distribution shifts cause significant performance drops for state-of-the-art OOD detectors, exposing weaknesses in existing testing frameworks.New benchmarks with semantically preserved shifts and high-resolution images over thousands of classes are proposed to redirect evaluation toward real-world applications.
  • Open-World Recognition: OOD systems must continuously discover novel categories and add them to the recognition function while minimizing operational downtime.This extends detection beyond controlled laboratory settings into open-world recognition and system updating.

11 Conclusion

OOD detection, OSR, novelty detection, and anomaly detection address overlapping unknown-sample scenarios but have largely developed independently. This survey unifies their methods, datasets, evaluation criteria, limitations, and future challenges to support cross-domain research.

  • Motivation: OOD detection, OSR, novelty detection, and anomaly detection share an intention and substantial overlap but have been pursued independently.The survey frames these settings as ubiquitous when testing includes classes that cannot feasibly be modeled in advance.
  • Contribution: The paper comprehensively reviews techniques, datasets, evaluation criteria, and future challenges across unfamiliar-sample detection domains.It also discusses limitations and identifies promising research directions.
  • Impact: The survey aims to help researchers develop a broader cross-domain perspective.Its stated goal is to bring related research fields closer together.

A Appendix

The appendix indicates that additional sections may be included.

  • Additional sections may be included in the document.
  • The passage provides no further substantive content for this appendix.
  • The appendix is reserved for optional material beyond the main sections.
Loading 2110.14051v5…