A Unifying Review of Deep and Shallow Anomaly Detection

Lukas Ruff; Jacob R. Kauffmann; Robert A. Vandermeulen; Grégoire Montavon; Wojciech Samek; Marius Kloft; Thomas G. Dietterich; Klaus-Robert Müller

Source-linked AI summary

A Unifying Review of Deep and Shallow Anomaly Detection

Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, Klaus-Robert Müller

arXiv:2009.11732v3Sep 24, 2020cs.LGcs.AIstat.ML

TL;DR

Anomaly detection lacks a systematic perspective connecting its diverse shallow and deep methods despite broad practical importance. This review unifies these approaches, empirically compares them with explainability, and finds heterogeneous performance in which simple models can sometimes outperform deep ones.

Problem
Anomaly detection needs a systematic perspective because distinguishing normal from anomalous events is difficult amid substantial normal-data variability.
Method
The review organizes probabilistic, one-class, reconstruction, shallow, and deep approaches within a unifying view and assesses them empirically using explainability techniques.
Results
Performance is heterogeneous across anomaly types: simple Gaussian, PCA, and KDE models sometimes outperform deeper counterparts on specific corruption or defect classes.
Takeaways & Limitations
Connecting classical kernel methods with deep learning and representation learning provides a shared perspective for understanding anomaly detection methods.
Takeaways & Limitations
Benchmark accuracy may not reflect generalization to broader anomaly types because models can exploit non-representative test-set features and exhibit Clever Hans effects.

Abstract

from arXiv · show

Deep learning approaches to anomaly detection have recently improved the state of the art in detection performance on complex datasets such as large collections of images or text. These results have sparked a renewed interest in the anomaly detection problem and led to the introduction of a great variety of new methods. With the emergence of numerous such methods, including approaches based on generative models, one-class classification, and reconstruction, there is a growing need to bring methods of this field into a systematic and unified perspective. In this review we aim to identify the common underlying principles as well as the assumptions that are often made implicitly by various methods. In particular, we draw connections between classic 'shallow' and novel deep approaches and show how this relation might cross-fertilize or extend both directions. We further provide an empirical assessment of major existing methods that is enriched by the use of recent explainability techniques, and present specific worked-through examples together with practical advice. Finally, we outline critical open challenges and identify specific paths for future research in anomaly detection.

I. INTRODUCTION … 1) What is an Anomaly?:

Anomaly detection studies observations that deviate considerably from normality, a problem with broad practical relevance and longstanding cross-disciplinary foundations. This review unifies shallow and deep approaches while formalizing anomalies probabilistically and organizing their main types.

1) What is an Anomaly?:: An anomaly is an observation that deviates considerably from some concept of normality, and anomaly detection develops methods to identify such observations.Anomalies are also called outliers or novelties and may be unusual, irregular, unexpected, rare, erroneous, fraudulent, or malicious.
A. Why Should We Care About Anomaly Detection?: Anomaly detection is broadly relevant across cybersecurity, finance, healthcare, industry, infrastructure monitoring, markets, scientific sensing, and disease detection.Applications include intrusion and fraud detection, industrial fault detection, medical diagnosis, acoustic novelty detection, and disease outbreak detection.
I. INTRODUCTION: The review connects traditional shallow and novel deep approaches, categorizing methods into one-class classification, probabilistic, reconstruction, and distance-based models.Its aims include summarizing developments, providing theoretical insights, and highlighting best practices; deep methods include autoencoders, one-class classification, generative models, and self-supervised methods.
A. Why Should We Care About Anomaly Detection?: Anomaly detection can support fail-safe deep-learning systems and reveal unknown patterns that generate scientific insights and hypotheses.The paper highlights medical and autonomous-driving applications and the accidental discovery of unknown unknowns in science.
A. Why Should We Care About Anomaly Detection?: Large normal-data variability, especially in biomedical data, can cause both type I errors and type II errors when distinguishing normal from anomalous events.Normal variation may be as large as or larger than the distance to anomalies, making reliable discrimination difficult.
A. Why Should We Care About Anomaly Detection?: Rare anomalies, unlabeled datasets, noise, irrelevant features, and masking effects make anomaly detection predominantly an unsupervised task.The objective is typically to learn a valid model of the majority of data points because labeled anomalous data is often unavailable or insufficient.
B. A Formal Definition of Anomaly Detection: The review introduces anomaly detection probabilistically, defines anomaly types, distinguishes anomalies from outliers and novelties, and presents the concentration assumption and density level set formulation.This establishes a formal foundation for the subsequent treatment of anomaly detection.
1) What is an Anomaly?:: Formally, normality is the ground-truth distribution P+ of normal behavior, while anomalies are points x in low-probability regions under P+.The anomaly set is defined using a threshold τ chosen so that its probability under P+ is sufficiently small.

2) Types of Anomalies: · 3) Anomaly, Outlier, or Novelty?: · 4) The Concentration Assumption:

The paper distinguishes contextual, collective, and hierarchical semantic anomaly types, while noting that anomaly, outlier, and novelty labels can reflect different application objectives. It also assumes that normal data concentrate in a bounded high-density region, even though the full normal support and anomaly set may be unbounded, and usually requires estimating normality from data.

2) Types of Anomalies:: Contextual anomalies depend on a variable such as time, space, spatio-temporal context, or graph structure.Time-series anomalies are the most prominent example.
2) Types of Anomalies:: Collective anomalies are related or dependent sets of points that are anomalous as a group, including clusters of similar network attacks.They may also be contextual, such as anomalous time series or biological subsequences.
2) Types of Anomalies:: A collective series or sequence can be anomalous under its joint conditional density even when each individual point appears normal under a marginal distribution.The joint density properly describes the distribution of the collective series or sequence.
2) Types of Anomalies:: Deep learning has emphasized a distinction between low-level sensory anomalies and high-level semantic anomalies across feature hierarchies.Examples range from edges and textures to objects and scenes, or from characters and words to concepts and topics.
3) Anomaly, Outlier, or Novelty?:: Anomalies, outliers, and novelties all occupy low-probability regions under P+, but the terms can correspond to distinct application objectives.Anomalies may be points of interest, outliers may be treated as noise or measurement error, and novelties may require updating models to a new normal.
4) The Concentration Assumption:: The concentration assumption states that normal data occupy a non-empty, small-volume bounded region within an otherwise potentially unbounded data space.This is also called the concentration or cluster assumption.
4) The Concentration Assumption:: The full support of the normal distribution need not be bounded, whereas a high-density subset may be concentrated and anomalies may remain unbounded.Approximately 95% of a standard univariate Gaussian’s mass lies in [−1.96, 1.96] despite its full-real-axis support.

5) Density Level Set Estimation: … C. Dataset Settings and Data Properties

The paper frames anomaly detection as estimating low-density regions under the normal distribution, using density level sets and related threshold or score models. It also emphasizes that practical performance depends on modeling choices, dataset settings, data properties, and assumptions about how normal data and anomalies are distributed.

5) Density Level Set Estimation:: Anomaly detection aims to determine whether a new instance belongs to an anomalous region by estimating low-density regions under the normal law P+.The objective may be implemented explicitly or implicitly through high-density regions and their complements.
5) Density Level Set Estimation:: For a fixed α, the α-density level set Cα is the smallest density level set having probability at least 1 − α under P.The corresponding threshold is τα, with λ typically denoting Lebesgue measure.
5) Density Level Set Estimation:: The extreme cases α = 0 and α → 1 yield the full support of p and the most likely modes of P, respectively.When the concentration assumption holds, some bounded level set Cα exists; Fig. 3 illustrates such sets for a standard Gaussian distribution.
6) Density Estimation for Level Set Estimation:: Density estimation constructs a plug-in level-set estimator by estimating p(x), deriving an empirical threshold ˆτα, and applying both to obtain a detector.Estimating the full density is generally the most costly route in samples, motivating methods that estimate level sets directly.
6) Density Estimation for Level Set Estimation:: One-class classification methods provide an alternative by estimating a collection or a single level set directly rather than estimating the full density.This approach is introduced as a response to the sample cost of density estimation.
7) Threshold vs. Score:: Density estimation produces a more informative model that can rank observations by estimated density, whereas a pure threshold detector yields only binary predictions.Menon and Williamson propose a compromise that learns a density outside the level-set boundary.
7) Threshold vs. Score:: Choosing α is an application-dependent modeling decision, and increasing α makes the detector focus only on the most likely regions of P.The passage notes that such a detector may be desirable when missed anomalies are costly, including medical diagnosis or fraud detection.
C. Dataset Settings and Data Properties: Real-world anomaly detection problems vary in dataset setting and data properties, including unsupervised, semi-supervised, and supervised settings and differences in data type or dimensionality.The review characterizes these settings and properties as relevant to modeling anomaly detection problems.

1) A Distribution of Anomalies?: … 4) The Supervised Setting:

The paper organizes anomaly detection by assumptions about anomaly distributions and available labels, emphasizing that noise and contamination shape the unsupervised problem. It distinguishes semi-supervised settings with scarce or uncertain labels from supervised learning, which may reduce to classification when both classes are representative.

1) A Distribution of Anomalies?:: Uniform anomaly distributions are an implicit assumption of methods such as KDE and OC-SVM, treating anomalies as equally likely across bounded data space X.This corresponds to an uninformative prior with no anomalous modes.
2) The Unsupervised Setting:: Unsupervised anomaly detection trains models using only unlabeled data, typically assuming i.i.d. samples and, in the clean case, P ≡ P+.The text identifies this as arguably the most common anomaly detection setting.
2) The Unsupervised Setting:: Noise obscures informative normal features and makes the ground-truth level sets of P+ harder to estimate.Samples are modeled as x + ε, where x ∼ P+; increasing noise expands the regions covered by the data.
2) The Unsupervised Setting:: Undetected anomalies contaminate unlabeled training data through the mixture P ≡ (1−η)P+ + ηP-, with η ∈ (0, 1).The paper presents noise and contamination jointly as disturbances requiring assumptions about ε and η, motivating robust methods.
3) The Semi-Supervised Setting:: Semi-supervised detection combines many unlabeled examples with few labeled normal or anomalous instances, commonly because labels are costly to obtain.The setting usually has m ≪ n, with labels encoded as +1 for normal and −1 for anomalous points.
3) The Semi-Supervised Setting:: Learning exclusively from confidently labeled normal examples is rare, and when most remaining data is assumed normal, it is effectively equivalent to unsupervised modeling.This special case may involve weakened assumptions about noise or contamination rather than absolute certainty that all training data is normal.
4) The Supervised Setting:: With completely labeled and representative normal and anomalous data, supervised anomaly detection becomes supervised binary classification rather than anomaly detection in the strict sense.The distinction persists because labeled anomalies are rarely fully representative of an anomaly class, whereas anomaly detection estimates a bound on the normal density level set.

5) Further Data Properties: … C. Neural Generative Models (VAEs and GANs)

The paper argues that anomaly-detection models should reflect intrinsic data properties and application-specific assumptions. It then surveys probabilistic, one-class, reconstruction, and neural generative approaches while emphasizing representation, optimization, interpretability, and transparency challenges.

5) Further Data Properties:: Intrinsic data properties should guide anomaly-detection modeling choices, including context incorporation and the selection of deep or shallow feature maps.The paper presents relevant data properties in Table I and a toy dataset in Fig. 4.
D. Challenges in Anomaly Detection: Anomaly detection is fundamentally challenging because its mostly unsupervised nature requires assumptions about the application, domain, data, anomaly types, and anomaly distribution.Labeled instances, when available, must also be incorporated in a way that generalizes.
D. Challenges in Anomaly Detection: Deep anomaly detection additionally requires choosing suitable representations, network architectures, learning rates, batch sizes, and other optimization parameters.Increasing model or data complexity also makes interpretability and decision transparency more difficult.
D. Challenges in Anomaly Detection: The review organizes anomaly-detection methods into density estimation and probabilistic models, one-class classification, and reconstruction models.It also examines connections between deep and shallow approaches across these sections.
III. DENSITY ESTIMATION AND PROBABILISTIC MODELS: Probabilistic anomaly-detection methods estimate the normal data probability distribution using classic density estimators or deep statistical models.Anomalies are predicted through the estimated normal-data probability distribution.
A. Classic Density Estimation: Mahalanobis distance is equivalent to fitting a multivariate Gaussian and evaluating a test point’s log-likelihood, while capturing linear interactions between feature pairs.This contrasts with modeling each data dimension independently.
A. Classic Density Estimation: Classic nonparametric density estimators work fairly well in low dimensions but suffer from the curse of dimensionality, motivating deep statistical models.The required sample size for fixed accuracy grows exponentially with feature-space dimension.
C. Neural Generative Models (VAEs and GANs): Neural generative models learn a network that maps samples from a simple source distribution Q, usually Gaussian or uniform, toward the actual input distribution P+, including VAEs and GANs.Formally, training seeks φω(Q) ≈ P+, where φω(Q) is the distribution obtained by pushing Q through the network φω.

1) VAEs: … IV. ONE-CLASS CLASSIFICATION

The review connects deep generative and discriminative approaches for anomaly detection through likelihood estimation, reconstruction, and decision-boundary learning. It highlights practical trade-offs involving approximation, invertibility, dimensionality, and the scope of the learned density representation.

1) VAEs:: VAEs estimate normal-data likelihood through latent-variable models optimized with the ELBO and stochastic gradient variational Bayes.The decoder models pθ(x | z), while the encoder approximates the latent posterior qθ′(z | x).
1) VAEs:: VAE reconstruction probability often performs better for anomaly detection than likelihood estimated directly from prior sampling.Direct likelihood uses Monte Carlo sampling from p(z), whereas reconstruction probability conditions on x to estimate Eqθ′(z|x) [log pθ (x|z)].
2) GANs:: GANs learn the target distribution through adversarial competition between a generator and discriminator, but their alternating optimization is notoriously finicky.Common variants include Wasserstein GAN and StyleGAN.
2) GANs:: Because GANs assign no input-space likelihood, anomaly detection uses discriminator scores or latent-space reconstruction, often with intermediate discriminator features.Explicit encoding networks in later adaptations also identify latent points for reconstruction-based detection.
D. Normalizing Flows: Normalizing flows map equal-dimensional latent samples to data through invertible layers, enabling exact density calculation by change of variables.Their structured Jacobians make determinant evaluation tractable, and training typically maximizes the likelihood of the data.
D. Normalizing Flows: Normalizing flows can apply likelihood-based anomaly detection directly because point densities are computed exactly while sampling remains reasonably efficient.This avoids the approximation required by some other generative approaches.
D. Normalizing Flows: Flow models do not reduce dimensionality and may assign high likelihood to anomalies, apparently because current architectures let low-level features dominate likelihood.These limitations argue against applying them to images whose effective dimensionality is much smaller than their image dimensionality.
E. Discussion: The review contrasts full density estimation with learning only one density level set, motivating one-class classification as a direct discriminative alternative.Conditional variants of GANs, VAEs, and normalizing flows support conditional density estimation across additional data settings.

A. One-Class Classification Objective · B. One-Class Classification in Input Space

One-class classification learns a boundary from predominantly normal data while balancing false alarms against missed anomalies. In input space, this principle yields minimum-volume enclosing models such as soft hyperspheres, whose assumptions and extensions determine the distributions they can represent.

A. One-Class Classification Objective: One-class classification treats anomaly detection as binary classification with access mainly to the normal class, minimizing false alarms and missed anomalies.The normal class is designated Y = +1, while anomalies are Y = −1.
A. One-Class Classification Objective: The objective can be formulated through binary classification risk and empirical risk minimization using unlabeled data, optionally supplemented by labeled examples.The empirical formulation estimates losses from available normal and additional labeled data.
A. One-Class Classification Objective: Without constraints or regularization, the unsupervised empirical objective is trivial, so R(f) encodes assumptions about the scoring function and anomaly distribution P-.Regularization generally targets the miss rate, including through volume minimization.
B. One-Class Classification in Input Space: A data-enclosing ball finds a hypersphere with radius R > 0 and center c ∈X that contains the data while minimizing R2 to control the miss rate.This construction seeks a tight spherical boundary around normal observations.
B. One-Class Classification in Input Space: Slack variables ξi ≥0 permit observations outside the sphere, while ν ∈(0, 1] balances boundary softness against tightness.The soft boundary allows some points to violate the enclosing constraint.
B. One-Class Classification in Input Space: The hypersphere objective is Support Vector Data Description in input space and can also be derived from a shifted, cost-weighted hinge-loss classification risk.For fθ(x) = ∥x −c∥2 −R2, the risk connects the geometric model to one-class classification.
B. One-Class Classification in Input Space: Assuming anomalies are uniformly distributed on X makes minimizing the expected miss rate equivalent to minimizing the sphere’s volume, while ν approximates the false alarm rate.Specifically, ν is an upper bound on the ratio of points outside and a lower bound on the ratio inside or on the boundary.
B. One-Class Classification in Input Space: Spherical input-space models fit limited distributions such as isotropic Gaussians, whereas ellipsoidal, nonparametric, and general-feature-space methods support broader distributional shapes.MVE and MCD address elliptical support, while One-Class Neighbor Machines can model multimodal distributions with non-convex support.

C. Kernel-based One-Class Classification · D. Deep One-Class Classification · E. Negative Examples

Kernel one-class methods use feature-space boundaries to model complex, potentially non-convex supports, while deep variants learn task-relevant representations but require regularization against collapse and often more data or informative priors. One-class classifiers can also incorporate labeled or generated negative examples, enabling direct binary discrimination and empirical miss-rate estimation.

C. Kernel-based One-Class Classification: Kernel OC-SVM and SVDD formulate one-class boundaries in an implicit RKHS feature space using linear and hypersphere models, respectively.Both methods can be solved through dual quadratic programs involving only kernel dot products.
C. Kernel-based One-Class Classification: For Gaussian kernels with constant norm, OC-SVM and SVDD are equivalent and yield an asymptotically consistent ν-density level set estimator.The equivalence applies to the standard Gaussian kernel or any kernel satisfying k(x, x) = c > 0.
C. Kernel-based One-Class Classification: Kernel-induced feature spaces improve one-class modeling for multimodal, non-convex, and nonlinear data settings.The review notes variants including nested density level-set formulations, Multi-Sphere SVDD, multiple-kernel OC-SVM, group-anomaly OC-SVM, and boosting.
D. Deep One-Class Classification: Deep one-class methods learn or transfer neural feature maps, applying hypersphere models such as Deep SVDD or linear models such as deep OC-SVM variants.They address the difficulty of selecting kernels and hand-crafting features for complex data.
D. Deep One-Class Classification: One-Class Deep SVDD minimizes mean squared feature distance to a center, often converges faster, and can be interpreted as a single-prototype deep clustering method.The learned transformation minimizes the mean squared distance of all data points to c ∈Z.
D. Deep One-Class Classification: Deep one-class objectives require regularization because a constant feature map φω ≡c can trivially solve minimum-volume or maximum-margin objectives.Proposed remedies include reconstruction terms, architectural constraints, frozen embeddings, and inverse penalties on embedding variance.
D. Deep One-Class Classification: Deep methods provide greater modeling flexibility and task-relevant features but generally need more data or an informative domain prior, while retaining the one-class boundary principle.This principle remains unchanged across deep and shallow feature mappings.
E. Negative Examples: One-class classifiers can directly incorporate labeled negative examples, which facilitate empirical miss-rate estimation; studied negatives are artificial, auxiliary, and true examples.Artificial negatives can be generated under assumptions about the anomaly distribution, and a uniform prior yields an asymptotically consistent density level set estimator.

V. RECONSTRUCTION MODELS … B. Principal Component Analysis

Reconstruction-based anomaly detection learns representations that reconstruct predominantly normal data well, then flags instances with high reconstruction error. Its foundations include manifold and prototype assumptions, while PCA and kernel PCA provide linear and nonlinear reconstruction baselines with geometric and probabilistic interpretations.

V. RECONSTRUCTION MODELS: Reconstruction methods learn to reconstruct normal instances accurately and detect anomalies through reconstruction failure.Most approaches are geometrically motivated, although probabilistic variants connect reconstruction to density or level-set estimation.
A. Reconstruction Objective: The reconstruction objective learns an encoder-decoder pair that maps x to a latent code and back to an approximation x̂.Regularization can constrain parameters, transformation structure, or latent-space geometry.
1) The Manifold Assumption:: The manifold assumption models data as lying approximately on a lower-dimensional manifold M embedded in the ambient space X.Encoders and decoders are trained so reconstructed data approximates M, typically using a latent dimension d much smaller than the data dimension D.
3) The Reconstruction Anomaly Score:: A reconstruction model must extract salient features and patterns so predominantly normal training data can be reconstructed with low error.If the learned representation truthfully captures manifold structure or prototypes, high reconstruction error indicates off-manifold or non-prototypical instances.
B. Principal Component Analysis: PCA finds an orthogonal basis maximizing empirical variance, with principal components given by eigenvectors associated with the largest covariance eigenvalues.From a reconstruction perspective, PCA minimizes mean squared error for projection onto a d-dimensional linear subspace and is the default reconstruction baseline.
B. Principal Component Analysis: Linear PCA exactly solves the reconstruction objective using a linear encoder and transposed linear decoder under the constraint WW ⊤= I.Its probabilistic interpretation uses a d-dimensional latent Gaussian with optional isotropic Gaussian noise.
B. Principal Component Analysis: Kernel PCA extends component analysis to nonlinear kernel feature maps, enabling reconstruction from kernel-induced nonlinear subspaces in input space.For RBF kernels, a prototype reconstruction model is equivalent up to a multiplicative constant to kernel density estimation; Robust PCA variants address contamination or noise.

C. Autoencoders · D. Prototypical Clustering · VI. A UNIFYING VIEW OF ANOMALY DETECTION

The section presents autoencoders and prototypical clustering as reconstruction-based approaches to anomaly detection, then unifies diverse methods through shared modeling components and transferable algorithmic ideas.

C. Autoencoders: Autoencoders use neural-network encoders and decoders for reconstruction and remain among the most widely adopted deep anomaly-detection methods.They were originally introduced for nonlinear dimensionality reduction and were also studied early for anomaly detection.
C. Autoencoders: A lower-dimensional bottleneck representation regularizes autoencoders by compressing data and limiting the dimensionality of the learned manifold or subspace.This objective optimizes the neural-network encoder and decoder weights.
C. Autoencoders: Denoising autoencoders reconstruct clean inputs from noise-corrupted inputs, enabling explicit noise modeling and noise-robust anomaly detection.Robust deep autoencoders address training data containing noise or unknown anomalies by separating well-represented data from other components.
C. Autoencoders: Variational autoencoders connect reconstruction-based detection to density estimation by approximately maximizing data likelihood through the ELBO.A standard Gaussian VAE minimizes mean reconstruction error over samples from an encoded latent Gaussian regularized toward a standard isotropic Gaussian.
C. Autoencoders: Other autoencoder approaches include RNN-based, convolutional, ensemble, gradient-constrained, topology-controlled, and two-step dimensionality-reduction methods.In two-step approaches, autoencoders produce embeddings for subsequent traditional anomaly-detection methods.
D. Prototypical Clustering: Prototypical clustering treats reconstruction error as a point’s distance to its nearest prototype, ideally representing a distinct mode of the normal data distribution.Vector Quantization methods include k-means, k-medians, and k-medoids, which define a Voronoi partition.
D. Prototypical Clustering: Deep clustering methods, including k-means-based approaches, have been adopted for anomaly detection but must be regularized against feature-map collapse.Their reconstruction error is measured in latent space Z, whereas deep autoencoders measure it in input space.
VI. A UNIFYING VIEW OF ANOMALY DETECTION: The unifying view characterizes anomaly-detection methods through specific modeling components and exposes connections for transferring algorithmic ideas between methods.The authors identify transfer from kernel-based anomaly detection to deep methods as a promising direction for future research.

A. Modeling Dimensions of the Anomaly Detection Problem … B. Evaluating Anomaly Detectors

The review unifies anomaly detection methods through five modeling dimensions and uses this framework to compare probabilistic, reconstruction, one-class, shallow, and deep approaches. It also examines benchmark construction and evaluation, emphasizing realistic anomalies, application-specific errors and constraints, and threshold-independent measures.

A. Modeling Dimensions of the Anomaly Detection Problem: Five dimensions organize anomaly detection methods: loss, model, feature map, regularization, and inference mode.The loss may incorporate labels or be unsupervised, while the model maps inputs to scalar values; inference may be frequentist or Bayesian.
B. Comparative Discussion: The unifying framework connects shallow and deep methods and supports systematic transfer of concepts between them.The review gives examples of potential combinations, including discrete codebooks in VQ-VAE, which can improve reconstruction quality and coherence in some settings.
C. Distance-based Anomaly Detection: Distance-based anomaly detection methods are studied separately from loss-based methods and often use lazy learning without prior model training.New test points are evaluated with respect to the available data rather than a model learned in an a priori training phase.
VII. EVALUATION AND EXPLANATION: Evaluating anomaly detectors requires meaningful anomaly-containing datasets and robust procedures because real-world anomalies are rare and may be previously unobserved.The review frames evaluation around both benchmark construction and robust assessment of detector performance on real data.
A. Building Anomaly Detection Benchmarks: Common benchmark strategies include k-classes-out and synthetic anomalies, but their semantic homogeneity or controllability may fail to represent real and adversarial variation.Real anomalies can exhibit wider and finer variation, including maliciously designed cases intended to evade detection.
B. Evaluating Anomaly Detectors: Application-specific costs and constraints determine decision thresholds and metrics, such as precision@k for analyst-limited alarms and recall@k under false-alarm constraints.When costs and constraints are unspecified or variable, AUROC or AUC evaluates performance across the full range of decision thresholds.

C. Comparison on MNIST-C and MVTec-AD · D. Explaining Anomalies

The review finds highly heterogeneous anomaly-detection performance across MNIST-C corruptions and MVTec-AD defect classes, with shallow models sometimes outperforming deeper ones. Explanations expose differing detection strategies and vulnerabilities, addressing limitations of potentially misleading quantitative validation.

C. Comparison on MNIST-C and MVTec-AD: AUC comparisons cover probabilistic, one-class, and reconstruction methods using raw-input, kernel, and neural-network representations on MNIST-C and MVTec-AD.MNIST-C contains fifteen corruptions, while MVTec-AD contains fifteen industrial image sets with manufacturing defects.
C. Comparison on MNIST-C and MVTec-AD: Performance is heterogeneous across corruptions and defect classes: AGAN generally performs well on MNIST-C but is systematically outperformed by DOCC on MVTec-AD.The review also notes that nonlinear models are not superior on every class and that simple shallow models occasionally outperform deeper counterparts.
C. Comparison on MNIST-C and MVTec-AD: Test performance may not reflect generalization to broader anomalies because methods can benefit from specific, possibly non-representative test-set anomalies.The section asks whether 100% accuracy on a corruption such as MNIST-C:stripe implies strong performance across a broad anomaly range.
D. Explaining Anomalies: Explanation techniques augment anomaly predictions to reveal models’ generalization properties and detection strategies while addressing limitations of classical validation procedures.Explainable AI methods have been extended from supervised learning to unsupervised anomaly detection.
D. Explaining Anomalies: Two explanation strategies are model-agnostic techniques and neuralization, which converts non-neural models into functionally equivalent neural networks for applying neural-network explainers.The review demonstrates neuralization for models including KDE and one-class SVMs, while noting that equivalent network structures may not be unique.
D. Explaining Anomalies: Neuralized KDE, DOCC, and autoencoder models use feature extraction, distance computation, and pooling, enabling LRP to explain anomaly predictions.LRP propagates scores through pooling winners and assigns relevance to input or feature-space directions contributing most to distance.
D. Explaining Anomalies: On an MNIST-C:stripe digit, KDE, DOCC, and AE detect anomalies accurately, but their LRP explanations reveal substantially different strategies.The example compares model explanations with a ground-truth explanation highlighting pixels changed by corruption.
D. Explaining Anomalies: Even with 100% accuracy on current data, models can have different generalization properties and vulnerabilities, making explanations important for comprehensive assessment.The review concludes that standard quantitative evaluation can be imprecise or misleading when available data is not fully representative.

VIII. WORKED-THROUGH EXAMPLES · A. Example 1: Thyroid Disease Detection · B. Example 2: MVTec Industrial Inspection

The section works through thyroid disease detection and MVTec wood-image inspection to illustrate modeling, evaluation, best practices, and explanation-guided improvement. The examples emphasize matching model choices to data properties, accounting for error costs, and identifying or correcting spurious prediction strategies.

VIII. WORKED-THROUGH EXAMPLES: Two real-world examples demonstrate the modeling and evaluation process alongside practical best practices.
A. Example 1: Thyroid Disease Detection: The Thyroid dataset contains 3772 instances, 6 real-valued features, and 93 (∼2.5%) anomalies, with a 60:10:30 train-validation-test split preserving anomaly ratios.
A. Example 1: Thyroid Disease Detection: OC-SVM with a standard RBF kernel is selected because the thyroid data are real-valued and low-dimensional, while ν approximately controls the false alarm rate α and implicitly the miss rate.
A. Example 1: Thyroid Disease Detection: Detection design should assess false alarms and missed anomalies because type I and type II errors can carry substantial costs, especially in medical disease detection.
A. Example 1: Thyroid Disease Detection: The initial best validation AUC was 83.9% at γ = (2−5D)−1, indicating that feature scaling had been omitted.
B. Example 2: MVTec Industrial Inspection: MVTec wood images are high-dimensional pixel arrays whose inputs already lie between −1 and +1, so feature rescaling is unnecessary.
B. Example 2: MVTec Industrial Inspection: The MVTec benchmark trains models, selects hyperparameters on held-out data, evaluates the remaining test data, and compares nine models using AUC.
B. Example 2: MVTec Industrial Inspection: KDE performs best because the wood anomalies involve small perforations and stains, but explanations reveal reliance on ubiquitous vertical stripes rather than the true anomalies.In the wood example, replacing the Gaussian kernel with a Mahalanobis kernel raises explanation accuracy from an average cosine similarity of 0.34 to 0.38.

IX. CONCLUSION AND OUTLOOK · A. Unexplored Combinations of Modeling Dimensions · B. Bridging Related Lines of Research on Robustness

The review establishes a systematic, unifying view of anomaly detection and identifies open challenges and future research directions. It highlights unexplored combinations across modeling dimensions and opportunities to connect anomaly detection with related robustness research.

IX. CONCLUSION AND OUTLOOK: The review synthesizes anomaly detection research into a systematic unifying view while addressing theoretical, practical, and empirical aspects.It presents a balanced and comprehensive snapshot of the field and identifies critical open challenges and future avenues.
A. Unexplored Combinations of Modeling Dimensions: A zoo of algorithms spans multiple modeling dimensions, with conceptual similarities linking kernel methods and deep learning.Differences in how the two domains have explored novel algorithms create opportunities for new methodology.
A. Unexplored Combinations of Modeling Dimensions: Deep anomaly detection has scarcely addressed robustness to noise, contamination, and signal-to-noise ratio, despite effective shallow-method solutions.The review identifies robustness as one specific opportunity for transferring ideas across modeling paradigms.
A. Unexplored Combinations of Modeling Dimensions: Bayesian methods provide another opportunity for combining modeling dimensions because Bayesian inference has been considered mostly for shallow methods.Further analysis of the framework may expose additional directions for innovation beyond the listed opportunities.
B. Bridging Related Lines of Research on Robustness: Robust deep learning research on out-of-distribution detection, calibration, uncertainty estimation, and adversarial attacks is closely related to anomaly detection.Clarifying differences among problem formulations could connect concepts and transfer ideas across these lines of research.
B. Bridging Related Lines of Research on Robustness: Classification with a reject option combines classification and anomaly detection by rejecting uncertain or potentially misclassified inputs.Related approaches calibrate class probabilities or quantify Bayesian uncertainty, while standard anomaly detection lacks class-label information.
B. Bridging Related Lines of Research on Robustness: Open set recognition fits the review’s framework because unseen classes can be regarded as generated by a distribution P−, yet it has developed separate methods.Bridging open set recognition with anomaly detection is identified as an important goal for future research.
B. Bridging Related Lines of Research on Robustness: Adversarial attacks can be interpreted as extremely hard-to-detect out-of-distribution samples targeting a classifier’s decision boundary and confidence.An imperceptibly perturbed input may remain normal under PX while becoming anomalous under the joint PX,Y when paired with the attacker-induced label.

C. Interpretability and Trustworthiness … F. Foundation and Theory

The review argues that anomaly detection must address interpretability, trustworthy evaluation, and challenging datasets alongside accuracy. It also highlights gains from limited supervision while identifying unresolved theoretical problems in likelihood- and reconstruction-based detection.

C. Interpretability and Trustworthiness: Accuracy alone is insufficient in anomaly detection, where interpretability and trustworthiness are equally critical.Anomaly heterogeneity complicates explanations, especially when anomalies result from missing rather than abnormal features.
D. The Need for Challenging and Open Datasets: Publicly available benchmark datasets with clearly defined evaluation criteria are essential for measuring progress in deep anomaly detection.The review invokes ImageNet and its competitions as a precedent from computer vision and supervised deep learning.
E. Weak Supervision and Self-Supervised Learning: Weak supervision uses imperfect or scarce labels, and even a few labeled anomalies can substantially improve performance on complex data.Labels may be inaccurate or incomplete, including coverage of only a few normal modes or specific anomalies.
E. Weak Supervision and Self-Supervised Learning: Self-supervised methods learn representations through auxiliary prediction tasks and can derive anomaly scores from classifier uncertainty.Examples include masked-word prediction, future-frame prediction, and image transformations such as colorization, cropping, and rotation.
E. Weak Supervision and Self-Supervised Learning: Self-supervised and geometric-transformation methods improve performance on common k-classes-out image benchmarks, with GOAD extending the approach beyond images.The review categorizes GOAD and geometric-transformation methods as classification-based approaches.
F. Foundation and Theory: Deep generative models can assign higher likelihood to anomalies than to in-distribution samples, challenging likelihood-based anomaly detection.Fashion-MNIST-trained models can systematically assign higher likelihood to MNIST digits, a finding replicated in subsequent work.
F. Foundation and Theory: Reconstruction-based models can well reconstruct simpler out-of-distribution points, including anomalous all-black MNIST images and points in PCA subspaces.These observations motivate further theoretical work on anomaly definitions, out-of-distribution generalization, and information-theoretic interpretations.

APPENDIX A NOTATION AND ABBREVIATIONS … C. Explaining KDE

The appendix defines the paper’s notation and abbreviations, reports Average Precision results for MNIST-C and MVTec-AD, documents training details, and explains KDE heatmap computation through neuralization and Taylor-type decomposition.

APPENDIX A NOTATION AND ABBREVIATIONS: Notation and abbreviations used throughout the work are provided in Tables VII and VIII.Table VII lists notation conventions, while Table VIII lists abbreviations.
A. Average Precision on MNIST-C and MVTec-AD: Average Precision detection performance for MNIST-C and MVTec-AD is reported in Tables IX and X, respectively.The tables supplement the experimental evaluation from Section VII-C.
A. Average Precision on MNIST-C and MVTec-AD: AP shows the same trends as AUC because the MNIST-C and MVTec-AD test sets are not highly imbalanced.The passage attributes this expected correspondence to the test-set class balance.
B. Training Details: PCA and kPCA reconstruction errors retain 90% of the training-data variance, while kPCA uses a kernel width capturing 50% of similarity scores with 50% neighbors.These settings specify the reconstruction and kernel-width choices used for the experiments.
B. Training Details: MVE uses the fast minimum covariance determinant estimator with support fraction 0.9 and contamination rate parameter 0.01.For MVTec-AD, dimensionality is first reduced with PCA retaining 90% of variance to facilitate MVE computation.
C. Explaining KDE: The KDE model is neuralized by replacing the first layer’s squared Euclidean distance with a squared Mahalanobis distance.The resulting KDE and Mahalanobis KDE models are used to compute heatmaps.
C. Explaining KDE: The KDE explanation implements a Taylor-type decomposition, with ⊙ denoting element-wise multiplication.This decomposition is described in reference.