PaperlayerView exact revision on arXiv

Source-linked AI summary

ADBench: Anomaly Detection Benchmark

Songqiao Han, Xiyang Hu, Hailiang Huang, Mingqi Jiang, Yue Zhao

arXiv:2206.09426v2cs.LGcs.AI

TL;DR

Anomaly-detection benchmarks lacked comprehensive evidence across supervision levels, anomaly types, and robustness to noisy or corrupted data. ADBench addresses this gap by evaluating 30 algorithms on 57 datasets across 98,436 experiments, yielding insights for algorithm selection and future benchmark design.

  • Problem

    Existing benchmarks provide limited evidence on semi-supervised methods, anomaly types, robustness, and complex CV/NLP datasets.

  • Method

    ADBench benchmarks 30 algorithms on 57 datasets across supervision levels, four anomaly types, and three data-corruption settings.

  • Results

    98,436 experiments show that supervision and anomaly-specific prior knowledge matter, while semi-supervised methods show potential robustness on noisy and corrupted data.

  • Takeaways & Limitations

    ADBench supports algorithm selection and fair, accessible evaluation of newly proposed anomaly-detection methods.

  • Takeaways & Limitations

    ADBench focuses on inductive tabular anomaly detection and identifies extensions to mixed anomaly types, varying anomaly ratios, and more modalities.

Abstract

from arXiv · show

Given a long list of anomaly detection algorithms developed in the last few decades, how do they perform with regard to (i) varying levels of supervision, (ii) different types of anomalies, and (iii) noisy and corrupted data? In this work, we answer these key questions by conducting (to our best knowledge) the most comprehensive anomaly detection benchmark with 30 algorithms on 57 benchmark datasets, named ADBench. Our extensive experiments (98,436 in total) identify meaningful insights into the role of supervision and anomaly types, and unlock future directions for researchers in algorithm selection and design. With ADBench, researchers can easily conduct comprehensive and fair evaluations for newly proposed methods on the datasets (including our contributed ones from natural language and computer vision domains) against the existing baselines. To foster accessibility and reproducibility, we fully open-source ADBench and the corresponding results.

1 Introduction

ADBench addresses gaps in tabular anomaly-detection evaluation by comparing methods across supervision, anomaly types, and data corruption. Its large-scale results highlight the importance of algorithm selection, supervision, and anomaly-specific knowledge.

  • Existing tabular benchmarks largely omit semi-supervised methods, anomaly-type analysis, robustness testing, statistical comparisons, and complex CV/NLP datasets.
  • The benchmark compares unsupervised, semi-supervised, and supervised methods while varying supervision, anomaly types, and data corruption.
  • None of the benchmarked unsupervised algorithms is statistically better than the others, emphasizing the importance of algorithm selection.
  • With 1% labeled anomalies, most semi-supervised methods can outperform the best unsupervised method, supporting the value of supervision.
  • ADBench evaluates 30 algorithms on 57 datasets, including 47 existing datasets and 10 newly created datasets.
  • ADBench is open-sourced with its results to support fair and accessible evaluation of newly proposed methods.

2 Related Work

Prior anomaly-detection benchmarks mainly emphasize unsupervised methods and limited datasets, while related work spans supervised, semi-supervised, and modality-specific settings. ADBench broadens evaluation across algorithms, datasets, learning paradigms, and comparison angles.

  • Anomaly Detection Algorithms: Unsupervised methods rely on assumptions about data distributions, with shallow methods often more interpretable and deep methods better suited to large, high-dimensional data.
  • Anomaly Detection Algorithms: Supervised anomaly detection treats the task as binary classification but may miss unknown anomaly types when labels are incomplete.
  • Anomaly Detection Algorithms: Semi-supervised methods use partial labels while retaining the ability to detect unseen anomaly types, often leveraging unlabeled data for representation learning.
  • Existing Datasets and Benchmarks: Compared with existing benchmarks, ADBench combines benchmark and synthetic datasets, shallow and deep methods, and multiple comparison angles.
  • Existing Datasets and Benchmarks: ADBench includes almost all publicly available datasets and adds larger CV and NLP datasets for a more holistic evaluation.
  • Existing Datasets and Benchmarks: Anomaly-detection benchmarks also exist for time series, graphs, CV, and NLP, but these tasks differ from tabular anomaly detection.

3 ADBench: AD Benchmark Driven by Research and Application Needs

ADBench defines an inductive evaluation framework spanning supervision levels, anomaly types, and robustness to noisy or corrupted data. It combines broad algorithm and dataset coverage with realistic synthetic anomaly generation.

  • Preliminaries and Problem Definition: In the inductive setting, an anomaly-detection model maps training data to anomaly scores for test samples, with higher scores indicating greater outlyingness.
  • Preliminaries and Problem Definition: Supervised methods use complete labels, whereas semi-supervised methods train on features with only partial labels.
  • Algorithms: ADBench includes newer unsupervised and semi-supervised algorithms, adapted ResNet and FTTransformer architectures, and ensemble methods.
  • Datasets: The benchmark gathers more than 40 public datasets and adds 10 complex CV and NLP datasets with richer features.
  • Types of Anomalies: To study anomaly types, ADBench injects realistic synthetic anomalies into datasets using generative models built from normal samples.
  • Types of Anomalies: The four generated anomaly types are local, global, dependency, and clustered anomalies, each constructed by modifying distributional or dependency properties.
  • Model Robustness with Noisy and Corrupted Data: ADBench evaluates robustness under three noisy and corruption settings, including duplicated anomalies that can produce anomaly masking.

4 Experiment Results and Analyses

Across 98,436 experiments, ADBench evaluates anomaly detection under varying supervision, anomaly types, and noisy or corrupted data. Results show that performance depends on label availability, alignment between algorithm assumptions and anomaly types, and robustness to specific corruptions.

  • Experimental setup: 98,436 experiments compare anomaly detection algorithms across supervision levels, anomaly types, and noisy or corrupted data.ADBench uses 57 datasets and evaluates unsupervised, semi-supervised, and supervised methods.
  • Varying supervision: No unsupervised algorithm statistically outperforms the others across the 57 datasets.The benchmark uses critical difference diagrams and statistical testing for comparisons.
  • Varying supervision: For γl ≤5%, semi-supervised methods exceed fully supervised methods, with median AUCROC of 75.56% versus 60.84% at γl = 1%.At γl = 5%, the corresponding median AUCROC values are 80.95% and 72.69%; most semi-supervised methods surpass the best unsupervised method with only 1% labeled anomalies.
  • Anomaly types: Unsupervised performance depends on matching algorithm assumptions to anomaly types: LOF is best for local anomalies, while KNN is best for global anomalies.No algorithm performs well across all anomaly types; LOF performs poorly on clustered anomalies.
  • Anomaly types: Label-informed methods are generally inferior to the best unsupervised methods for local, global, and dependency anomalies, but outperform OCSVM for clustered anomalies.Partially labeled anomalies may bias learning, whereas a few labels can represent similar behaviors in clustered anomalies.
  • Noisy and corrupted data: When anomalies are duplicated six times, median ΔAUCROC is −16.43% for unsupervised methods, compared with −0.05% for semi-supervised and 0.13% for supervised methods.Supervised methods degrade by no more than 5% when 50% of input features are corrupted, while unsupervised and semi-supervised methods can degrade by up to 10%; label information supports feature selection.

5 Conclusions and Future Work

ADBench provides a comprehensive tabular anomaly detection benchmark and identifies extensions involving mixed anomaly types, anomaly ratios, and additional data modalities.

  • ADBench benchmarks 30 algorithms on 57 datasets while providing insights into supervision, anomaly types, robust detection design, and future research directions.
  • Future extensions should evaluate mixed anomaly types, different intrinsic anomaly ratios, and more data modalities.

Aknowledgement

The authors describe their contributions across problem scoping, experiments, analysis, drafting, and revision, and acknowledge reviewers, contributors, and fellowship support.

  • The authors divide contributions among problem scoping, experiment and implementation, result analysis, paper drafting, and paper revision.
  • The paper acknowledges anonymous reviewers, named contributors, and Norton Graduate Fellowship support for Y.Z.

Supplementary Material for ADBench: Anomaly Detection Benchmark

The supplementary material provides additional information on related work, algorithms, datasets, and experimental settings, with further details on existing AD algorithms and benchmarks.

  • Supplementary material contains additional information on related works, algorithms, datasets, and experiment settings.
  • It provides more details on existing anomaly detection algorithms and benchmarks discussed in Section 2.

A.1 Unsupervised Methods

The paper reviews unsupervised, semi-supervised, and supervised anomaly detection methods, emphasizing distribution assumptions, representative algorithms, and the broader benchmark landscape.

  • Unsupervised Methods: Unsupervised methods often assume particular data distributions, so performance depends on agreement between the data and those assumptions.
  • Unsupervised Methods: Representative shallow methods include IForest, OCSVM, and ECOD, using isolation, margin-based boundaries, and empirical cumulative distributions.
  • Unsupervised Methods: DAGMM jointly optimizes autoencoder reconstruction, latent-density estimation, and regularization in an end-to-end neural network.
  • Supervised Methods: Fully supervised anomaly detection is often impractical because it requires large-scale labeled training data containing both normal and anomalous samples.
  • Semi-supervised Methods: Semi-supervised methods use partial labels while retaining the ability to detect unseen anomaly types, also leveraging unlabeled data for representation learning.
  • Semi-supervised Methods: Representative semi-supervised methods include XGBOD, DeepSAD, REPEN, DevNet, and PReNet, spanning feature augmentation, representation learning, ranking, deviation scoring, and pairwise relations.
  • Existing AD Benchmarks: Existing benchmarks differ in algorithms, datasets, scalability, robustness, synthetic-data generation, and theoretical versus experimental emphasis.
  • Benchmark Organization: ADBench organizes its algorithms into 14 unsupervised, 7 semi-supervised, and 9 supervised methods, with reported hyperparameter settings.

B.2 ADBench Dataset List

ADBench expands its dataset coverage with 57 datasets spanning application domains, including 10 newly added complex CV and NLP datasets. The benchmark standardizes dataset sizes and provides extracted representations for downstream tabular anomaly detection.

  • Dataset coverage: ADBench includes 57 datasets across healthcare, audio and language processing, image processing, finance, and other application domains.Datasets smaller than 1,000 samples are resampled to 1,000, while datasets larger than 10,000 use 10,000-sample subsets.
  • Dataset coverage: Ten newly added datasets from computer vision and natural language processing provide larger samples and richer features than many existing public datasets.
  • Representation construction: Pretrained neural networks extract representations from CV and NLP datasets so shallow tabular anomaly detection methods can process these complex data.The benchmark uses representations because some shallow methods cannot directly handle large, high-dimensional CV datasets.
  • Anomaly construction: CV and NLP datasets define anomalies through corrupted images, manufacturing defects, negative or low-rated reviews, and downsampled non-normal classes.For several multiclass datasets, one class is treated as normal and the remaining classes are downsampled to 5% as anomalies.
  • Representation construction: The released embeddings use ImageNet-pretrained ResNet18 for CV and pretrained language models for NLP, while backbone effects remain a future analysis direction.

C Details on Experiment Setting

ADBench evaluates algorithms in an inductive train-test setting using fixed original-paper hyperparameters and extensive experiments spanning supervision, anomaly types, and data corruption.

  • General settings: All algorithms are adapted to the inductive setting, using 70% of data for training and the remaining 30% for testing.This enables prediction on incoming data rather than only scoring the input data transductively.
  • Hyperparameters: Default hyperparameters from each algorithm’s original paper are used for fair comparison, without hold-out tuning for semi- or fully-supervised methods.
  • Experiment scale: 98,436 experiments evaluate algorithm performance across real-world datasets, CV and NLP subclasses, anomaly types, and robustness settings.
  • Robustness evaluation: 53,580 experiments assess robustness under three data-noise and corruption settings, including duplicated anomalies, irrelevant features, and annotation errors.
  • Computational resources: Classical models run on an Intel i7-8700 workstation, while deep learning models use an NVIDIA Tesla V100 GPU.

D.1 Additional Results for Overall Model Performance on Benchmark Datasets in §4.2

Additional ADBench results show consistent AUCROC and AUCPR patterns: no unsupervised method statistically dominates, while label-informed methods benefit from limited supervision.

  • Overall performance: No unsupervised algorithm statistically outperforms the others on AUCPR across the 57 benchmark datasets.The AUCPR conclusion is consistent with the corresponding AUCROC results.
  • Overall performance: When labeled anomalies are below 5%, semi-supervised methods perform better than supervised methods on AUCPR.
  • Overall performance: Semi-supervised GANomaly performs worse than anomaly-informed models using labeled anomalies, indicating that modeling normal behavior alone is insufficient for these detections.The reported explanation links missing true-anomaly knowledge to false positives and false negatives.
  • Stability and competitiveness: DeepSVDD and RF are relatively stable detectors but are inferior to most other algorithms, whereas IForest and CatBoost are competitive but have relatively large performance variances.
  • Runtime: PCA, HBOS, GANomaly, and NB have the shortest inference times, while SOD and XGBOD spend more time searching feature subspaces.

D.2 Additional Results for Different Types of Anomalies §4.3

Performance depends strongly on anomaly type: unsupervised methods excel when their distributional assumptions match the anomalies, while label-informed methods usually lag except in clustered cases.

  • Anomaly-type dependence: Unsupervised methods are significantly better when their model assumptions conform to the underlying anomaly types.
  • Anomaly-type dependence: Prior knowledge of anomaly types can be more important than label information for local, global, and dependency anomalies.
  • Comparative performance: Semi- and supervised algorithms generally underperform the best unsupervised methods for local, global, and dependency anomalies, except for clustered anomalies.
  • Ensemble exception: XGBOD is comparable to or can outperform the best unsupervised model when more labeled anomalies are available.Its stacking ensemble combines heterogeneous unsupervised methods with supervised XGBoost, improving adaptability to different data assumptions while using labels.

D.3 Additional Results for Algorithm Robustness in §4.4

Additional robustness results compare anomaly detection algorithms under duplicated anomalies, irrelevant features, and annotation errors using AUCROC and AUCPR. They reinforce the main robustness findings, including unsupervised susceptibility to duplicated anomalies, label-informed resilience to irrelevant features, and robustness to minor annotation errors.

  • AUCROC robustness: AUCROC results under duplicated anomalies, irrelevant features, and annotation errors are consistent with the relative performance changes reported in §4.4.Figure D9 varies duplication times or noise ratios while keeping the AUCROC range consistent across algorithms.
  • Feature corruption: AUCPR results confirm that supervised methods are robust to irrelevant features.The robustness analysis uses additional AUCPR results in Figures D10 and D11.
  • Annotation errors: Both semi-supervised and fully supervised methods are robust to minor annotation errors below 10%.This robustness finding is reported for the AUCPR analysis of noisy and corrupted data.
  • AUCROC robustness: Unsupervised methods are more susceptible to duplicated anomalies than label-informed methods when evaluated with AUCROC.The duplicated-anomaly observation is reported for both the absolute AUCROC results and relative performance changes.
  • AUCPR robustness: AUCPR improves under duplicated anomalies because the metric emphasizes the positive anomaly class, an effect observed for both unsupervised and label-informed methods.When anomaly and normal classes should be weighted equally, the AUCROC results remain the relevant comparison.
Loading 2206.09426v2…