PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Anomaly Detection with Deviation Networks

Guansong Pang, Chunhua Shen, Anton van den Hengel

arXiv:1911.08623v1cs.LGstat.ML

TL;DR

Deep anomaly detection is limited by scarce labels and methods that indirectly optimize scores through representation learning. The paper introduces DevNet, which directly learns scores using few labeled anomalies, a prior, and deviation loss, and reports better scoring with substantially fewer labels than competing methods.

  • Problem

    Deep anomaly detection has limited labeled data, while existing methods typically learn representations and optimize anomaly scores indirectly or unsupervised.

  • Method

    DevNet directly learns anomaly scores end-to-end using a few labeled anomalies, a prior probability, and Z-Score-based deviation loss.

  • Results

    DevNet significantly outperforms four state-of-the-art methods in AUC-ROC and AUC-PR, with 3%-29% and 21%-309% average improvements, respectively.

  • Takeaways & Limitations

    DevNet achieves comparable or better accuracy while using 75%-88% fewer labeled anomalies than competing methods.

  • Takeaways & Limitations

    The authors identify extending DevNet to settings with only one or two labeled anomalies as future work.

Abstract

from arXiv · show

Although deep learning has been applied to successfully address many data mining problems, relatively limited work has been done on deep learning for anomaly detection. Existing deep anomaly detection methods, which focus on learning new feature representations to enable downstream anomaly detection methods, perform indirect optimization of anomaly scores, leading to data-inefficient learning and suboptimal anomaly scoring. Also, they are typically designed as unsupervised learning due to the lack of large-scale labeled anomaly data. As a result, they are difficult to leverage prior knowledge (e.g., a few labeled anomalies) when such information is available as in many real-world anomaly detection applications. This paper introduces a novel anomaly detection framework and its instantiation to address these problems. Instead of representation learning, our method fulfills an end-to-end learning of anomaly scores by a neural deviation learning, in which we leverage a few (e.g., multiple to dozens) labeled anomalies and a prior probability to enforce statistically significant deviations of the anomaly scores of anomalies from that of normal data objects in the upper tail. Extensive results show that our method can be trained substantially more data-efficiently and achieves significantly better anomaly scoring than state-of-the-art competing methods.

1 INTRODUCTION

Deep anomaly detection faces limited labeled anomalies and difficult high-dimensional, nonlinear data, while existing methods indirectly optimize anomaly scores. The paper proposes direct end-to-end score learning with few labeled anomalies and a prior, reporting stronger and more data-efficient detection.

  • High-dimensionality and nonlinear feature relations make many traditional anomaly detection methods ineffective.
  • Deep anomaly detection remains challenging because large-scale labels are costly and anomalies can exhibit dissimilar behaviors.
  • Existing methods typically learn representations first and derive anomaly scores afterward, indirectly optimizing the scores and potentially producing irrelevant representations.
  • The proposed framework directly learns anomaly scores from original data while using a few labeled anomalies, a prior reference score, and deviation loss.
  • DevNet combines neural networks, a Gaussian prior, and Z-Score-based deviation loss to produce data-efficient and interpretable anomaly scores.
  • 3%-29% average AUC-ROC improvement and 21%-309% average AUC-PR improvement were obtained over four state-of-the-art methods across nine real-world data sets.

2 RELATED WORK

Related work largely uses unsupervised representation learning followed by reconstruction- or distance-based scoring. DevNet instead directly differentiates anomaly-score learning and extends limited-label approaches beyond graph-specific methods.

  • Autoencoder and GAN methods learn representations and then use reconstruction errors as anomaly scores.
  • Existing deep anomaly detectors optimize feature representations indirectly rather than directly optimizing anomaly scores.
  • Some few-labeled-anomaly methods use belief propagation, but they apply only to graph data.
  • Few-shot classification and positive-unlabeled learning are related research areas because they leverage limited labeled examples, but address different learning settings.

3 END-TO-END ANOMALY SCORE LEARNING

The framework learns anomaly scores directly from inputs using a neural scorer, a prior-based reference score, and deviation loss. Its objective separates anomalies into the upper tail while keeping normal scores near the reference.

  • The goal is to learn a scoring function that ranks anomalies above normal objects using a small labeled anomaly set and unlabeled data.
  • The framework combines neural networks, a prior probability distribution, and a loss function for end-to-end anomaly-score learning.
  • The procedure uses an anomaly-scoring network to produce one scalar score for each input.
  • A reference-score generator computes µR as the mean score of randomly selected normal objects or determines it from a prior probability F.
  • Deviation loss pushes anomaly scores into the upper tail relative to µR while keeping normal-object scores close to µR.
  • A central training problem is obtaining enough normal objects because unlabeled training data have unknown class labels.

4 DEVIATION NETWORKS

DevNet directly learns anomaly scores end to end by combining a neural representation learner with a scoring function, a Gaussian reference prior, and a Z-Score-based deviation loss. The framework uses labeled anomalies and unlabeled data to push normal scores toward the reference while separating anomalies into the upper tail with interpretable probabilistic meaning.

  • End-to-end Anomaly Scoring Network: DevNet combines a feature representation learner and an anomaly scoring function to map inputs directly to scalar anomaly scores for end-to-end training.The feature learner can use architectures suited to multidimensional, image, or sequence data, while the scoring learner uses a single linear output unit.
  • Gaussian Prior-based Reference Scores: A Gaussian prior supplies a reference score computed as the mean of sampled scores for random normal objects, chosen for interpretability and efficient constant generation.The experiments use µ = 0, σ = 1, and l = 5000; performance was empirically insensitive to these choices when σ was not too large and l was sufficiently large.
  • Z-Score-based Deviation Loss: The deviation loss uses a Z-Score to pull normal scores toward the reference and enforce at least a deviation between the reference and anomaly scores.Negative deviations for anomalies incur particularly large loss, encouraging positive upper-tail deviations.
  • Training Procedure: Unlabeled training objects are treated as normal, allowing DevNet to train without labeled normal data even when the unlabeled set contains many anomalies.The paper attributes the observed robustness partly to anomalies being rare and having limited impact on stochastic-gradient optimization, while describing this as training with noisy data.
  • Training Procedure: Training samples combine labeled anomalies and unlabeled objects through stratified mini-batches, while each iteration samples Gaussian reference scores before gradient descent updates.Algorithm 1 initializes the weights, samples a batch with half its objects from each set, computes the reference mean and standard deviation, and optimizes the deviation loss.
  • Interpretability of Anomaly Scores: At test time, DevNet ranks objects by optimized anomaly scores, and scores outside µ ± z_pσ have probability 2(1 − p) under the Gaussian reference.With µ = 0, σ = 1, and p = 0.95, the upper-tail threshold is 1.96 and corresponds to probability 0.05 under the reference distribution.

5 EXPERIMENTS

Experiments on nine real-world data sets compare DevNet with four competing methods using AUC-ROC, AUC-PR, data-efficiency, robustness, ablations, and scalability. DevNet generally achieves stronger anomaly scoring with few labeled anomalies, remains robust to contamination, and scales linearly with data size and dimensionality.

  • 5.1 Data Sets and Competing Methods: Nine diverse real-world data sets evaluate DevNet against REPEN, DSVDD, FSNet, and iForest using AUC-ROC and AUC-PR.The data sets cover intrusion detection, fraud detection, malicious URLs, disease detection, and other domains.
  • 5.5 Performance: DevNet performs best on eight data sets for AUC-ROC and nine for AUC-PR, improving over competitors by 3%-29% and 21%-309%, respectively.The reported improvements are statistically significant across the comparisons described.
  • 5.6 Data Efficiency: 75%-88% less labeled data lets DevNet achieve performance comparable to or better than the strongest competing method in several cases.Examples include 83% less data than FSNet on donors, and 88% and 75% less than DSVDD on news20 and thyroid.
  • 5.6 Data Efficiency: With only five labeled anomalies, DevNet and DSVDD improve average performance over iForest by more than 400% on most data sets.Campaign requires slightly more labeled data because its anomalies may have very intricate distributions.
  • 5.7 Robustness to Contamination: DevNet remains consistently stronger across contamination levels, achieving average AUC-PR improvements of 200% over REPEN, 28% over DSVDD, and 336% over FSNet.All deep detectors degrade as contamination rises because anomalies sampled as normal objects can mislead optimization.
  • 5.9 Scalability: DevNet runtime increases linearly with data size and dimensionality and is 10 to 20 times faster than REPEN, FSNet, and iForest on large data sets.The paper attributes this advantage to DevNet’s computationally efficient loss function.

6 CONCLUSIONS

The paper presents DevNet as an end-to-end anomaly-score learning framework that uses few labeled anomalies and a prior, reporting improved data efficiency and detection performance. It also identifies extensions to image and sequence data as future work.

  • DevNet uses a few labeled anomalies with a prior for end-to-end differentiable anomaly-score learning.The method directly optimizes anomaly scores rather than feature representations.
  • DevNet is reported to be more data-efficient and to perform better on AUC-ROC and AUC-PR than two-step deep anomaly detectors.
  • The experiments indicate that deep detectors can use randomly sampled negative examples from anomaly-contaminated unlabeled data and positive examples from a small labeled anomaly set.
  • The authors plan to test DevNet on image and sequence data using convolutional and recurrent architectures.
  • They also plan a hybrid data-driven and prior-driven reference-score approach for applications with only one or two labeled anomalies.

A.1 Data Accessing and Preprocessing

The appendix describes the benchmark datasets and the data-access table, along with preprocessing for missing and categorical values.

  • The datasets cover project excitement, income, fraud, celebrity attributes, backdoor attacks, and malicious URLs as anomaly-detection tasks.
  • Donors treats exceptionally exciting K-12 teacher projects as anomalies, comprising 5.92% of the data.
  • CelebA uses bald celebrities as anomalies, with fewer than 3% of celebrities in the target class.
  • Table 3 provides links for accessing the datasets.
  • Missing feature values are replaced by the corresponding feature mean, while categorical features use one-hot encoding.

A.2 Algorithm Implementation

The algorithm-implementation appendix identifies the software sources and implementation modifications used for the competing methods.

  • iForest uses the implementation available in the scikit-learn Python package.
  • REPEN is taken directly from its authors, whose code is publicly accessible.
  • DSVDD is enhanced with a margin term separating the center c from labeled anomalies in representation space.
  • FSNet training episodes randomly sample equal numbers of objects from unlabeled data and limited labeled anomalies.

A.2.1 Implementation of Competing Methods.

The competing deep methods use default Keras layer and optimizer settings unless otherwise specified, with particular regularization and activation choices for hidden layers.

  • The implementations use Keras default layer and optimizer settings unless Section 5.3 specifies otherwise.
  • Hidden layers use dense layers with uniform Glorot initialization and an ℓ2-norm weight-decay regularizer.
  • The regularizer uses λ = 0.01, and no constraints are applied to kernels or biases.
  • The hidden-layer activation function is the default ReLu.

A.2.2 Optimization Settings.

The implementation section states that the relevant packages and their versions are listed for the algorithm implementation.

  • The algorithm implementation uses a specified set of relevant software packages and versions.
  • The package list is presented as implementation information rather than as an optimization procedure.
  • The listed versions provide the stated software environment for the algorithm implementation.

A.2.3 Packages Used in Our Implementation.

The implementation uses Python and a collection of machine-learning, numerical, data-processing, and monitoring packages with specified versions.

  • Python 3.6.6 and TensorFlow-GPU 1.10.0 are included in the implementation environment.
  • Keras 2.2.4, Keras Applications 1.0.6, and Keras Preprocessing 1.0.5 support the implementation.
  • Scikit-learn 0.20.0, NumPy 1.14.5, pandas 0.23.4, and SciPy 1.1.0 are listed among the used packages.
  • TensorBoard 1.10.0 is included in the listed implementation packages.
Loading 1911.08623v1…