PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Semi-Supervised Anomaly Detection

Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, Marius Kloft

arXiv:1906.02694v2cs.LGstat.ML

TL;DR

Semi-supervised anomaly detection must use both mostly normal unlabeled data and limited labeled normal or anomalous examples, but existing methods often omit labeled anomalies. The paper introduces Deep SAD, a general deep method and information-theoretic framework, and reports strong performance across benchmark settings, while assuming most unlabeled data is normal.

  • Problem

    Most semi-supervised anomaly-detection methods use labeled normals but not labeled anomalies, despite practical access to both types of verified examples.

  • Method

    Deep SAD generalizes Deep SVDD to use unlabeled data and labeled normal or anomalous samples in an end-to-end deep anomaly-detection objective.

  • Results

    Across MNIST, Fashion-MNIST, CIFAR-10, and classic benchmarks, Deep SAD is on par with or outperforms shallow, hybrid, and deep competitors.

  • Takeaways & Limitations

    The evaluation suggests preferring general semi-supervised anomaly detection when labeled information is available for both normal samples and anomalies.

  • Takeaways & Limitations

    Deep SAD assumes that most unlabeled data is normal, and its entropy interpretation assumes an isotropic Gaussian latent distribution for one derivation.

Abstract

from arXiv · show

Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous. Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples. Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific. In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection. We further introduce an information-theoretic framework for deep anomaly detection based on the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution, which can serve as a theoretical interpretation for our method. In extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, we demonstrate that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

1 INTRODUCTION

Semi-supervised anomaly detection uses unlabeled data together with verified normal and anomalous examples, addressing limitations of methods that use only labeled normals. Deep SAD is introduced as a general deep approach for this setting.

  • Anomaly detection identifies unusual samples, commonly by learning a compact description of mostly normal data without labels.Shallow methods can require manual feature engineering and may scale poorly on high-dimensional or large datasets.
  • Semi-supervised anomaly detection combines mostly normal unlabeled samples with a small set of labeled normal or anomalous samples.The objective is to learn a model that compactly characterizes the normal class.
  • Most existing semi-supervised methods use labeled normal samples but not labeled anomalies, corresponding more precisely to Learning from Positive and Unlabeled Examples.Only a few prior works incorporate labeled anomalies.
  • Classification-based semi-supervised learning is ill-suited to anomaly detection because anomalies need not resemble one another, invalidating the cluster assumption for the anomaly class.
  • Deep SAD is introduced as an end-to-end, general semi-supervised anomaly-detection method.The paper generalizes unsupervised Deep SVDD and presents an information-theoretic framework alongside experiments with novel baselines.

2 AN INFORMATION-THEORETIC VIEW ON DEEP ANOMALY DETECTION

The paper frames deep anomaly detection through information theory, contrasting supervised compression with unsupervised Infomax. It uses this perspective to motivate incorporating labels without treating anomalies as a coherent class.

  • The Information Bottleneck principle views supervised representation learning as compressing X into Z while retaining information about labels Y.Its trade-off is between minimizing I(X; Z) and maximizing I(Z; Y).
  • Infomax instead maximizes mutual information between data X and latent representation Z under a constraint or regularizer.The regularizer imposes statistical properties useful for a downstream task.
  • Autoencoders can be understood as implicitly maximizing I(X; Z) through reconstruction while regularizing the latent code.
  • Because anomalies are not necessarily similar, classification principles based on the cluster assumption are ill-defined for anomaly detection.The paper instead builds on Infomax while incorporating label information through a novel representation.

3 DEEP SEMI-SUPERVISED ANOMALY DETECTION

Deep SAD generalizes Deep SVDD to semi-supervised anomaly detection by combining unlabeled data with labeled normal and anomalous samples. Its objective concentrates normal representations while separating anomalies, with an entropy-based interpretation of latent compactness.

  • Unsupervised Deep SVDD: Deep SVDD trains a neural transformation to minimize the volume of a hypersphere enclosing data in latent space.The hypersphere is centered at a predetermined point c.
  • Unsupervised Deep SVDD: Penalizing mapped-sample distances to c extracts stable common variation, placing normal points near the center and anomalies farther away.Weight decay provides the objective’s standard regularization term.
  • Anomaly scoring: The trained model scores test samples by their distance from the hypersphere center.Deep SVDD optimizes the representation with SGD and backpropagation.
  • Entropy interpretation: Deep SVDD also admits an entropy interpretation: minimizing empirical latent variance minimizes an upper bound on latent Gaussian entropy.For fixed dimensionality, Gaussian entropy is proportional to log-variance.
  • Deep SAD objective: Deep SAD combines the unlabeled Deep SVDD loss with a weighted labeled-data term and assumes most unlabeled samples are normal.The weighting hyperparameter η controls the balance between labeled and unlabeled data.
  • Deep SAD interpretation: Deep SAD models normal latent data with low entropy and anomalous latent data with high entropy, while retaining distance-based scoring and autoencoder pre-training.The method is optimized with SGD and backpropagation, and autoencoder pre-training supports mutual-information maximization.

4 EXPERIMENTS

The experiments evaluate Deep SAD across image and classic anomaly-detection benchmarks, varying labeled-anomaly availability, unlabeled-data pollution, anomaly-class diversity, and hyperparameters. Deep SAD performs especially well on complex CIFAR-10 data, remains robust to pollution and η changes, while shallow methods retain a slight edge on small, low-dimensional benchmarks.

  • Evaluation setup: Deep SAD is evaluated on MNIST, Fashion-MNIST, CIFAR-10, and classic anomaly-detection benchmark datasets against shallow, hybrid, and deep competitors.The comparison includes unsupervised, semi-supervised, and supervised methods.
  • Experimental scenarios: The experiments vary labeled-data ratio γl, unlabeled anomaly pollution ratio γp, and the number of labeled anomaly classes kl.These parameters define the three experimental scenarios.
  • Scenario (i): labeled anomalies: Deep SAD performs best on the most complex CIFAR-10 dataset as labeled anomalies increase, while supervised classification is vulnerable to novel test-time anomalies.Deep SAD both uses labeled examples and generalizes to novel anomalies in this scenario.
  • Scenario (ii): polluted data: Detection performance decreases as unlabeled-data pollution increases, but Deep SAD is especially robust on CIFAR-10.The pollution scenario fixes the labeled-training ratio at γl = 0.05 and varies γp.
  • Scenario (iii): anomaly-class diversity: Detection performance improves with more diverse labeled anomalies, while supervised performance is sensitive to the number of anomaly classes.Scenario (iii) increases the number of known anomaly classes kl in the labeled training data.
  • Sensitivity analysis: Deep SAD is fairly robust to η changes, and detection performance increases with representation dimension d toward an upper bound.The η analysis uses γl = 0.05, γp = 0.1, and kl = 1; the dimensionality analysis compares against the hybrid SSAD baseline.
  • Classic benchmarks: Shallow kernel methods have a slight edge on relatively small, low-dimensional benchmark datasets, while Deep SAD remains competitive.The paper contrasts this result with stronger deep-method performance on complex data with hierarchical structure.

5 CONCLUSION AND FUTURE WORK

The paper concludes that Deep SAD generalizes Deep SVDD to general semi-supervised anomaly detection and interprets it through an information-theoretic framework. Its experiments suggest preferring semi-supervised anomaly detection when labeled normal or anomalous samples are available, while the framework is intended to support future methods and theoretical analyses.

  • Contribution: Deep SAD is a deep method for general semi-supervised anomaly detection and a generalization of unsupervised Deep SVDD.The conclusion explicitly presents this as the paper’s central methodological contribution.
  • Conclusion: Experimental results suggest preferring general semi-supervised anomaly detection when labeled information on normal samples or anomalies is available.This conclusion concerns settings where labeled information is available alongside the anomaly-detection task.
  • Information-theoretic interpretation: The information-theoretic framework interprets Deep SAD as minimizing normal-data latent entropy and maximizing anomalous-data latent entropy.The framework is based on the Infomax principle.
  • Future work: The information-theoretic framework is intended as a basis for new methods and rigorous future analyses, including studying deep anomaly detection under the rate-distortion curve.The paper presents these as future directions rather than established results.

A ADDITIONAL RESULTS ON MNIST, FASHION-MNIST, AND CIFAR-10

This appendix examines how Deep SAD and hybrid SSAD respond to changes in network representation dimensionality d, reporting average AUC and standard deviation across 90 experiments.

  • Figure 6 evaluates sensitivity to representation dimensionality d for Deep SAD and the closest competitor, hybrid SSAD.Results are reported as average AUC with standard deviation over 90 experiments.

A.2 AUC SCATTERPLOTS OF BEST VS. SECOND BEST METHODS ON CIFAR-10

Figures 7–9 compare the best and second-best methods on CIFAR-10 across three experimental scenarios. The scenarios vary labeled-anomaly availability, unlabeled-data pollution, and the number of labeled anomaly classes.

  • Figures 7–9 use AUC scatterplots comparing the best and second-best methods on CIFAR-10.The plots cover experimental scenarios (i)–(iii).
  • The three scenarios vary the ratio of labeled training data, the pollution ratio of unknown anomalies, and the number of labeled anomaly classes.These parameters correspond to γl, γp, and kl, respectively.

B RESULTS ON CLASSIC ANOMALY DETECTION BENCHMARK DATASETS

On classic, non-image anomaly-detection benchmarks, shallow kernel methods perform slightly better on small, low-dimensional datasets, while Deep SAD remains competitive under the reported evaluation setting.

  • The benchmark evaluation uses 60:40 random train-to-test splits, preserves anomaly proportions, and runs 10 seeds with γl = 0.01 and γp = 0.The benchmarks are evaluated as non-image, tabular datasets.
  • Shallow kernel methods perform slightly better on the rather small, low-dimensional benchmarks, while Deep SAD remains competitive.The authors note that shallow methods receive a strong hyperparameter-selection advantage.
  • Table 2 reports average AUC with standard deviation over 10 seeds for classic anomaly-detection benchmark datasets.The table setting has no pollution, γp = 0, and a labeled-anomaly ratio of γl = 0.01.

C OPTIMIZATION OF DEEP SAD

Deep SAD is optimized with mini-batch SGD and backpropagation, using L2 weight decay and initialization procedures for the network weights and hypersphere center. The method is designed to scale to large datasets while avoiding trivial hypersphere collapse.

  • Deep SAD optimizes its generally non-convex objective with mini-batch SGD, backpropagation, and L2 weight decay regularization.Algorithm 1 summarizes the optimization routine.
  • SGD gives Deep SAD linear scaling with the number of training batches, parallelizable batch computations, and low memory complexity at prediction time.Prediction requires only a forward pass using the trained network parameters.
  • Network weights are initialized from an autoencoder encoder trained with reconstruction loss.This initialization follows the Infomax principle for unsupervised representation learning.
  • The hypersphere center c is fixed from the mean of initial network representations, excluding labeled anomalies.The authors report smoother and faster SGD convergence when c is fixed near the initial representations.
  • Deep SAD prevents hypersphere collapse by adopting bias-free networks with unbounded activation functions.With sufficiently many labeled anomalies, opposing labeled and unlabeled objectives also prevent collapse.

D NETWORK ARCHITECTURES

The experiments use LeNet-type CNNs for image datasets and standard MLPs for classic anomaly-detection benchmarks. Autoencoder decoders mirror the encoder architectures with upsampling and deconvolutions replacing pooling and convolutions.

  • Image datasets: LeNet-type CNNs process MNIST, Fashion-MNIST, and CIFAR-10 experiments.Each convolutional module combines convolution, leaky ReLU activation with α = 0.1, and (2×2)-max-pooling.
  • Classic benchmarks: Standard MLP feed-forward architectures process the classic anomaly-detection benchmark datasets.The reported networks use three layers, with dataset-specific widths for arrhythmia, cardio, satellite, satimage-2, shuttle, and thyroid.
  • Autoencoders: Autoencoder decoders are constructed symmetrically from the corresponding encoder architectures.Max-pooling is replaced by simple upsampling, while convolutions are replaced by deconvolutions.

E DETAILS ON COMPETING METHODS

The competing methods span kernel, tree, density, autoencoding, deep one-class, semi-supervised generative, and supervised approaches. The evaluation uses specified benchmark settings, anomaly scores, optimization schedules, and deliberately favorable tuning for OC-SVM/SVDD and SSAD.

  • Classical and kernel methods: OC-SVM and SVDD are treated as equivalent under the employed Gaussian/RBF kernel and receive test-subset hyperparameter tuning.The best γ and ν values are selected to maximize AUC on 10% of the test set.
  • Classical and kernel methods: Isolation Forest uses t = 100 trees and subsampling size ψ = 256, while KDE selects bandwidth by five-fold cross-validation.KDE chooses h from the specified Gaussian-kernel bandwidth grid using log-likelihood.
  • Classical and kernel methods: SSAD also receives favorable test-subset tuning of its RBF scale parameter γ, with remaining hyperparameters set according to the original work.The method uses κ = 1, κ = 1, ηu = 1, and ηl = 1 as reported.
  • Autoencoding and hybrids: Autoencoders use MSE reconstruction loss as both the training objective and anomaly score, while hybrid variants apply classical detectors to their bottleneck representations.The hybrid methods include OC-SVM, IF, KDE, and SSAD.
  • Deep methods: Unsupervised Deep SVDD reports the better of Soft-Boundary and One-Class variants, while Deep SAD uses λ = 10−6 and η = 1 by default.Deep SVDD removes network bias terms to prevent hypersphere collapse.
  • Deep methods: SS-DGM reports the better-performing M2 or M1+M2 model, whereas the supervised classifier assigns y = +1 to all unlabeled examples.Deep methods use Adam, Batch Normalization, and a two-phase learning-rate schedule of 10−4 followed by 10−5.

F COMPLETE TABLES OF EXPERIMENTAL RESULTS

The complete tables organize results across three controlled experimental scenarios and classic benchmark datasets. Reported values are average AUCs accompanied by standard deviations computed over repeated experiments or seeds.

  • Experimental scenarios: Table 3 varies the ratio of labeled anomalies γl in the training set and reports average AUC with standard deviations.The standard deviations are computed over 90 experiments at various γl ratios.
  • Experimental scenarios: Table 4 evaluates pollution of the unlabeled training data with unknown anomalies and reports average AUC with standard deviations.The standard deviations are computed over 90 experiments at various γp ratios.
  • Experimental scenarios: Table 5 varies the number of anomaly classes kl in labeled training data, reporting average AUC with standard deviations over 100 experiments.The repetitions cover multiple values of kl.
  • Classic benchmarks: Table 6 reports complete results on classic benchmarks with γp = 0 and γl = 0.01 in training, using averages over 10 seeds.The table fixes the evaluation setting to no pollution and a 0.01 labeled-anomaly ratio.
Loading 1906.02694v2…