Anomaly Detection using One-Class Neural Networks

1 Anomaly detection: motivation and challenges

• Anomalies are unusual instances that may reflect data errors or a different generating mechanism; unsupervised detection is used in fraud, intrusion, and fault-diagnosis applications.The passage cites Chandola et al. [2007] for anomaly detection and Hawkins [1980] for the outlier definition.Source: §1 · p. 1
• OC-SVMs are widely used and effective, but their performance is suboptimal on complex, high-dimensional datasets, motivating deep-learning alternatives.The passage cites Sch¨olkopf and Smola [2002], Tax and Duin [2004], and Vapnik and Vapnik [1998], Vishwanathan et al. [2003], Bengio et al. [2007], respectively.Source: §1 · p. 1
• Autoencoder methods assess anomalies by reconstruction error, while hybrid methods feed learned features to separate detectors and cannot shape hidden representations for anomaly detection.Although generic pre-trained representations are efficient, learning from scratch on moderately sized task-specific datasets can perform better Andrews et al. [2016b].Source: §1 · pp. 1–2
• OC-NN uses a one-class-SVM-like loss to train neural networks, making progressively richer representations serve the anomaly-detection objective.Its objective finds a hyperplane separating normal data points from the origin.Source: §1 · p. 2
• OC-NN parameters are learned by alternating minimization, with an objective subproblem equivalent to quantile selection.Source: §1 · p. 2
• The paper reports that OC-NN outperforms other state-of-the-art deep-learning approaches on complex image and sequence datasets.Source: §1 · p. 2

2 Background and related work on anomaly detection

• Deep-feature/OC-SVM hybrids use task-agnostic representations, whereas OC-NN combines progressively rich deep features with a one-class objective that separates normal points from the origin.The paper contrasts this approach with a deep model that minimizes the volume of a hypersphere around network representations.Source: §2 · p. 2

2.1 Robust Deep Autoencoders for anomaly detection

• Robust deep autoencoders decompose X into latent representation LD and noise or outliers S that are hard to reconstruct, optimizing the decomposition with backpropagation and ADMM [Boyd and Vandenberghe 2004].The paper names Robust Deep Autoencoder (RDA) and Robust Deep Convolutional Autoencoder (RCAE) as examples [Zhou and Paffenroth 2017], [Chalapathy et al. 2017].Source: §2.1 · pp. 2–3

2.2 One-Class SVM for anomaly detection

• OC-SVM maps unlabeled data into an RKHS and maximizes the separating hyperplane’s distance from the origin, while ν trades off that distance against allowed crossings [Sch¨olkopf and Smola 2002].The origin acts as the sole negative instance, while mapped data points are treated as positive instances.Source: §2.2 · p. 3

3.1 One-Class Neural Networks (OC-NN)

• The model uses a feed-forward network with one hidden layer, linear or sigmoid activation, and one output node; deeper architectures are possible.Source: §3.1 · p. 3
• OC-NN replaces OC-SVM’s input-feature dot product with a dot product over neural-network representations, allowing hidden features to be refined for anomaly detection.The change makes the objective non-convex, so parameter inference is not guaranteed to find a global optimum.Source: §3.1 · p. 3

3.2 Training the model

• Training alternates between optimizing network parameters w and V and setting r to the υ-quantile of the network scores.The theorem identifies the υth quantile as the solution for r given w and V.Source: §3.2 · pp. 3–4

3.3 OC-NN Algorithm

• The OC-NN algorithm learns w and V with backpropagation, updates r using the score quantile, and classifies points as normal when their score minus r is nonnegative.It returns decision scores for the input points.Source: §3.3 · p. 5

4.1 Methods compared

• Comparisons included OC-SVM-SVDD, Isolation Forest, Kernel Density Estimation (KDE), Deep Convolutional Autoencoder (DCAE), AnoGAN, and Soft-Bound and One Class Deep SVDD, each as specified by cited formulations.The passages cite Schölkopf and Smola [2002], Liu et al. [2008], Parzen [1962], Masci et al. [2011], Radford et al. [2015], and Ruff et al. [2018], respectively.Source: §4.1 · p. 6
• Soft-Bound and One Class Deep SVDD were included, following the formulation in Ruff et al. [2018].Source: §4.1 · p. 6
• OC-NN, DCAE, and RCAE were implemented with Keras and TensorFlow, while publicly available implementations were used for OC-SVM and Isolation Forest.The cited implementation sources are Chollet et al. [2015] and Abadi et al. [2016].Source: §4.1 · p. 6

4.2 Datasets

• The study used synthetic data and four real-world datasets: MNIST, GTSRB, and CIFAR-10 among them; each dataset was further processed to define an anomaly-detection task.Synthetic data comprised 190 normal and 10 anomalous points drawn from a normal distribution; MNIST and CIFAR-10 contain 60,000 images each, while the supplied GTSRB description concerns 32×32 colour images.Source: §4.2 · p. 6

4.3.1 Baseline Model Parameters:

• For shallow baselines, OC-SVM/SVDD used a Gaussian kernel with γ selected by holdout search, KDE bandwidth by five-fold cross-validation, and Isolation Forest used 100 trees and subsampling size 256.OC-SVM/SVDD ran with ν = 0.1; the search ranges were γ ∈ 2^-10 through 2^-1 and h ∈ 2^0.5 through 2^5.Source: §4.3.1 · p. 6

4.3.2 Deep Baseline Models:

• DCAE used MSE loss and an encoder architecture shared with Deep SVDD, RCAE, and OC-NN, while AnoGAN used latent dimension 256 and Deep SVDD used a two-phase learning-rate schedule.DCAE trained for 250 + 100 epochs and Deep SVDD for 150 + 100; leaky ReLU used α = 0.1, and RCAE followed its cited robust-loss settings, Chalapathy et al. [2017].Source: §4.3.2 · p. 7

4.3.3 One-class neural Networks (OC-NN):

• OC-NN copied encoder layers from a pretrained deep autoencoder into a one-hidden-layer feed-forward network, continued training encoder weights, and set ν according to each dataset’s outlier proportion.A linear-activation hidden layer gave the best results; the architecture summary is identified in Table 2.Source: §4.3.3 · p. 7, Table 2 · p. 7

5 Experimental Results

• OC-NN performed on par with state-of-the-art methods on complex datasets and outperformed conventional shallow methods in some scenarios.Source: §5 · p. 8

5.1 Synthetic Data

• On synthetic data, OC-NN identified all 10 anomalous points with negative decision scores and performed on par with classical OC-SVM.The experiment used 190 normal training points and 10 anomalies in 512 dimensions.Source: §5.1 · p. 8

5.2 OC-NN on MNIST and CIFAR-10

• On MNIST, RCAE outperformed shallow and deep competitors; on CIFAR-10, shallow methods beat deep models on two classes, while OC-NN, Soft, and One-class Deep SVDD performed robustly overall.OC-NN seemed to outperform shallow methods, Soft, and One-class Deep SVDD on the lower-global-contrast AUTOMOBILE, BIRD, and DEER classes; performance on strongly structured FROG and TRUCK classes depended substantially on network architecture.Source: §5.2 · p. 9, Table 3 · p. 11

5.3 Detecting Adversarial attacks on GTSRB stop signs using OC-NN

• For GTSRB stop-sign adversarial detection, RCAE outperformed the other deep models.The experiment used Boundary Attack examples and a training set of 1,050 normal stop signs plus 100 adversarial examples.Source: §5.3 · p. 11, Table 4 · p. 11

• OC-NN uses an OC-SVM-like loss to train a neural network for anomaly detection.Source: §6 · p. 11
• Unlike hybrid approaches with generic features, OC-NN constructs hidden-layer features specifically for anomaly detection.Source: §6 · p. 11
• The paper proposes alternating minimization to learn OC-NN parameters and shows that an optimization subproblem is equivalent to quantile selection.Source: §6 · p. 11