DROCC: Deep Robust One-Class Classification

Sachin Goyal; Aditi Raghunathan; Moksh Jain; Harsha Vardhan Simhadri; Prateek Jain

Source-linked AI summary

DROCC: Deep Robust One-Class Classification

Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, Prateek Jain

arXiv:2002.12718v2Feb 28, 2020cs.LGstat.ML

TL;DR

One-class methods struggle with feature engineering, transformation design, and representation collapse, especially across complex domains. DROCC uses a low-dimensional manifold assumption to generate adaptive anomalous examples without side-information, and it performs strongly across one-class settings and domains, with up to 20% greater accuracy reported for anomaly detection. Its scope depends on the manifold and sampling assumptions, and the authors note unresolved efficiency and theoretical-analysis issues.

Problem
One-class methods need accurate special-class discrimination, but classical methods require difficult feature engineering and deep methods can depend on domain-specific transformations or suffer representation collapse.
Method
DROCC assumes the positive class lies on a well-sampled, locally linear low-dimensional manifold and adaptively generates anomalous examples for discriminative training without side-information.
Results
DROCC succeeds across anomaly detection and OCLN benchmarks spanning vision, audio, time-series, and tabular data, with up to 20% greater accuracy than baselines for anomaly detection.
Takeaways & Limitations
DROCC provides a general approach for modeling positive classes across multiple domains and for OCLN settings requiring low false-positive rates against arbitrary negatives.
Takeaways & Limitations
DROCC assumes sufficiently sampled low-curvature manifold structure, while its simplified projection may slow convergence and its rigorous analysis remains future work.

Abstract

from arXiv · show

Classical approaches for one-class problems such as one-class SVM and isolation forest require careful feature engineering when applied to structured domains like images. State-of-the-art methods aim to leverage deep learning to learn appropriate features via two main approaches. The first approach based on predicting transformations (Golan & El-Yaniv, 2018; Hendrycks et al., 2019a) while successful in some domains, crucially depends on an appropriate domain-specific set of transformations that are hard to obtain in general. The second approach of minimizing a classical one-class loss on the learned final layer representations, e.g., DeepSVDD (Ruff et al., 2018) suffers from the fundamental drawback of representation collapse. In this work, we propose Deep Robust One-Class Classification (DROCC) that is both applicable to most standard domains without requiring any side-information and robust to representation collapse. DROCC is based on the assumption that the points from the class of interest lie on a well-sampled, locally linear low dimensional manifold. Empirical evaluation demonstrates that DROCC is highly effective in two different one-class problem settings and on a range of real-world datasets across different domains: tabular data, images (CIFAR and ImageNet), audio, and time-series, offering up to 20% increase in accuracy over the state-of-the-art in anomaly detection. Code is available at https://github.com/microsoft/EdgeML.

1. Introduction

The introduction frames one-class learning around modeling a special class despite limited or difficult negative information. DROCC addresses feature-learning limitations with a manifold-based, adaptive discriminative method and reports strong results across domains and one-class settings.

Motivation: Classical anomaly-detection methods struggle on vision and speech because they depend on appropriately designed input features.Deep methods learn features automatically, but existing approaches have limitations including representation collapse and dependence on domain-specific transformations.
Approach: DROCC assumes typical data lie on a well-sampled, locally linear low-dimensional manifold and generates adaptive anomalous examples during training.It classifies points outside small ℓ2 neighborhoods of training typical points and uses gradient ascent to find effective anomalies.
Problem setting: OCLN requires accurate positive classification and low false-positive rates against arbitrary negatives when only limited negative instances are available.The paper highlights wake-word detection as an example where close but non-identical inputs should not trigger the system.
Results: DROCC is evaluated on vision, audio, time-series, and tabular benchmarks, where it successfully models the positive class and outperforms baselines.The experiments cover multiple domains and two one-class problem settings.
Results: 20% greater accuracy is reported for DROCC over baseline methods on CIFAR-10 anomaly detection.The cited comparison names DeepSVDD, Autoencoder, and GAN-based methods as baselines.
Contributions: DROCC is extended to OCLN, where low false-positive rates on arbitrary negatives are crucial, with an experimental setup for evaluating this less-studied problem.The extension uses limited negative information while targeting generalization beyond the sampled negatives.

2. Related Work

Related work spans classical, generative, transformation-based, and side-information-based anomaly detection. DROCC is positioned as directly modeling manifold membership without requiring side-information and as avoiding representation collapse.

Generative modeling: Deep autoencoders and GANs learn anomaly-detection representations by reconstructing or generating the entire typical input distribution.DROCC instead directly identifies whether a point lies on the typical-data manifold.
Deep one-class methods: DeepSVDD extends one-class objectives to learned representations but suffers from representation collapse.DROCC requires representations to discriminate typical points from off-manifold perturbations, making it robust to this collapse issue.
Transformation-based methods: Transformation-based methods classify which transformation was applied and require appropriate transformations that may vary substantially across domains.The reported MNIST comparison shows horizontal flips work better for typical class 3 than class 8.
Transformation-based methods: DROCC's low-dimensional manifold assumption is presented as more generic across domains, with approximately 0.97 AUROC reported for MNIST typical classes 8 and 3.The cited related-work passage contrasts this with the transformation-based results for those classes.
Side-information-based methods: Some anomaly-detection methods use out-of-distribution samples or labeled anomalies as side-information, whereas DROCC does not assume such information.The paper describes these approaches as complementary to DROCC.

3. Anomaly Detection

DROCC models typical data as a well-sampled low-dimensional locally linear manifold and trains a classifier using adaptively generated off-manifold negatives. Its optimization uses a saddle-point formulation solved with gradient descent-ascent and projection constraints.

Anomaly Detection: The anomaly detector learns a function that labels typical points positive and points outside the typical set negative.The training data consist primarily of typical samples, with only a small anomaly fraction allowed by the stated assumption.
Anomaly Detection: DROCC assumes typical points lie on a well-sampled low-dimensional locally linear manifold, so nearby points can be compared using ℓ2 distance.Points outside a small radius around training typical points are treated as anomalous.
Optimization: DROCC synthetically generates anomalous examples by searching for points outside the radius-r neighborhood of each training typical point.The generated negatives are used in a classification loss together with regularization terms.
Optimization: The training objective is a saddle-point problem analogous to adversarial training, replacing an ℓp perturbation ball with the off-manifold set N_i(r).Projected gradient descent-ascent is used to optimize the objective.
Optimization: Projection onto the full constraint set is computationally challenging because it involves all training points, motivating a computationally easier redefinition.The simplified projection may slow convergence, according to the conclusion.
Optimization: Adversarial search maximizes the negative-label loss and then projects the generated point onto the constrained set N_i(r).The projection limits the point to a local annulus-like region relative to the training point while enforcing distance from training samples.

4. One-class Classification with Limited Negatives (OCLN)

DROCC is extended to one-class classification with limited negatives by incorporating negative examples and learning to ignore noisy coordinates. The resulting methods are evaluated in wake-word and digit-classification settings designed around difficult negatives and low false-positive rates.

Problem: OCLN identifies arbitrary negative-class outliers despite limited access to the negative distribution.The setting assumes the positive distribution is much more thoroughly represented in training data than the negative distribution.
Methods: DROCC–OE uses limited negatives through an outlier-exposure-style approach, whereas standard anomaly detection ignores this side-information.Binary classifiers may generalize only to the training distribution of negatives and therefore have high false-positive rates on negatives far from that distribution.
Methods: DROCC–LF learns to ignore noisy input coordinates by replacing local Euclidean comparisons with a Mahalanobis distance.Its parameters are estimated by minimizing a regularized loss, with λ > 0 and µ > 0 as regularization parameters.
Methods: The DROCC–LF optimization uses projected gradient descent-ascent, with projection onto the local negative region as its key step.The projection is generally not available in closed form and requires a separate optimization procedure.
Experiments: DROCC–LF significantly outperforms standard anomaly detection, binary classification, and DROCC–OE on practically relevant metrics in wake-word and digit-classification experiments.The wake-word setup generates difficult negatives such as “Arvin” and “Marvelous” and emphasizes low false-positive rates on these examples.

5. Empirical Evaluation

DROCC is evaluated on two one-class problems across synthetic and real-world datasets spanning tabular data, time series, images, and audio. It consistently outperforms strong baselines, including on close-negative classification.

Experimental setup: Experiments cover anomaly detection and One-Class Classification with Limited Negatives across synthetic and real-world datasets.The evaluation includes tabular, time-series, image, and audio domains.
Synthetic data: DROCC accurately captures a one-dimensional sine-wave manifold, whereas OC-SVM and DeepSVDD learn poorly fitting minimum-enclosing-ball boundaries.The positive data lie on a 1-D manifold in the synthetic visualization.
Tabular data: 18% gains are observed for DROCC on the Arrhythmia tabular benchmark under F1-Score comparison with classical and deep baselines.The reported gain is the highest across the three tabular datasets discussed.
Image data: DROCC consistently achieves the best performance on most CIFAR-10 classes, with gains as high as 20% over DeepSVDD on some classes.Nearest Neighbour is competitive for Bird and Deer and beats the other baselines on those classes.
Image data: DROCC remains near 72% while GEOM drops from 87.8% to 65.8% AUC on augmented CIFAR-10 deer data with flips and ±30° rotations.The augmentation tests robustness when normal images vary in orientation.
Image data: On ImageNet, DROCC achieves the best results for every class except golf ball and is 23% more accurate than DeepSVDD on French-Horn versus rest.DROCC and DeepSVDD use MobileNetv2 for this experiment.
OCLN: 99.4% recall at 3% FPR is achieved by DROCC–LF on MNIST close-negative classification, improving on DROCC–OE and DeepSAD.DROCC–OE reaches 98.16% recall and DeepSAD 90.91%; BCE reaches 54%.
OCLN: 10% higher accuracy is reported for DROCC–LF than baselines at 3% FPR in wake-word detection, with a similar trend for both Marvin and Seven.The evaluation uses synthesized close negatives such as Mar, Vin, and Marvelous that were absent from training.

6. Conclusions

The paper introduces DROCC for deep anomaly detection and extends it to OCLN, where positives are well sampled and only a small number of negatives are available. Both methods outperform strong baselines, while simplified projection sets may slow convergence.

6. Conclusions: DROCC models normal data with a low-dimensional manifold and uses a saddle-point optimization solved by gradient descent-ascent.The method compares nearby points using Euclidean distance under the manifold assumption.
6. Conclusions: DROCC–LF extends DROCC to OCLN with well-sampled positives and limited negative examples.OCLN requires generalization against arbitrary negatives while maintaining low false-positive rates.
6. Conclusions: Both methods perform significantly better than strong baselines in their respective problem settings.
6. Conclusions: Simplifying the projection set improves computational efficiency but can slow convergence.The authors identify designing optimization algorithms for stricter projection sets as future work.

A.1. DROCC–LF Proof

The proof derives the DROCC–LF projection using KKT conditions and expresses the optimizer as a covariance-weighted displacement from the original point. The remaining lemma follows by feasibility and the diagonal structure of Σ.

A.1. DROCC–LF Proof: The proof considers mutually exclusive active constraints before analyzing the relevant constrained optimization case.The two constraints cannot be active simultaneously.
A.1. DROCC–LF Proof: The Lagrangian introduces a multiplier τ ≤ 0 for the selected active constraint.
A.1. DROCC–LF Proof: KKT first-order conditions yield ˜x = (I + τΣ)^-1(z + τ · Σx) = x + (I + τ · Σ)^-1δ, where δ = z − x.
A.1. DROCC–LF Proof: Primal and dual feasibility complete the bound needed for the lemma.Substitution of ˜x and the diagonal entries Σ(i, i) = σi gives the final result.

B. Synthetic Experiments

The synthetic experiments compare DROCC with classical and deep baselines on manifold-shaped data. They visualize decision boundaries, vary nearby negative sampling, and include a spherical-manifold setup.

B. Synthetic Experiments: The sine-wave experiment uses 1024 positive points sampled from a 1-D manifold and compares four decision boundaries.The methods are DROCC, OC-SVM with RBF and polynomial kernels, and DeepSVDD.
B. Synthetic Experiments: Negative points are sampled from vertically displaced versions of the sine manifold to evaluate performance near the positive data.Table 7 reports AUC values for these varied negative displacements.
B. Synthetic Experiments: DROCC–LF training combines normal batches, adversarial search, gradient updates, and projection steps.The algorithm initializes perturbations h, updates them using normalized gradients, and projects them according to Proposition 1.
B. Synthetic Experiments: The spherical experiment samples normal points uniformly from a unit-sphere volume and OOD points from sphere surfaces with varying radii.AUC is evaluated as the radius changes.

B.2. Spherical Manifold

The spherical-manifold experiment tests whether DROCC distinguishes points inside a unit sphere from OOD points sampled on surfaces at varying radii. Its evaluation focuses on AUC as OOD points approach the normal distribution.

B.2. Spherical Manifold: Positive points are sampled uniformly from the volume of a unit sphere, while OOD points are sampled from surfaces of varying radius.
B.2. Spherical Manifold: Table 6 reports average AUC for spherical-manifold experiments with varying OOD radii.
B.2. Spherical Manifold: DROCC outperforms all baselines when OOD points are close to the normal distribution.
B.2. Spherical Manifold: For the neural-network setting, DROCC achieves 100% AUC at radius 1.6 rather than at 1 + ϵ for very small ϵ.The learned boundary is essentially a minimum enclosing ball with a slightly larger radius.

C. LFOC Supplementary Experiments

The OCLN supplementary experiments compare methods on keyword detection under low false-positive rates and document synthesized near-negatives and experiment settings.

C. LFOC Supplementary Experiments: The supplied passages identify hyperparameter tables for the tabular and CIFAR-10 experiments.These are referenced as Table 10 and Table 11, respectively.
C. LFOC Supplementary Experiments: The experiments use synthesized near-negatives for each keyword.Table 9 lists the close negatives synthesized for the evaluated keywords.

D.1. Hyper-Parameters

The experiments analyze how DROCC’s radius, adversarial-negative loss weight, and negative-sampling strategy affect performance, with results emphasizing appropriate local sampling and gradient ascent-descent.

D.1. Hyper-Parameters: DROCC samples negative points at distance r from positive points, while µ weights the loss from adversarially generated negatives.Figure 8 studies the effect of µ on CIFAR-10 performance.
D.1. Hyper-Parameters: The supplied passages reference hyperparameter tables for ImageNet, time-series, and LFOC experiments.These are listed as Tables 12, 13, and 14, respectively.
D.1. Hyper-Parameters: Large sampling radii loosen the modeled manifold and increase false positives, whereas very small radii can place the boundary too close to positives and promote overfitting.The radius study varies r on CIFAR-10 with γ = 1.
D.1. Hyper-Parameters: DROCC-Rand replaces gradient ascent with random sampling in the negative set N_i(r).The formulation is introduced to analyze DROCC without the gradient-ascent component.
D.1. Hyper-Parameters: Random negative sampling reduces CIFAR-10 performance because high-dimensional sampling does not find points sufficiently close to the positive manifold.This result emphasizes the importance of gradient ascent-descent.

E. Experiment details and Hyper-Parameters for Reproducibility

The reproducibility details specify shared architectures and task-specific scoring setups for baseline comparisons across deep-learning experiments.

E. Experiment details and Hyper-Parameters for Reproducibility: Deep-learning baselines use a single fully connected layer with 128 units, while classical algorithms receive input features directly.Table 10 lists the associated hyperparameters.
E. Experiment details and Hyper-Parameters for Reproducibility: For CIFAR-10, DROCC and DeepSVDD use the same LeNet-based architecture, with DROCC adding a final fully connected layer for binary scores.DeepSVDD optimizes its one-class objective on penultimate-layer representations.
E. Experiment details and Hyper-Parameters for Reproducibility: For ImageNet, MobileNetv2 is the base architecture for both DeepSVDD and DROCC, using width multiplier 1.0.Both methods use penultimate-layer representations for their respective objectives.
E. Experiment details and Hyper-Parameters for Reproducibility: Time-series and Audio Commands experiments use a single-layer LSTM, with DROCC scoring a fully connected layer over the final hidden state.DeepSVDD instead optimizes its one-class objective on that final hidden state.
E. Experiment details and Hyper-Parameters for Reproducibility: Low-FPR Audio Commands experiments train on 1,000 keyword examples and 2,000 randomly sampled examples from other classes.Validation and test sets include keyword examples, other-class words, and 600 synthesized close negatives.