PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data

Guansong Pang, Anton van den Hengel, Chunhua Shen, Longbing Cao

arXiv:2009.06847v2cs.LGcs.AIcs.CRcs.CVstat.ML

TL;DR

Anomaly detection often has only a small set of labeled examples that may not cover all anomaly classes, alongside large unlabeled datasets. This paper uses deep reinforcement learning to learn from labeled anomalies while exploring unlabeled data, and reports that DPLAN outperforms five state-of-the-art competing methods across 48 datasets.

  • Problem

    Anomaly classes are diverse and rare, so labeled examples may not cover them all, while existing approaches either fit limited labeled anomalies or learn unsupervised from unlabeled data.

    Source: §1 · p. 1

  • Method

    The DRL approach interacts with an anomaly-biased simulation environment and combines labeled-anomaly rewards with novelty rewards to learn from labeled examples and explore unlabeled anomalies.

    Source: §1 · p. 2, §3.2.1 · p. 3

  • Results

    Across 48 datasets generated from four real-world datasets, DPLAN significantly and more stably outperforms five state-of-the-art weakly or unsupervised methods, with at least 7%-12% improvement in precision-recall rates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework jointly optimizes detection of known anomalies from labeled examples and unknown anomalies explored in unlabeled data.

    Source: §1 · p. 2

    The problem setting presumes a generic dataset without sequential decision processes, such as multidimensional data.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

We consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset. This is a common scenario in many important applications. Existing related methods either exclusively fit the limited anomaly examples that typically do not span the entire set of anomalies, or proceed with unsupervised learning from the unlabeled data. We propose here instead a deep reinforcement learning-based approach that enables an end-to-end optimization of the detection of both labeled and unlabeled anomalies. This approach learns the known abnormality by automatically interacting with an anomaly-biased simulation environment, while continuously extending the learned abnormality to novel classes of anomaly (i.e., unknown anomalies) by actively exploring possible anomalies in the unlabeled data. This is achieved by jointly optimizing the exploitation of the small labeled anomaly data and the exploration of the rare unlabeled anomalies. Extensive experiments on 48 real-world datasets show that our model significantly outperforms five state-of-the-art competing methods.

1 INTRODUCTION

The introduction frames anomaly detection with a small, incomplete set of labeled anomalies and large unlabeled data as a challenge of learning known and unknown anomalies together. It proposes a deep reinforcement learning approach and evaluates its DPLAN instantiation across varied anomaly scenarios.

1 INTRODUCTION

  • Anomalies are rare, unpredictable, and diverse, making labeled training data that covers every anomaly class difficult or impossible to obtain.The introduction notes that this makes fully supervised methods impractical.Source: §1 · p. 1
  • Unsupervised methods can detect diverse anomalies but may produce false positives, while semi- or weakly supervised methods fit limited labeled examples and can ignore unlabeled-anomaly signals.Pseudo-labeling unlabeled data can introduce false positives and decouples labeling from detection modeling.Source: §1 · pp. 1–2
  • The proposed DRL approach interacts with a simulation environment to learn from labeled anomalies while exploring unlabeled data for known and novel anomalies.An anomaly-biased environment and combined reward balance exploitation of labeled examples with exploration of possible anomalies.Source: §1 · p. 2
  • DPLAN instantiates the approach with an adapted deep Q-network, proximity-dependent observation sampling, and synthesized labeled-anomaly and isolation-based rewards.These components support joint optimization for detecting known and unknown anomalies.Source: §1 · p. 2
  • Across 48 datasets generated from four real-world datasets, DPLAN significantly and more stably outperforms five state-of-the-art weakly or unsupervised methods, with at least 7%-12% improvement in precision-recall rates.The scenarios vary known-abnormality coverage and anomaly contamination rates.Source: §1 · p. 2

2 RELATED WORK

Related work spans unsupervised and weakly supervised anomaly detection, positive-unlabeled learning, active anomaly detection, and reinforcement learning applications. The paper distinguishes its setting by combining partially labeled anomalies with automatic exploration of unlabeled data without requiring sequential input or human feedback.

2 RELATED WORK

  • Unsupervised methods can detect diverse anomalies but may struggle with intricate or high-dimensional data and mismatched anomaly definitions; weakly supervised methods use labeled anomalies but risk overfitting them.Weakly supervised examples include label propagation and end-to-end feature learning.Source: §2 · p. 2
  • Unlike positive-unlabeled learning, this problem allows anomalies from different manifolds; unlike active anomaly detection, it explores unlabeled data without human feedback.The paper also distinguishes its non-sequential setting and predefined reward functions from inverse-reinforcement-learning anomaly detection.Source: §2 · p. 2

3 THE PROPOSED APPROACH

The approach seeks anomaly scores that rank known and unknown anomalies above normal instances in a generic dataset without sequential decision processes. It casts joint detection as a DRL task, using an anomaly-biased environment and combined rewards to learn from labeled examples and explore unlabeled data.

3.1 Problem Statement

  • The scoring objective ranks either known or unknown anomalies above normal instances in a dataset containing a small labeled set and mostly normal unlabeled data.The labeled set spans k known anomaly classes; the unlabeled set may contain anomalies from known and unknown classes.Source: §3.1 · p. 3
  • The problem assumes a generic dataset without sequential decision processes, such as multidimensional data.Source: §3.1 · p. 3

3.2 DRL Tailored for Anomaly Detection

  • The DRL approach uses an anomaly-detection agent to exploit labeled anomaly signals while exploring possible known and unknown anomalies in unlabeled data.The framework comprises agent A, intrinsic reward f, and an anomaly-biased environment E containing sampler g and labeled-anomaly reward h.Source: §3.2 · p. 3, §3.2.1 · p. 3, Figure 1 · p. 3

3.2.1 Foundation of the Proposed Approach.

  • Each data instance is an observation, the agent chooses normal or anomalous, and the environment samples the next observation with a bias toward anomalies.The agent is implemented as a neural network that selects between the two actions.Source: §3.2.1 · p. 3

3.2.2 Procedure of Our DRL-based Anomaly Detection.

  • A combined reward pairs high external reward for correctly recognizing labeled anomalies with intrinsic reward for discovering novel unlabeled observations.The agent iterates through observations and rewards over episodes to maximize cumulative reward and detect known and unknown anomalies.Source: §3.2.1 · p. 3, §3.2.2 · pp. 3–4

4 THE INSTANTIATED MODEL: DPLAN

DPLAN instantiates anomaly detection as a DQN agent interacting with a simulation environment, using proximity-based sampling and combined rewards to learn from labeled anomalies and unlabeled data.

4.1 DQN-based Anomaly Detection Agent 𝐴

  • The DQN agent learns an action-value function from discounted rewards and uses the estimated value of labeling an observation anomalous as its anomaly score.At inference, Q(ŝ, a1; θ*) is used as the anomaly score.Source: §4.1 · p. 4, §4.4 · p. 5

4.2 Proximity-driven Observation Sampling 𝑔

  • The simulator samples labeled anomalies uniformly or selects unlabeled observations by proximity, choosing nearest neighbors after a1 and farthest neighbors after a0.The proximity is measured in the agent’s feature-embedding space; the two sampling functions are each used with probability 0.5.Source: §4.2 · p. 4

4.3 Combining External and Intrinsic Rewards

  • The combined reward pairs labeled-data feedback, which rewards correctly detected known anomalies and penalizes false positives and negatives, with an iForest-based signal for exploring unlabeled anomalies.With the external reward alone, DPLAN leveraged limited anomaly examples better than existing semi-supervised methods.Source: §4.3 · pp. 4–5

4.4 Theoretical Analysis of DPLAN

  • When iForest captures abnormality and the agent sufficiently approximates Q, the expected action-a1 values rank labeled anomalies above unlabeled anomalies and unlabeled normals.The paper presents this ordering as the rationale for using larger Q(ŝ, a1; θ*) values to identify anomalies of interest.Source: §4.4 · p. 5

5 EXPERIMENTS

Experiments use four real-world datasets to create 48 datasets spanning different anomaly-coverage and contamination scenarios. DPLAN is compared with five competing detectors using AUC-PR and AUC-ROC.

5.1 Datasets

  • Four datasets from different domains form the base pool for 48 scenarios; the one-class-known setup uses 60 labeled anomalies and 2% unlabeled-data contamination.The four datasets contain two to seven anomaly classes; test sets contain one known and one to six unknown classes.Source: §5.1 · p. 5, §5.4.1 · p. 6

5.2 Competing Methods and Their Settings

  • The five competitors include anomaly-informed DevNet and Deep SAD, unsupervised REPEN and iForest, and DUA, a DevNet variant trained with iForest-selected pseudo-anomalies.Deep methods use one hidden layer with 20 units by default; the reported DUA result uses a 0.05% pseudo-anomaly threshold.Source: §5.2 · pp. 5–6

5.3 Performance Evaluation Measures

  • AUC-ROC and AUC-PR are averaged over 10 runs, with paired Wilcoxon signed-rank tests across datasets used to assess significance.AUC-PR focuses on anomaly-class precision and recall, while AUC-ROC can present an overoptimistic view of detection performance.Source: §5.3 · p. 6

5.4.2 Scenario II: Increasing the Number of Known Anomaly Classes.

  • As known anomaly classes increase, DPLAN’s AUC-PR rises from 0.438-0.768 to 0.826-0.853 across datasets, with maximal relative improvement of 91%.This experiment uses the seven NB15 datasets; increasing known-class coverage adds supervision, particularly when the first known class generalizes poorly.Source: §5.4.2 · p. 6

5.4.1 Scenario I: Known Anomalies from One Class.

  • DPLAN achieves the best performance on 10 of 13 datasets in both metrics, with AUC-PR gains of 7% over DevNet, 11% over Deep SAD, and 12% over DUA.Its AUC-PR improvement is significant at the 99% confidence level; AUC-ROC improvement is significant at least at the 90% level.Source: §5.4.1 · p. 6, Table 2 · p. 7

5.4.3 Tolerance to Increasing Anomaly Pollution.

  • Across contamination rates from 2% to 12%, DPLAN’s superiority over the three competing methods is consistent, though performance trends vary by dataset.DPLAN, DevNet, and Deep SAD remain stable on some datasets but decline on others; DUA fluctuates substantially.Source: §5.4.3 · p. 7

5.4.4 Comparison Summary.

  • Against DevNet and Deep SAD, DPLAN’s agent-environment interactions support learning beyond the labeled anomaly examples; it achieves 7%-11% higher AUC-PR and more stable performance across scenarios.The comparison covers Table 2 and Figures 2 and 3.Source: §5.4.4 · p. 7
  • DPLAN improves on unsupervised detectors by nearly 100%, while its joint optimization outperforms DUA’s decoupled pseudo-labeling approach across most datasets.DUA improves DevNet on a few datasets but degrades and fluctuates on most, especially in AUC-PR; unsupervised methods can still do better in some cases.Source: §5.4.4 · p. 7

5.5 Empirical Analysis of DPLAN

The ablations favor DPLAN’s anomaly-biased environment and combined rewards, while longer training and larger representations help selectively. Its training is slower than competing methods, but test-time scoring is fast.

5.5.1 Ablation Study.

  • Over REnv, Org gains more than 47% average AUC-PR and 17% average AUC-ROC; DQN+ also improves over Org.DQN+’s improvement is reported for settings with large unlabeled datasets, including seven UNSW_NB15 and two Covertype datasets.Source: Table 3 · p. 8, §5.5.1 · p. 8
  • Org outperforms ERew on most datasets, especially in AUC-ROC at the 90% confidence level, supporting the contribution of intrinsic reward.ERew alone also significantly outperforms DevNet and Deep SAD in the comparison reported in Table 2.Source: §5.5.1 · p. 8

5.5.2 Learning with More Training Steps.

  • AUC-PR and episode reward often converge around 20,000 training steps, but 100,000 steps raise AUC-PR from 0.490 to 0.604 on Hypothyroid and 0.436 to 0.863 on Subnormal.Those later gains are 23% and 98% over 20,000 steps, respectively; performance may move oppositely on Fuzzers, Generic, and Reconnaissance, so 20,000 steps is generally recommended.Source: Figure 4 · p. 8, §5.5.2 · p. 8
  • DPLAN performs stably across representation sizes 20, 40, 80, 160, and 320, while 10 dimensions is less effective; Subnormal improves as dimensionality increases.The authors suggest the usable supervisory information may become bounded, with Subnormal as an exception where performance continues to improve.Source: §5.5.3 · pp. 8–9, Figure 5 · p. 9

5.5.4 Computational Efficiency.

  • Training takes 15–120 seconds on most datasets and scales linearly with training steps, while one-pass scoring takes under three seconds across the test instances.Training is often slower than competing methods; online scoring is faster than REPEN and iForest, which take about 20 and 40 seconds, respectively.Source: §5.5.4 · p. 9

6 CONCLUSIONS

The conclusion presents DPLAN as a deep reinforcement-learning approach that uses labeled anomalies to learn known abnormality while exploring unlabeled data for unknown anomalies.

6 CONCLUSIONS

  • DPLAN combines labeled-anomaly external reward with active exploration of unlabeled data to refine learned abnormality and improve generalization.The conclusion reports 23%–98% further AUC-PR improvement over the default model on some datasets with more training steps, and describes inference as computationally efficient to scale.Source: §6 · p. 9

A.1 Data Accessing and Preprocessing

The appendix describes four real-world datasets and their anomaly definitions, along with preprocessing and selected downsampling procedures.

A.1 Data Accessing and Preprocessing

  • The four datasets cover network attacks, thyroid disease, human activities, and forest-cover types, with different classes designated as anomalies in each.For UNSW_NB15, attacks are anomalies against normal network flows; Annthyroid uses hypothyroid and subnormal patients, HAR uses downstairs and upstairs activities, and Covertype uses cottonwood and douglas-fir against lodgepole pine.Source: §A.1 · p. 10
  • Selected anomaly classes are randomly downsampled to preserve rarity, with UNSW_NB15 attack classes set to 3,000 instances each and downstairs/upstairs set to 2% of the dataset.The paper says the attack-class downsampling supports class balance for fair evaluation across anomaly classes.Source: §A.1 · p. 10
  • Preprocessing one-hot encodes categorical features, replaces missing values with feature means, and normalizes all features to [0, 1].Source: §A.1 · p. 10

A.2 The Algorithm of DPLAN

DPLAN trains an action-value agent by sampling observations from labeled and unlabeled data, combining rewards, and updating Q-values. At detection time, the action-value determines anomaly scores.

A.2 The Algorithm of DPLAN

  • Training initializes action-value functions and an experience set, then runs episodes of fixed-length observation sequences beginning with a random unlabeled observation.Each step chooses randomly with probability epsilon, otherwise selecting the action with the largest current action-value.Source: §A.2 · p. 10
  • The environment returns a labeled anomaly with probability p; otherwise it samples an unlabeled observation using the proximity-dependent function, and supplies an external reward.An intrinsic reward is also calculated, and the two rewards are combined for each observation, whether labeled or unlabeled.Source: §A.2 · p. 10
  • DPLAN stores experience records, samples minibatches, and performs gradient descent on the Q-value target, periodically updating target parameters and the target action-value function.Source: §A.2 · p. 10
  • At test time, one forward pass estimates action-values for each observation, and the maximum action-value is returned as its anomaly score.Selecting action a1 with a large value indicates an anomaly; a0 with a small value indicates otherwise.Source: §A.2 · pp. 10–11

A.3 Implementation Details

The implementation uses Python across the compared methods and balances labeled or pseudo-labeled anomalies against normal instances in mini-batches through oversampling.

A.3 Implementation Details

  • All methods are implemented in Python, with DevNet and REPEN taken from their authors and iForest from scikit-learn.DevNet, DUA, REPEN, and Deep SAD use Keras with TensorFlow as the backend.Source: §A.3.1 · p. 11
  • Oversampling gives DevNet, DUA, REPEN, and Deep SAD the same proportion of labeled or pseudo anomalies and pseudo normal instances in each mini-batch.The passage identifies this as a common way to address class imbalance.Source: §A.3.1 · p. 11
  • DevNet and REPEN use author-provided code, while iForest comes from scikit-learn.Source: §A.3.1 · p. 11

A.3.1 Algorithm Implementation.

DPLAN builds on Keras-rl and OpenAI Gym, with a default shallow network and task-specific adaptations to DQN training settings.

A.3.1 Algorithm Implementation

  • DPLAN uses a Keras-rl deep Q-network implementation and an anomaly-biased simulation environment implemented under OpenAI Gym.Source: §A.3.1 · p. 11
  • By default, DPLAN, DevNet, Deep SAD, DUA, and REPEN use one hidden layer with 20 ReLU units.DQN+ adds two hidden layers of 500 and 100 ReLU units, each followed by dropout at rate 0.9.Source: §A.3.2 · p. 11
  • DPLAN trains for 20,000 steps, including 10,000 warm-up steps, and updates its target network every 10,000 steps.Episodes last 2,000 steps; epsilon-greedy exploration is annealed from 1.0 to 0.1 during training and is not used in evaluation.Source: §A.3.2 · p. 11
  • The experience replay memory holds 100,000 entries, and other DQN parameters follow the original defaults, with key settings shown in Table 5.The listed package versions include gym 0.12.5, keras 2.3.1, keras-rl 0.4.2, and tensorflow 1.14.0; runtime results use an Intel Core i7-8700 CPU and 16 GB RAM.Source: §A.3.2 · p. 11, Table 5 · p. 11, §A.3.1 · p. 11
Loading 2009.06847v2…