Source-linked AI summary
Deep Weakly-supervised Anomaly Detection
Guansong Pang, Chunhua Shen, Huidong Jin, Anton van den Hengel
TL;DR
Semi-supervised anomaly detectors trained on a few labeled anomalies can struggle to generalize beyond those examples. PReNet predicts relations between sampled training instances to learn pairwise patterns, and it outperforms nine competing methods across 12 real-world datasets for detecting seen and unseen anomalies.
Problem
Semi-supervised methods often fit the small set of labeled anomaly examples and may not generalize to new anomaly types unseen during training.
Source: §1 · p. 1Method
PReNet learns pairwise relation features and anomaly scores by predicting whether sampled training instances form anomaly-anomaly, anomaly-unlabeled, or unlabeled-unlabeled pairs.
Source: §1 · p. 2, §3.1.2 · p. 3Results
Takeaways & Limitations
Pairwise patterns let PReNet target abnormalities that fit learned abnormal patterns or deviate from normal patterns, while augmenting training anomaly data from limited labeled examples.
Source: §1 · p. 2The pairwise interpretation relies on unlabeled data being typically dominated by normal instances, although the authors report robustness to noisy pairs from anomaly contamination.
Source: §3.2.1 · p. 3
Abstract
from arXiv · showhide
Recent semi-supervised anomaly detection methods that are trained using small labeled anomaly examples and large unlabeled data (mostly normal data) have shown largely improved performance over unsupervised methods. However, these methods often focus on fitting abnormalities illustrated by the given anomaly examples only (i.e.,, seen anomalies), and consequently they fail to generalize to those that are not, i.e., new types/classes of anomaly unseen during training. To detect both seen and unseen anomalies, we introduce a novel deep weakly-supervised approach, namely Pairwise Relation prediction Network (PReNet), that learns pairwise relation features and anomaly scores by predicting the relation of any two randomly sampled training instances, in which the pairwise relation can be anomaly-anomaly, anomaly-unlabeled, or unlabeled-unlabeled. Since unlabeled instances are mostly normal, the relation prediction enforces a joint learning of anomaly-anomaly, anomaly-normal, and normal-normal pairwise discriminative patterns, respectively. PReNet can then detect any seen/unseen abnormalities that fit the learned pairwise abnormal patterns, or deviate from the normal patterns. Further, this pairwise approach also seamlessly and significantly augments the training anomaly data. Empirical results on 12 real-world datasets show that PReNet significantly outperforms nine competing methods in detecting seen and unseen anomalies. We also theoretically and empirically justify the robustness of our model w.r.t. anomaly contamination in the unlabeled data. The code is available at https://github.com/mala-lab/PReNet.
1 INTRODUCTION
Anomaly detection methods face a trade-off between avoiding costly labels and learning what anomalies of interest look like. PReNet uses pairwise relations to address the limited generalization of methods trained on a small set of seen anomalies.
1 INTRODUCTION
- Semi-supervised methods use a small labeled anomaly set, but fitting its examples can leave new anomaly types undetected and limits seen-anomaly performance when training anomaly data are scarce.The paper identifies zeroday attacks and novel defects or planet surfaces as examples of unseen anomalies [34] [8] [31].Source: §1 · p. 1
- The paper frames weakly supervised anomaly detection as detecting both seen and unseen anomalies from an incomplete illustration of anomaly classes.It describes this setting alternatively as open-set supervised anomaly detection.Source: §1 · p. 1
- PReNet predicts whether sampled instance pairs are anomaly-anomaly, anomaly-unlabeled, or unlabeled-unlabeled, then scores test instances by their pair relations with random training instances.Pairs that fit the first two relation types or deviate from the last are treated as anomalous.Source: §1 · p. 2
- Because unlabeled data are often mostly normal, the three pair types support joint learning of anomaly-anomaly, anomaly-normal, and normal-normal patterns rather than fitting seen abnormalities alone.Pairwise labels also generate surrogate anomaly-informed training labels, extending training anomaly data with limited labeled examples.Source: §1 · p. 2
- PReNet significantly outperforms nine state-of-the-art methods on seen and unseen anomalies across 12 real-world datasets, with 50%-87.5% less labeled anomaly data needed for comparable or better performance.The paper also reports substantially better sample efficiency.Source: §1 · p. 2
2 RELATED WORK
Prior work spans unsupervised and semi-supervised anomaly detection, including approaches that use labeled normal or anomaly instances. The paper distinguishes its pairwise relation approach from methods focused on fitting specific examples or optimizing representations for particular anomaly measures.
2 RELATED WORK
- Earlier semi-supervised methods use labeled normal instances to learn normal-class patterns, while newer methods use small labeled anomaly sets through propagation, representation learning, classification, or loss functions [9].DevNet and Deep SAD fit a Gaussian prior-driven anomaly-score distribution and a one-class hypersphere, respectively [31] [33] [25].Source: §2 · p. 2
- Weakly supervised and open-set supervised AD target seen and unseen anomalies, addressing the tendency of anomaly-example methods to overfit without regularization or large, diverse anomaly samples [16] [31] [34] [8] [31] [33] [25].The paper follows this line with pairwise relation learning.Source: §2 · p. 2
- Few-shot and positive-unlabeled learning share the setting of limited labeled positive instances, but assume same-class examples share intrinsic structure, unlike seen and unseen anomalies, which may have different structures.Source: §2 · p. 2
- Deep AD methods use pretrained, autoencoder-, or GAN-based representations, while some newer methods tailor representations to anomaly measures; PReNet instead unifies representation learning and anomaly scoring, and differs from DevNet's deviation loss [12] [22] [9] [31] [33].The paper also notes that many deep methods focus on unsupervised or exclusively normal training data.Source: §2 · p. 2
- The paper's three-way ordinal relation prediction differs from unsupervised AD using self-trained ordinal regression [35]; anomaly-contamination estimation may provide a prior for weakly supervised AD [37].Source: §2 · p. 2
3 THE PROPOSED APPROACH
PReNet learns to rank instances by predicting ordinal relations among randomly paired labeled anomalies and unlabeled data. These pairings expand the training examples and jointly model abnormal and normal patterns.
3.1.1 Problem Statement.
- The method frames anomaly detection as learning a scoring function that ranks both seen and unseen anomalies above normal instances.The training data comprises a small labeled anomaly set and a larger unlabeled set.Source: §3.1.1 · p. 3
3.1 Overview of Our Approach PReNet
- PReNet learns pairwise relations among anomaly–anomaly, anomaly–unlabeled, and unlabeled–unlabeled pairs to score seen and unseen anomalies against normal instances.The goal is to assign higher scores to anomalies, whether seen or unseen, than to normal instances.Source: §3.1.2 · p. 3, §3.1.1 · p. 3
3.1.2 Anomaly-informed Pairwise Relation Prediction.
- Ordinal labels rank anomaly–anomaly pairs above anomaly–unlabeled pairs, which in turn rank above unlabeled–unlabeled pairs.End-to-end learning uses these three pair types to learn pairwise anomaly scores.Source: §3.1.2 · p. 3
3.2 The Instantiated Model
- PReNet combines random instance pairing with pairwise anomaly-score learning, using anomaly and unlabeled instances to form a labeled pair dataset.Pairs receive ordered target values c1 > c2 > c3 ≥ 0 for the three pair types.Source: §3.2.1 · p. 3
3.2.2 Pairwise Relation-based Anomaly Score Learning.
- A shared-weight two-stream network maps each instance to a representation, then learns a simple linear pairwise relation and anomaly score from the paired features.The paper reports no clear advantage from adding nonlinear layers to the pairwise relation in this setting.Source: §3.2.2 · pp. 3–4
3.3 Anomaly Detection Using PReNet
- At inference, PReNet pairs a test instance with randomly sampled anomaly and unlabeled instances and ensembles the resulting pair scores.The model is reported to perform stably with sufficiently large E; E = 30 is the default.Source: §3.3 · p. 4
4 THEORETICAL ANALYSIS
Pairwise augmentation jointly learns patterns of abnormality, normality, and their interactions. The analysis relates unlabeled-data contamination to expected pair composition and anomaly-score ordering.
4.1 Pairwise Relation Feature Learning
- Pairing expands the training data theoretically from N + K instances to (N + K)^2 examples, while shared weights keep feature learning in the original data space without extra model complexity.The three pair types support joint learning of anomaly–anomaly, anomaly–normal, and normal–normal patterns rather than fitting seen abnormalities alone.Source: §4.1 · p. 4
4.2 Robust Anomaly Score Learning
- With default pair-label values, expected true-anomaly scores exceed normal-instance scores when contamination is below 2/3; the analysis also suggests good regression performance at contamination ≤ 2.5%.The ≤ 2.5% estimate invokes tolerance to about 5% outliers.Source: §4.2 · p. 5
5 EXPERIMENTS
The experiments evaluate PReNet on seen and unseen anomalies, then examine its robustness and design choices. Across both detection settings, PReNet compares favorably with competing methods, including when trained with fewer labeled anomalies.
5.1 Datasets
- Evaluation uses 12 real-world datasets for seen-anomaly detection and 28 datasets for unseen-anomaly detection, each holding out one attack class for testing while training uses anomalies from the remaining three.For seen-anomaly evaluation, each dataset is stratified into 80% training and 20% holdout-test data; unlabeled training data may contain randomly selected anomalies.Source: §5.1 · p. 5
5.2 Competing Methods and Their Settings
- PReNet is compared with semi-supervised, few-shot, and unsupervised detectors; results average 10 runs, with paired Wilcoxon tests for significance.The deep baselines use one hidden layer with 20 units; additional comparisons with XGBOD, PUMAD, and FEAWAD are in the appendix.Source: §5.2 · p. 5, §5.3 · p. 5
5.3 Performance Evaluation Metrics
- The evaluation reports AUC-ROC and AUC-PR, with AUC-PR emphasizing precision and recall for the anomaly class.Reported results are averaged over 10 independent runs, and paired Wilcoxon signed-rank tests assess significance.Source: §5.3 · p. 5
5.4 Detection of Seen Anomalies
- Across 12 datasets, PReNet’s AUC-PR average gains were DevNet 3.7%, DSAD 19.6%, FSNet 54.0%, cFSNet 59.1%, REPEN 114.2%, and iForest 331.85%.On multiple datasets, PReNet matched or exceeded DevNet using 50%-75% fewer labeled anomalies, and outperformed DSAD using 87.5% fewer.Source: §5.4 · p. 6, Table 2 · p. 7
5.6 Further Analysis of PReNet
- AUC-PR is generally robust to margin values, including m=0.25, and ensemble size; larger λ values lower scores on some datasets, and λ=0.01 is generally recommended.Increasing ensemble size often offers only marginal improvement.Source: §5.6 · pp. 7–9
5.5 Detection of Unseen Anomalies
- Across 28 unseen-class datasets, PReNet’s AUC-PR average gains were DevNet >11%, DSAD 17%, FSNet 30%, cFSNet 20%, and REPEN 27%.It remained the most sample-efficient method, performing substantially better than the best competitors with 50%-87.5% fewer labeled examples.Source: §5.5 · p. 6, Table 3 · p. 8
5.6 Further Analysis of PReNet
- With |A|=60, PReNet is generally stable below 10% unlabeled contamination, except news20, whose over one million features may require improved relation learning.The tested contamination rates were 0%, 2%, 5%, and 10%.Source: §5.6 · p. 6, Figure 4 · p. 9
- Three-way relation modeling beats two-way TRM by 5.6% on average; removing the feature layer loses over 10% average AUC-PR, while added depth often fails and nonlinear learning often overfits.PReNet is compared with four ablated variants in Table 4.Source: §5.6 · pp. 6–7, Table 4 · p. 9
6 CONCLUSIONS
PReNet learns pairwise relations to detect seen and unseen anomalies, and substantially improves over nine competing methods on 12 real-world datasets. Its precision-recall performance in unseen anomaly detection improves by 10%-30%.
6 CONCLUSIONS
- PReNet substantially outperforms nine competing methods on 12 real-world datasets, with 10%-30% improved precision-recall performance in unseen anomaly detection.The paper notes that improving this metric is especially challenging for unseen anomalies.Source: §6 · p. 9
- The results suggest labeled anomaly data can support accurate anomaly detection regardless of its scale or coverage of anomaly classes.The authors plan to explore supervision from different data domains in future work.Source: §6 · p. 9
A PROOF OF THEOREM
The proof compares expected pairwise anomaly scores for anomalous and normal instances. The theorem states a guarantee under anomaly contamination and ordered pair-relation labels.
A PROOF OF THEOREM
- Theorem A.1 states that, for a given test instance xk, one expectation is guaranteed to exceed another under anomaly contamination rate ε≥0 and pair-label values y{a,a}=c1, y{a,u}=c2, y{u,u}=c3 with c1>c2>c3≥0.Source: A PROOF OF THEOREM · pp. 9–10
- The proof derives the expected pairwise anomaly scores for true anomalies and normal instances from the regression model, then invokes the scoring function in Eqn. (6).The supplied proof passages identify the two expectations and the scoring-function step.Source: A PROOF OF THEOREM · pp. 9–10
B.1 Datasets
The evaluation uses 12 anomaly-detection datasets spanning financial, census, image, intrusion, web, marketing, text, and disease data.
B.1 Datasets
- The 12 datasets cover donation-project excitement, high-income census records, credit-card fraud, celebrity attributes, four intrusion attacks, web pages, marketing campaigns, text, and thyroid disease.Anomalies include rare high-income persons, fraudulent transactions, bald celebrities, and hypothyroid patients.Source: §B.1.1 · p. 10
- Anomaly prevalence ranges from 0.2% for fraud to 15.0% for DoS attacks across the datasets.The passage reports individual anomaly proportions, including 13.1% for reconnaissance, 3.1% for fuzzers, and 2.4% for backdoors.Source: §B.1.1 · p. 10
- Seven datasets contain real anomalies, while five contain semantically real anomalies that are rare and differ substantially from the majority.The seven real-anomaly datasets are donors, fraud, dos, rec, fuz, bac, and thyroid.Source: §B.1.1 · p. 10
B.1.1 Seen Anomaly Detection Datasets.
The appendix describes seen- and unseen-anomaly dataset construction and the implementation and training settings used for PReNet and comparison methods.
B.1.1 Seen Anomaly Detection Datasets.
- Seen-dataset training uses an 80% training split, combines randomly selected anomalies with normal training instances in the unlabeled set, and samples a limited labeled anomaly set.The remaining 20% is held out for testing.Source: §B.1.1 · p. 10
- The unseen-anomaly evaluation holds out one of four intrusion classes, trains with combinations of the other three, and tests on the held-out class plus held-out normal instances.The four attack datasets yield 28 seen/unseen permutations; datasets with only one anomaly class cannot serve this evaluation.Source: §B.1.2 · p. 10
B.2 Implementation Details
- PReNet is implemented with TensorFlow/Keras and uses a one-hidden-layer MLP with 20 units, ReLU, ℓ2 regularization at λ=0.01, and RMSprop at learning rate 0.001.The package list includes TensorFlow 1.14.0 and Keras 2.3.1; the authors report one-hidden-layer architectures performed better and more stably than deeper ones.Source: §B.2.1 · p. 10, §B.2.2 · pp. 10–11
B.2.2 Hyperparameter Settings.
- Deep detectors train for 50 epochs with 20 batches per epoch; default batch sizes are 512 for PReNet, DevNet, and DSAD, and 256 for FSNet and REPEN.The comparison methods also oversample the labeled anomaly set.Source: §B.2.2 · p. 11
C.1 Comparison to Three More Methods
PReNet outperforms XGBOD, PUMAD, and FEAWAD on most datasets, especially relative to FEAWAD; the comparison reports AUC-PR results for seen anomaly-detection datasets.
C.1 Comparison to Three More Methods
- The comparison evaluates all three methods on seen and unseen anomaly-detection datasets, with results reported in Tables 6 and 7.The XGBOD and FEAWAD official implementations were used; PUMAD was implemented by the authors using metric learning similar to REPEN.Source: §C.1 · p. 11
- For XGBOD, the authors used one-class SVM and iForest to generate features, excluding kNN and LOF because of computational cost on large datasets.Other XGBOD settings remained unchanged.Source: §C.1 · p. 11
- PReNet substantially outperforms XGBOD, PUMAD, and FEAWAD on most datasets, with FEAWAD especially underperforming.Table 6 reports AUC-PR results for seen anomaly-detection datasets.Source: §C.1 · p. 11, Table 6 · p. 11
C.2 Performance Ranking of All Methods
PReNet ranks best across seen and unseen anomaly-detection settings, with stronger post-hoc significance against competitors in the unseen setting.
C.2 Performance Ranking of All Methods
- The ranking compares methods across seen and unseen AD; ordinal rank one or percentile 1.00 represents perfect performance, and lower ranks or higher percentiles are better.Table 8 presents the average ordinal and percentile ranks based on AUC-PR.Source: §C.2 · p. 11, Table 8 · p. 11
- 94.2% of cases on 12 seen datasets and 90.2% on 28 unseen datasets favor PReNet over all nine competitors, making it the best performer in both settings.Table 8 summarizes average ordinal and percentile ranks based on AUC-PR across the seen and unseen datasets.Source: §C.2 · p. 11, Table 8 · p. 11
- At the 95% confidence level, PReNet significantly outperforms all nine methods in unseen AD, but not DevNet or DSAD in seen AD.The comparison uses a Conover post-hoc test.Source: §C.2 · p. 11