PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Encoding with AutoEncoders for Weakly-supervised Anomaly Detection

Yingjie Zhou, Xucheng Song, Yanru Zhang, Fanxing Liu, Ce Zhu, Lingqiao Liu

arXiv:2105.10500v3cs.LGcs.NI

TL;DR

Weakly-supervised anomaly detection must learn from few labeled anomalies and abundant normal data, making direct discriminative training difficult. The paper uses an autoencoder to encode samples with latent representation, residual direction, and reconstruction error, and designs a detector around these factors. Experiments report superior performance over competitive methods, including strong gains over unsupervised detection with few labels.

  • Problem

    Few annotated anomalies and abundant normal data make weakly-supervised anomaly detection difficult for directly trained discriminative models.

  • Method

    An autoencoder provides hidden representation, reconstruction residual vector, and reconstruction error, which an anomaly detection model incorporates for scoring.

  • Results

    The proposed method achieves superior performance over competitive methods and surpasses DAGMM by more than 30.39% in AUC-PR and 16.66% in AUC-ROC when using 30 labels.

  • Takeaways & Limitations

    The encoding strategy can effectively leverage limited labeled abnormal data together with normal data for anomaly detection.

Abstract

from arXiv · show

Weakly-supervised anomaly detection aims at learning an anomaly detector from a limited amount of labeled data and abundant unlabeled data. Recent works build deep neural networks for anomaly detection by discriminatively mapping the normal samples and abnormal samples to different regions in the feature space or fitting different distributions. However, due to the limited number of annotated anomaly samples, directly training networks with the discriminative loss may not be sufficient. To overcome this issue, this paper proposes a novel strategy to transform the input data into a more meaningful representation that could be used for anomaly detection. Specifically, we leverage an autoencoder to encode the input data and utilize three factors, hidden representation, reconstruction residual vector, and reconstruction error, as the new representation for the input data. This representation amounts to encode a test sample with its projection on the training data manifold, its direction to its projection and its distance to its projection. In addition to this encoding, we also propose a novel network architecture to seamlessly incorporate those three factors. From our extensive experiments, the benefits of the proposed strategy are clearly demonstrated by its superior performance over the competitive methods.

I. INTRODUCTION

The paper addresses anomaly detection when annotated anomalies are scarce but normal data are abundant. It proposes autoencoder-based feature encoding and a model that incorporates the resulting information for weakly-supervised detection.

  • Few annotated anomalies make detection challenging despite the availability of abundant normal data.
  • The proposed strategy uses an autoencoder to derive hidden representation, reconstruction residual, and reconstruction error features.These factors characterize a sample’s projection onto the normal-data manifold, its residual direction, and its distance from that manifold.
  • An anomaly detector is designed to incorporate the encoded representation, including reconstruction error throughout its subnetwork.
  • The paper introduces a novel input-encoding strategy intended to provide a more effective feature representation for anomaly detection.
  • Experiments on various datasets and ablation studies evaluate the method and its component contributions.

II. RELATED WORKS

Related work spans supervised, unsupervised, and weakly supervised anomaly detection. Supervised methods depend on sufficient, balanced anomaly labels; unsupervised methods learn from unlabeled data but can suffer from distribution and label-assignment errors, motivating methods that exploit limited labeled anomalies.

  • Supervised anomaly detection performs well with sufficient, balanced annotated abnormal and normal data, but anomaly data are difficult to collect and usually rare.
  • Unsupervised methods learn feature representations and prior knowledge from unlabeled samples, including spatial-temporal features and nonlinear subspace structure.
  • Unsupervised approaches may degrade when modeled data-distribution assumptions fail or assign incorrect labels, which may require labeled anomalous samples for correction.
  • Weakly supervised approaches incorporate limited anomalous information through mechanisms such as analyst feedback to improve anomaly scores under extremely unbalanced data.
  • DevNet uses limited labeled anomalies, deviation-based score distributions, and end-to-end anomaly-score learning, whereas the present framework focuses on a different representation strategy.

III. PROPOSED MODEL

The proposed model uses an autoencoder-based feature encoder and an anomaly score generator. Its representation combines latent coordinates, normalized residual direction, and reconstruction distance to describe how samples deviate from the normal-data manifold.

  • A. System Overview: The model contains a feature encoder and an anomaly score generator that jointly support anomaly scoring from encoded representations.The encoder transforms inputs into three factors, while the generator calculates an anomaly score.
  • B. Feature Encoder: The autoencoder extracts hidden representation, reconstruction residual vector, and reconstruction error as the input feature representation.
  • B. Feature Encoder: The reconstruction error measures the distance between an input and its reconstruction, and the residual vector is normalized so it represents residual direction.Reconstruction error is also commonly used as an anomaly indicator.
  • B. Feature Encoder: The three factors uniquely characterize an input, allowing anomalies to deviate through coordinates, reconstruction errors, residual directions, or combinations of these.
  • B. Feature Encoder: Geometrically, the hidden representation gives intrinsic manifold coordinates, while reconstruction approximates a test point’s projection onto the normal-data manifold.

C. Anomaly Score Generator

The anomaly score generator maps the autoencoder’s three-factor representation into an anomaly score. Its architecture explicitly preserves the one-dimensional reconstruction error throughout the MLP so that it is not overwhelmed by higher-dimensional features.

  • The generator produces an anomaly score from the autoencoder-derived feature representation, with normal scores encouraged toward a prior distribution and anomaly scores deviating from it.
  • The representation combines hidden features, reconstruction residuals, and reconstruction error before being mapped to a scalar score.
  • The proposed MLP adds reconstruction error as an extra dimension at every layer to emphasize its role in anomaly detection.
  • The first MLP layer receives the residual and hidden representations, while the final layer outputs the anomaly score s0.

D. Objective Function

The objective jointly trains the feature encoder and anomaly score generator by reconstructing normal data while making reconstruction errors and anomaly scores discriminate anomalies from normal samples.

  • Training jointly minimizes autoencoder reconstruction error and encourages discriminative anomaly scores for separating anomalies from normal samples.
  • The network contains a feature encoding autoencoder and an anomaly score generator that consumes hidden representation, residual vector, and reconstruction error.
  • The reconstruction loss minimizes expected error while encouraging anomaly reconstruction errors to exceed a predefined threshold.
  • The score loss keeps normal-sample scores near 0 and pushes outlier scores away from 0 by a predefined margin.
  • The joint objective combines the component losses using a hyper-parameter λ that balances their contributions.
  • Because normal annotations are unavailable, all unlabeled training samples are treated as normal, relying on anomalies being rare in real-world data.

E. Training procedure

Training uses a two-stage procedure: autoencoder pre-training on unlabeled data followed by end-to-end optimization with unlabeled samples and a few labeled anomalies.

  • Mini-batches contain equal numbers of abnormal and normal samples, so abnormal samples are over-sampled during stochastic-gradient training.
  • The method first pre-trains the feature encoding network using reconstruction loss on all unlabeled training samples.
  • The second stage jointly optimizes the complete network using all unlabeled data and a few annotated anomalies, initialized from pre-training.
  • Algorithm 1 specifies random initialization, repeated loss computation and parameter updates, and termination upon convergence for both stages.

IV. EXPERIMENTS

The experiments evaluate the proposed method on datasets using performance metrics, report its main results, and analyze the effects of its components through ablation studies.

  • The evaluation covers datasets, performance metrics, main algorithm results, and ablative analyses of the method’s components.

A. Datasets Description

The study evaluates anomaly detectors on eight datasets spanning network security, finance, medical diagnosis, and other application scenarios, using standardized preprocessing and AUC-based metrics. It compares the proposed method with weakly supervised, unsupervised, and supervised baselines under a limited-label setting.

  • Datasets Description: Eight real-world datasets cover network security, finance fraud detection, medical diagnosis, and related application scenarios.The datasets include NSL-KDD, Spambase, Arrhythmia, Fraud, Cardio, Shuttle, Satellite, and Mammography.
  • Datasets Description: Each dataset is preprocessed by mean imputation, one-hot encoding of categorical attributes, and normalization of every attribute into [0, 1].
  • Performance Metrics: AUC-ROC and AUC-PR evaluate detector performance, with higher values indicating better performance.AUC-PR places more emphasis on anomalies, and results are averaged over ten independent runs with standard deviations.
  • Experiment Design: The comparison includes DevNet, Deep SAD, DAGMM, and FCN, representing weakly supervised, unsupervised, and supervised detection approaches.FCN directly uses labeled anomalies as positives and unlabeled data as negatives.
  • Experiment Results: The proposed method achieves the best average performance, improving AUC-ROC over DevNet, Deep SAD, DAGMM, and FCN by 1.6%, 1.1%, 15.4%, and 10.7%, respectively.Its corresponding AUC-PR improvements are 3.0%, 5.4%, 30.1%, and 14.1%.

D. Sample Efficiency

The proposed method is evaluated with varying numbers of labeled anomalies and shows strong sample efficiency, especially on AUC-PR with few labels. It also outperforms fully supervised and unsupervised comparators using the same limited labels.

  • The experiment varies the number of labeled anomalies among 30, 60, 90, and 120; Arrhythmia is excluded because too few abnormal samples are available.
  • The proposed method demonstrates significantly better sample efficiency than competing methods on AUC-PR, while its AUC-ROC performance is comparably well.
  • With 30 or 60 labeled anomalies, the proposed method is best on AUC-PR across datasets, while its AUC-ROC is comparable to competitors’ best performance.
  • Compared with FCN using the same number of labeled anomalies, the proposed method consistently achieves superior performance on both AUC-ROC and AUC-PR.
  • With 30 labels, the proposed method surpasses DAGMM by more than 30.39% in AUC-PR and 16.66% in AUC-ROC on average.

E. Ablative Analysis

Ablation studies examine the encoding strategy and whether its gains arise from extra encoder parameters. The results support using all three factors and indicate that the gains persist beyond parameter-count differences.

  • The ablations evaluate combinations of the three encoding factors, the objective function, hyper-parameters, pre-training, and reconstruction-error strength.
  • The three factors are indispensable: using all three yields 1.5%-3.1% higher AUC-ROC and 1.4%-6.8% higher AUC-PR than single- or two-factor variants.
  • The second ablation compares the proposed encoder with an encoder that outputs only the reconstructed vector while retaining the same network structure.
  • The proposed method improves average performance by 3.2% in AUC-ROC and 3.0% in AUC-PR, showing gains are not due to additional network parameters.

2) Effect of Having Reconstruction Error Regularization Term on Anomaly Score Generator:

Further ablations test reconstruction-error regularization, hyper-parameter settings, and pre-training. Reconstruction-error regularization and pre-training improve performance, while the discriminative hyper-parameter has a dataset-consistent preferred setting.

  • Removing reconstruction-error minimization from the objective causes average performance drops of 2.3% in both AUC-ROC and AUC-PR.
  • The reconstruction-error regularization term plays a crucial role in the anomaly score generator.
  • The model performs better when a0 is near a specific value; values that are too small reduce discrimination, while values that are too large complicate training, with 5 used consistently.
  • Compared with omitting pre-training, pre-training improves average AUC-ROC by 3.1% and average AUC-PR by 4.6%.

5) Effect of Feeding Reconstruction Error to Each Layer of Anomaly Score Generator:

The study evaluates whether reconstruction error should be supplied to every layer of the anomaly score generator. Feeding it throughout the generator preserves information and improves performance over concatenating it only at the first layer.

  • The proposed design feeds reconstruction error e to each layer of the anomaly score generator.A variant instead concatenates e with the other two factors only at the first layer.
  • 1.5% and 1.4% average performance drops occur on AUC-ROC and AUC-PR, respectively, when using the variant strategy.The comparison is reported against the model using the original design.
  • The result demonstrates that feeding reconstruction error to each layer avoids losing information carried by the reconstruction error.
  • The broader model combines a feature encoder and anomaly score generator, jointly optimized with a specially designed deviation loss.The paper reports that extensive experiments show superior performance over competitive methods.
Loading 2105.10500v3…