PaperlayerView exact revision on arXiv

Source-linked AI summary

An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series

Astha Garg, Wenyu Zhang, Jules Samaran, Savitha Ramasamy, Chuan-Sheng Foo

arXiv:2109.11428v1cs.LGcs.AIstat.ML

TL;DR

Existing metrics and comparisons for multivariate time-series anomaly detection and diagnosis are insufficiently systematic and can be misleading. This paper evaluates models and scoring functions independently, proposes the composite F-score, and identifies dynamic scoring with a channel-wise autoencoder as a strong candidate.

  • Problem

    Multivariate time-series monitoring lacks systematic evidence about appropriate scoring functions and their importance relative to model choice.

  • Method

    The study evaluates deep-learning algorithms for anomaly detection and independent diagnosis across seven multivariate time-series datasets, varying models and scoring functions independently.

  • Results

    The composite F-score addresses shortcomings in existing event-wise metrics, while dynamic scoring functions and a Univariate Fully-Connected Auto-Encoder perform strongly across detection and diagnosis.

  • Takeaways & Limitations

    The findings provide guidance for designing and evaluating multivariate time-series anomaly detection and diagnosis methods.

  • Takeaways & Limitations

    UAE may perform less well on systems with multiple operating states, and its performance may partly reflect the prevalence of temporal anomalies.

Abstract

from arXiv · show

Several techniques for multivariate time series anomaly detection have been proposed recently, but a systematic comparison on a common set of datasets and metrics is lacking. This paper presents a systematic and comprehensive evaluation of unsupervised and semi-supervised deep-learning based methods for anomaly detection and diagnosis on multivariate time series data from cyberphysical systems. Unlike previous works, we vary the model and post-processing of model errors, i.e. the scoring functions independently of each other, through a grid of 10 models and 4 scoring functions, comparing these variants to state of the art methods. In time-series anomaly detection, detecting anomalous events is more important than detecting individual anomalous time-points. Through experiments, we find that the existing evaluation metrics either do not take events into account, or cannot distinguish between a good detector and trivial detectors, such as a random or an all-positive detector. We propose a new metric to overcome these drawbacks, namely, the composite F-score ($Fc_1$), for evaluating time-series anomaly detection. Our study highlights that dynamic scoring functions work much better than static ones for multivariate time series anomaly detection, and the choice of scoring functions often matters more than the choice of the underlying model. We also find that a simple, channel-wise model - the Univariate Fully-Connected Auto-Encoder, with the dynamic Gaussian scoring function emerges as a winning candidate for both anomaly detection and diagnosis, beating state of the art algorithms.

I. INTRODUCTION

The paper systematically evaluates deep-learning methods for multivariate time-series anomaly detection and diagnosis, focusing on how models, scoring functions, and metrics affect performance. It finds that dynamic scoring and a simple channel-wise autoencoder are especially effective, while existing metrics can be misleading.

  • Study scope: The study evaluates semi-supervised and unsupervised deep-learning methods for anomaly detection and diagnosis on real-world multivariate CPS data.The evaluation targets streaming anomaly detection and channel-level diagnosis under limited anomalous-operation labels.
  • Evaluation design: The authors vary 10 models and 4 scoring functions independently to characterize their effects and compare the resulting combinations with existing algorithms.This modular evaluation separates the contribution of the underlying model from post-processing choices.
  • Main findings: Dynamic scoring functions outperform static scoring functions overall and may matter more than the underlying model for anomaly-detection performance.The paper presents dynamic scoring as a previously unexplored design choice for deep MVTS anomaly detection.
  • Main findings: The Univariate Fully-Connected Autoencoder with dynamic scoring outperforms other algorithms overall on both anomaly detection and diagnosis.UAE trains independent fully connected autoencoders for individual channels.
  • Evaluation metrics: Existing metrics either ignore anomalous events or are insufficiently robust, motivating the composite F1 score, Fc_1.The point-adjusted F1 score can assign a score of 0.96 to a detector making random predictions on one dataset.

III. PROBLEM SETUP

The problem setup defines streaming anomaly detection and independent channel-level diagnosis for multivariate time series, then places reconstruction or prediction methods within a modular scoring and thresholding framework. The evaluation uses publicly available CPS datasets with predominantly temporal anomalies.

  • Problem setup: Anomaly detection predicts whether each observed test time-point is anomalous using training multivariate time series assumed to be anomaly-free.The setup simulates streaming by restricting predictions at time t to data observed through t.
  • Problem setup: Anomaly diagnosis predicts the channel or channels responsible for each anomalous event, without implying that these channels are the system’s root causes.The ground-truth event start and end times are assumed known for this independent diagnosis task.
  • Datasets: The study uses seven publicly available multivariate time-series datasets from real-world cyberphysical systems, with regular sampling and temporal and cross-channel structure.The datasets include multi-entity and single-entity systems, with separate models trained per entity for multi-entity datasets.
  • Datasets: The tested datasets primarily contain temporal anomalies, while open datasets with known strictly cross-channel anomalies were not found.Multiple-operating-state systems may contain more cross-channel anomalies than the steady-state CPS datasets evaluated here.
  • Modular framework: The modular framework combines a reconstruction or prediction model, a scoring function, and a thresholding function.The model produces errors; scoring transforms and aggregates channel-level information into time-point anomaly scores, which thresholding converts into binary labels.

VI. ALGORITHMS

The evaluation treats model, scoring function, and thresholding function as separate components, comparing algorithms across these design choices. It includes autoencoder, forecasting, adversarial, probabilistic, and baseline approaches.

  • VI. ALGORITHMS: The framework separates each algorithm into a model, scoring function, and thresholding function for independent comparison.The study varies scoring functions for selected models and evaluates other models with predefined scoring functions.
  • VI. ALGORITHMS: Raw Signal reconstructs every signal as 0, while PCA performs lossy reconstruction using components explaining 90% of the variance.These provide simple reconstruction baselines for comparison.
  • 2) Generic normality feature learning models:: The evaluated set includes UAE, fully connected, LSTM, TCN, and variational autoencoders, alongside BeatGAN, NASA LSTM, DAGMM, OC-SVM, OCAN, and OmniAnomaly.The models cover reconstruction, forecasting, adversarial, shallow, and probabilistic normality-learning approaches.
  • 2) Generic normality feature learning models:: UAE trains a separate fully connected autoencoder for each channel, whereas FC AE concatenates channels to capture cross-channel relationships.LSTM AE, TCN AE, and LSTM VAE provide additional sequence and latent-variable architectures.

1) Anomaly measure-dependent feature learning models:

The study evaluates several normality-learning models and four scoring functions, including static and dynamic Gaussian approaches. Dynamic scoring adapts channel statistics during testing to accommodate long-term changes.

  • C. Scoring functions: The scoring-function comparison includes normalized errors, static Gaussian scoring, dynamic Gaussian scoring, and Gaussian-kernel smoothing of dynamic scores.The functions post-process model errors into anomaly scores, with smoothing applied after Gauss-D.
  • C. Scoring functions: Normalized errors aggregate channel-wise reconstruction errors after subtracting each channel’s mean training reconstruction error.This adjustment accounts for differences in training error across channels before the root-mean-square aggregation.
  • C. Scoring functions: Gauss-S fits a Gaussian distribution to training errors and uses a cumulative-distribution-based score rather than −log pdf, avoiding high scores at both distribution tails.The channel-wise parameters are the empirical mean and standard deviation, and Φ denotes the standard normal cdf.
  • C. Scoring functions: Gauss-D replaces Gauss-S’s static mean and variance with dynamic estimates to adapt to long-term changes during testing.The dynamic estimates use a window of size W, with training values prepended to initialize the test sequence.
  • C. Scoring functions: Gauss-D-K convolves the Gauss-D channel-wise scores with a Gaussian kernel before aggregation.Smoothing can amplify total anomaly scores when multiple channels respond at slightly different times.

D. Thresholding functions

The evaluation compares three thresholding strategies while emphasizing event-aware anomaly metrics. The proposed composite F-score combines time-wise precision with event-wise recall to better reflect practical detection goals.

  • D. Thresholding functions: The study evaluates best-F-score, top-k, and tail-p thresholding functions without assuming an anomalous validation set.Best-F-score uses test labels and is therefore not applicable in practice; top-k is non-streaming, while tail-p supports streaming for Gaussian scores.
  • D. Thresholding functions: Best-F-score selects the static threshold maximizing F1, point-adjusted F1, or Fc_1 for a given anomaly score.It provides an upper limit for static-threshold performance but requires test-set labels.
  • D. Thresholding functions: Top-k labels exactly k time-points as anomalous, using the actual anomaly count, but requires scores for the full test set before thresholding.A separate top-k threshold is set for each entity.
  • D. Thresholding functions: Tail-p labels probability scores below ε as anomalous and can operate in streaming settings for Gauss-S, Gauss-D, and Gauss-D-K.For multivariate scores, the threshold is −m log10(ε), with a single-channel formulation for SMAP and MSL.
  • A. Composite F-score: Fc_1 combines time-wise precision with event-wise recall, rewarding detected events while penalizing false-positive time-points.An event counts as detected when at least one of its time-points is a true positive.

B. Comparison of metrics for anomaly detection

The comparison shows that existing F-score metrics reward different labeling behaviors, while Fc1 better distinguishes desirable event detections from spurious or trivial predictions.

  • Metric behavior: F1 can be too pessimistic, whereas Fpa1 can be too optimistic when evaluating anomaly predictions.The metrics differ because they reward different balances of false positives, true positives, and event overlap.
  • Metric behavior: Fc1 is the only compared metric that gives low scores to undesirable predictions and higher scores to desirable predictions while identifying the ideal detector.In the synthetic case, Fc1 scores undesirable predictions below 0.6 and correctly ranks the ideal detector.
  • Metric behavior: Fpa1 assigns RAD scores above 0.6 on every dataset, even though RAD randomly labels time-points as anomalous.On WADI, RAD reaches Fpa1 = 0.9613 while labeling only 0.17% of time-points anomalous.
  • Diagnosis metrics: Anomaly diagnosis evaluation uses HitRate@150 and RC-top-3, assuming anomalous time-points are known but their causes are unknown.HitRate@150 rewards identifying all true causes, while RC-top-3 rewards identifying at least one true cause among the top three.

IX. RESULTS

Results show that scoring functions substantially affect Fc1 performance, often more than model choice, with dynamic scoring generally outperforming static scoring. UAE leads across datasets, although threshold choice and scoring-function assumptions shape the comparisons.

  • Effect of scoring function: Both model and scoring-function choices significantly affect Fc1 performance across datasets.Fig. 3 varies models and scoring functions under the top-k threshold to expose these effects.
  • Effect of scoring function: For DMDS, SMD, SWaT, and SKAB, scoring-function choice generally matters more than model choice.For UAE on SWaT, Fc1 ranges from approximately 0.1–0.6 across scoring functions, while Gauss-D-K yields approximately 0.4–0.6 across most models.
  • Effect of scoring function: Dynamic scoring functions outperform static scoring functions overall by adapting to changing normal behavior during testing.Dynamic scoring can also adapt to test-set anomalies; Gauss-D performs worse than Gauss-S on SMAP when event length is comparable to the window size.
  • Effect of scoring function: Gauss-D-K performs better overall than Gauss-D by amplifying scores from multiple channels responding to the same anomaly at slightly different times.Its performance is sensitive to σk, which the study sets empirically through a non-rigorous search.
  • Model comparison: UAE is the best-performing model on five of seven datasets and leads overall by mean Fc1 and average rank.Its advantage over Raw Signal, DAGMM, OCAN, OmniAnomaly, NASA LSTM, and MSCRED is statistically significant, while some remaining comparisons are not.
  • Model comparison: UAE’s channel-wise models may help detect temporal anomalies by retaining per-channel information before aggregating scores.FC AE is second by average rank, while FC AE and TCN AE outperform several RNN-based methods.
  • Threshold comparison: Every dataset and algorithm has time-wise precision below 0.5 in the top-k comparison, despite high event-wise recall for the best algorithms.Best-F-score and streaming thresholds generally produce higher precision or higher Fc1 than the top-k threshold.

B. Anomaly Diagnosis Results

Independent anomaly diagnosis evaluates whether channel-level anomaly scores rank the true causes highly. UAE performs strongly, while dataset characteristics and limited statistical power constrain some comparisons.

  • Diagnosis performance: RC-top-3 scores of ∼0.95 for DMDS and SMD indicate that ranking channels by anomaly score can support independent diagnosis.SWaT and WADI score lower because non-causal channels may also be affected and diagnosed as causes.
  • Scoring functions: Gauss-D has the highest average scoring-function rank, 2.0, but differences between scoring functions are statistically insignificant.HitRate@150 shows similar trends across datasets, models, and scoring functions.
  • Model comparison: UAE is the top model with an average RC-top-3 rank of 1.5 when evaluated with Gauss-D.Comparable patterns appear for HitRate@150.
  • Cross-task consistency: Anomaly detection and diagnosis results are consistent: methods ranking anomaly scores well across time-points also rank channel scores well before aggregation.The overall model comparison for diagnosis is not statistically significant, likely because many comparisons use only four datasets.
  • Implications: The evaluation identifies UAE with dynamic scoring as a strong candidate for both anomaly detection and diagnosis, while multiple operating conditions remain an open scope boundary.The study calls for more challenging CPS datasets containing temporal and cross-channel anomalies.

A. SWaT

The evaluation spans heterogeneous CPS datasets, including water-treatment, water-distribution, industrial-actuator, spacecraft, and server-machine data. UAE models each channel independently, whereas FC AE models all channels jointly.

  • SWaT: SWaT contains 51 sensor and actuator channels, including 14 pump signals constant during training but allowed to change during testing.Its initial seven days provide normal-operation training data.
  • Related CPS datasets: WADI is a scaled-down water-distribution network connected to the SWaT plant and organized into three PLC-controlled processes.The processes are the primary grid, secondary grid, and return-water grid.
  • Related CPS datasets: DAMADICS uses real process data with induced faults from three benchmark actuators at the Lublin Sugar Factory.The study uses the publicly available real dataset rather than the simulator.
  • Related CPS datasets: SMD contains five weeks of data from 28 server entities sampled every minute, with interpretation labels for anomaly diagnosis.The authors use a 50% train-test split.
  • Compared models: UAE trains a separate autoencoder for each channel, while FC AE uses one model over a flattened multichannel subsequence.LSTM AE instead uses principal components explaining 90% of variance as input.

S4. HYPERPARAMETERS AND IMPLEMENTATION

The supplementary materials document training, algorithmic comparisons, and implementation details for the evaluation. They also show that published-score comparisons depend on metric and dataset alignment.

  • Implementation: Deep methods other than OmniAnomaly and OCAN train for at most 100 epochs with early stopping based on held-out validation error.Patience is set to 10, and hyperparameters are tuned on SWaT only.
  • Implementation: UAE is the third-slowest model to train, but its independent channel-wise models are easily parallelizable; FC AE is the second-fastest.The timing comparison uses single-entity datasets and one RTX 2080 Ti GPU.
  • Published comparisons: UAE attains a WADI point-wise F1 of 0.4740 with Gauss-D-K, improving 46.5% over MAD-GAN, while LSTM VAE reaches 0.5025.The reported comparison uses the best-F-score threshold.
  • Published comparisons: UAE is reported as the top algorithm in point-adjusted F1 comparisons on SMAP, MSL, and SMD, although the study does not use that metric for its main comparisons.The comparison uses UAE with a Gauss-D threshold against published OmniAnomaly and USAD results.
  • Published comparisons: SWaT comparisons with published point-wise F1 scores are not directly comparable because this study shortens a long anomaly and therefore uses a different test set.Some tested algorithms exceed literature F1 scores but are not top-performing by Fc1.

S7. STATISTICAL TESTS

Statistical tests support differences among scoring functions and some model comparisons, but not every apparent ranking difference is significant. The analysis uses Friedman tests followed by Hochberg post-hoc comparisons.

  • Scoring-function tests: Friedman testing rejects equal scoring-function performance across 70 dataset-model combinations: statistic 57.13, p-value 2.4e −12.Hochberg post-hoc tests find Gauss-D-K significantly different from all other methods.
  • Model tests: Friedman testing rejects equal algorithm performance across seven datasets and 13 methods: statistic 43.53, p-value 1.83e −5.Only comparisons between UAE and the bottom six algorithms are statistically significant in post-hoc testing.
  • Evaluation setup: The scoring-function comparison uses Fc1 means and standard deviations over five seeds with the top-k threshold.Gauss-D is used unless an algorithm is marked with its own scoring function.
  • Evaluation setup: The supplementary figures compare precision-recall behavior and Fc1 effects under top-k or best-Fc1 thresholds across scoring functions and models.These visualizations complement the tabulated scoring-function comparisons.

C. Fc1 score with tail-p threshold

This section reports Fc1 results using tail-p thresholding, with specified exceptions for certain algorithms. Supplementary comparisons also relate Fc1 to other metrics and examine scoring-function effects across algorithms and datasets.

  • C. Fc1 score with tail-p threshold: Table S9 reports mean Fc1 scores and standard deviations over five seeds, averaged across datasets, using the tail-p threshold and Gauss-D scoring function.The selected threshold is the value producing the best Fc1 among −log10(ϵ) values from 1 to 5.
  • C. Fc1 score with tail-p threshold: NASA LSTM NPT uses a non-parametric threshold, while OC-SVM scores are thresholded at 0.5 instead of using the tail-p threshold.
  • C. Fc1 score with tail-p threshold: Tables S10 and S11 report point-wise F1 scores with Gauss-D and Gauss-D-K scoring functions, respectively, using the best-F1 threshold unless algorithms specify their own scoring functions.
  • C. Fc1 score with tail-p threshold: Tables S12–S15 compare AU-ROC and AU-PRC across models using Gauss-D or Gauss-D-K scoring functions, while Figure S9 compares average algorithm ranks from Fc1 and AU-ROC.Figure S9 states that the two metrics are generally in agreement because scatter points lie close to the x=y line.
  • C. Fc1 score with tail-p threshold: Tables S16 and S17 report root-cause diagnosis results using RC-Top3-all and Hitrate@150, and Figure S10 examines scoring-function effects on Hitrate@150.
Loading 2109.11428v1…