PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Rigorous Evaluation of Time-series Anomaly Detection

Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, Sungroh Yoon

arXiv:2109.05257v2cs.LGcs.AIstat.ME

TL;DR

The paper investigates whether reported TAD improvements are properly evaluated, focusing on point adjustment and the lack of baseline comparisons. It combines theoretical and experimental analysis, finding severe PA overestimation and limited advancement over an untrained-model baseline, then proposes a new protocol and baseline.

  • Problem

    Current TAD studies often report high F1 after point adjustment, but the paper questions whether these evaluations properly establish meaningful improvements.

  • Method

    The paper theoretically and experimentally examines PA, compares existing methods with an untrained-model baseline, and proposes a new evaluation protocol.

  • Results

    PA can severely overestimate TAD capability, while without PA existing methods show no or mostly insignificant improvement over the proposed baseline.

  • Takeaways & Limitations

    Rigorous TAD evaluation should use the proposed baseline and PA%K protocol alongside existing metrics.

  • Takeaways & Limitations

    Without PA, F1 can unexpectedly underestimate detection capability when incompletely labeled anomalies share statistics with normal signals.

Abstract

from arXiv · show

In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets, giving the impression of clear improvements in TAD. However, most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; that is, even a random anomaly score can easily turn into a state-of-the-art TAD method. Therefore, the comparison of TAD methods after applying the PA protocol can lead to misguided rankings. Furthermore, we question the potential of existing TAD methods by showing that an untrained model obtains comparable detection performance to the existing methods even when PA is forbidden. Based on our findings, we propose a new baseline and an evaluation protocol. We expect that our study will help a rigorous evaluation of TAD and lead to further improvement in future researches.

1 Introduction

The paper argues that common TAD evaluations can overstate progress because point adjustment makes random and informative anomaly scores appear equivalent. It proposes a new baseline and evaluation protocol for more rigorous comparisons.

  • Most TAD studies report high F1 scores after applying point adjustment (PA), creating an impression of clear improvements.PA is widely used before scoring, and some studies report only F1PA.
  • PA counts an entire contiguous anomaly segment as correctly detected when at least one moment in that segment is flagged.Scores computed after this adjustment are denoted F1PA, whereas scores without PA are denoted F1.
  • PA can make predictions from random and well-trained anomaly scores identical, thereby greatly overestimating detection performance.The paper reports that random anomaly scores can overturn most state-of-the-art TAD methods under this evaluation.
  • Without PA, existing methods show no or mostly insignificant improvement over the proposed baseline.The baseline is based on detection performance from an untrained model, including a randomly initialized reconstruction model with simple architecture.
  • The paper proposes a new baseline and an evaluation protocol to support rigorous evaluation of TAD.Its contributions specifically address PA’s overestimation and the absence of baseline comparisons.

2 Background

TAD covers several anomaly types in multivariate sensor data and commonly uses unsupervised reconstruction-based methods trained on normal signals. Prior work also questions whether benchmark datasets and evaluation practices adequately support reliable comparisons.

  • 2.1 Types of anomaly in time-series signals: TAD datasets contain contextual, collective, and point anomalies, with point anomalies currently dominant.Contextual anomalies differ in signal shape, collective anomalies accumulate small noise over time, and point anomalies are temporary large deviations.
  • Unsupervised TAD methods learn shared patterns from normal signals and assign higher anomaly scores to more abnormal inputs.This setting assumes that only normal data are available during training.
  • Reconstruction-based anomaly detection trains a model to minimize the distance between a normal input and its reconstruction.Anomalous test inputs are expected to produce different reconstruction behavior and consequently different anomaly scores.
  • Related work identifies limitations in benchmark datasets and shows that simple detectors can suffice for some datasets.Other work also proposes anomaly taxonomies and synthetic datasets, whereas this paper focuses on PA overestimation and missing baseline comparisons.

3 Pitfalls of the TAD evaluation

The paper formalizes TAD scoring and shows that point adjustment can inflate performance, while untrained reconstruction models can still produce informative anomaly scores without training.

  • 3.1 Problem formulation: TAD predicts anomaly labels for test windows by thresholding anomaly scores computed from time-series inputs.The formulation uses normalized windows and compares anomaly scores A(w_t) with threshold δ.
  • 3.1 Problem formulation: Reconstruction-based scoring can use mean squared error between an input window and its reconstructed version.The reconstruction model output is denoted by ˆw_t = f_θ(w_t).
  • 3.1 Problem formulation: Point adjustment labels an entire anomaly segment as correctly detected when any point in that segment exceeds the threshold.The adjusted labeling scheme sets ˆy_t to 1 for all t in S_m after at least one threshold exceedance within the segment.
  • 3.2 Random anomaly score with high F1PA: PA increases true positives and decreases false negatives while preserving false positives, so precision, recall, and F1 can only increase after adjustment.This mechanism explains why adjusted scores can overestimate detection capability.
  • 3.2 Random anomaly score with high F1PA: For uniform random anomaly scores, F1PA can approach 1 as the threshold changes when anomaly segments are sufficiently long.The result holds across typical anomaly ratios and segment lengths except when anomaly segments are short.
  • 3.3 Untrained model with comparably high F1: Untrained reconstruction models can yield anomaly scores proportional to input-window magnitude, producing comparable F1 to current TAD methods for point anomalies.Randomly initialized outputs are close to zero, while abrupt sensor increases make anomalous windows have larger Euclidean magnitude; longer windows further increase F1.

4 Towards a rigorous evaluation of TAD

The paper proposes more rigorous TAD evaluation by combining an untrained-model baseline with PA%K, which balances PA’s overestimation against possible underestimation by F1 without PA.

  • 4.1 New baseline for TAD: The proposed baseline measures F1 from predictions of a randomly initialized reconstruction model, such as an untrained single-layer LSTM autoencoder.The input itself can alternatively serve as the anomaly score when the model consistently outputs zero.
  • 4.2 New evaluation protocol PA%K: F1 without PA avoids PA’s overestimation but can underestimate detection capability when incompletely labeled anomalies resemble normal signals.The SWaT t-SNE visualization shows some ground-truth anomalies closer to normal data than to other anomalies.
  • 4.2 New evaluation protocol PA%K: PA%K applies point adjustment to an anomaly segment only when the fraction of correctly detected anomalies exceeds threshold K.It is intended for use alongside existing TAD metrics rather than as their replacement.
  • 4.2 New evaluation protocol PA%K: K can be selected manually from 0 to 100 according to prior knowledge about test-label reliability, or performance can be summarized by the area under the F1PA%K curve.A larger K is allowable when test-set labels are reliable.

5 Experimental results

Experiments compare established TAD methods with dataset statistics, correlation analyses, and simple baselines. The results show that PA can make random scores appear highly effective, while longer input windows and baseline comparisons materially affect unadjusted F1.

  • 5.1 Benchmark TAD datasets: The evaluation covers five benchmark datasets, including SWaT, WADI, and SMD, and seven representative TAD methods.SMD results are averaged across 28 machine-specific models.
  • 5.3 Correlation between F1PA and F1: For SWaT, F1PA and F1 have PCC -0.59 and KRC 0.07; for WADI, they have PCC 0.41 and KRC 0.43.These correlations do not assure that F1PA reliably reflects detection performance or supports valid method comparisons.
  • 5.4 Comparison results: A random anomaly score has low F1 but can achieve state-of-the-art F1PA on most datasets, sometimes far exceeding existing methods.For MSL and SMAP, low F1 alongside high F1PA makes successful detection indistinguishable from random scoring when only PA-based results are reported.
  • 5.4 Comparison results: PA’s overestimation depends on dataset distribution, becoming less conspicuous for SMD’s shorter anomaly segments and lower anomaly ratio.SMD anomaly segments average 90 points, compared with hundreds to thousands in other datasets; the maximum achievable F1PA in the corresponding case is approximately 0.8.
  • 5.4 Comparison results: Across datasets, existing methods’ F1 is mostly inferior to the Case 2 and Case 3 baselines, with only GDN consistently exceeding them.Case 2 uses the input itself as an anomaly score, while Case 3 uses a randomized model’s anomaly score.
  • 5.4 Comparison results: As window size τ increases, F1 generally rises after a short decrease, then may saturate or decline when windows become too large.Longer windows are more likely to contain point anomalies, but can also cause nominal windows to include anomalies unexpectedly.

6 Discussion

The discussion identifies PA-based overestimation and missing baseline comparisons as central evaluation pitfalls, then recommends PA%K, untrained-model baselines, and threshold-robust metrics.

  • PA can overestimate detection performance, so higher F1PA does not necessarily indicate better detection capability.
  • PA%K varies between F1PA at K = 0 and F1 at K = 100, providing an alternative that can reduce overestimation without modifying datasets.The motivation for PA originates from incomplete test-dataset labeling, while fine-grained relabeling may be infeasible.
  • TAD methods should report improvement relative to an untrained-model baseline rather than only comparing absolute performance with existing methods.The window size should be selected carefully because it affects baseline performance.
  • Threshold selection strongly affects detection results because thresholds are often chosen after inspecting the test dataset or optimizing F1.AUROC and AUPR are suggested as metrics with reduced threshold dependence.

7 Conclusion

The conclusion reports that PA can severely overestimate TAD capability and presents a new baseline, PA%K protocol, and directions for more rigorous evaluation.

  • PA can severely overestimate a TAD model’s capability and may not reflect its true modeling performance.
  • The paper proposes a new TAD baseline and reports that only a few methods have achieved significant advancement relative to it.
  • PA%K is proposed to mitigate PA overestimation while remaining applicable with existing metrics.
  • The paper suggests baseline selection and other directions for evaluating TAD methods more rigorously.

A1 Relaxation of assumptions

The appendix relaxes the single-segment and uniform-score assumptions, showing that PA can still yield F1PA close to 1 with multiple segments and Gaussian anomaly scores.

  • A1.1 Multiple anomaly segments: Multiple anomaly segments do not prevent F1PA from approaching 1 under the generalized analysis.
  • A1.1 Multiple anomaly segments: When γ = 0.05 and M = 6, F1PA approaches 1 if the average anomaly-segment length is sufficiently large.
  • A1.2 Gaussian anomaly scores: The analysis extends to Gaussian anomaly scores A(wt) ∼ N(0, σ), where 0 < σ << 1.
  • A1.2 Gaussian anomaly scores: For σ = 0.02, F1PA also approaches 1 when the average anomaly-window length is sufficiently large.Gaussian-distributed anomaly scores correspond to an untrained model initialized with the same distribution.

A2 Related works

The related-work section distinguishes reconstruction, forecasting, and normal-distribution modeling approaches to TAD, with reconstruction and forecasting methods using prediction errors as anomaly scores.

  • Reconstruction-based AD trains models to minimize the distance between normal inputs and reconstructions, using reconstruction error as the anomaly score.Architectures include autoencoders and GANs, with varied distance metrics.
  • Forecasting-based AD predicts future signal values and uses the distance between predictions and ground truth as the anomaly score.LSTM and gated recurrent unit models are examples of forecasting architectures.
  • Other approaches model the normal-data distribution using one-class classification, hierarchical temporal memory, Bayesian networks, and graph neural networks.
  • Figure 8 examines F1PA for Gaussian anomaly scores with γ = 0.05 and M = 6.

A3 Experimental details

The experiments used published code for reimplementations and specified hardware/software settings, while implementing the untrained model as a simple single-layer LSTM encoder-decoder that remained untrained after initialization.

  • Computational environment: The experimental environment included Intel Xeon Gold 6242R CPUs, NVIDIA RTX GPUs, PyTorch, TensorFlow, and CUDA 11.0.MSCRED, THOC, and GDN used NVIDIA GeForce RTX 2080Ti GPUs; other methods used Quadro RTX 8000 hardware.
  • Reimplementation and preprocessing: The experiments used public code from the original authors to reproduce existing methods.Downsampling was omitted in the main manuscript because it could modify the test-data distribution.
  • Hyperparameter settings: Hyperparameter searches covered model-specific settings including context-vector size, latent-vector dimensions, hidden units, clustering, embedding, output-layer, and adjacent-attribute choices.The listed ranges included context vector sizes [10, 50, 100], latent dimensions [15, 30, 50] and [3, 5, 10], embedding dimensions [64, 128, 256], and adjacent attributes [5, 15].
  • Untrained-model architecture: The study implemented Case 3 with a simple untrained encoder-decoder consisting of single-layer LSTM encoder and decoder components.The architecture used hidden-unit dimension 25 and context dimension 25.
  • Untrained-model architecture: The untrained model was not trained after initialization.Its anomaly scores were generated from the randomized model used in Case 3.
Loading 2109.05257v2…