Source-linked AI summary
Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress
Renjie Wu, Eamonn J. Keogh
TL;DR
The paper argues that widely used time-series anomaly-detection benchmarks have severe limitations that undermine evaluation and comparison. It introduces new benchmark datasets largely free of these flaws and calls for broader community recommendations.
Problem
Common benchmark datasets have severe limitations, including unrealistic anomaly density, mislabeled ground truth, and trivial detection cases, constraining reliable algorithm evaluation.
Method
The paper examines benchmark flaws and introduces a new set of benchmark datasets designed to be largely free of the current benchmarks’ flaws.
Results
No reported F1 score, including 1.0 or 0.9, or claimed 20% improvement, would establish algorithm utility or general superiority on these flawed datasets.
Takeaways & Limitations
Meaningful comparisons and progress gauges require benchmark datasets that avoid the flaws identified in the classic archives.
Takeaways & Limitations
The authors caution that the new benchmark resource is not the last word and recommend gathering diverse viewpoints to develop crowdsourced benchmark recommendations.
Abstract
from arXiv · showhide
Time series anomaly detection has been a perennially important topic in data science, with papers dating back to the 1950s. However, in recent years there has been an explosion of interest in this topic, much of it driven by the success of deep learning in other domains and for other time series tasks. Most of these papers test on one or more of a handful of popular benchmark datasets, created by Yahoo, Numenta, NASA, etc. In this work we make a surprising claim. The majority of the individual exemplars in these datasets suffer from one or more of four flaws. Because of these four flaws, we believe that many published comparisons of anomaly detection algorithms may be unreliable, and more importantly, much of the apparent progress in recent years may be illusionary. In addition to demonstrating these claims, with this paper we introduce the UCR Time Series Anomaly Archive. We believe that this resource will perform a similar role as the UCR Time Series Classification Archive, by providing the community with a benchmark that allows meaningful comparisons between approaches and a meaningful gauge of overall progress.
1 INTRODUCTION
Recent growth in time series anomaly detection has concentrated on a small set of popular benchmarks and deep-learning approaches. The paper argues that flaws in these datasets can make published comparisons unreliable and illustrates how complex methods may be matched by one-line solutions.
- Recent anomaly-detection interest has surged, partly driven by attempts to transfer deep learning’s success from other domains and time series classification.
- Most studies evaluate on a handful of benchmarks from Yahoo, Numenta, NASA, or Pei’s Lab, whose exemplars often contain four stated flaws.The flaws are triviality, unrealistic anomaly density, mislabeled ground truth, and run-to-failure bias.
- A supervised deep-learning detector with convolution, LSTM, and VAE components reported accuracy exceeding 0.90 on a Yahoo subset.
- Much of that complex approach’s reported result can be duplicated with a single line of code and a few minutes of effort.
- The one-line result does not establish that original papers lack value, but it strongly suggests testing new datasets that stump trivial solutions.
- The authors distinguish dataset limitations from criticism of the groups that originally created and released these resources.
2 A TAXONOMY OF BENCHMARK FLAWS
The paper identifies four flaws in popular anomaly-detection benchmarks: triviality, unrealistic anomaly density, mislabeled ground truth, and run-to-failure bias. Examples and brute-force tests show that many benchmark tasks are easily solved or scored inconsistently, undermining algorithm comparisons.
- Triviality: 316 out of 367 Yahoo time series (86.1%) can be solved with a one-liner using individually searched k, c, and b parameters.The authors used brute-force search over these parameters across the Yahoo Benchmark.
- Triviality: 193 out of 367 Yahoo time series can be solved using individual offset values b alone, while some A3 cases share k = 5 and c = 0.The authors argue that the overall 86.1% result is competitive with many published results, although scoring-function differences complicate precise comparisons.
- Unrealistic anomaly density: Benchmark anomaly density is often unrealistic, with contiguous anomalous regions covering more than half or one-third of some series and many anomalies packed into short regions.The paper argues that such density complicates both the detection task and the interpretation of scores.
- Mislabeled ground truth: Rapidly toggling labels can penalize algorithms for placing subsequence-based detections just outside labeled regions, even when the detected event is effectively the same.The paper notes that detectors may place labels at the beginning, middle, or end of a subsequence.
3 INTRODUCING THE UCR ANOMALY ARCHIVE
The paper introduces the UCR Time Series Anomaly Archive to address flaws in established benchmarks, using diverse datasets and carefully constructed or externally validated anomalies.
- Archive design: The UCR Time Series Anomaly Archive was created to avoid the flaws identified in existing anomaly-detection benchmarks, while retaining a spectrum from easy to very hard problems.A small fraction may remain solvable with a one-liner because some real-world anomalies are inherently easy to detect.
- Archive design: The archive spans medicine, sports, entomology, industry, space science, and robotics after outreach sought datasets from the broader research community.The authors’ call for contributions produced no submissions, so the resulting collection still reflects their own assembled datasets.
- Archive design: Each dataset is intended to contain one anomaly in testing, with detailed provenance and metadata documenting how the data were created.The paper presents representative examples because the archive contains many datasets.
- Natural anomalies confirmed out-of-band: A subtle plethysmography anomaly is validated by a PVC occurring in a parallel ECG, providing out-of-band evidence of a true anomaly.The ECG and plethysmography signals have a slight timing lag, accommodated by scoring-function tolerance.
- Natural anomalies confirmed out-of-band: The authors generalize this strategy by using obvious out-of-band data to annotate subtle anomalies rather than directly using an overly simple ECG benchmark.This technique supports semantic validation of anomalies across many archive datasets.
- Plausible synthetic anomalies: Synthetic anomalies are inserted into anomaly-free series at random locations while preserving plausible behavior, such as swapping one foot cycle into a gait series.The replacement models a sudden spasm while retaining natural-looking gait dynamics.
- Plausible synthetic anomalies: The gait example preserves ordinary speed changes by including them in both training and test data, and nine of ten volunteers identified the anomaly after careful inspection.This reflects the authors’ attempt to avoid examples that are either trivial or impossibly cryptic.
4 RECOMMENDATIONS
The paper recommends abandoning flawed legacy benchmarks, explaining algorithms through invariances and visualizations, and reconsidering unsubstantiated assumptions favoring deep learning.
- Benchmark datasets: The authors recommend abandoning the Yahoo, Numenta, NASA, and OMNI benchmarks because they consider them irretrievably flawed for algorithm evaluation and comparison.They also recommend discounting or reevaluating papers that rely primarily or exclusively on these datasets.
- Algorithms and invariances: Algorithms should be described by the invariances they support, such as amplitude scaling, offset, occlusion, noise, trend, warping, and uniform scaling.The paper presents invariances as a useful lens for practitioners choosing algorithms for particular data characteristics.
- Algorithms and invariances: Many anomaly-detection papers do not clearly state the circumstances under which their proposed algorithms should work for practitioners.The paper illustrates this concern with questions about changing baselines and the anomaly-relevant aspects of data.
- Visualization: Fig. 13 compares anomaly scores from Telemanom and Discord on a clean ECG and after substantial Gaussian noise is added.Both methods peak at the obvious anomaly in the clean signal; under noise, Discord still peaks correctly while Telemanom peaks at the wrong location.
- Visualization: Visualization can summarize predictions across 12,000 locations, whereas precision, recall, and F1 may summarize predictions at only two or three locations.The authors do not propose replacing standard metrics, but recommend using plots to reveal behavior and robustness.
- Visualization: Many papers show few or no examples of the time series, despite time series analytics being inherently visual and benchmark flaws often being visually apparent.The authors connect this omission to reliance on black-box evaluation and aggregate scores.
- Scoring functions: Scoring functions can unfairly penalize methods whose anomaly peaks occur at different positions within the same anomalous subsequence.The paper recommends visualizing both algorithm outputs and the acceptable answer range to make evaluation fairer.
- Deep-learning assumptions: The authors report no forceful reproducible evidence that deep learning outperforms much simpler methods, while older methods can be competitive and are usually faster and simpler.They emphasize that this is not evidence that deep learning is never competitive, but urge consideration of existing methods.
5 CONCLUSIONS
The paper concludes that common anomaly-detection benchmarks are unsuitable for comparison and presents the UCR Archive as a more promising resource, while calling for broader community recommendations.
- Conclusions: The authors conclude that commonly used anomaly-detection benchmarks contain flaws that make them unsuitable for evaluating or comparing algorithms.They present the UCR Time Series Anomaly Archive as largely free of those flaws, but not as the final solution.
- Conclusions: The paper calls for a committee or conference workshop to develop recommendations for a crowdsourced, diverse benchmark collection.The authors hope their work will prompt community action.