Source-linked AI summary
TAB: Unified Benchmarking of Time Series Anomaly Detection Methods
Xiangfei Qiu, Zhe Li, Wanghui Qiu, Shiyan Hu, Lekui Zhou, Xingjian Wu, Zhengyu Li, Chenjuan Guo, Aoying Zhou, Zhenli Sheng, Jilin Hu, Christian S. Jensen, Bin Yang
TL;DR
TSAD research lacks broad datasets and consistent evaluation protocols, making reliable comparison difficult. TAB addresses this gap with a comprehensive benchmark and unified pipeline covering diverse datasets, methods, tasks, strategies, and metrics. Its experiments identify task- and anomaly-specific performance patterns, while foundation methods still require further development.
Problem
Existing TSAD studies often use limited datasets and inconsistent protocols, while foundation methods and generalization abilities remain insufficiently evaluated.
Method
TAB combines 29 multivariate datasets, 1,635 univariate time series, broad TSAD method coverage, and a unified evaluation pipeline with multiple tasks, strategies, and metrics.
Results
TAB experiments show that strong methods differ across UTSAD and MTSAD and across anomaly types, while foundation methods currently perform relatively poorly.
Takeaways & Limitations
Reliable TSAD comparison requires broad datasets, consistent experimental procedures, and evaluation across tasks, anomaly types, and supervision settings.
Takeaways & Limitations
Existing benchmarks remain limited in extensibility, restricting support for new datasets, methods, and evaluation metrics.
Abstract
from arXiv · showhide
Time series anomaly detection (TSAD) plays an important role in many domains such as finance, transportation, and healthcare. With the ongoing instrumentation of reality, more time series data will be available, leading also to growing demands for TSAD. While many TSAD methods already exist, new and better methods are still desirable. However, effective progress hinges on the availability of reliable means of evaluating new methods and comparing them with existing methods. We address deficiencies in current evaluation procedures related to datasets and experimental settings and protocols. Specifically, we propose a new time series anomaly detection benchmark, called TAB. First, TAB encompasses 29 public multivariate datasets and 1,635 univariate time series from different domains to facilitate more comprehensive evaluations on diverse datasets. Second, TAB covers a variety of TSAD methods, including Non-learning, Machine learning, Deep learning, LLM-based, and Time-series pre-trained methods. Third, TAB features a unified and automated evaluation pipeline that enables fair and easy evaluation of TSAD methods. Finally, we employ TAB to evaluate existing TSAD methods and report on the outcomes, thereby offering a deeper insight into the performance of these methods. Besides, all datasets and code are available at https://github.com/decisionintelligence/TAB.
1 INTRODUCTION
TAB addresses shortcomings in TSAD evaluation by combining broader datasets, consistent protocols, extensive method coverage, and multiple evaluation strategies. Its benchmark-based experiments reveal performance patterns across tasks and anomaly types.
- Motivation: Existing TSAD studies often use few datasets and domains, limiting the comprehensiveness of performance evaluations.More than half of surveyed studies include at most four multivariate datasets, while only one covers 17 datasets.
- Motivation: Inconsistent dataset splitting, batch handling, and point adjustment make comparisons across TSAD methods unreliable.Point adjustment can allow random methods to score well on large anomaly windows, and evaluation metrics such as AUC-ROC should not be recalculated afterward.
- TAB contributions: TAB provides 29 public multivariate datasets and 1,635 univariate time series spanning diverse domains.The datasets are selected and organized to support broader TSAD evaluation.
- TAB contributions: TAB unifies preprocessing, method integration, training, testing, and post-processing so univariate and multivariate methods undergo consistent evaluation.Its pipeline uses standardized inputs, a universal interface for self-implemented and third-party methods, and fixed evaluation procedures.
- TAB contributions: TAB covers non-learning, machine learning, deep learning, LLM-based, and time-series pretrained methods, with label-based and score-based metrics.It also supports univariate and multivariate tasks plus zero-shot, few-shot, and full-shot evaluation strategies.
- Experimental findings: Experiments find that method performance depends on task and anomaly type, while foundation methods currently remain relatively poor overall.Non-learning and machine learning methods perform strongly on UTSAD; deep and machine learning methods perform strongly on MTSAD, with point, subsequence, trend, and shapelet anomalies favoring different method families.
2 RELATED WORKS
Related TSAD work spans multiple method families, but existing benchmarks provide incomplete coverage of datasets, evaluation strategies, foundation methods, generalization, and extensibility. TAB addresses these gaps through a broader, unified, and extensible benchmark.
- TSAD methods comprise non-learning, machine learning, deep learning, LLM-based, and time-series pre-trained categories.
- Benchmark limitations: Existing benchmarks often cover only univariate or multivariate data, while many omit fine-grained dataset classification and mixtures of real-world and synthetic datasets.
- Benchmark limitations: Most benchmarks do not systematically compare foundation methods or evaluate zero-shot and few-shot generalization, focusing instead on full-shot settings.
- Benchmark limitations: Several benchmarks lack flexible interfaces or pipelines for adding datasets, methods, and evaluation metrics.
- TAB: TAB evaluates both univariate and multivariate datasets, 11 foundation methods, and zero-shot, few-shot, and full-shot strategies within a fair, extensible pipeline.
3 HETEROGENEITY AMONG TSAD DATASETS
TAB organizes TSAD datasets by intrinsic time-series characteristics and anomaly types. These dimensions capture substantial heterogeneity in real-world data and support more structured benchmark analysis.
- The dataset taxonomy considers inherent time-series characteristics and the anomaly types they contain.
- Time-series characteristics: Real-world time series may exhibit trend, seasonality, shifting, transition, and stationarity.
- Time-series characteristics: Stationarity means that the mean of observations is constant and the variance is finite.
- Anomaly types: Anomalies are grouped into point anomalies and subsequence anomalies, including global, contextual, shapelet, seasonal, trend, and mixed types.
4 TAB: BENCHMARK DETAILS
TAB’s benchmark design covers dataset collection and filtering, supported TSAD methods, evaluation strategies and metrics, and the complete benchmark pipeline.
- TAB’s design is organized around dataset collection and filtering, supported methods, evaluation strategies and metrics, and the full benchmark pipeline.
4.1 Datasets
TAB provides a consistently formatted collection of multivariate and univariate time series spanning diverse domains and characteristics. Its univariate construction filters series for anomaly presence and data quality, while acknowledging concerns about legacy ground-truth labels.
- Dataset Collection: TAB includes 220 multivariate time series derived from 29 datasets and 1,635 univariate time series from 15 datasets.The collection is sourced from community literature and formatted consistently.
- Multivariate Datasets: The 29 multivariate datasets span 14 domains, with feature dimensions from 2 to 248 and sequence lengths from 5,040 to 1,416,825.Some datasets contain multiple multivariate time series, and results are averaged across series within each dataset.
- Univariate Datasets: The univariate collection combines seven originally univariate datasets with eight datasets transformed from multivariate time series.The first seven are commonly used for univariate anomaly detection.
- Univariate Datasets: TAB derives univariate series from multivariate data by running anomaly detection separately on each series and retaining series where at least one method achieves AUC-ROC > 0.85.The filtering also excludes series without anomalies and those with anomaly ratios above 10%.
- Dataset Limitations: TAB uses widely studied legacy datasets, but their ground-truth anomaly labels remain subject to data-quality concerns.The paper identifies addressing these label-quality issues as an open area of interest.
4.2 Comparison Methods
TAB compares a broad range of TSAD approaches across methodological families, using separate method sets for univariate and multivariate evaluation.
- Comparison Scope: TAB evaluates 46 methods for univariate anomaly detection and 40 methods for multivariate anomaly detection.Eight methods are removed from the multivariate comparison because of task-specific applicability.
- Method Families: The compared methods span non-learning, machine learning, deep learning, LLM-based, and time-series pre-trained approaches.This classification supports comparisons across established and newer TSAD paradigms.
4.3 Evaluation Settings
TAB combines multiple evaluation strategies with label-based and score-based metrics to assess TSAD methods under different data-availability conditions.
- Evaluation Strategies: TAB supports zero-shot, few-shot, and full-shot evaluation strategies that can produce anomaly scores or anomaly labels.These strategies cover different degrees of adaptation and supervision.
- Zero-Shot Evaluation: Zero-shot evaluation uses only test data to assess foundation methods’ generalization to new datasets.It is intended to assess whether pre-training produced general knowledge transferable to unseen datasets.
- Few-Shot Evaluation: Few-shot evaluation fine-tunes with a subset of training data and full validation data to represent low-data learning scenarios.It assesses generalization and reasoning with minimal data support.
- Full-Shot Evaluation: Full-shot evaluation fine-tunes with all training and validation data to reveal upper-bound performance.This setting evaluates performance when all available data is used.
- Evaluation Metrics: TAB computes both label-based metrics, which depend on thresholds, and score-based metrics, which use raw anomaly scores.The benchmark supports all metrics to provide a complete performance assessment and allows additional metrics to be customized.
4.4 Unified Pipeline
TAB uses a unified, layered pipeline to standardize data processing, method execution, evaluation, and reporting. The pipeline is extensible, scalable, and designed to support fair comparisons across methods.
- Pipeline Architecture: TAB’s data, method, and evaluation layers standardize inputs, method workflows, and post-processing across compared anomaly detection methods.The data layer standardizes preprocessing and splitting, the method layer uses a universal interface, and the evaluation layer applies fixed post-processing.
- Data Layer: The data layer stores diverse univariate and multivariate series in a unified format organized by characteristics, anomaly ratios, and anomaly types.This organization supports dataset selection and comparability.
- Method Layer: The method layer supports non-learning, machine learning, and deep learning methods through a simple interface compatible with third-party libraries.Users can integrate external anomaly-detection libraries for faster comparison.
- Evaluation and Reporting: The evaluation and reporting layers provide multiple strategies, customizable metrics, logging for experimental traceability, and visualization of method performance.The reporting layer tracks experimental settings and helps users inspect results.
- Extensibility and Scalability: The pipeline supports third-party libraries, customizable strategies and metrics, characteristic-based dataset selection, parallel operation, and CPU or GPU execution.These features make the benchmark flexible across datasets and computing environments.
- Use in Practice: By integrating a new method into the model layer, users can evaluate it quickly and compare it for specific application scenarios.TAB is intended to assist method selection through standardized benchmarking.
5 EXPERIMENTS
TAB standardizes datasets, splitting, post-processing, and reporting to support reproducible and comparable TSAD experiments.
- TAB uses 29 multivariate datasets and 1,635 univariate time series, with anomaly and characteristic types available in the repository.
- TAB unifies experimental inconsistencies and documents hyperparameter tuning and the experimental environment to improve reproducibility.
- The benchmark follows raw-data splits when available and otherwise uses a 50% training-validation and 50% test split with a validation subset.
- Table 6 reports V-PR, Aff-F1, training and inference time, and CPU and GPU memory across methods and datasets.
- Post-processing comparisons show that overlapping and non-overlapping windows usually have little effect, but some cases differ, requiring one consistent choice.
5.2 Overall Performance
TAB evaluates broad method classes across univariate and multivariate datasets, revealing different performance patterns by dimensionality and learning setting.
- 46 methods are evaluated across 1,635 univariate time series using V-PR and Aff-F1.
- Machine learning and non-learning methods achieve the best average V-PR and Aff-F1 on univariate datasets, while classic methods remain competitive.
- Prediction-based methods usually outperform other methodological classes on univariate TSAD, followed by density-based methods, while contrast-based methods perform weakest.
- 40 methods are evaluated on 29 public multivariate datasets, where time-series pre-trained models improve substantially from zero-shot to full-shot and few-shot settings.
- KNN, KMeans, DAGMM, TsNet, and AE show excellent multivariate performance, with deep learning and machine learning methods demonstrating clear advantages.
5.3 Method Recommendations
Method performance varies with anomaly type and time-series characteristics, so TAB uses these patterns to support more targeted method selection.
- The benchmark classifies univariate datasets by six anomaly types and five time-series characteristics, whose distributions are not uniform.
- Trend, stationarity, and shifting characteristics generally yield better Aff-F1, whereas seasonality and transition make anomaly detection more difficult.
- Users can calculate five characteristic values for an unseen time series and use the observed performance relationships to choose and optimize methods.
- Runtime, memory, and accuracy jointly favor S2G and DWT for univariate detection and lightweight CATCH or TsNet for multivariate detection when resources permit.
- Non-learning methods are generally fastest, while most deep learning methods require relatively long training times.
6 CONCLUSIONS
TAB is a unified benchmark for comparing TSAD methods across diverse datasets, method families, evaluation settings, and metrics.
- TAB includes 1,635 univariate time series and 29 multivariate datasets spanning multiple domains.
- The benchmark covers non-learning, machine learning, deep learning, LLM-based, and time-series pre-trained methods.
- TAB provides a unified, extensible evaluation pipeline for fair comparisons under consistent experimental settings.
- The authors evaluate 40 multivariate and 46 univariate methods using multiple evaluation strategies and metrics, and provide an online leaderboard.