TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data

Shreshth Tuli; Giuliano Casale; Nicholas R. Jennings

Source-linked AI summary

TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data

Shreshth Tuli, Giuliano Casale, Nicholas R. Jennings

arXiv:2201.07284v6Jan 18, 2022cs.LG

TL;DR

Detecting and diagnosing anomalies in increasingly multimodal multivariate time series remains challenging. TranAD combines transformer encoders with self-conditioning, adversarial training, and meta-learning, improving F1 scores by up to 17% while identifying root causes for up to 75% of detected anomalies.

Problem
Anomaly detection becomes increasingly challenging in large-scale databases as data modality grows, complicating multivariate time-series analysis.
Method
TranAD uses transformer-based encoder-decoder networks with self-conditioning, adversarial training, and meta-learning for multivariate time-series anomaly detection and diagnosis.
Results
17% and 11% F1-score improvements on complete and limited training data, respectively, with root-cause identification reaching up to 75% of detected anomalies.
Takeaways & Limitations
TranAD supports fast anomaly detection and diagnosis across the evaluated datasets, including settings with limited training data.
Takeaways & Limitations
The authors propose extending TranAD to other transformer models and evaluating component cost-benefit tradeoffs for deployment settings with expensive computation.

Abstract

from arXiv · show

Efficient anomaly detection and diagnosis in multivariate time-series data is of great importance for modern industrial applications. However, building a system that is able to quickly and accurately pinpoint anomalous observations is a challenging problem. This is due to the lack of anomaly labels, high data volatility and the demands of ultra-low inference times in modern applications. Despite the recent developments of deep learning approaches for anomaly detection, only a few of them can address all of these challenges. In this paper, we propose TranAD, a deep transformer network based anomaly detection and diagnosis model which uses attention-based sequence encoders to swiftly perform inference with the knowledge of the broader temporal trends in the data. TranAD uses focus score-based self-conditioning to enable robust multi-modal feature extraction and adversarial training to gain stability. Additionally, model-agnostic meta learning (MAML) allows us to train the model using limited data. Extensive empirical studies on six publicly available datasets demonstrate that TranAD can outperform state-of-the-art baseline methods in detection and diagnosis performance with data and time-efficient training. Specifically, TranAD increases F1 scores by up to 17%, reducing training times by up to 99% compared to the baselines.

1 Introduction

The introduction frames multivariate time-series anomaly detection as increasingly important yet difficult because data are high-dimensional, volatile, limited for training, and require ultra-fast inference. TranAD addresses these challenges with a transformer-based architecture incorporating self-conditioning and adversarial training.

Background: Modern IT operations generate high-dimensional sensor data for continuous monitoring, making anomaly detection important for system management and fault tolerance.Traditional data-mining approaches identify deviations from usual trends to report faults.
Challenges: Large-scale anomaly detection is challenged by increasing data modality, data volatility, limited training-data availability, and demands for ultra-fast inference.Synchronizing geographically distant databases is expensive, while accurate inference requires substantial data and rapid recovery supports Quality of Service.
Existing solutions: Existing unsupervised reconstruction methods model temporal trends and use prediction deviations from ground truth as anomaly scores.These methods classify time-series observations using extreme value analysis techniques.
New insights: Recurrent models are slow, computationally expensive, and ineffective at modeling long-term trends because inference proceeds sequentially across prior timestamps.Transformers enable single-shot inference over complete input series and faster detection through GPU parallelization.
Our contributions: TranAD is a transformer-based anomaly detection model that adapts transformer networks and model-agnostic meta learning with self-conditioning and adversarial training.The proposed architecture is designed to be fast for training and testing.

2 Related Work

Related work spans univariate and multivariate time-series anomaly detection, including classical statistical, signal-processing, graph, and discord-discovery methods. Recent approaches increasingly use deep recurrent, autoencoding, adversarial, convolutional, and attention-based architectures, but face trade-offs in training cost, generalization, or applicability.

Scope of prior work: Time-series anomaly detection research addresses both univariate data from a single source and multivariate data comprising multiple time series.The literature includes methods designed for each setting.
Classical methods: Classical approaches model time-series distributions using k-means, SVMs, regression, wavelets, Hilbert transforms, PCA, process regression, hidden Markov chains, graphs, isolation forests, or ARIMA.GraphAn detects outliers through graph-distance metrics, while isolation forests recursively partition feature space.
Discord discovery: Time-series discord methods identify subsequences maximally different from others, with matrix profiling and variants supporting anomaly and motif discovery.Discord discovery has also been proposed for fault prediction.
Deep learning methods: Deep learning methods include LSTM-based forecasting, recurrent Gaussian-mixture autoencoders, ConvLSTM image encoders, and LSTM-based GANs for temporal modeling and anomaly detection.These methods learn sequential dependence, latent representations, inter-modal correlations, temporal information, or time-series distributions.
Limitations and comparison: Prior methods retain important limitations: some require high training times, insufficient-data settings, or specialized inputs, while vanilla-transformer HitAnomaly targets natural-language logs rather than generic continuous time series.MSCRED cannot generalize to insufficient training data, and HitAnomaly is not appropriate for generic continuous time-series inputs.
Efficient architectures: Recent USAD, GDN, and openGauss methods avoid resource-hungry recurrent models by using attention-based architectures to improve training speeds.USAD combines an autoencoder with two decoders and adversarial game-style training, reducing training times compared with prior art.

3 Methodology

TranAD formulates multivariate anomaly detection and diagnosis over normalized, windowed time series, reconstructing local contexts to derive anomaly scores. Its two-phase transformer architecture uses temporal attention, focus-score self-conditioning, adversarial training, and online thresholding for detection and diagnosis.

Problem formulation: TranAD represents each timestamp as an m-dimensional observation and predicts binary anomaly labels for timestamps or dimensions.Detection labels whether a datapoint is anomalous, while diagnosis identifies anomalous modes within the datapoint.
Data preprocessing: The method normalizes each modality using training-series minima and maxima into [0, 1), then converts the series into length-K sliding windows with replication padding.Training and testing use the resulting window sequences so each datapoint is modeled with local context.
Transformer architecture: TranAD uses two transformer encoders and two decoders whose attention operations combine input windows with the complete sequence to capture temporal trends and process batches in parallel.Multi-head attention attends across representation subspaces, while masking hides future timestamps within the same input batch.
Self-conditioning: Focus scores from the first reconstruction phase modify second-phase attention weights, increasing activation for subsequences that capture short-term temporal trends.The focus scores indicate deviations between reconstructed outputs and inputs and serve as a self-conditioning prior.
Inference and thresholding: 7.2% higher F1 scores were observed for TranAD with Peak Over Threshold than with annual maximum thresholding, while test-time inference runs sequentially online.Anomaly scores are compared with dynamically determined thresholds to produce timestamp and per-dimension labels.

4 Experiments

Experiments compare TranAD with established anomaly-detection baselines across seven public datasets and evaluate detection, statistical significance, and diagnosis. TranAD generally achieves the strongest detection and diagnosis results, including under limited training data, while retaining dataset-specific weaknesses.

Experimental Setup: TranAD is compared with ten state-of-the-art baselines across seven publicly available datasets, with Isolation Forest omitted because of low F1 scores.Most baseline implementations use publicly available code, while other models were re-implemented.
Anomaly Detection: TranAD averages 0.8802 F1 and 0.8012 F1*, outperforming baselines on every dataset except MSL for F1 and WADI for F1*.GDN achieves the highest F1 on MSL at 0.9591, while OmniAnomaly achieves the highest F1* on WADI at 0.1017.
Anomaly Detection: Self-conditioning on complete traces combined with local windows helps TranAD capture long-term trends, overcoming local-context limitations affecting USAD and MTAD-GAT.TranAD uses self-attention for inter-dimensional correlations and outperforms GDN overall, although GDN scores higher on MSL and MSDS.
Anomaly Detection: TranAD achieves the best overall model rank with a significant statistical difference under critical-difference analysis using Wilcoxon tests after Friedman-test rejection.The analysis evaluates F1 and AUC scores across all datasets at α=0.05.
Anomaly Diagnosis: TranAD detects 46.3%−75.3% of anomaly root causes and improves diagnosis scores by up to 6% on SMD and 30% on MSDS.Its average diagnosis-score improvement is 4.25%.

5 Analyses

Analyses show that TranAD’s transformer architecture is the most important component, while its training is substantially faster than baselines. Performance and efficiency vary with training-set and window size, with a window of 10 balancing F1 score and training time.

Ablation study: Replacing the transformer encoder-decoder causes the largest ablation drop, nearly 11% in F1 score and 56% on WADI.The WADI result demonstrates the importance of attention-based transformers for large-scale datasets.
Training efficiency: TranAD’s training time is 75%−99% lower than that of baseline methods.The comparison uses average training times in seconds per epoch, while MERLIN’s time measures discord discovery in test data.
Training-set sensitivity: F1, AUC, and training time are evaluated as the training-data ratio varies from 20% to 100%, excluding MERLIN because it uses no training data.Other deep reconstruction models receive randomly sampled subsequences matching each training-data ratio, with 90% confidence bounds reported.
Window-size sensitivity: Smaller windows reduce inference time, but overly small windows lose local context and overly large windows can hide short anomalies.Window size affects both anomaly-detection scores and training times.
Window-size sensitivity: A window size of 10 provides a reasonable balance between F1 score and training time and is used in the experiments.The analysis motivates this choice by contrasting reduced inference time with the risks of insufficient context or obscured short anomalies.

6 Conclusions

TranAD detects and diagnoses anomalies in multivariate time-series data using a transformer encoder-decoder with self-conditioning, adversarial training, and meta-learning. Future work targets broader temporal-trend generalization and deployment-specific component cost-benefit analysis.

Contributions: TranAD detects and diagnoses anomalies in multivariate time-series data using a transformer-based encoder-decoder.The model supports quick training and high detection performance across the datasets considered.
Contributions: Self-conditioning and adversarial training amplify errors and improve training stability, while meta-learning identifies data trends with limited data.These mechanisms address error amplification, stability, and limited-data learning.
Future Work: Future work will explore bidirectional neural networks and other transformer models to generalize across diverse temporal trends.The proposed extension is intended to broaden model generalization.
Future Work: Future work will apply cost-benefit analysis to each model component according to deployment settings to avoid expensive computation.The analysis is intended to guide component choices under deployment constraints.

A MERLIN Implementation

The paper uses a custom Python implementation of MERLIN for Section 4 comparisons, selecting discord-length hyperparameters by grid-search to maximize F1. Its benchmark scores closely match the original MATLAB implementation, while training time improves by up to 96%.

Implementation: The Section 4 comparisons use a custom Python implementation of the MERLIN baseline.The original MERLIN implementation is in MATLAB.
Implementation: Grid-search selects MinL and MaxL hyperparameters for discord lengths by maximizing F1 score.The original code does not specify how to select these hyperparameters.
Results: Up to 96% training time gains are achieved while the Python implementation’s benchmark scores remain very close to the original MATLAB implementation.Precision, Recall, AUC, F1 score, and training time are compared, with deviation calculated as (y−x)/x.