PaperlayerView exact revision on arXiv

Source-linked AI summary

Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

Jiehui Xu, Haixu Wu, Jianmin Wang, Mingsheng Long

arXiv:2110.02642v5cs.LG

TL;DR

Unsupervised time-series anomaly detection needs a distinguishable criterion for rare anomalies despite complex dynamics and limited labels. The paper proposes Anomaly Transformer with Anomaly-Attention and minimax association learning to measure and amplify association discrepancy. It achieves state-of-the-art results on six benchmarks across three applications.

  • Problem

    Unsupervised time-series anomaly detection must learn informative representations and distinguish rare anomalies from abundant normal points without labeled data.

  • Method

    Anomaly Transformer uses two-branch Anomaly-Attention to model prior-association and series-association, with minimax learning to amplify their discrepancy.

  • Results

    Anomaly Transformer achieves state-of-the-art anomaly detection results on six benchmarks covering three real applications.

  • Takeaways & Limitations

    Association discrepancy provides an association-based anomaly criterion that collaborates with reconstruction performance for detection.

  • Takeaways & Limitations

    The model may fail with windows that are too small for association learning, while Transformer complexity is quadratic in window size, requiring a trade-off.

Abstract

from arXiv · show

Unsupervised detection of anomaly points in time series is a challenging problem, which requires the model to derive a distinguishable criterion. Previous methods tackle the problem mainly through learning pointwise representation or pairwise association, however, neither is sufficient to reason about the intricate dynamics. Recently, Transformers have shown great power in unified modeling of pointwise representation and pairwise association, and we find that the self-attention weight distribution of each time point can embody rich association with the whole series. Our key observation is that due to the rarity of anomalies, it is extremely difficult to build nontrivial associations from abnormal points to the whole series, thereby, the anomalies' associations shall mainly concentrate on their adjacent time points. This adjacent-concentration bias implies an association-based criterion inherently distinguishable between normal and abnormal points, which we highlight through the \emph{Association Discrepancy}. Technically, we propose the \emph{Anomaly Transformer} with a new \emph{Anomaly-Attention} mechanism to compute the association discrepancy. A minimax strategy is devised to amplify the normal-abnormal distinguishability of the association discrepancy. The Anomaly Transformer achieves state-of-the-art results on six unsupervised time series anomaly detection benchmarks of three applications: service monitoring, space & earth exploration, and water treatment.

1 INTRODUCTION

The paper frames unsupervised time-series anomaly detection as learning informative temporal representations and a distinguishable criterion for rare anomalies. It introduces Anomaly Transformer, Anomaly-Attention, and minimax association learning to exploit association discrepancy, achieving state-of-the-art results on six benchmarks across three applications.

  • Motivation: Unsupervised detection must identify rare abnormal time points in large-scale, continuously monitored systems without expensive labels.The motivation spans industrial equipment, space probes, and other multi-sensor systems where failures threaten security and finances.
  • Problem: Existing approaches separately emphasize pointwise representations or explicit associations, limiting their ability to capture complex temporal dynamics.Classic density and clustering methods neglect temporal information, while deep models commonly use reconstruction or prediction criteria.
  • Approach: Transformers provide unified modeling of global representations and long-range relations through self-attention distributions over time points.The self-attention map exposes each time point’s temporal association with the series.
  • Approach: Anomaly Transformer uses Anomaly-Attention to model prior-association and series-association simultaneously for computing Association Discrepancy.The prior-association uses a learnable Gaussian kernel for adjacent concentration, while series-association uses self-attention weights from the raw series.
  • Approach: A minimax strategy amplifies the normal-abnormal distinguishability of Association Discrepancy and yields an association-based detection criterion.The design exploits the observation that anomalous associations tend to concentrate on adjacent points whereas normal points associate more broadly.
  • Results: Anomaly Transformer achieves state-of-the-art anomaly detection results on six benchmarks spanning three real applications.The paper also reports extensive ablations and case studies.

2 RELATED WORK

Prior time-series anomaly-detection work includes density-estimation, clustering-based, reconstruction-based, autoregression-based, and association-oriented paradigms. The paper distinguishes its approach through a new association-based criterion co-designed with temporal models for informative time-point associations.

  • Existing paradigms: Prior work broadly determines anomalies through density estimation, clustering, reconstruction, or autoregression.These paradigms use local density, distance to cluster centers, reconstruction error, or prediction error, respectively.
  • Density estimation: Density-estimation methods compare local density or connectivity, while mixture models estimate density in learned representations.Examples include LOF, COF, DAGMM, and MPPCACD.
  • Clustering: Clustering-based methods score points by distance to compact normal-data clusters or multi-scale cluster structures.Examples include SVDD, Deep SVDD, THOC, and ITAD.
  • Reconstruction and autoregression: Reconstruction-based and autoregression-based methods detect anomalies using reconstruction or prediction errors.LSTM-VAE, OmniAnomaly, and InterFusion represent reconstruction-based examples, while VAR and LSTM predictors represent autoregression-based methods.
  • Paper distinction: This paper introduces a new association-based criterion embodied by co-designing temporal models to learn more informative time-point associations.It differs from random-walk and subsequence-based methods by making association learning central to the criterion.

3 METHOD

The Anomaly Transformer detects unlabeled time-series anomalies by jointly modeling local prior-associations and global series-associations, then measuring their discrepancy. A minimax objective makes this discrepancy more distinguishable while combining it with reconstruction error.

  • The model targets unlabeled anomaly detection by learning informative representations and a distinguishable criterion for rare abnormal points.
  • Anomaly-Attention uses two branches to model prior-association and series-association simultaneously.The prior branch uses a learnable Gaussian kernel over relative temporal distance, while the series branch uses self-attention.
  • Association Discrepancy is the layer-averaged symmetrized KL divergence between prior- and series-associations.Its point-wise output assigns one discrepancy value to each time point; anomalies are expected to have smaller values than normal points.
  • The minimax strategy makes the prior-association approximate the series-association before optimizing the series-association to enlarge their discrepancy under reconstruction loss.Stop-gradient prevents the optimization phases from directly collapsing the prior-association while strengthening normal-abnormal distinguishability.
  • The final anomaly criterion combines normalized association discrepancy with reconstruction error so both signals contribute to detection.Because anomalies usually decrease association discrepancy, the design can still assign them higher anomaly scores while improving reconstruction.

4 EXPERIMENTS

The experiments evaluate Anomaly Transformer across six benchmarks and show strong performance, including consistent state-of-the-art results on real-world datasets. Ablations and visualizations indicate that the association-based criterion, learnable prior-association, and minimax strategy contribute to distinguishability.

  • Main results: Anomaly Transformer is evaluated on six benchmarks spanning service monitoring, space and earth exploration, and water treatment.
  • Main results: Anomaly Transformer achieves consistent state-of-the-art results on five real-world datasets and the highest AUC values across all five ROC comparisons.The ROC curves assess false-positive and true-positive rates under multiple predefined threshold proportions.
  • Main results: Anomaly Transformer achieves state-of-the-art performance on the NeurIPS-TS benchmark covering pointwise and pattern-wise anomalies.
  • Ablation study: 18.76% averaged absolute F1-score promotion results from the association-based criterion over the reconstruction criterion, increasing 76.20 to 94.96.
  • Ablation study: 8.43% and 7.48% averaged absolute F1-score promotions result from the learnable prior-association and minimax strategy, respectively.The full model also surpasses the pure Transformer by 18.34% absolute improvement, increasing 76.62 to 94.96.
  • Model analysis: The association-based criterion produces smaller values for normal points and more distinct anomaly values than reconstruction in point-contextual and pattern-seasonal cases.The learned scale parameter σ is generally smaller for anomalies than normal points, matching the adjacent-concentration bias.
  • Model analysis: The minimax strategy makes series-associations attend more to non-adjacent areas while reconstruction preserves larger adjacent weights for anomalies than normal points.

5 CONCLUSION AND FUTURE WORK

The paper concludes that Anomaly Transformer uses association discrepancy to improve unsupervised time series anomaly detection. It reports state-of-the-art empirical results, notes stable performance across window sizes, and identifies theoretical analysis as future work.

  • Conclusion: Anomaly Transformer combines two-branch Anomaly-Attention with minimax optimization to distinguish normal and abnormal time points through association discrepancy.
  • Conclusion: The association-based criterion makes reconstruction performance and association discrepancy collaborate for anomaly detection.
  • Conclusion: Anomaly Transformer achieves state-of-the-art results across an exhaustive set of empirical studies.
  • Future work: Future work includes theoretical study of Anomaly Transformer through classic analyses for autoregression and state space models.
  • Sensitivity: The model is stable to window-size choices over extensive datasets, although larger windows increase memory cost and reduce the sliding number.

D.1 ABLATION OF MULTI-LEVEL QUANTIFICATION

The ablations examine how association discrepancy is quantified and how prior-association kernels affect performance. Multi-layer averaging and the Gaussian prior-association kernel are favored over the tested alternatives.

  • Averaging association discrepancy across multiple layers achieves the best performance, supporting multi-level quantification.
  • The association discrepancy is calculated from statistical distances between prior-association and series-association distributions.
  • Symmetrized Kullback–Leibler divergence gives the best performance among the tested association-discrepancy definitions.Jensen–Shannon divergence and cross-entropy provide fairly good results, while L2 and Wasserstein distance are less suitable in the reported experiments.
  • The Gaussian kernel consistently surpasses the power-law kernel for modeling prior-association.Both kernels are unimodal and use a learnable parameter, but the Gaussian scale parameter is easier to optimize.

E.2 ABLATION OF CRITERION DEFINITION

This ablation compares anomaly criteria and optimization behavior. The proposed multiplication-based combination of reconstruction and association discrepancy performs best, while both loss components converge on the real-world datasets.

  • The multiplication combination of association discrepancy and reconstruction performance performs best among the tested anomaly criteria.Direct association discrepancy also surpasses the competitive baseline THOC consistently.
  • The model’s total loss combines reconstruction loss with association discrepancy and uses minimax optimization to control association learning.
  • Both reconstruction loss and association discrepancy converge within limited iterations on all five real-world datasets.The reported convergence behavior is described as essential for optimizing the model.
  • Increasing the number of layers or hidden channels generally improves results but increases memory and computation costs.

H PROTOCOL OF THRESHOLD SELECTION

The threshold protocol selects anomaly-score thresholds without test labels by using unlabeled validation scores. It can either target a dataset-specific anomaly ratio or use fixed threshold values with close performance.

  • The protocol uses the Gap Statistic method on unlabeled validation anomaly scores to select hyper-parameters in the unsupervised setting.
  • The higher-score validation cluster contains approximately 0.1% of points for SWaT, 0.5% for SMD, and 1% for other datasets.
  • A fixed threshold δ can be chosen so that the selected number of validation anomalies exceeds δ and is detected as anomalous.
  • Setting δ to 0.1 for SMD, MSL, and SWaT, and 0.01 for SMAP and PSM yields performance close to selecting by the anomaly ratio r.
  • Selecting the number of detected anomalies by ratio r is more practical when available human resources determine review capacity.
  • Anomaly Transformer achieves the best performance when compared with change-point detection and time-series segmentation baselines.

J LIMITATIONS AND FUTURE WORK

The paper identifies two limitations: association learning can fail with very small windows, and the theoretical understanding of the Anomaly Transformer remains underdeveloped.

  • Very small window sizes may cause the model to fail at association learning.
  • Transformers have quadratic complexity with respect to window size, creating a trade-off for real-world applications.
  • The theory of complex deep models such as the Anomaly Transformer remains under-explored.The paper proposes future work on theorem-level justification using analyses related to autoregression and state space models.

K DATASET

This section introduces the statistical details of the experiment datasets and points to a benchmark-details table.

  • The benchmark details are presented in Table 13.

L UCR DATASET

The UCR dataset provides a broad, challenging evaluation setting, and the Anomaly Transformer achieves state-of-the-art performance on it.

  • The UCR dataset contains 250 one-dimensional sub-datasets, each with one anomaly segment, spanning lengths from 6,684 to 900,000.The sub-datasets are pre-divided into training and test sets.
  • The Anomaly Transformer achieves state-of-the-art results on the challenging UCR benchmark.
  • Table 14 reports UCR precision, recall, and F1-score percentages, comparing IsolationForest with the Anomaly Transformer.In the table, “Ours” denotes the Anomaly Transformer.
Loading 2110.02642v5…