PaperlayerView exact revision on arXiv

Source-linked AI summary

CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection

Zahra Zamanzadeh Darban, Geoffrey I. Webb, Shirui Pan, Charu C. Aggarwal, Mahsa Salehi

arXiv:2308.09296v4cs.LGcs.NE

TL;DR

TSAD often lacks labelled data, and tightly defined normal boundaries can classify slight deviations as anomalous. CARLA uses a two-stage contrastive and self-supervised approach, and the authors report superior performance across seven real-world datasets against ten state-of-the-art methods.

  • Problem

    Tightly defined normal boundaries can classify slight deviations as anomalies, while TSAD often faces a lack of labelled data.

    Source: §1 · p. 2

  • Method

    CARLA is a two-stage contrastive approach that leverages generic anomaly knowledge and a pretext stage.

    Source: §1 · pp. 3–4, §3.2 · p. 10

  • Results

    CARLA shows superior performance across seven real-world datasets over ten state-of-the-art unsupervised, semi-supervised, and self-supervised methods.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The authors report a consistent balance between FPR and AU-PR across multivariate and univariate datasets, which they characterize as precision and reliability.

    Source: §1 · p. 5

    Future work will refine the selection of positive and negative samples for abnormal anchors in the unsupervised setting.

    Source: §5 · p. 30

Abstract

from arXiv · show

One main challenge in time series anomaly detection (TSAD) is the lack of labelled data in many real-life scenarios. Most of the existing anomaly detection methods focus on learning the normal behaviour of unlabelled time series in an unsupervised manner. The normal boundary is often defined tightly, resulting in slight deviations being classified as anomalies, consequently leading to a high false positive rate and a limited ability to generalise normal patterns. To address this, we introduce a novel end-to-end self-supervised ContrAstive Representation Learning approach for time series Anomaly detection (CARLA). While existing contrastive learning methods assume that augmented time series windows are positive samples and temporally distant windows are negative samples, we argue that these assumptions are limited as augmentation of time series can transform them to negative samples, and a temporally distant window can represent a positive sample. Our contrastive approach leverages existing generic knowledge about time series anomalies and injects various types of anomalies as negative samples. Therefore, CARLA not only learns normal behaviour but also learns deviations indicating anomalies. It creates similar representations for temporally closed windows and distinct ones for anomalies. Additionally, it leverages the information about representations' neighbours through a self-supervised approach to classify windows based on their nearest/furthest neighbours to further enhance the performance of anomaly detection. In extensive tests on seven major real-world time series anomaly detection datasets, CARLA shows superior performance over state-of-the-art self-supervised and unsupervised TSAD methods. Our research shows the potential of contrastive representation learning to advance time series anomaly detection.

1. Introduction

CARLA addresses unlabelled TSAD and weaknesses in conventional contrastive sampling with a two-stage approach. The paper reports superior performance across real-world benchmarks and a balance between false-positive rate and precision-recall performance.

1. Introduction

  • CARLA targets the lack of labelled data and the risk that tightly defined normal boundaries classify slight deviations as anomalies.The authors connect this issue to false positives when models cannot distinguish normal from anomalous data.Source: §1 · p. 2
  • Existing contrastive TSAD methods risk treating augmented windows that become negative samples as positive and temporally distant positive windows as negative.The authors argue these sampling assumptions can lead to ineffective anomaly detection.Source: §1 · p. 3
  • CARLA injects known types of time-series anomalies as negatives, drawing similar representations for temporally close windows and dissimilar ones for injected anomalies.A second self-supervised stage classifies windows using their nearest and furthest neighbours in representation space.Source: §1 · p. 4
  • CARLA outperforms ten state-of-the-art unsupervised, semi-supervised, and self-supervised methods across seven real-world datasets.The authors also report a consistent balance between False Positive Rate (FPR) and Area Under the Precision-Recall Curve (AU-PR) across multivariate and univariate datasets.Source: §1 · p. 5

2. Related Work

The related-work discussion spans TSAD methods, unsupervised time-series representations, and contrastive learning. It describes approaches to scarce labels, representation challenges, and contrastive embedding methods.

2. Related Work

  • The related-work section organizes its review around deep-learning TSAD, unsupervised time-series representation, and contrastive representation learning.These are the three areas identified in its overview.Source: §2 · p. 5

2.1. Time Series Anomaly Detection

  • TSAD research includes statistical, classical machine-learning, and deep-learning approaches, with unsupervised and semi-supervised methods suited to different label availability.The discussion cites deep models including autoencoders, VAEs, RNNs, LSTMs, GANs, and Transformers.Source: §2.1 · pp. 5–6

2.2. Unsupervised Time Series Representation

  • Unsupervised representation methods use time-based negative sampling, triplet loss, or local signal smoothness, while some face scalability or complex-series modelling challenges.The discussion names TKAE and TST among methods with challenges, and TNC and T-Loss among proposed approaches to address them.Source: §2.2 · p. 6

2.3. Contrastive Representation Learning

  • Contrastive representation learning places similar samples close and dissimilar samples far apart, with InfoNCE and SimCLR using positive-negative pairs.For TSAD, the discussion cites TS2Vec's hierarchical multi-level semantic representation and DCdetector's dual-attention asymmetric design.Source: §2.3 · pp. 6–7

3. CARLA

CARLA uses anomaly-injected contrastive pretraining to learn time-series window representations that bring temporally proximate windows together. It then uses representation-space neighbour information for self-supervised normal/anomalous classification.

3. CARLA

  • CARLA’s end-to-end pipeline combines a Pretext Stage with a Self-supervised Classification Stage that classifies windows using their representation-space neighbors.The pipeline learns representations with anomaly injection, then uses nearest and furthest neighbors to support anomaly detection.Source: §3 · p. 7, Figure 2 · p. 8

3.1. Anomaly Injection

  • Anomaly injection enables self-supervised learning without ground-truth labels, using generic heuristics for prevalent out-of-distribution instances rather than attempting to represent every anomaly type.The authors describe exhaustive anomaly coverage as unattainable.Source: §3.1 · p. 8

3.1.1. Anomaly Injection Steps

  • Training randomly injects point or subsequence anomalies into selected dimensions, with injected portions ranging from one data point to 90% of a window.Randomly selecting a subset of dimensions is intended to reflect that multivariate anomalies may not affect every dimension and to diversify injections.Source: §3.1.1 · pp. 8–9

3.1.2. Point Anomalies

  • CARLA injects single-point spike anomalies at randomly selected instances, using Global and Contextual types.Source: §3.1.2 · p. 10

3.1.3. Subsequence Anomalies

  • Subsequence injections create contextual out-of-distribution examples motivated by Outlier Exposure, using Seasonal, Shapelet, and Trend pattern anomalies.Source: §3.1.3 · p. 10

3.2. Pretext Stage

  • The Pretext Stage pairs contrastive representation learning with a post-processing step that identifies semantically meaningful nearest and furthest neighbors for each learned representation.Source: §3.2 · pp. 10–11

3.2.1. Part One: Contrastive Representation Learning

  • Part One uses a multi-channel ResNet with varied kernel sizes and triplet loss to bring an anchor closer to a preceding window and separate it from its anomaly-injected counterpart.Different kernel sizes capture features at multiple temporal scales; the preceding positive window is randomly selected from the previous y windows.Source: §3.2.1 · pp. 11–12
  • In Part One, CARLA learns similar representations for temporally proximate windows and uses injected anomalies to improve the decision boundary, reducing false positives and enhancing precision.The passage compares precision with current state-of-the-art models but gives no quantitative results.Source: §3.2.1 · p. 12

3.2.2. Part Two: Nearest and Furthest Neighbours

  • Part Two uses Q nearest and furthest neighbours of learned representations as semantic similarity and dissimilarity priors; nearest neighbours belong to the same class in most cases.These neighbour sets are then used to enhance the self-supervised classification method.Source: §3.2.2 · p. 12

3.3. Self-supervised Classification Stage

  • The self-supervised classifier starts from pretext features, aligns anchors with nearest neighbours, separates them from furthest neighbours, and uses entropy loss to encourage class diversity.The combined loss learns discriminative representations for distinguishing normal and anomalous windows.Source: §3.3 · pp. 13–15

3.4. CARLA’s Inference

  • At inference, CARLA labels a new window by whether it belongs to the majority class of original windows, and its majority-class probability serves as an anomaly score, with lower values indicating higher anomaly likelihood.The majority class is the class containing the greatest number of anchors.Source: §3.4 · pp. 15–16

4. Experiments

The experiments assess CARLA against benchmark methods across seven real-world TSAD datasets. The datasets cover univariate and multivariate series, with unlabeled training data and predefined train/test splits except for Yahoo.

4. Experiments

  • The evaluation compares CARLA with alternative methods across seven commonly used real benchmark datasets for time-series anomaly detection.The experiments include both univariate and multivariate datasets.Source: §4.1 · p. 16, §4 · p. 16

4.1. Benchmark Datasets

  • All datasets except Yahoo have predefined train/test splits with unlabeled training data.Source: §4.1 · p. 16
  • The datasets include NASA’s MSL and SMAP spacecraft-monitoring data, SMD from 28 servers with 38 sensors, and SWaT water-treatment data gathered over 11 days using a 51-sensor platform.Source: §4.1 · p. 17

4.2. Benchmark Methods

CARLA is compared with ten TSAD models spanning semi-supervised, unsupervised, and self-supervised approaches. The comparison also includes a random anomaly-score model as an evaluation reference.

  • The benchmark includes ten TSAD models from semi-supervised, unsupervised, and self-supervised categories.Source: §4.2 · p. 18
  • The comparison includes LSTM-VAE, reconstruction-based Donut, OmniAnomaly, AnomalyTransformer and TranAD, and forecasting-based THOC and TimesNet.Source: §4.2 · p. 18
  • MTAD-GAT, TS2Vec, and DCdetector represent hybrid and representation-based self-supervised approaches, while a random anomaly-score model illustrates evaluation metrics and strategies.Source: §4.2 · p. 18

4.3. Evaluation Setup

CARLA uses a shared architecture and hyperparameter configuration across datasets, with separate pretext and self-supervised classification training stages. The evaluation reports conventional F1 without point adjustment.

  • CARLA uses a three-layer ResNet with kernel sizes 8, 5, and 3, and a 128-dimensional representation.The authors use these kernel sizes to capture temporal dependencies.Source: §4.3 · p. 18
  • Across datasets, CARLA uses window size 200, 10 classes, five nearest and furthest neighbours, entropy coefficient 5, and trains for 30 and 100 epochs in its two stages.Source: §4.3 · p. 18
  • The evaluation excludes point adjustment and reports conventional F1 without it.The authors relegate point-adjusted results to Appendix B.Source: §4.3 · p. 18

4.4. Benchmark Comparison

CARLA achieves higher F1 and AU-PR than other models across nearly all evaluated datasets, with stated exceptions. It has the best average F1 rank on both multivariate and univariate benchmarks and generally combines low false-positive rates with competitive AU-PR.

  • For datasets with multiple time series, the authors sum per-series confusion-matrix counts before calculating overall precision, recall, and F1 rather than averaging individual F1 scores.Source: §4.4 · p. 20
  • CARLA’s average F1 rank is 1.00 on univariate and 1.60 on multivariate datasets, and the authors report significant differences from other models.Source: Figure 4 · p. 21, §4.4 · p. 22
  • CARLA outperforms other models in F1 and AU-PR across datasets except SWaT for both metrics and Yahoo-A1 for AU-PR, where it ranks third.Source: Table 3 · p. 20, §4.4 · p. 21
  • CARLA has the highest precision across all seven datasets and the highest F1 except on SWaT, while OmniAnomaly’s median precision is below 19.67%.Source: §4.4 · p. 21
  • On multivariate datasets, CARLA generally has lower FPR while remaining near the low-FPR, high-AU-PR region; on SWaT it has the lowest FPR and third-highest AU-PR.Source: Figure 5 · p. 22, §4.4 · pp. 22–23
  • On Yahoo-A1, CARLA has a low-FPR, high-AU-PR balance and ranks third, while on KPI it has low FPR and the highest AU-PR.Source: §4.4 · p. 23

4.5. Ablation Study

The ablation study examines CARLA’s stages, positive-pair selection, injected anomaly types, and loss components. Across these tests, the second stage improves representation separation, while the choice of pairs, anomaly types, and losses also matters.

4.5.1. Effectiveness of CARLA’s Stages

  • On M-6 in MSL, the Self-supervised Classification stage improves normal/anomalous separation and yields more clearly separated anomaly scores closer to 0 or 1.The authors interpret the latter distribution as indicating improved discrimination between normal and anomalous windows.Source: Figure 6 · p. 24, §4.5.1 · p. 25

4.5.2. Positive Pair Selection

  • Random temporal neighbours outperform weak noise augmentation for positive pairs; the authors favor temporal proximity because anomalies are rare.The comparison uses the MSL benchmark with identical configurations and hyperparameters.Source: §4.5.2 · p. 25, Table 4 · p. 25

4.5.3. Effectiveness of Different Anomaly Types

  • Removing trend anomalies significantly harms precision, while removing contextual or shapelet anomalies also lowers precision, with recall remaining high or improving.Contextual-anomaly removal slightly decreases precision and improves recall; shapelet-anomaly removal moderately decreases precision while recall remains relatively high.Source: §4.5.3 · p. 26
  • Removing global and seasonal anomalies reduces precision while recall improves or remains high, and the authors identify these types as relatively more significant to CARLA’s performance.The reported analysis associates global-anomaly removal with improved recall and seasonal-anomaly removal with high recall.Source: §4.5.3 · p. 26

4.5.4. Effectiveness of Loss Components

  • Including both Linconsistency and Lentropy yields the best reported precision, recall, F1, and AU-PR among the tested loss-component settings.Settings using only one loss component perform relatively lower across the metrics.Source: Table 6 · p. 27, §4.5.4 · p. 27

4.6. Parameter Sensitivity

The parameter-sensitivity experiments vary window size, class count, neighbour count, and the entropy coefficient. The authors report preferred settings for each, while noting stability across several tested values.

4.6.1. Effect of Window Size

  • Comparisons across MSL, SMD, and Yahoo show that window size 200 consistently outperforms other tested sizes based on both F1 score and AU-PR.Performance is stable for window sizes between 150 and 250.Source: §4.6.1 · pp. 27–28, Figure 7 · p. 28

4.6.2. Effect of Number of Classes

  • On MSL, Figure 8 reports F1 score and AU-PR across class counts; the model performs relatively well across counts, and increasing the count beyond 10 does not significantly improve performance.Source: Figure 8 · p. 28, §4.6.2 · p. 28

4.6.3. Effect of Number of neighbours in N and F

  • For the MSL dataset, the Figure 8b comparison shows that employing 5 nearest/furthest neighbours outperforms the other options, consistently achieving higher F1 scores and AU-PR values.Source: Figure 8 · p. 28, §4.6.3 · p. 29

4.6.4. Effect of Entropy Coefficient

  • On MSL, entropy coefficient 5 yields the highest F1 and AU-PR scores; coefficients 2 and 10 are competitive but slightly lower, while coefficient 0 significantly reduces model performance.Source: Figure 8 · p. 28, §4.6.4 · p. 29

5. Conclusion

CARLA combines contrastive representation learning with anomaly injection and neighbour information for time-series anomaly detection. Its evaluation spans 208 datasets from seven commonly used benchmarks, and the authors identify sampling and anomaly-injection choices as areas for further work.

5. Conclusion

  • CARLA combines contrastive representation learning with anomaly injection and semantically meaningful nearest and furthest neighbours to learn representations aligned with time-series patterns.The conclusion associates these neighbour-based priors with enhanced detection accuracy.Source: §5 · pp. 29–30
  • Evaluation on seven commonly used time-series benchmarks encompassing 208 datasets with diverse real-world anomalies shows promising anomaly-detection results.The authors describe the results as demonstrating CARLA’s effectiveness in this domain.Source: §5 · p. 30
  • Future work is proposed on refining positive and negative sample selection and broadening injected anomaly forms; varying anomaly severity is hypothesized to improve representation-space detection.The authors present severity variation as a hypothesis rather than an established result.Source: §5 · p. 30
  • Datasets with many abnormal samples can produce more anomalous windows during the Pretext Stage, making positive and negative selection for abnormal anchors important to revisit.The authors identify this as relevant to the unsupervised setting.Source: §5 · p. 30
  • The authors advocate conventional F1 without point adjustment and AU-PR for evaluating TSAD models, with findings intended to support benchmarking and comparison.They specifically mention datasets such as WADI and KPI.Source: §5 · p. 30

Appendix A. List of Symbols and Definitions

Table A.8 demonstrates the symbols used in the paper.

Appendix A. List of Symbols and Definitions

  • Table A.8 demonstrates the symbols used in the paper.The supplied passage does not enumerate the symbols or their definitions.Source: Appendix A · p. 30

Appendix B. Point Adjustment (PA)

The appendix describes point adjustment (PA), which many TSAD methods rely on, and reports that the Random model outperforms all state-of-the-art models when PA is incorporated.

Appendix B. Point Adjustment (PA)

  • Figure B.9 illustrates PA on 12 samples, showing anomaly scores, thresholded detections, ground-truth labels, adjusted detections, and evaluation of both adjusted and unadjusted results.Source: Figure B.9 · p. 31
  • PA treats every earlier segment in a detected anomalous subsequence as anomalous, a strategy many existing TSAD methods rely on.Source: Appendix B · pp. 30–31
  • The Random model outperforms all state-of-the-art models under PA; the reported comparison covers PA precision, recall, and F1 across multivariate and univariate datasets.Source: Appendix B · pp. 31–32, Table A.9 · p. 32
Loading 2308.09296v4…