Source-linked AI summary
DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection
Yiyuan Yang, Chaoli Zhang, Tian Zhou, Qingsong Wen, Liang Sun
TL;DR
Time-series anomaly detection requires representations that separate anomalies from complex normal data, while anomalies can interfere with reconstruction-based learning. DCdetector uses asymmetric dual-attention contrastive learning with multi-scale processing and no reconstruction component. It achieves the best or comparable performance on eight benchmark datasets and avoids trivial collapse even without negative samples or stop-gradient operation.
Problem
Time-series anomaly detection needs representations that discriminate anomalies, but reconstruction-based learning can be hindered by anomalies and suitable reconstruction models are difficult to build.
Method
DCdetector uses a dual-branch asymmetric attention structure with multi-scale and channel-independent patching, trained by a pure contrastive loss without reconstruction error.
Results
The model achieves the best or comparable performance to state-of-the-art methods on eight benchmark datasets.
Takeaways & Limitations
DCdetector learns permutation-invariant representations that enlarge differences between normal points and anomalies while avoiding trivial collapse without negative samples.
Takeaways & Limitations
The method uses an inductive bias that anomalies have less interaction with the whole series than with adjacent points.
Abstract
from arXiv · showhide
Time series anomaly detection is critical for a wide range of applications. It aims to identify deviant samples from the normal sample distribution in time series. The most fundamental challenge for this task is to learn a representation map that enables effective discrimination of anomalies. Reconstruction-based methods still dominate, but the representation learning with anomalies might hurt the performance with its large abnormal loss. On the other hand, contrastive learning aims to find a representation that can clearly distinguish any instance from the others, which can bring a more natural and promising representation for time series anomaly detection. In this paper, we propose DCdetector, a multi-scale dual attention contrastive representation learning model. DCdetector utilizes a novel dual attention asymmetric design to create the permutated environment and pure contrastive loss to guide the learning process, thus learning a permutation invariant representation with superior discrimination abilities. Extensive experiments show that DCdetector achieves state-of-the-art results on multiple time series anomaly detection benchmark datasets. Code is publicly available at https://github.com/DAMO-DI-ML/KDD2023-DCdetector.
1 INTRODUCTION
Time series anomaly detection is important but difficult because anomalies are rare, varied, and embedded in complex normal patterns. DCdetector addresses these challenges with dual-branch contrastive representation learning and reports comparable or superior benchmark performance without reconstruction loss.
- Motivation: Anomaly detection supports safety and loss prevention across industrial, financial, and maintenance applications.Examples include wind-turbine failure prevention and financial-fraud detection.
- Motivation: Complex time series make anomalies difficult to define, while their rarity and coexistence with normal points complicate detection.Normal system behavior can vary across conditions, such as changing weather patterns for wind turbines.
- Limitations of Reconstruction: Reconstruction methods can be obstructed by anomalies when learning a clean model of normal time-series data.Normal and abnormal points may appear within one instance, making reconstruction-based learning harder.
- DCdetector: DCdetector uses a dual-branch contrastive structure with shared weights, patching-based attention, and multi-scale design for time-series representation learning.The model uses similarity between branches and multi-scale processing to capture temporal dependencies while reducing patching information loss.
- DCdetector: DCdetector is trained purely contrastively without reconstruction loss and learns permutation-invariant representations that enlarge differences between normal points and anomalies.Channel-independence patching is also proposed to enhance local semantic information.
- Results: DCdetector achieves comparable or superior performance to state-of-the-art methods on seven multivariate and one univariate benchmark datasets.The paper also discusses why the model avoids collapse without negative samples.
2 RELATED WORK
Related work spans statistical, machine-learning, deep-learning, supervised, unsupervised, and self-supervised approaches to time-series anomaly detection. DCdetector uses contrastive representation learning without negative samples and reports a concise alternative to reconstruction and association-modeling approaches.
- Time Series Anomaly Detection: Time-series anomaly detection methods include statistical, classical machine-learning, and deep-learning approaches.Examples include moving averages, exponential smoothing, ARIMA, clustering, density-based methods, and classification algorithms.
- Time Series Anomaly Detection: Supervised methods can perform better when anomaly labels are available, whereas unsupervised methods address settings where labels are difficult to obtain.Self-supervised methods have also been proposed to improve generalization in unsupervised anomaly detection.
- Architecture Comparison: Compared with reconstruction-based and Anomaly Transformer architectures, DCdetector omits reconstruction loss, specially designed Gaussian kernels, and MinMax association learning.The comparison presents DCdetector as a more concise architecture than the two alternatives.
- Contrastive Representation Learning: Contrastive representation learning places similar samples near one another and dissimilar samples farther apart in an embedding space.Classical contrastive models typically use positive and negative sample pairs.
- DCdetector: DCdetector applies multi-scale patching-based dual attention to distinguish time-series anomalies from normal points without negative samples.The model is reported not to collapse even without stop-gradient operation.
3 METHODOLOGY
DCdetector learns anomaly-discriminative representations from two permuted attention views of multivariate time series, avoiding reconstruction. Its pipeline uses channel-independent patching, multi-scale dual attention, contrastive optimization, representation discrepancy, and thresholded anomaly scores.
- Problem: The task is to distinguish anomalous from normal points in complex, mostly unlabeled time series, where anomalies are rare and normal behavior can vary across conditions.The paper motivates unsupervised and self-supervised representation learning because labels are costly and anomalies may be dynamic or previously unseen.
- Forward Process: DCdetector treats each multivariate channel independently, divides channels into patches, applies shared self-attention, and concatenates the channel representations.Instance normalization precedes this process, while channel independence is used to reduce parameters and overfitting.
- Dual Attention Contrastive Structure: The model creates patch-wise and in-patch representations as two permuted views, exploiting stable latent patterns for normal points and weaker shared patterns for anomalies.Patch-wise attention combines same-position points across patches, whereas in-patch attention combines points within each patch.
- Optimization: The two branches share weights and are trained with a pure contrastive loss using KL divergence and stop-gradient, without a reconstruction component.Stop-gradient trains the branches asynchronously, while the representation-learning design avoids interference from anomalous reconstruction losses.
- Anomaly Criterion: Anomaly scores are computed from representation discrepancy, with anomalies receiving higher point-wise scores than normal points and a threshold determining the binary output.The architecture therefore connects dual-view representation differences to the final anomaly criterion.
4 EXPERIMENTS
Experiments evaluate DCdetector across multivariate, univariate, and challenging anomaly-detection benchmarks using multiple metrics and comparisons. Results generally show state-of-the-art or competitive performance, robustness across settings, and sensitivity to selected design choices.
- Datasets: DCdetector is evaluated on seven benchmark datasets spanning multivariate and univariate time series scenarios.The evaluation includes MSL, SMAP, PSM, SMD, SWaT, NIPS-TS-SWAN, NIPS-TS-GECCO, and UCR.
- Baselines and Evaluation Criteria: The experiments compare DCdetector with nineteen competitive baselines on five real-world multivariate datasets and use multiple anomaly-detection metrics.Metrics include accuracy, precision, recall, F1-score, affiliation precision/recall, and VUS-related measures.
- Main Results: DCdetector achieves state-of-the-art F1 results on most benchmark datasets and performs better or comparably to Anomaly Transformer across most metrics.The comparison includes commonly used and recently proposed evaluation criteria.
- Multivariate Anomaly Detection: DCdetector achieves state-of-the-art results on NIPS-TS-SWAN and NIPS-TS-GECCO, including datasets with anomaly ratios ranging from 1.1% to 32.6%.The paper reports complete outperformance of other methods on these two datasets and better performance in most multi-metric comparisons.
- Univariate Anomaly Detection: On UCR univariate datasets, DCdetector correctly identifies and locates the single subsequence-anomaly segment and achieves optimal results.The evaluation is performed separately for each UCR sub-dataset, with average results reported in Table 4.
- Model Analysis: Ablations show that two stop-gradient modules give the best performance, while simultaneous denoising and normalization degrade performance.The final model therefore retains instance normalization but omits bilateral filtering.
- Model Analysis: DCdetector robustly detects point-wise and pattern-wise synthetic anomalies, while performance remains stable across window sizes from 30 to 210.Within the window-size range [45, 195], performance fluctuates by less than 2.3%; multi-scale patching also contributes to final performance.
5 CONCLUSION
DCdetector is a contrastive learning-based anomaly detector that uses dual-branch attention to learn permutation-invariant representations and improve separation between normal and abnormal points. Multiscale and channel-independence patching, together with a pure contrastive loss, support its reported benchmark performance.
- DCdetector uses a dual-branch attention structure to learn a permutation-invariant representation.
- The learned representation enlarges differences between normal points and anomalies, improving detection accuracy.
- Multiscale and channel-independence patching enhance the model’s time-series representation.
- The model uses a pure contrastive loss without reconstruction error.
- Extensive experiments report best or comparable performance on eight benchmark datasets against various state-of-the-art algorithms.
B DATASET DESCRIPTION
The evaluation uses seven adopted benchmark datasets covering univariate and multivariate time-series scenarios with different anomaly types and ratios.
- Seven benchmark datasets are used for evaluation, including MSL, SMAP, PSM, SMD, SWaT, NIPS-TS-SWAN, and NIPS-TS-GECCO.
- The datasets include both univariate and multivariate time series with different anomaly types and ratios.
- MSL, SMAP, PSM, SMD, SWaT, NIPS-TS-SWAN, and NIPS-TS-GECCO are multivariate, while UCR is univariate.
C EXTRA STUDIES
The extra studies examine parameter sensitivity while also reporting training memory usage and iteration time.
- Additional ablation experiments investigate the sensitivity of DCdetector to its parameters.
- The extra studies provide more detailed results than those reported in Section 4.5.1.
- The studies report memory usage and iteration time during training.
C.1 Study on Metrics in Loss Function
The study compares statistical distances for measuring discrepancy between patch-wise and in-patch representations. The proposed loss achieves state-of-the-art performance across all benchmarks, while Jensen–Shannon divergence degrades performance, especially on MSL.
- The proposed loss function achieves state-of-the-art performance on all benchmarks.
- Simple KL divergence produces a comparable result despite being an asymmetrical loss function.
- Jensen–Shannon divergence causes visible performance degradation, especially on the MSL benchmark.
C.2 Study on Multi-scale Patching
DCdetector evaluates seven multi-patching scales, using odd patch sizes to prevent information loss during upsampling. Multi-scale configurations increase memory use, and the best scale varies across datasets.
- Seven multi-patching scales are tested:,,,,,, and.
- Odd patch sizes are preferred to prevent information loss during upsampling.
- Multi-scale designs generally require more memory, while different datasets have different best-performing patching scales.The passage attributes this variation possibly to differences in information density and anomaly types.
C.3 Study on Window Size
Window size is an important hyperparameter because it splits time series into instances. DCdetector remains robust across a broad range of window sizes, with only slightly lower performance than the best settings from 45 to 195.
- Window size splits a time series into instances because a single point usually cannot serve as a sample.
- DCdetector is robust to different window sizes across the evaluated benchmarks.
- Across window sizes from 45 to 195, performance is only slightly below the best result for all benchmarks.
C.4 Study on Attention Head
The study varies the attention-head count, embedding dimension, and encoder depth to assess their effects on DCdetector. The model generally performs well with small attention-head counts and no more than three encoder layers, while embedding effects vary by benchmark.
- Attention Head: DCdetector achieves good performance with a small number of attention heads.The tested head counts are H∈{1, 2, 4, 8}.
- Embedding Dimension: Embedding dimension has little effect on SMAP and PSM, while MSL performs best with a small d_model and small memory.The tested embedding dimensions are d_model∈{128, 256, 512, 1024}.
- Encoder Layers: Different benchmarks have different optimal encoder-layer counts, but DCdetector reaches its best performance with no more than three layers.The tested layer counts are L∈{1, 2, 3, 4, 5}.
- Encoder Layers: The model does not fail with too few encoder layers or over-fit with too many encoder layers.
C.7 Study on Anomaly Threshold
The anomaly threshold has limited impact on DCdetector within the range 0.5–1. PSM and SMAP are more robust than MSL, and the best results occur at thresholds of 0.7 or 0.8.
- Anomaly threshold δ determines whether a sample is classified as anomalous based on Eq. 12.
- Thresholds from 0.5 to 1 have little effect on final performance across the three benchmarks.
- PSM and SMAP are more robust to anomaly threshold changes than MSL.
- The best results for the three benchmarks occur when δ equals 0.7 or 0.8.