Source-linked AI summary
MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection
Junho Song, Keonwoo Kim, Jeonglyul Oh, Sungzoon Cho
TL;DR
Reconstruction-based anomaly detectors can reconstruct anomalies too well, while real-world multivariate time series also pose challenges from imbalance and unlabeled anomalies. MEMTO combines a Transformer with gated normal-pattern memory, two-phase K-means initialization, and input- and latent-space anomaly scoring. Across five benchmarks, its average F1-score is 95.74%, up from 93.62% for Anomaly Transformer.
Problem
Reconstruction-based methods may over-generalize by reconstructing anomalies too well, while multivariate time-series detection also faces severe data imbalance and unlabeled anomalies.
Source: §1 · pp. 1–2Method
MEMTO is a reconstruction-based Transformer with gated normal-pattern memory, K-means initialization in a two-phase training paradigm, and a detection criterion using input and latent spaces.
Source: §1 · p. 2, §3.3 · p. 5Results
Average F1-score rises from 93.62% for Anomaly Transformer to 95.74% for MEMTO across five multivariate time-series benchmarks.
Source: §4.2 · p. 7Takeaways & Limitations
Stored normal-pattern features can make anomaly reconstructions resemble normal samples, making anomalies difficult for MEMTO to reconstruct.
Source: §1 · p. 2A thorough theoretical proof of the efficacy of the memory update stages and bi-dimensional criterion remains to be established.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Detecting anomalies in real-world multivariate time series data is challenging due to complex temporal dependencies and inter-variable correlations. Recently, reconstruction-based deep models have been widely used to solve the problem. However, these methods still suffer from an over-generalization issue and fail to deliver consistently high performance. To address this issue, we propose the MEMTO, a memory-guided Transformer using a reconstruction-based approach. It is designed to incorporate a novel memory module that can learn the degree to which each memory item should be updated in response to the input data. To stabilize the training procedure, we use a two-phase training paradigm which involves using K-means clustering for initializing memory items. Additionally, we introduce a bi-dimensional deviation-based detection criterion that calculates anomaly scores considering both input space and latent space. We evaluate our proposed method on five real-world datasets from diverse domains, and it achieves an average anomaly detection F1-score of 95.74%, significantly outperforming the previous state-of-the-art methods. We also conduct extensive experiments to empirically validate the effectiveness of our proposed model's key components.
1 Introduction
The introduction frames multivariate time-series anomaly detection as an unsupervised task with normal-only training data, while noting the difficulty of modeling temporal dependencies and inter-variable correlations. It presents MEMTO’s gated normal-pattern memory, two-phase training, and bi-dimensional detection criterion as responses to over-generalization in reconstruction-based methods.
1 Introduction
- The task is to classify each multivariate time-series timestamp as normal or abnormal, despite severe data imbalance and unlabeled anomalies.The stated unsupervised-learning assumption is that training data contains only normal samples.Source: §1 · p. 1
- Reconstruction-based methods can over-generalize by reconstructing anomalies too well, for example when encoders capture anomaly-specific features or decoders have excessive capacity.The introduction also notes that traditional methods struggle to capture dynamic nonlinear temporal dependencies and complex inter-variable correlations.Source: §1 · pp. 1–2
- MEMTO combines a reconstruction-based Transformer with gated memory items that represent normal patterns and learn how much to update from each input.The two-phase training paradigm initializes memory items with K-means, and the detection criterion considers both input and latent spaces.Source: §1 · p. 2
- The contributions include a data-driven gated memory module, a two-phase training paradigm intended to stabilize memory-based models, and a bi-dimensional detection criterion.The authors also report state-of-the-art results on five real-world benchmark datasets and ablation studies of key components.Source: §1 · p. 2
2 Related work
Related work describes memory networks that retrieve and write memory, including attention-based retrieval, and contrasts MEMTO’s learned update intensity with earlier memory-based anomaly detection approaches.
2 Related work
- An improved Memory Network [34] uses attention to retrieve entries in an end-to-end manner, requiring less training supervision than earlier approaches [38].The passage identifies external memory as supporting retrieval and writing [11] [38].Source: §2 · p. 2
- Unlike MNAD [24], which adds weighted queries without controlling their injection, MEMTO learns how intensively each memory item updates for new normal patterns.MemAE [8] is described as lacking an explicit memory-updating process.Source: §2 · p. 3
3 Method
MEMTO encodes time-series sub-series into latent queries, uses prototypical memory items to shape their reconstruction, and scores anomalies using deviations in both latent and input space. Its training combines reconstruction and sparsity losses with a two-phase procedure that initializes memory items by clustering queries.
3.1 Overall description of MEMTO
- MEMTO encodes each input sub-series, uses the resulting queries to retrieve or update gated-memory items, and decodes combined query-memory features into a reconstructed sub-series.The memory update stage operates during training; the updated queries are fed to the decoder.Source: §3.1 · p. 3, §3.1.2 · p. 4
3.1.1 Encoder and decoder
- A Transformer encoder projects input sub-series into latent space, while a weak two-layer fully connected decoder reconstructs them.The decoder is kept weak so reconstruction depends on the encoder's ability to represent the input.Source: §3.1.1 · pp. 3–4
3.1.2 Gated memory module
- Memory items represent prototypical normal patterns, and a learned update gate controls how much newly acquired normal patterns change each item.The memory regularizes latent representations toward normal prototypes, making anomalous inputs harder to reconstruct.Source: §3.1.2 · pp. 4–5
3.2 Training
- Training minimizes reconstruction loss and adds an entropy loss to encourage sparse retrieval of closely relevant memory prototypes.The reconstruction loss is the L2 distance between input and reconstructed sub-series.Source: §3.2 · p. 5
3.3 Two-phase training paradigm
- The two-phase paradigm first trains MEMTO to reconstruct inputs, then clusters queries from a random 10% of training data and uses centroids to initialize memory items before the second training phase.The proposed clustering initialization addresses the risk of instability from randomly initialized, incrementally updated memory items; the paper does not claim K-means is universally optimal.Source: §3.3 · pp. 5–6
3.4 Anomaly criterion
- The anomaly criterion combines latent-space deviation from the nearest memory item with input-space deviation, using normalized latent deviation to weight input deviation.The resulting anomaly score is calculated at each time point.Source: §3.4 · p. 6
4 Experiment
MEMTO is evaluated on five real-world multivariate time-series datasets, with a 95.74% average F1-score reported against the previous state-of-the-art. Ablations and analyses examine its detection criterion, memory module, training paradigm, and computational trade-offs.
4.2 Main results
- 95.74% average F1-score versus 93.62% for Anomaly Transformer across five real-world tasks.The datasets span server, NASA rover and satellite, water-treatment, and pooled-server settings; inputs use non-overlapping windows of length 100 and an 80/20 training-validation split.Source: Table 1 · p. 6, §4.1 · pp. 6–7, §4.2 · p. 7
4.3 Ablation study
- 69.23% and 60.16% average F1-scores with ISD-only and LSD-only, respectively; using both achieves the highest performance across datasets with lower variance.Source: Table 2 · p. 7, §4.3 · pp. 7–8
- Removing Gated memory cuts average F1 by 32.56p%; MemAE and MNAD perform comparably, while update-gated memory improves results.Source: Table 3 · p. 7, §4.3 · p. 8
- Without K-means initialization, MEMTO’s average F1 decreases by 8.4p%, versus 14.7p% for MemAE and 15.4p% for MNAD.The authors present the two-phase training paradigm as applicable to memory-module-based models.Source: Table 3 · p. 7, §4.3 · p. 8
4.4 Discussion and analysis
- Gated memory yields smaller normal-to-abnormal mean-LSD ratios on most datasets, indicating greater separation of abnormal and normal queries than with the compared modules.Source: Table 4 · p. 8, §4.4 · pp. 8–9
- On randomly sampled 150-point segments, the combined criterion detects anomalies robustly with fewer false alarms than Anomaly Transformer; LSD-only scores fail to exceed threshold at abnormal points.Source: Figure 3 · p. 9, §4.4 · p. 9
- Two-phase training takes approximately 2.45 times longer but enhances performance by 10.4%p; against Anomaly Transformer, MEMTO trains over twice as long yet infers 1.17 seconds faster.Source: Table 5 · p. 9, §4.4 · pp. 9–10
- Performance varies little across memory-item counts; ten items are selected to balance performance and computational complexity.Source: Figure 4 · p. 10, §4.4 · p. 10
5 Conclusion
MEMTO is an unsupervised reconstruction-based model for multivariate time-series anomaly detection, combining adaptive normal-pattern memory, two-phase training, and a bi-dimensional anomaly criterion. The authors report state-of-the-art performance on real-world benchmarks while noting that the method lacks a thorough theoretical proof and does not visually inspect stored prototypes.
5 Conclusion
- MEMTO combines adaptive memory for normal patterns, robust two-phase training, and a bi-dimensional anomaly criterion that considers both latent and input spaces.The authors report state-of-the-art performance against competitive models on real-world multivariate time-series benchmarks.Source: §5 · p. 10
- A thorough theoretical proof of the efficacy of the memory update stages and bi-dimensional criterion remains to be established.The authors also did not visually inspect the prototypical normal patterns stored in memory items.Source: §5 · p. 10
- MEMTO is tailored to multivariate time-series anomaly detection in complex cyber-physical systems, but the authors discourage uses related to financial crimes or other applications with negative societal consequences.Source: §5 · p. 10
A Training details
The experiments use shared hyperparameters across datasets, with several settings selected through grid search. Training details also specify the anomaly threshold, optimizer, and hardware.
A Training details
- The anomaly threshold is the top-p% of combined training and validation anomaly scores, with dataset-specific p values; training uses Adam at 5e-5 and early stopping after 10 epochs without validation-loss improvement.The setup uses 10 memory items, K-means clusters, PyTorch, and four NVIDIA GTX 1080 Ti 12GB GPUs.Source: A Training details · p. 14
A.1 Hyperparameter settings
- Important hyperparameters were selected by grid search, while other values were set to commonly used defaults based on empirical observations.Source: §A.1 · p. 14
- All datasets use the same settings: λ = 1e-2, lr = 5e-5, τ = 0.1, and M = 10, with M matching the number of K-means clusters and memory items.Source: §A.1 · p. 14
A.2 Dataset
The appendix describes the five benchmark datasets and gives an algorithm-level account of MEMTO’s forward process for an input sub-series.
A.2 Dataset
- The dataset table reports train, validation, and test sample counts, anomaly ratios, and data dimensions for five benchmarks.Source: Table 6 · p. 14
- SWaT was obtained by submitting a request through the iTrust dataset site.Source: §A.2 · p. 14
B Algorithm for MEMTO
- Algorithm 2 presents the matrix-operation forward process for a single input sub-series, with the algorithm returning a reconstructed sub-series.The listed steps include a gated memory update and a query update.Source: B Algorithm for MEMTO · p. 15
C Additional experiments
The additional experiments examine objective-function terms and anomaly criteria, while also describing the reporting and significance-testing procedure for repeated runs.
C.1 Objective function and anomaly criterion
- Table 7 reports F1-score ablations of the anomaly criterion and objective function, where Lrec denotes reconstruction loss and Lentr entropy loss.Source: Table 7 · p. 15
C Additional experiments
- Table 8 reports mean and standard deviation over 10 runs for Anomaly Transformer and MEMTO, with t-tests at p < 0.05 used to indicate statistical significance.Source: Table 8 · p. 15
C.1 Objective function and anomaly criterion
- MEMTO variants using only Lrec or Lentr with the bi-dimensional deviation-based criterion show competitive average F1, while using only ISD or LSD as anomaly criterion significantly reduces performance, supporting their combination.Source: §C.1 · p. 16
C.2 Statistical significance test
A t-test comparing MEMTO with Anomaly Transformer found a statistically significant performance difference across all datasets.
C.2 Statistical significance test
- The t-test compares MEMTO’s results with those of Anomaly Transformer.Source: §C.2 · p. 16
- Across all datasets, the comparison yielded p-values below 0.05, confirming a significant performance difference between the models.Source: §C.2 · p. 16
- The statistical test was conducted against Anomaly Transformer, identified as the latest state-of-the-art model.Source: §C.2 · p. 16
C.3 Number of decoder layers
The decoder-layer analysis balances reconstruction capacity against over-generalization and resource cost, while the gated memory mechanism separates normal from abnormal samples by LSD.
C.3 Number of decoder layers
- The decoder-layer figure reports F1-score as bars and parameter counts in millions as a blue line.Source: Figure 5 · p. 16
- A two-layer decoder is most suitable in the studied tasks, balancing detection performance against resource cost.A one-layer decoder lacks reconstruction capacity, whereas a ten-layer decoder may reconstruct anomalies too accurately and increase computational and memory costs.Source: §C.3 · p. 16, Figure 5 · p. 16
D.1 LSD values
- The proposed gated memory module generally yields lower mean LSD for normal than abnormal samples, with a larger relative difference than other memory mechanisms.The authors interpret this pattern as evidence that the mechanism captures prototypical normal-pattern features.Source: §D.1 · p. 16, Table 9 · p. 17
D.2 Anomaly score
The anomaly-score visualization covers four benchmark datasets using randomly sampled test segments. The authors report that MEMTO detects anomalies precisely with a low false-positive rate compared with other baselines.
D.2 Anomaly score
- MEMTO consistently detects anomalies precisely with a low false-positive rate compared with other baselines under point adjustment.Source: §D.2 · p. 17, Figure 6 · p. 17
- The figure plots anomaly scores for randomly sampled 150-length test segments from MSL, PSM, SMAP, and SWaT.Source: §D.2 · p. 17, Figure 6 · p. 17