Source-linked AI summary
Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, Anton van den Hengel
TL;DR
Autoencoders may reconstruct anomalies too well, weakening reconstruction error as an anomaly criterion. MemAE augments the autoencoder with a memory of prototypical normal patterns and retrieves relevant items for reconstruction. Experiments across datasets and applications support its generalization and effectiveness.
Problem
Autoencoders can generalize to abnormal inputs and reconstruct them well, so higher reconstruction error is not always a reliable anomaly criterion.
Method
MemAE uses the encoder output as a query to retrieve relevant items from a memory trained to record prototypical normal patterns, then reconstructs from those items.
Results
Experiments on various datasets from different applications support MemAE's generalization and effectiveness for unsupervised anomaly detection.
Takeaways & Limitations
Using fixed normal-pattern memory at test time strengthens reconstruction errors on anomalies as an anomaly-detection criterion.
Takeaways & Limitations
The paper identifies future work in integrating the memory module into more complicated base models and applying it to more challenging applications.
Abstract
from arXiv · showhide
Deep autoencoder has been extensively used for anomaly detection. Training on the normal data, the autoencoder is expected to produce higher reconstruction error for the abnormal inputs than the normal ones, which is adopted as a criterion for identifying anomalies. However, this assumption does not always hold in practice. It has been observed that sometimes the autoencoder "generalizes" so well that it can also reconstruct anomalies well, leading to the miss detection of anomalies. To mitigate this drawback for autoencoder based anomaly detector, we propose to augment the autoencoder with a memory module and develop an improved autoencoder called memory-augmented autoencoder, i.e. MemAE. Given an input, MemAE firstly obtains the encoding from the encoder and then uses it as a query to retrieve the most relevant memory items for reconstruction. At the training stage, the memory contents are updated and are encouraged to represent the prototypical elements of the normal data. At the test stage, the learned memory will be fixed, and the reconstruction is obtained from a few selected memory records of the normal data. The reconstruction will thus tend to be close to a normal sample. Thus the reconstructed errors on anomalies will be strengthened for anomaly detection. MemAE is free of assumptions on the data type and thus general to be applied to different tasks. Experiments on various datasets prove the excellent generalization and high effectiveness of the proposed MemAE.
1. Introduction
Unsupervised anomaly detection is difficult in high-dimensional data because normal-only training lacks human supervision and autoencoders may reconstruct anomalies too well. MemAE addresses this by retrieving prototypical normal patterns from memory for reconstruction, strengthening anomaly reconstruction errors.
- Unsupervised anomaly detection learns a normal profile from normal examples alone and identifies samples that do not conform to it.
- High-dimensional data makes normal-profile modeling especially challenging because such data are notoriously difficult to model.
- Autoencoders use an encoding bottleneck and normally detect anomalies through higher reconstruction error than for normal inputs.
- Autoencoders can reconstruct abnormal inputs well, especially when anomalies share normal compositional patterns or the decoder is too strong.
- MemAE queries a memory with the input encoding, sparsely retrieves relevant items, and sends their aggregate to the decoder instead of directly decoding the encoding.
- During training, sparse addressing encourages limited memory slots to capture prototypical normal patterns; during testing, fixed memory items provide reconstruction from normal-pattern neighborhoods.
2. Related Work
Related methods learn normal-data profiles through one-class classification, clustering, reconstruction, or memory networks. These approaches address unsupervised anomaly detection from different modeling perspectives, with high-dimensional processing remaining a challenge for some methods.
- One-class classification methods learn a discriminative boundary around normal samples, while clustering methods build detailed normal-data profiles for anomaly identification.
- One-class and clustering methods usually suffer from suboptimal performance when processing high-dimensional data.
- Reconstruction-based methods assume anomalies cannot be accurately represented by models learned only from normal data, using techniques such as PCA and sparse representation.
- Video anomaly detection methods use probabilistic PCA, dynamic textures, sparse coding, online dictionaries, or spatial-temporal deep-learning information.
- Memory-augmented networks use external memory for attention-based retrieval, one-shot learning, multimodal generation, and preserving detailed data structure.
3. Memory-augmented Autoencoder
MemAE inserts a memory module between the encoder and decoder, using sparse attention to reconstruct inputs from relevant prototypical memory patterns. During training, this memory is learned from normal data; during testing, restricting reconstruction to retrieved normal patterns strengthens anomaly reconstruction errors.
- Encoder and Decoder: MemAE encodes each input into a query, retrieves relevant memory items through attention-based addressing, aggregates them, and sends the result to the decoder.The encoder generates the query, while the decoder reconstructs samples from retrieved memories rather than directly from the encoding.
- Memory-based Representation: The memory is a matrix of N fixed-dimensional vectors that stores prototypical encoded patterns for reconstruction.Each memory item is a row vector, and N defines the memory's maximum capacity.
- Attention for Memory Addressing: Attention computes memory weights from similarity between the input encoding and memory items, producing a weighted memory representation.The addressing weights are non-negative and sum to one, and the memory retrieves items most similar to the query.
- Training: Training updates the encoder, decoder, and memory jointly so restricted reconstruction encourages memory slots to capture representative normal patterns.The training objective combines reconstruction error with a sparsity regularizer on memory addressing weights.
- Testing: At test time, fixed memory contents reconstruct inputs from selected normal patterns, so abnormal encodings are replaced by normal patterns and produce larger reconstruction errors.Anomaly detection uses the reconstruction MSE as its criterion.
- Hard Shrinkage for Sparse Addressing: Hard shrinkage and entropy regularization promote sparse addressing, encouraging representations to use fewer relevant memory items.The shrinkage threshold λ can be selected in the interval [1/N, 3/N], after which the weights are re-normalized.
4. Experiments
Experiments across image, video, and cybersecurity datasets show that MemAE generally improves anomaly detection over comparable autoencoder-based and other baseline methods. Visualizations and ablations attribute this behavior to prototypical normal memory patterns and sparse memory addressing.
- Image anomaly detection: MemAE generally outperforms the compared methods on image anomaly detection, with sparse addressing improving results over AE without memory.The image experiments average AUC values across 10 datasets sampled from MNIST and CIFAR-10.
- Visualizing the memory: Memory slots learned from normal MNIST digits record different prototypical patterns of the normal training samples.A decoded individual slot may appear blurry and noisy because reconstructions usually combine several addressed items.
- Visualizing the memory: For anomalous MNIST inputs, MemAE retrieves normal memory patterns and can reconstruct a different normal digit, producing significant reconstruction error.An input “9” with training on “5” is reconstructed as a “5”, while AE without memory may reconstruct abnormal samples well.
- Video anomaly detection: On video datasets, MemAE produces much better results than TSC and StackRNN, while sparse addressing remains beneficial compared with AE and MemAE-nonSpar.On UCSD-Ped2, its error map significantly highlights an abnormal vehicle and bicycle event, whereas AE reconstructs the anomaly well.
- Video anomaly detection: MemAE achieves better or comparative performance than other methods across video settings while using only reconstruction error and supporting different data types.The authors emphasize general anomaly detection rather than maximizing accuracy on a particular application, and report 0.0262 seconds per UCSD-Ped2 frame (38 fps).
- Cybersecurity anomaly detection: On KDDCUP, the proposed models obtain superior performance after evaluation with average precision, recall, and F1 score over 20 runs.The authors attribute the result to explicitly memorizing behavior patterns of attack samples.
- Ablation studies: Removing either hard shrinkage or entropy loss degenerates UCSD-Ped2 AUC performance, indicating both components contribute to effective sparse addressing.Without hard shrinkage, testing may use noisy non-sparse weights; entropy loss helps optimize addressing weights early in training.
5. Conclusion
The paper concludes that MemAE improves autoencoder-based unsupervised anomaly detection by retrieving prototypical normal patterns for reconstruction, strengthening anomaly reconstruction errors. Experiments across datasets and applications support its generalization and effectiveness, while future work targets more complex base models and challenging applications.
- MemAE uses encoded inputs as queries to retrieve relevant memory patterns for reconstruction rather than feeding encodings directly to the decoder.The memory is trained to record prototypical normal patterns.
- MemAE reconstructs normal samples well and enlarges reconstruction errors for anomalies, strengthening reconstruction error as the detection criterion.
- Experiments on various datasets from different applications demonstrate the generalization and effectiveness of MemAE.
- Future work will integrate the general, encoder-decoder-agnostic memory module into more complicated base models and apply it to more challenging applications.