Source-linked AI summary
Learning Memory-guided Normality for Anomaly Detection
Hyunjong Park, Jongyoun Noh, Bumsub Ham
TL;DR
The paper addresses unsupervised video anomaly detection when CNN proxy-task methods overlook diverse normal patterns and can reconstruct abnormal frames. It uses a prototype-recording memory module, compactness and separateness losses, and a new update scheme; experiments report state-of-the-art performance and efficiency on standard benchmarks.
Problem
CNN proxy-task methods for unsupervised anomaly detection do not explicitly model diverse normal patterns and can reconstruct abnormal video frames.
Method
A memory module records prototypical normal patterns, while feature compactness and separateness losses train discriminative memory items and features.
Results
The model outperforms the state of the art on standard video anomaly-detection benchmarks, with reported gains in ablation studies and efficient operation.
Takeaways & Limitations
Multiple prototypes and explicitly trained memory items provide a way to represent diverse normal patterns while reducing CNN representation capacity.
Abstract
from arXiv · showhide
We address the problem of anomaly detection, that is, detecting anomalous events in a video sequence. Anomaly detection methods based on convolutional neural networks (CNNs) typically leverage proxy tasks, such as reconstructing input video frames, to learn models describing normality without seeing anomalous samples at training time, and quantify the extent of abnormalities using the reconstruction error at test time. The main drawbacks of these approaches are that they do not consider the diversity of normal patterns explicitly, and the powerful representation capacity of CNNs allows to reconstruct abnormal video frames. To address this problem, we present an unsupervised learning approach to anomaly detection that considers the diversity of normal patterns explicitly, while lessening the representation capacity of CNNs. To this end, we propose to use a memory module with a new update scheme where items in the memory record prototypical patterns of normal data. We also present novel feature compactness and separateness losses to train the memory, boosting the discriminative power of both memory items and deeply learned features from normal data. Experimental results on standard benchmarks demonstrate the effectiveness and efficiency of our approach, which outperforms the state of the art.
1. Introduction
Video anomaly detection is typically unsupervised because anomalous events are context-dependent, labor-intensive to annotate, and difficult to collect. The paper addresses limitations of CNN proxy-task methods by modeling diverse normal patterns with memory prototypes and specialized losses.
- Anomaly detection aims to model normality without anomalous samples and identify test-time events not described by that model.
- CNN-based methods commonly reconstruct normal input frames and treat frames with large reconstruction errors as anomalies.
- Proxy reconstruction and prediction tasks extract general feature representations rather than directly modeling normal patterns.
- The proposed memory module represents diverse normal video patterns using multiple prototypical items, reducing reliance on CNN representation capacity.
- Feature compactness and separateness losses respectively cluster features with nearby memory items and prevent all memory items from becoming similar.
- The paper reports a new state of the art on standard video anomaly-detection benchmarks and provides extensive ablation analyses.
2. Related work
Prior work uses reconstructive or discriminative models to describe normality, but CNN reconstructive methods can also reconstruct anomalous samples. This paper combines a memory module with CNN features to represent diverse normal patterns while reducing CNN capacity.
- Unsupervised anomaly-detection methods learn normality without anomalous training data using reconstructive or discriminative formulations.
- CNN-based reconstructive approaches have advanced anomaly detection but can reconstruct anomalous samples.
- The framework reconstructs input frames or predicts future frames using an encoder, memory module, and decoder.
- For prediction, the model uses four successive frames to predict the fifth and avoids optical-flow computation, making it faster than methods requiring optical flow.
- Memory items record diverse and discriminative normal patterns through feature compactness and separateness losses, enabling 10 items instead of MemAE's 2,000.
3. Approach
The approach combines an encoder, memory module, and decoder to reconstruct input frames or predict future frames using prototypical normal patterns. It trains the memory with compactness and separateness objectives and filters abnormal test frames before memory updates.
- Framework: The framework uses an encoder, memory module, and decoder, trained end-to-end with reconstruction, compactness, and separateness losses.The encoder extracts queries, the memory retrieves and updates prototypical patterns, and the decoder reconstructs the frame.
- Abnormality detection: The model quantifies frame abnormalities using reconstruction discrepancies and distances between query features and their nearest memory items.These two discrepancies provide complementary signals for estimating abnormality.
- Memory: Memory reading computes query–item similarities, converts them into matching probabilities, and forms each retrieved feature as a weighted average of memory items.Using all items lets the model represent queries through combinations of diverse normal patterns.
- Memory: Memory updating assigns each query to its nearest item and updates that item using a weighted average of the corresponding queries.The weighted average emphasizes queries near the item rather than summing all assigned queries.
- Memory: At test time, a weighted regular score excludes frames above threshold γ from memory updates, limiting incorporation of abnormal patterns.The score focuses more on regions with large reconstruction errors.
- Training losses: Feature compactness pulls queries toward their nearest memory items, while feature separateness keeps memory items sufficiently distant to represent varied normal patterns.The separateness loss uses a margin α and treats the nearest and second-nearest items as positive and hard-negative samples.
4. Experiments
Experiments on three benchmark datasets evaluate the method through quantitative comparisons, runtime, qualitative localization, and ablations. The memory module and separateness loss improve anomaly detection while maintaining efficient inference.
- Experimental setup: The evaluation uses UCSD Ped2, CUHK Avenue, and ShanghaiTech, measuring average AUC against state-of-the-art methods.The benchmarks contain varied abnormal events and scene configurations.
- Quantitative results: 97.0% and 88.5% average AUC are achieved by Ours-P w/ Mem. on UCSD Ped2 and CUHK Avenue, respectively.The prediction model gives the best results on those two datasets, while Frame-Pred outperforms it on ShanghaiTech.
- Quantitative results: The memory module improves AUC across all datasets, with prediction-task gains of 2.7%, 4.0%, and 3.7% on UCSD Ped2, CUHK Avenue, and ShanghaiTech.The reconstruction model is competitive with reconstructive methods and outperforms them on other datasets except MemAE.
- Runtime: 67 fps is achieved for anomaly detection, exceeding the reported 20 fps for Unmasking, 50 fps for StackRNN, 25 fps for Frame-Pred, and 45 fps for MemAE.The implementation takes 0.015 seconds per 256 × 256 image on an Nvidia GTX TITAN Xp.
- Qualitative results: Qualitative results show normal regions predicted well while abnormal regions, including vehicles, jumping, and fighting, are highlighted.The visualization presents input frames, prediction errors, and abnormal regions for all three datasets.
- Ablation and representation analysis: A 3.8% AUC gain from the feature separateness loss, together with separated memory items and sparse query access, supports more discriminative representations.The feature compactness and separateness losses are complementary, and the combined abnormality score outperforms PSNR alone.
5. Conclusion
The paper presents an unsupervised video anomaly-detection approach that models diverse normal patterns with multiple prototypes stored in a memory module. Feature compactness and separateness losses, together with a new update scheme, support memory separation and improved anomaly-detection performance.
- Multiple prototypes represent diverse normal video patterns through prototypical features stored in memory items.
- Feature compactness and separateness losses separate memory items and enable sparse memory access.
- A new memory update scheme for mixed normal and abnormal test samples significantly boosts anomaly-detection performance.
- Extensive evaluations on standard benchmarks show that the model outperforms the state of the art.