Learning Memory-guided Normality for Anomaly Detection

1. Introduction

• Anomaly detection is typically unsupervised because anomalous events are context-dependent, costly to annotate, and rare in real-world data.At test time, events and activities not described by a model of normality are treated as anomalies.Source: §1 · p. 1
• Reconstruction and prediction methods use proxy tasks and can reconstruct abnormal frames, while a single-center hypersphere approach does not represent varied normal patterns.CNN features from abnormal frames may be reconstructed by combining features of normal ones, resulting in low reconstruction error.Source: §1 · pp. 1–2
• The approach outperforms the state of the art on UCSD Ped2, CUHK Avenue, and ShanghaiTech.The paper also reports effectiveness and efficiency on these standard benchmarks.Source: §1 · p. 2
• A memory module represents diverse normal video patterns with multiple prototypical features, limiting CNNs to representations built from memory items.The paper assumes a single prototype cannot capture the multiple modes or centroids present in normal-frame feature space.Source: §1 · p. 2
• Feature compactness pulls normal-frame features toward their nearest memory items, while feature separateness distinguishes the nearest from the second-nearest item.Together, the losses discourage a degenerate solution in which all memory items become similar and make features and items more discriminative.Source: §1 · p. 2
• A weighted regular score estimates anomalies in each test frame, and the memory updates only when a frame is judged normal.This update scheme is intended to prevent memory items from recording anomalous samples at test time.Source: §1 · p. 2

• Earlier unsupervised methods learn normality reconstructively with AEs, dictionaries, or generative models, or discriminatively with statistical boundaries, but may miss complex image and video distributions [3].Source: §2 · p. 2
• CNN-based reconstructive methods outperform classical approaches but may reconstruct anomalies; prediction can mitigate this, though [22] requires optical-flow estimation and ground-truth flow for training.Source: §2 · p. 2
• Unlike Deep SVDD [35], which maps normal data to a hypersphere center, this method reconstructs or predicts frames from memory-item combinations; unlike Deep-Cascade, it records normal patterns explicitly, including in test sequences.Its future-frame prediction avoids optical flow and is described as faster than [22].Source: §2 · p. 3
• Memory items explicitly record diverse, discriminative normal patterns; feature compactness and separateness losses enable 10 items versus 2,000 for MemAE, and the memory updates at test time.The model also discriminates anomalies while updating, allowing it to memorize normal patterns in test data.Source: §2 · p. 3
• Memory networks provide global read/write memory beyond compressed LSTM states, but this work applies them to anomaly detection by assigning normal patterns to individual prototypical items with a different update strategy.Source: §2 · p. 3

3. Approach

• The encoder retrieves and updates prototypical normal patterns in a memory, whose read features guide frame reconstruction alongside the query features.The model is trained end-to-end with reconstruction, feature compactness, and feature separateness losses; a weighted regular score prevents memory updates from abnormal frames at test time.Source: §3 · p. 3

3.1.1 Encoder and decoder

• The model uses a U-Net encoder-decoder with L2-normalized features; it removes encoder batch normalization and ReLU, and removes skip connections for reconstruction but retains them for future-frame prediction.For prediction, four successive frames are used to predict the fifth; the paper uses nearly the same architecture and losses for reconstruction and prediction.Source: §3 · p. 3, §3.1.1 · p. 4

3.1.2 Memory

• Memory reads combine all items using query-to-item matching probabilities, while updates use weighted averages of queries assigned to each item.At test time, a weighted reconstruction-error score above threshold γ marks a frame abnormal and excludes it from memory updates; the weighting emphasizes regions with large reconstruction errors.Source: §3.1.2 · pp. 4–5

3.2. Training loss

• Feature compactness pulls queries toward their nearest memory items, while feature separateness pushes the second-nearest item away to keep memory items distinct.The separateness loss uses the nearest item as a positive and the second-nearest as a hard negative, addressing the tendency of compactness alone to make all items similar.Source: §3.2 · pp. 5–6

3.3. Abnormality score

• The final abnormality score sums min-max-normalized query-to-nearest-memory distance and reconstruction error measured by PSNR, balanced by λ.Abnormal frames are assumed to have queries dissimilar to normal-pattern memory items and low PSNR because their patterns are not reconstructed well.Source: §3.3 · p. 6

4.1. Implementation details

• The experiments use UCSD Ped2, CUHK Avenue, and ShanghaiTech, resizing frames to 256 × 256 and training end-to-end for 1, 15, and 36 hours, respectively.The setup uses 10 memory items and batch size 4, with 60, 60, and 10 training epochs for the three datasets, respectively.Source: §4.1 · p. 6

4.2. Results

• Ours-P with memory achieves 97.0% AUC on UCSD Ped2 and 88.5% on CUHK Avenue, while Frame-Pred [22] outperforms it on ShanghaiTech.Frame-Pred [22] uses optical-flow and adversarial-learning modules; the paper reports 67 fps for its model versus 25 fps for Frame-Pred.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• Adding memory to prediction raises AUC by 2.7% on UCSD Ped2, 4.0% on CUHK Avenue, and 3.7% on ShanghaiTech.The authors report that memory boosts AUC regardless of task across all three datasets.Source: §4.2 · p. 7
• Ours-R with memory is competitive with reconstructive methods on UCSD Ped2 and outperforms them on other datasets except MemAE [8].Ours uses 2D convolutions and 10 memory items of size 512, versus MemAE’s 3D convolutions and 2,000 items of size 256; the reported speeds are 67 fps and 45 fps, respectively.Source: §4.2 · p. 7
• Inference takes 0.015 seconds per 256 × 256 image, or 67 fps, faster than Unmasking [41] at 20 fps, StackRNN at 50 fps, Frame-Pred [22] at 25 fps, and MemAE [8] at 45 fps.Source: §4.2 · p. 7

4.3. Discussions

• On UCSD Ped2 reconstruction, feature compactness alone reduces AUC, while feature separateness raises it by 3.8%; the losses are complementary.The ablations also find that updating memory with normal-frame features at test time boosts AUC, and the abnormality score combining PSNR with memory items performs better than PSNR alone.Source: §4.3 · pp. 7–8, Table 2 · p. 8
• Feature separateness makes query features activate only a few memory items and separates memory prototypes, while normal regions are predicted well and abnormal regions are highlighted.The qualitative examples show a vehicle, jumping, and a fight highlighted on UCSD Ped2, CUHK Avenue, and ShanghaiTech, respectively; using sixteen frames to reconstruct the ninth yields 91.0% AUC on UCSD Ped2, a 0.8% gain with about 4 MB more parameters.Source: §4.2 · p. 7, §4.3 · p. 8, Figure 5 · p. 8

• The model outperforms the state of the art in extensive evaluations on standard benchmarks.Source: §5 · p. 8
• The approach uses memory items to record multiple prototypes of normal data patterns for video anomaly detection.Source: §5 · p. 8
• Feature compactness and separateness losses separate memory items, enabling sparse access to the memory.Source: §5 · p. 8
• A new memory update scheme for settings with both normal and abnormal samples significantly boosts anomaly-detection performance.Source: §5 · p. 8