Source-linked AI summary
Learning Normal Dynamics in Videos with Meta Prototype Network
Hui Lv, Chen Chen, Zhen Cui, Chunyan Xu, Yong Li, Jian Yang
TL;DR
Video anomaly detection must learn normality from normal data despite AE overgeneralization, memory costs, and differences between training and test scenes. The paper introduces attention-based Dynamic Prototype Units and meta-learns them as few-shot Meta Prototype Units for rapid scene adaptation. Across multiple benchmarks, the approach achieves state-of-the-art unsupervised anomaly-detection performance and demonstrates scene adaptation without extra memory consumption.
Problem
Video anomaly detection lacks anomaly coverage, while AE overgeneralization, memory-bank costs, and scene-specific normality limit existing normal-data methods.
Method
DPU learns compact normal-dynamics prototypes through differentiable attention, while MPU meta-learns an initialization and updates only its parameters for few-shot adaptation.
Results
The DPU-based AE achieves state-of-the-art performance on various unsupervised anomaly-detection benchmarks, while experiments validate MPU’s few-shot adaptation capability.
Takeaways & Limitations
Prototype-based normal-dynamics modeling provides an end-to-end, memory-efficient approach with demonstrated fast adaptation to new scenes.
Takeaways & Limitations
Memory-bank approaches are constrained by the substantial memory required to store normal patterns across the training set.
Abstract
from arXiv · showhide
Frame reconstruction (current or future frame) based on Auto-Encoder (AE) is a popular method for video anomaly detection. With models trained on the normal data, the reconstruction errors of anomalous scenes are usually much larger than those of normal ones. Previous methods introduced the memory bank into AE, for encoding diverse normal patterns across the training videos. However, they are memory-consuming and cannot cope with unseen new scenarios in the testing data. In this work, we propose a dynamic prototype unit (DPU) to encode the normal dynamics as prototypes in real time, free from extra memory cost. In addition, we introduce meta-learning to our DPU to form a novel few-shot normalcy learner, namely Meta-Prototype Unit (MPU). It enables the fast adaption capability on new scenes by only consuming a few iterations of update. Extensive experiments are conducted on various benchmarks. The superior performance over the state-of-the-art demonstrates the effectiveness of our method.
1. Introduction
Video anomaly detection learns regular patterns from normal data, but AE methods can overgeneralize, memory banks consume substantial resources, and scene-specific normality varies. The paper proposes DPU and its meta-learned MPU extension to encode normal dynamics compactly and adapt to unseen scenes.
- Motivation: Video anomaly detection is typically trained only on normal data because the space of possible anomalies is too broad to enumerate.During inference, patterns inconsistent with learned regularities are treated as anomalies.
- Limitations of existing methods: AE-based methods reconstruct current or future frames, but anomalous inputs can sometimes produce prediction errors similar to normal inputs.This undermines the assumption that abnormal scenes necessarily yield larger reconstruction errors.
- Limitations of existing methods: Memory-bank methods explicitly store shared normal patterns, but storing items across the training set is extremely memory-consuming.These methods aim to strengthen normal-region prediction while suppressing abnormal regions.
- Proposed approach: DPU uses differentiable attention over consecutive normal-frame encodings to mine diverse normal dynamics as compact prototypes within an AE.Normalcy weights form a map that guides prototype construction.
- Scene adaptation: Normal patterns differ across scenes, making a uniform threshold for updating memory-bank items unreliable for distinguishing normal from abnormal frames.The paper therefore targets adaptation to unseen surveillance scenarios.
- Scene adaptation: MPU applies meta-learning to DPU, learning an initialization that adapts to new scenes by updating only MPU parameters for a few iterations.The pretrained AE remains frozen, reducing the parameters involved in adaptation.
- Contributions: The paper reports state-of-the-art performance on multiple unsupervised anomaly-detection benchmarks and validates MPU’s few-shot adaptation capability.Its contributions include differentiable prototype learning and fast adaptation using few parameters and update iterations.
2. Related Work
Prior work uses AEs to model regular video patterns, but reconstruction or prediction can overgeneralize, while memory banks add storage costs. This paper instead combines attention-based dynamic prototypes with a scene-adaptive meta-learning formulation.
- AE-based methods: AE variants model regular spatial and temporal patterns and reconstruct or predict video frames for anomaly detection.Recurrent architectures and adversarial prediction have also been explored.
- AE-based methods: Future-frame prediction methods can suffer overgeneralization because anomalous frames may also produce small prediction errors.Thus, abnormal and normal frames are not always separated by reconstruction error.
- Memory-based methods: MemAE and LMN store normal patterns in memory banks, whereas this work dynamically learns normalcy through an attention mechanism and prototypes.The proposed prototypes adapt to the current scene spatially and temporally while avoiding the extra memory cost of stored items.
- Proposed framework: The DPU-based framework plugs a Dynamic Prototype Unit into an AE, reconstructs a normalcy encoding map, and aggregates it with AE features for later frame prediction.DPU can be inserted at different AE locations and resolutions.
- Meta-learning: Earlier meta-learning work adapted the whole network for scene-adaptive anomaly detection, while this paper meta-learns a more limited parameter set.The related-work passage describes the prior approach as learning an initialization for the entire model.
3. Method
The method augments an auto-encoder with a Dynamic Prototype Unit that learns compact, diverse normal-dynamics prototypes and enriches encoded video representations. Meta-learning turns this unit into a few-shot learner that adapts to new scenes through a small number of updates.
- Network and Inference: The framework is an end-to-end AE network whose DPU can be inserted at different resolutions and whose attention mappings generate the dynamic prototype pool.The DPU output is passed to the AE decoder for frame prediction, and inference considers feature-reconstruction and frame-prediction cues for anomaly scoring.
- Dynamic Prototype Unit: The DPU takes consecutive observed frames, encodes them in the AE, and predicts the subsequent ground-truth frame through the remaining AE layers.The selected hidden encoding is passed through the DPU before decoding the upcoming frame.
- Dynamic Prototype Unit: The DPU processes AE encodings through attention, prototype ensemble, prototype retrieval, and aggregation to produce normalcy-enhanced representations.Attention assigns normalcy weights, ensemble forms dynamic prototypes, and retrieval reconstructs a normalcy encoding that is combined with the original encoding.
- Objective Functions: Feature reconstruction and frame prediction jointly train the model, while compactness and diversity terms shape the learned prototype pool.Frame prediction uses the distance between the predicted and ground-truth frames; feature reconstruction encourages compact and diverse prototypes.
- Few-shot Normalcy Learner: The MPU meta-training procedure seeks an initialization that adapts the DPU and decoder to a new scene using one or a few update iterations.Training uses scene-specific input-output pairs, evaluates the adapted model on another pair from the same scene, and sets the episode length T to 1 for fast adaptation.
4. Experiments
Experiments evaluate the model in unsupervised and few-shot cross-dataset settings, alongside efficiency and component analyses. Results examine anomaly-detection performance, scene adaptation, inference speed, DPU placement, and prototype quantity.
- Evaluation Settings: The evaluation covers unsupervised anomaly detection on standard datasets and few-shot cross-dataset testing for new-scene adaptation.The unsupervised setting uses seen test scenarios, whereas cross-dataset testing examines adaptation to a new camera.
- Comparisons with SOTA Methods: The DPU achieves superior performance to state-of-the-art methods in the standard unsupervised comparison, while remaining below rGAN on Ped1 and Shanghai Tech.The comparison reports average AUC (%) across UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech; the authors attribute rGAN’s advantage to its more complex ConvLSTM architecture.
- Model Complexity and Inference Speed: The model runs at 166.8 FPS on one Nvidia RTX-2080Ti GPU and uses 1.28K extra DPU parameters with 10 prototypes.Unlike memory-bank methods, it requires no extra memory space for prototypes.
- Model Complexity and Inference Speed: Adaptation with K = 1 takes 0.04 seconds, and inference is almost 80 × faster than rGAN.The corresponding K = 1 adaptation speed is reported as 23.9 FPS for this method versus 1.3 FPS for rGAN.
- DPU Resolution Analysis: Higher-resolution DPU placements improve AUC because higher-resolution AE layers contain more anomaly cues for feature reconstruction.Activation maps use the L2 norm of spatial encoding vectors, where higher activation indicates more information in the encoding vector.
- Prototype Quantity Analysis: Ten prototypes are an appropriate default: increasing their number introduces noise and reduces prototype diversity, causing a drastic performance decline.The prototype count is controlled by the number of attention mapping functions in DPU.
5. Conclusion
The paper introduces an attention-based prototype module that models normal video dynamics for unsupervised anomaly detection. It also extends this module with meta-learning for few-shot scene adaptation.
- The prototype module explicitly models normal dynamics in video sequences through an attention mechanism.
- The module is fully differentiable and trained end-to-end without extra memory consumption.
- The approach achieves state-of-the-art performance on various unsupervised anomaly-detection benchmarks.
- Meta-learning improves the prototype module into a few-shot normalcy learner for scene adaptation.