Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection

Hang Zhou; Junqing Yu; Wei Yang

Source-linked AI summary

Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection

Hang Zhou, Junqing Yu, Wei Yang

arXiv:2302.05160v1Feb 10, 2023cs.CV

TL;DR

Weakly supervised video anomaly detection needs representations of normality as well as anomaly-focused features, because neglecting normal data may increase false alarms. UR-DMU combines expressive global-local attention, dual normal/abnormal memory banks, and Gaussian-regulated normal uncertainty learning. It achieves state-of-the-art results on UCF-Crime and XD-Violence, including 86.97% AUC on UCF-Crime and 81.77% AP with fused features on XD-Violence.

Problem
Existing weakly supervised methods focus on discriminative anomaly representations while neglecting normal data, which may yield a high false alarm rate.
Method
UR-DMU uses GL-MHSA for expressive video embeddings, dual memory banks for normal and abnormal prototypes, and uncertainty learning to model normality as a Gaussian latent space.
Results
UR-DMU outperforms previous methods on UCF-Crime and XD-Violence, achieving 86.97% AUC on UCF-Crime and 81.77% AP with fused features on XD-Violence.
Takeaways & Limitations
Modeling both normal and abnormal prototypes while regulating normal uncertainty supports separation of anomaly from latent normality in WS-VAD.
Takeaways & Limitations
The predicted anomaly windows and scores remain relatively coarse and unstable because the method uses snippet-level features.

Abstract

from arXiv · show

Learning discriminative features for effectively separating abnormal events from normality is crucial for weakly supervised video anomaly detection (WS-VAD) tasks. Existing approaches, both video and segment-level label oriented, mainly focus on extracting representations for anomaly data while neglecting the implication of normal data. We observe that such a scheme is sub-optimal, i.e., for better distinguishing anomaly one needs to understand what is a normal state, and may yield a higher false alarm rate. To address this issue, we propose an Uncertainty Regulated Dual Memory Units (UR-DMU) model to learn both the representations of normal data and discriminative features of abnormal data. To be specific, inspired by the traditional global and local structure on graph convolutional networks, we introduce a Global and Local Multi-Head Self Attention (GL-MHSA) module for the Transformer network to obtain more expressive embeddings for capturing associations in videos. Then, we use two memory banks, one additional abnormal memory for tackling hard samples, to store and separate abnormal and normal prototypes and maximize the margins between the two representations. Finally, we propose an uncertainty learning scheme to learn the normal data latent space, that is robust to noise from camera switching, object changing, scene transforming, etc. Extensive experiments on XD-Violence and UCF-Crime datasets demonstrate that our method outperforms the state-of-the-art methods by a sizable margin.

1 Introduction

Weakly supervised video anomaly detection commonly learns anomaly-focused representations from video-level labels, but neglecting normality can increase false alarms. UR-DMU addresses this with dual prototype memories and uncertainty-regulated normal representations.

Existing weakly supervised detection: MIL-based methods treat videos as bags and infer instance-level anomaly labels by selecting top-K snippets from positive bags.Abnormal videos are positive bags, normal videos are negative bags, and anomaly snippets are positive instances.
Existing weakly supervised detection: Existing weakly supervised approaches emphasize discriminative anomaly features while neglecting normal data, contributing to false alarms.The introduction reports false alarms from RTFM and HL-Net on part of a video.
Proposed approach: UR-DMU stores normal and abnormal prototypes in separate memory units to better distinguish hard samples and reduce false alarms.The abnormal memory receives anomaly information, while the normal memory learns normal patterns from both normal and abnormal videos.
Proposed approach: The model regulates low-magnitude normal features with a Gaussian distribution to represent uncertainty from camera switching and subject changes.No additional regulation is applied to the anomaly feature space.
Proposed approach: GL-MHSA captures different associations in anomaly videos, while dual memory units and uncertainty learning improve normality and anomaly pattern representation.These components are presented as the paper’s main contributions for expressive embeddings and better separation.

2 Related Work

Related work covers weakly supervised and uncertainty-aware video anomaly detection, memory networks, and self-training approaches. UR-DMU extends memory-based methods by storing both normal and abnormal prototypes.

Weakly Supervised Video Anomaly Detection: Weakly supervised VAD uses video-level labels and MIL, with recent methods expanding temporal receptive fields or improving instance embeddings.RTFM is described as using multi-scale temporal modeling for long- and short-range dependencies.
Weakly Supervised Video Anomaly Detection: Self-training methods progressively generate pseudo labels or refine anomaly representations without requiring manually labeled training data.The cited approaches include anomaly regression, pseudo-label generation, and self-guided attention.
Memory Networks: Prior memory-network approaches mainly memorize normal patterns, whereas UR-DMU stores both normal and abnormal prototypes with a dual loss.Memory networks have been applied to anomaly suppression, test-time updates, and optical-flow reconstruction.
Data Uncertainty Learning: Data uncertainty learning models feature variance caused by noise and randomness, including uncertainty in face recognition and background frames in temporal localization.UR-DMU applies this line of work to normality latent embedding learning.

3 Method

UR-DMU combines expressive temporal feature learning, separate normal and abnormal prototype memories, and uncertainty-based modeling of the normal latent space. These components support distinguishing anomaly patterns from normality, including hard samples and noise-related variation.

3 Method: UR-DMU uses GL-MHSA, DMU, and NUL for feature learning, prototype storage, and normality latent embedding learning.GL-MHSA captures video associations, DMU stores normal and abnormal prototypes, and NUL models normality.
3.2 Global and Local Feature Learning: The GL-MHSA module combines global and local associations to obtain more expressive video embeddings.It is motivated by graph convolutional approaches for global and local structure and Transformer modeling of long-range temporal associations.
3.3 Dual Memory Units: Dual memory units use learnable normal and abnormal memory banks to store corresponding templates and assess feature–prototype similarities.Query and read operations produce query scores, memory augmentation features, and top-K memory matches.
3.3 Dual Memory Units: Memory query scores distinguish normal and abnormal inputs, while abnormal-memory activation remains sparse because anomalies occupy only several snippets.Normal inputs are constrained toward normal-memory matches and away from abnormal-memory matches; abnormal inputs are processed through both banks.
3.3 Dual Memory Units: A dual memory loss uses four BCE losses to constrain normal-memory learning on normal patterns and abnormal-memory learning on abnormal patterns.A triplet loss further separates normal and abnormal feature embeddings using top-K selected snippets from normal and abnormal videos.
3.4 Normal Data Uncertainty Learning: Normal Data Uncertainty Learning models normal representations with a Gaussian latent distribution and reparameterized noise, then separates anomaly embeddings by magnitude distance.The KLD regularization suppresses instability from excessive variance, and testing uses only the mean encoder for prediction.

4 Experiments

Experiments on UCF-Crime and XD-Violence evaluate UR-DMU with frame-level metrics and ablations of its memory units, memory counts, loss terms, and modules. UR-DMU achieves strong benchmark performance, while qualitative and ablation results support dual memories and uncertainty learning.

Results on UCF-Crime: UR-DMU achieves 86.97% AUC on UCF-Crime, improving 1.35% over MSL and 2.67% over RTFM.All compared methods use RGB features; UR-DMU uses 10-crop augmentation I3D features.
Results on XD-Violence: UR-DMU reaches 81.66% AP with RGB features and 81.77% AP after concatenating video and audio features on XD-Violence.The RGB result improves 3.85% over RTFM’s 77.81% AP.
Ablation Study: The dual memory configuration performs best because it stores both normal and abnormal patterns, whereas single memories provide weaker discrimination.The single abnormal memory lacks normal patterns and has a higher false alarm rate; the single normal memory performs better than the abnormal-only alternative.
Ablation Study: When Ma = 60 and Mn = 60, UR-DMU achieves the best performances on both datasets.The study searches nine configurations of abnormal and normal memory-bank counts.
Ablation Study: Ltrip most effectively separates normality and anomaly, while {Ldm, Ltrip, Lkl} are essential and Ldis has a slight effect.The loss-term study evaluates eight combinations and reports that the terms work best together.
Ablation Study: Adding the NUL module produces large metric improvements, but the full model performs best on both datasets.The NUL module cannot map all normal patterns into Gaussian space without memory-bank assistance.
Qualitative Results: Qualitative comparisons on eight videos show precise anomaly regions and lower false alarm rates on normal portions than RTFM.The comparison covers XD-Violence and UCF-Crime; RTFM fails on some changing scenes, while UR-DMU lowers FAR in examples (d) and (h).

5 Conclusion

UR-DMU separates normality and abnormality in WS-VAD using expressive temporal embeddings, dual prototype memories, and uncertainty learning. The approach remains limited by coarse and unstable predictions from snippet-level features.

UR-DMU captures long- and short-term temporal dependencies through the GL-MHSA module.This produces more expressive embeddings for video representation.
The DMU stores abnormal and normal prototypes and uses a separation loss to distinguish their representations.The memory units are learned under video-level labels.
The DUL scheme learns a latent normal space from rich normal data with noise perturbation.It is intended to improve representation of normal patterns.
Predicted anomaly windows and scores remain relatively coarse and unstable because the model uses snippet-level features.The authors propose exploring finer temporal and image-space locations with contrastive learning.