Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction

Xuanzhao Wang; Zhengping Che; Bo Jiang; Ning Xiao; Ke Yang; Jian Tang; Jieping Ye; Jingyu Wang; Qi Qi

Source-linked AI summary

Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction

Xuanzhao Wang, Zhengping Che, Bo Jiang, Ning Xiao, Ke Yang, Jian Tang, Jieping Ye, Jingyu Wang, Qi Qi

arXiv:2011.02763v2Nov 5, 2020cs.CVcs.LG

TL;DR

Video anomaly detection is challenging because anomalies are context-dependent and abnormal examples are scarce, while reconstruction errors may not sufficiently separate normal from abnormal frames. ROADMAP uses multi-path ConvGRU frame prediction with a noise tolerance loss to model multiscale semantic motion and reduce noise interference. It outperforms compared baselines across three datasets, achieving 88.3% frame-level AUROC on CUHK Avenue.

Problem
Video anomaly detection faces context-dependent anomaly definitions, scarce abnormal samples, and insufficient reconstruction-error differences between normal and abnormal frames.
Method
ROADMAP uses multi-path ConvGRU frame prediction for multiscale spatial-temporal modeling and introduces a noise tolerance loss during training.
Results
88.3% frame-level AUROC was achieved on CUHK Avenue, exceeding the frame-prediction baseline by 3.2 percentage points.
Takeaways & Limitations
Experiments on three datasets, together with ablation studies, support ROADMAP’s effectiveness and the proposed design choices.

Abstract

from arXiv · show

Video anomaly detection is commonly used in many applications such as security surveillance and is very challenging.A majority of recent video anomaly detection approaches utilize deep reconstruction models, but their performance is often suboptimal because of insufficient reconstruction error differences between normal and abnormal video frames in practice. Meanwhile, frame prediction-based anomaly detection methods have shown promising performance. In this paper, we propose a novel and robust unsupervised video anomaly detection method by frame prediction with proper design which is more in line with the characteristics of surveillance videos. The proposed method is equipped with a multi-path ConvGRU-based frame prediction network that can better handle semantically informative objects and areas of different scales and capture spatial-temporal dependencies in normal videos. A noise tolerance loss is introduced during training to mitigate the interference caused by background noise. Extensive experiments have been conducted on the CUHK Avenue, ShanghaiTech Campus, and UCSD Pedestrian datasets, and the results show that our proposed method outperforms existing state-of-the-art approaches. Remarkably, our proposed method obtains the frame-level AUROC score of 88.3% on the CUHK Avenue dataset.

I. INTRODUCTION

ROADMAP addresses ambiguous, context-dependent anomalies and scarce abnormal training data with a robust unsupervised frame-prediction framework. Its multi-path ConvGRUs and noise tolerance loss target multiscale semantic motion while reducing background-noise interference, and experiments on three datasets support its effectiveness.

Anomaly definitions vary across applications and contexts, while abnormal events are rare, unpredictable, and costly to label.
ROADMAP is an unsupervised video anomaly detection framework designed for robust performance across different scenarios.
Multi-path ConvGRUs model informative objects at different scales and temporal relationships while paying less attention to static background regions.
Noise tolerance loss mitigates interference from intrinsically noisy pixels in video frames, improving prediction-based anomaly detection robustness and performance.
Experiments on three publicly available datasets show ROADMAP outperforming many state-of-the-art baselines.

A. Traditional Video Anomaly Detection

Traditional methods rely on limited hand-crafted features, while newer approaches include classification and frame prediction. ROADMAP frames anomaly detection as predicting the next frame from history and assessing anomalies from prediction-based differences.

Hand-crafted and shallow-feature methods offer inferior performance because of limited representational power.
Frame-level anomaly detection assigns each video frame a binary label indicating whether its content is anomalous.
ROADMAP predicts the next frame from P preceding frames and uses an assessment model to determine anomalies from the predictions.
Its multi-path recurrent encoder-decoder captures spatial and temporal dependencies at different scales while noise tolerance loss reduces undesired noise influence.

B. Multi-Path Frame Prediction Network

The prediction network uses an encoder, parallel multiscale predictors, and a decoder to forecast the next surveillance-video frame. Its design emphasizes semantic objects and temporal dependencies while reducing noise effects.

The network contains an encoder, predictor, and decoder, with the overall prediction architecture shown in Fig. 1.
The encoder extracts multiscale spatial features from each input frame for subsequent predictor paths.
Three parallel prediction modules separately forecast next-frame features at different resolutions using non-local blocks and ConvGRUs.
The decoder fuses different-scale predictor features through upsampling and channel concatenation to reconstruct the predicted frame.

4) Design principle:

The design principle adapts frame prediction to surveillance-video characteristics by emphasizing semantic changes across scales while suppressing static background and unpredictable noise.

The predictor should generate plausible normal frames while enlarging the gap between abnormal frames and their predictions.
Multi-path prediction passes background through shallow modules and emphasizes semantically meaningful object motion and intrusion in deeper modules.
Parallel predictors model objects at varied sizes, while recurrent blocks capture temporal smoothness in normal events and behaviors.
Camera noise creates unpredictable pixel fluctuations that dominate training loss and conceal informative video changes.
Noise tolerance loss uses a pretrained loss network's perceptual features to reduce captured-video noise interference during prediction training.
The paper distinguishes its noise tolerance loss from a differently defined perceptual loss used in an existing video anomaly detection paper.

D. Training Frame Predictor

The frame predictor is trained with complementary intensity, gradient-difference, and noise-tolerance losses so predictions retain useful details while addressing noise.

Training combines intensity loss and gradient-difference loss with noise tolerance loss between predicted and ground-truth frames.
Intensity loss is the ℓ2 distance between predicted and ground-truth frames in intensity space.
Gradient-difference loss measures ℓ1 discrepancies between predicted and ground-truth frame gradients in vertical and horizontal directions.
The overall training objective is a weighted combination of the three losses controlled by λint, λgd, and λnt.

E. Calculating Anomaly Score

Anomaly scores are derived from prediction quality: lower PSNR relative to other frames yields a higher normalized anomaly score.

The assessment model computes anomaly scores from prediction quality using PSNR, following image-quality assessment practice.
PSNR is calculated from the mean squared error between each ground-truth frame and its predicted frame.
The score St normalizes each frame's PSNR across all T frames in the video and reverses the scale.
A higher St indicates that the frame is more likely to be anomalous.
ROADMAP was evaluated on CUHK Avenue, ShanghaiTech Campus, and UCSD Pedestrian datasets.

B. Evaluation Metrics

Evaluation uses frame-level ROC analysis with AUROC as the main metric, alongside AUPRC and F1, under standard unsupervised settings using only normal training videos.

B. Evaluation Metrics: Frame-level AUROC is the main evaluation metric, with AUPRC and F1 used as additional metrics; higher scores indicate better performance.
All frames were resized to 256×256, normalized to [−1, 1], and eight consecutive frames were used to predict the next frame.
The prediction model was trained for 50 epochs with AdamW-based stochastic gradient descent.
Evaluation compared ROADMAP with state-of-the-art baselines trained using only normal videos in the standard unsupervised setting.
D. Results: Table I reports AUROC scores for anomaly detection results across the evaluated methods and datasets.
D. Results: 88.3% frame-level AUROC was achieved on CUHK Avenue, 3.2 percentage points above the frame prediction-based anomaly detection baseline.

2) ShanghaiTech:

On ShanghaiTech Campus, ROADMAP achieved strong frame-level anomaly detection performance, while ablations showed that multi-path ConvGRU connections and non-local blocks improve AUROC.

76.6% frame-level AUROC exceeded the frame prediction-based baseline by 3.8 percentage points on ShanghaiTech Campus.
Ablation studies: The ablation studies covered all four datasets, with complete results reported in Table II.
Multi-path network and ConvGRU: 86.1% AUROC followed the addition of multi-path ConvGRU connections, improving 3.7 percentage points over the single-path model on Avenue.
Multi-path network and ConvGRU: 1.0–2.6 percentage-point improvements from the multi-path structure were observed across the other datasets.
Non-local blocks: 0.6 percentage points of AUROC improvement resulted from adding non-local blocks, increasing Avenue performance from 86.1% to 86.7%.

3) Noise tolerance loss:

Noise tolerance loss improved ROADMAP on the larger, challenging datasets, while score-gap analysis and anomaly curves assessed robustness and event sensitivity.

Noise tolerance loss: 88.3% AUROC on Avenue included a 1.6 percentage-point gain from noise tolerance loss.
Noise tolerance loss: 76.6% AUROC on ShanghaiTech included a 2.1 percentage-point gain from noise tolerance loss.
Noise tolerance loss: Noise tolerance loss was omitted on the grayscale, low-resolution UCSD Pedestrian datasets because it was little helpful there.
Score-gap analysis: Higher ΔS indicates a more robust network for distinguishing normal and abnormal events.
Score-gap analysis: ROADMAP produced larger score gaps consistently than simplified architectures such as multi-scale, autoencoder, and U-Net designs.
Anomaly score curves: Anomaly scores increased drastically for sudden intrusions, increased gradually for slowly emerging unusual activity, and quickly decreased after anomalous objects disappeared.

3) Investigations on the prediction module:

Visual investigations show that ROADMAP’s components focus on informative spatial and temporal cues, while its efficiency profile balances detection speed, model size, and memory use.

Case studies with visualizations: The non-local block concentrated attention on objects and locations of interest, while ConvGRU emphasized the running person and excluded static normal objects.
Case studies with visualizations: Attention visualizations demonstrate cooperation between non-local and ConvGRU components in capturing spatial-temporal dependencies and informative regions.
Multi-path structure: The shallow path encoded invariant background, the middle path focused on people, and the deep path emphasized blurred areas and differently colored bags.
Evaluation metrics: AUPRC and F1 scores were calculated on Avenue and ShanghaiTech because these were the only benchmarks with fewer abnormal than normal frames.
Performance trade-off: 0.109 seconds per frame, 102MB model size, and 619MB executing memory characterize the default configuration’s efficiency profile.

1) Comparisons on numbers of input frames:

The method balances input length against detection quality and speed, with P = 8 providing a suitable operating point. Additional tests show that prediction reuse can preserve competitive accuracy at real-time speed, while smaller inputs substantially reduce performance.

Input-frame length: P = 8 provides sufficiently fast running speed of about 100 milliseconds per frame with suitable detection quality.Using more than eight input frames offered limited improvement on Avenue and worsened performance on ShanghaiTech.
Input-frame length: More than 8 input frames improved Avenue AUROC by no more than 0.1 percentage points and produced worse performance on ShanghaiTech.These comparisons use AUROC changes relative to the initial setting P = 8.
Prediction-length reuse: 87.5% AUROC was maintained on Avenue at 32.2 fps with Q = 24, exceeding all other baseline results in Table I.Prediction outputs can be reused to detect multiple future frames, supporting the standard real-time requirement of more than 30 fps.
Prediction-length reuse: On ShanghaiTech, prediction reuse caused slightly larger degradation, but results remained competitive for Q ≤ 8.Q denotes the number of frames detected using reused prediction outputs during testing.
Input-frame size: Halving the input height and width caused a performance drop of more than 5 percentage points.The main experiments used 256×256 inputs for fair comparison with other baselines.
Overall findings: The experiments and ablation studies validated the proposed design’s effectiveness and reported superiority across three datasets.The conclusion attributes robustness improvements to the multi-path predictor and noise tolerance loss.