Source-linked AI summary
Learning Deep Representations of Appearance and Motion for Anomalous Event Detection
Dan Xu, Elisa Ricci, Yan Yan, Jingkuan Song, Nicu Sebe
TL;DR
Video anomaly detection remains difficult in complex scenes, especially when methods rely on hand-crafted appearance and motion features. AMDN learns separate and joint representations with stacked denoising autoencoders and combines anomaly scores through double fusion. Experiments on two challenging surveillance datasets show competitive performance against existing methods.
Problem
Complex video anomaly detection is difficult, while many existing approaches rely on hand-crafted appearance and motion features.
Method
AMDN uses stacked denoising autoencoders to learn separate appearance and motion representations plus a joint representation, then combines multiple one-class SVM anomaly scores through late fusion.
Results
Competitive performance against existing methods is demonstrated on two challenging video surveillance datasets.
Takeaways & Limitations
The framework provides an unsupervised deep-learning approach that combines appearance and motion information for anomalous event detection.
Abstract
from arXiv · showhide
We present a novel unsupervised deep learning framework for anomalous event detection in complex video scenes. While most existing works merely use hand-crafted appearance and motion features, we propose Appearance and Motion DeepNet (AMDN) which utilizes deep neural networks to automatically learn feature representations. To exploit the complementary information of both appearance and motion patterns, we introduce a novel double fusion framework, combining both the benefits of traditional early fusion and late fusion strategies. Specifically, stacked denoising autoencoders are proposed to separately learn both appearance and motion features as well as a joint representation (early fusion). Based on the learned representations, multiple one-class SVM models are used to predict the anomaly scores of each input, which are then integrated with a late fusion strategy for final anomaly detection. We evaluate the proposed method on two publicly available video surveillance datasets, showing competitive performance with respect to state of the art approaches.
1 Introduction
Video anomaly detection must identify abnormal events in complex, long surveillance streams, where scene complexity and deceptive behavior make the task difficult. AMDN addresses limitations of hand-crafted representations by learning appearance, motion, and joint features unsupervised, then combining early and late fusion.
- Complex scenes and deceptive abnormal behaviors make automatic detection of abnormal events in long video streams challenging.
- Earlier approaches commonly model local or global activity patterns using hand-crafted appearance and motion cues such as color, texture, HOG, and optical flow.
- Deep learning can learn rich and discriminative feature representations, but prior computer-vision applications largely use supervised convolutional networks.
- AMDN uses stacked denoising autoencoders to learn appearance and motion features separately, plus a joint representation capturing their correlations.
- The framework combines early fusion of image and optical-flow inputs with late fusion of anomaly scores from multiple one-class SVM classifiers.
2 AMDN for Abnormal Event Detection
AMDN learns appearance, motion, and joint representations with stacked denoising autoencoders, then combines three one-class SVM anomaly scores for patch-based detection.
- Representation learning: Three SDAE pipelines learn separate appearance and motion features plus a joint representation capturing their correlations.The joint pipeline uses pixel-level early fusion of gray-image and optical-flow patches.
- AMDN training: Stacked denoising autoencoders are trained through greedy layer-wise pretraining followed by whole-network fine-tuning.Pretraining corrupts inputs, while fine-tuning treats all SDAE layers as a single model optimized with backpropagation.
- Representation learning: Appearance features are learned from normalized dense multi-scale image patches, while motion features use normalized optical-flow patches.Both pipelines use sliding windows and four encoding layers with over-complete first-layer filters.
- AMDN training: The bottleneck hidden layer is selected to produce compact learned feature representations for anomaly detection.Features are extracted through a forward pass after fine-tuning the network.
- Anomaly detection: For each test patch, three one-class SVMs generate anomaly scores that are linearly combined into a final anomaly score.The method formulates detection as patch-based binary categorization and can optionally use dynamic background subtraction during testing.
3 Experimental Results
Experiments on the UCSD and Train datasets evaluate AMDN using frame-level, pixel-level, and precision/recall analyses against existing methods. AMDN achieves competitive or superior results, with double fusion outperforming its early- and late-fusion variants.
- Datasets and Experimental Setup: Experiments use the UCSD Ped1/Ped2 and Train datasets to evaluate the proposed method against existing anomaly-detection approaches.UCSD evaluation includes frame-level and pixel-level ground truths; the Train dataset contains unusual movements under dynamic illumination and camera shake.
- UCSD Dataset: UCSD evaluation measures frame-level detection on Ped1 and Ped2 and pixel-level localization on ten Ped1 test sequences.A detected region is counted as true when it overlaps more than 40% with the annotated region.
- UCSD Dataset: AMDN outperforms all competing approaches in pixel-level UCSD evaluation for both EER and AUC, while remaining competitive with the two best baselines at frame level.The comparison uses ROC curves and reports Area Under Curve and Equal Error Rate.
- Fusion Analysis: AMDN outperforms both early fusion and late fusion variants on UCSD, and late fusion outperforms early fusion in this application.Early fusion uses only the joint representation, whereas late fusion combines separate appearance and motion pipelines without the joint representation.
- Train Dataset: On the Train dataset, AMDN outperforms all listed baselines according to the precision/recall curve.The baselines include dominant behavior learning, spatio-temporal oriented energies, local optical flow, behavior templates, and mixture of Gaussian.
4 Conclusions
The paper presents an unsupervised deep representation approach for video anomaly detection using stacked autoencoders and double fusion. Experiments on two challenging datasets demonstrate effectiveness and competitive performance against existing methods.
- Conclusions: The approach learns appearance and motion representations of scene activities with multiple stacked autoencoder networks.A double fusion scheme combines the learned feature representations.
- Conclusions: A double fusion scheme combines the learned feature representations for video anomaly detection.The conclusion identifies fusion as part of the proposed framework.
- Conclusions: Experiments on two challenging datasets demonstrate the approach's effectiveness and competitive performance relative to existing methods.The conclusion reports this result without specifying individual dataset metrics.
- Future Work: Future work includes other network architectures, alternative SDAE multimodal fusion methods, and multi-task learning for heterogeneous scenes.These directions are proposed for extending the framework.