Source-linked AI summary
Object-centric Auto-encoders and Dummy Anomalies for Abnormal Event Detection in Video
Radu Tudor Ionescu, Fahad Shahbaz Khan, Mariana-Iuliana Georgescu, Ling Shao
TL;DR
Abnormal event detection is challenging because anomalous examples are scarce and existing outlier-based methods have limited prior information for distinguishing normal from abnormal events. The paper combines object-centric auto-encoders for motion and appearance with clustered one-versus-rest classification, achieving superior results across four data sets, including an 8.4% frame-level AUC gain on ShanghaiTech.
Problem
Abnormal event detection lacks representative anomalous training data, while common outlier-detection formulations provide limited prior information for distinguishing normal and abnormal events.
Method
The method learns object-centric motion and appearance features with convolutional auto-encoders, clusters normal samples, and trains one-versus-rest classifiers using other clusters as dummy anomalies.
Results
The approach achieves superior performance on Avenue, ShanghaiTech, UCSD, and UMN, including an 8.4% frame-level AUC gain on ShanghaiTech over the state-of-the-art method.
Takeaways & Limitations
Object-centric feature learning and multi-class normality classification are both reported as important contributors to the method’s superior results.
Takeaways & Limitations
Comparisons on Avenue require excluding five test videos with static abnormal objects that were considered improperly labeled by a prior study.
Abstract
from arXiv · showhide
Abnormal event detection in video is a challenging vision problem. Most existing approaches formulate abnormal event detection as an outlier detection task, due to the scarcity of anomalous data during training. Because of the lack of prior information regarding abnormal events, these methods are not fully-equipped to differentiate between normal and abnormal events. In this work, we formalize abnormal event detection as a one-versus-rest binary classification problem. Our contribution is two-fold. First, we introduce an unsupervised feature learning framework based on object-centric convolutional auto-encoders to encode both motion and appearance information. Second, we propose a supervised classification approach based on clustering the training samples into normality clusters. A one-versus-rest abnormal event classifier is then employed to separate each normality cluster from the rest. For the purpose of training the classifier, the other clusters act as dummy anomalies. During inference, an object is labeled as abnormal if the highest classification score assigned by the one-versus-rest classifiers is negative. Comprehensive experiments are performed on four benchmarks: Avenue, ShanghaiTech, UCSD and UMN. Our approach provides superior results on all four data sets. On the large-scale ShanghaiTech data set, our method provides an absolute gain of 8.4% in terms of frame-level AUC compared to the state-of-the-art method [Sultani et al., CVPR 2018].
1. Introduction
The paper addresses context-dependent, rare abnormal events by combining object-centric unsupervised feature learning with discriminative classification of normality clusters. It reports superior performance across four benchmarks.
- Motivation: Abnormal events are difficult to detect because their interpretation depends on context and anomalous examples are rare compared with normal events.These conditions make it difficult to obtain representative anomalies for traditional supervised learning.
- Contributions: The detection task is reformulated as a multi-class classification problem by clustering normal training samples and separating each cluster from the rest.The framework uses other normality clusters as surrogate negatives for one-versus-rest classification.
- Contributions: The proposed representation focuses on detected objects and learns deep features for both motion and appearance using convolutional auto-encoders.This object-centric design also supports localizing anomalies within frames.
- Results: 1.5% and 8.4% are the reported absolute gains on Avenue and ShanghaiTech over state-of-the-art methods, respectively.Experiments cover Avenue, ShanghaiTech, UCSD, and UMN, with superior performance reported on all four data sets.
- Evaluation: The paper evaluates the approach on four benchmarks and compares it with state-of-the-art abnormal event detection methods.The paper structure includes related work, the proposed approach, experiments, and conclusions.
2. Related Work
The related-work review characterizes abnormal event detection primarily as normality modeling and surveys dictionary, hashing, deep-learning, and auto-encoder approaches. It positions this paper as distinct through object-detection-based features and multi-class formulation.
- Existing approaches: Most prior methods learn a familiarity or normality model and label events outside that model as abnormal.Some methods represent normal events with dictionary atoms, while others use hashing or deep learning.
- Unsupervised and auto-encoder methods: Unsupervised learning and auto-encoders have been used in earlier abnormal event detection systems, including cascaded and end-to-end variants.The reviewed methods differ in their input features and network designs.
- Differences from related work: The proposed approach trains auto-encoders on object detections from a single-shot detector rather than on the feature pipelines used by related methods.Compared with Hinami et al., it uses SSD/FPN detections and unsupervised convolutional auto-encoder features instead of moving or geodesic proposals and fine-tuned Fast R-CNN semantic features.
- Differences from related work: The paper also differs by formalizing abnormal event detection as a discriminative multi-class classification problem.The authors state that they are the first, to their knowledge, to use this formulation.
3. Method
The method combines object-centric auto-encoder features for appearance and motion with clustered one-versus-rest classification using dummy anomalies derived from normal data.
- Pipeline: The framework uses supervised object detection, unsupervised feature learning, model training, and inference stages.Detected objects are processed through the subsequent feature-learning and classification pipeline.
- Object detection: An SSD detector identifies objects frame by frame, which are cropped, converted to grayscale, and passed to feature learning.The detector processes approximately 13 frames per second on a GPU.
- Feature learning: Three convolutional auto-encoders learn one appearance representation and two motion representations from cropped objects and their gradients.The motion auto-encoders capture movement before and after each detection moment.
- Feature learning: Latent appearance and motion features are concatenated into a 3072-dimensional vector, with abnormal objects expected to produce higher reconstruction errors.The latent representation of each auto-encoder has dimensions 8 × 8 × 16.
- Model training: K-means divides normal training samples into k normality clusters, allowing samples from other clusters to serve as dummy anomalies for each cluster.This creates binary training problems despite the absence of truly abnormal training samples.
- Model training and inference: Independent binary classifiers separate each cluster from the others, and inference uses the negative of the highest classifier score as the sample abnormality score.Object scores are combined into a pixel-level map; its maximum becomes the frame score before temporal Gaussian smoothing.
4. Experiments
Experiments evaluate the framework on Avenue, ShanghaiTech, UCSD Ped2, and UMN using frame-level AUC, ablations, parameter variation, and qualitative detections. The approach outperforms prior methods across the four evaluation sets, while results identify object-centric features and multi-class classification as important components.
- Evaluation: The evaluation uses frame-level AUC on Avenue, ShanghaiTech, UCSD Ped2, and UMN, counting a frame as correctly detected when it contains at least one abnormal pixel.Pixel-level detection maps are smoothed before final abnormality maps are obtained.
- Benchmark results: 90.4% frame-level AUC on Avenue surpasses the strongest cited baseline by 1.5% and is the only result above 90%.The comparison uses Ionescu et al.'s 88.9% as the best and most recent baseline.
- Benchmark results: 12.1% is the improvement over Liu et al.'s 72.8% state-of-the-art frame-level AUC on ShanghaiTech.The paper also reports strong score-to-ground-truth correlation on an example ShanghaiTech video, with false positives caused by multiple people sharing one bounding box.
- Benchmark results: 97.8% frame-level AUC on UCSD Ped2 is 2.4% above Liu et al. and more than 7% above auto-encoder-based frameworks.On one illustrated UCSD Ped2 video, the reported frame-level AUC exceeds 99%, alongside qualitative localization results.
- Benchmark results: The framework reaches 99.6% frame-level AUC on UMN, matching the top reported score, while the second scene is lower at 99.1%.The second scene also contains normal frames with high abnormality scores before the third abnormal event.
- Analysis: Parameter variation is limited: changing k alters accuracy by less than 1.1%, changing C alters performance by less than 0.3%, and removing components reduces ShanghaiTech performance.Frame-level features without the object detector achieve 72.4% AUC, while the corresponding one-class SVM result is 81.3%; replacing the multi-class approach drops performance by 5.7%.
5. Conclusion and Future Work
The paper concludes that its method combines object-centric convolutional auto-encoders with a multi-class formulation and outperforms state-of-the-art approaches across four data sets.
- The method is based on training object-centric convolutional auto-encoders and formalizing abnormal event detection as a multi-class problem.
- Experiments on four data sets indicate that the approach outperforms a series of state-of-the-art methods.
- Future work will focus on improving the framework by segmenting and tracking objects.