Source-linked AI summary
Unmasking the abnormal events in video
Radu Tudor Ionescu, Sorina Smeureanu, Bogdan Alexe, Marius Popescu
TL;DR
The paper tackles video abnormal-event detection without training sequences, a setting where conventional normality models cannot be built in advance. It adapts unmasking by repeatedly classifying consecutive frame groups while removing discriminant features, and reports better performance than a state-of-the-art unsupervised method and some supervised methods across four datasets, while processing video online.
Problem
Video abnormal-event detection is context-dependent, and representative anomaly training data or additional training sequences are generally unavailable.
Method
The framework applies unmasking to consecutive video sequences by iteratively training a binary classifier and removing the most discriminant features.
Results
The approach performs better than the compared unsupervised method and some supervised methods across four datasets, while processing video online at 20 frames per second.
Takeaways & Limitations
Unmasking provides a training-sequence-free framework for online abnormal-event detection in video.
Takeaways & Limitations
Late fusion of motion and appearance features produced no considerable improvements, motivating better fusion or learned unsupervised deep features.
Abstract
from arXiv · showhide
We propose a novel framework for abnormal event detection in video that requires no training sequences. Our framework is based on unmasking, a technique previously used for authorship verification in text documents, which we adapt to our task. We iteratively train a binary classifier to distinguish between two consecutive video sequences while removing at each step the most discriminant features. Higher training accuracy rates of the intermediately obtained classifiers represent abnormal events. To the best of our knowledge, this is the first work to apply unmasking for a computer vision task. We compare our method with several state-of-the-art supervised and unsupervised methods on four benchmark data sets. The empirical results indicate that our abnormal event detection framework can achieve state-of-the-art results, while running in real-time at 20 frames per second.
1. Introduction
The paper addresses context-dependent abnormal event detection when representative anomalies and additional training sequences are unavailable. It proposes unmasking-based online detection and evaluates it against supervised and unsupervised methods.
- Motivation: Abnormality depends on context, so the same action can be normal in one setting and abnormal in another.The paper gives driving into a pedestrian area and fleeing a bank as contextual examples.
- Motivation: Traditional supervised learning is usually unsuitable because sufficiently representative anomaly sets are generally impossible to obtain.Most existing approaches instead learn familiarity from training video and detect deviations.
- Approach: The framework detects short-lasting anomalies by measuring change from the immediately preceding event with a classifier over a sliding window.The first half is treated as normal reference, while the second half is tested as potentially abnormal using motion and appearance features.
- Approach: Unmasking repeatedly removes discriminant features and uses the resulting accuracy profile to score anomalies.Accuracy remains higher when the current event differs substantially from its predecessor; the mean profile value becomes the anomaly score.
- Results: Experiments on Avenue, Subway, UCSD, and UMN show better results than a state-of-the-art unsupervised method and parity or superiority over some supervised methods.Unlike the compared unsupervised approach, the framework processes video online at 20 frames per second.
2. Related Work
Prior abnormal-event detectors generally learn normality from training data, even when parts of their learning procedures are unsupervised. The paper positions its method among the few approaches requiring no training data.
- Existing approaches: Most abnormal-event detection methods learn a model or dictionary of normal events from training data and flag outliers as abnormal.Related approaches use normal-event dictionaries, deep features, or locality-sensitive hashing filters.
- Partially unsupervised methods: Several methods use unsupervised learning steps but still require training videos to construct or initialize normality models.Examples include incremental familiarity models, Growing Neural Gas, spectral-clustering dictionaries, and autoencoder features.
- Training-free detection: The cited approach in [6] is identified as the only prior method requiring no kind of training data.It detects changes by testing whether frames are distinguishable from all preceding frames after permuting frame order.
3. Method
The method applies unmasking to sliding windows without training data, comparing consecutive frame groups through iteratively pruned classifiers. Motion and appearance streams produce localized anomaly scores that are averaged over unmasking iterations.
- Framework overview: The framework uses eight sequential steps: sliding-window formation, labeling, feature extraction, unmasking, accuracy collection, and anomaly scoring.Each 2·w-frame window treats the first w frames as normal and the last w as abnormal for testing.
- Framework overview: Frames are divided into four 2 × 2 spatial bins, processed independently, and assigned the maximum bin score per frame.The framework is also applied separately to motion and appearance features before the final scoring stage.
- Feature extraction: Motion features use normalized 3D gradients from spatio-temporal cubes, retaining more dimensions to support gradual discriminant-feature removal.Frames are resized to 160 × 120 pixels and partitioned into non-overlapping 10 × 10 patches across five consecutive frames.
- Feature extraction: Appearance features come from the pre-trained VGG-f CNN, selected because it can process about 20 frames per second on CPU.The CNN is used without task-specific fine-tuning because the setting permits no training data.
- Feature extraction: The final feature vectors are normalized with the L2-norm.
- Unmasking: Unmasking trains a linear classifier on consecutive frame groups, repeatedly removes highly weighted features, and averages training accuracies into an anomaly score.The paper substitutes training accuracy for the original cross-validation accuracy to enable online detection.
4. Experiments
Experiments on four benchmark data sets evaluate the training-free unmasking framework using AUC-based detection metrics and qualitative examples. The method generally improves on unsupervised baselines, sometimes matches or surpasses supervised methods, and runs online at 20 frames per second, although performance varies by data set and setting.
- Experimental setup: Experiments use only test videos from four data sets, without training videos to build a normality model.Evaluation uses ROC curves and frame-level AUC, plus pixel-level AUC when Avenue and UCSD annotations are available.
- Experimental setup: A 2.5% improvement is obtained by including unmasking, while processing frames in 2 × 2 bins yields a further 1.4% gain on preliminary Avenue tests.The preliminary comparison also favors conv5 CNN features over fc6 and fc7 features; the unmasking procedure uses k = 10 loops and removes m = 50 features per loop.
- Avenue: 2.3% frame-level AUC and 2.0% pixel-level AUC improvements over the unsupervised baseline are reported on Avenue, with scores near the supervised method.The framework processes the video online, whereas the compared unsupervised approach operates offline.
- Avenue: A threshold around 0.5 identifies both abnormal events in Avenue test video 4 without false positives, while other Avenue videos still contain false detections.Reported false positives include people holding a large object or sitting on the ground.
- Subway: 1.5% and 3.3% improvements are obtained on Subway Entrance gate and Exit gate videos respectively, with the Exit gate result surpassing the supervised comparison.The late fusion strategy does not improve Subway results, and performance is much lower on the Entrance gate than on the Exit gate.
- UCSD and UMN: On UCSD, the framework reaches 68.4% frame-level AUC and 52.4% pixel-level AUC on Ped1, while UMN results vary substantially across scenes.UMN performance is affected by illumination changes; CNN appearance features improve the second-scene frame-level AUC from 84.9% with motion alone to 86.5%.
5. Conclusion and Future Work
The paper presents a training-sequence-free abnormal event detection framework based on unmasking, outperforming the compared unsupervised method and some supervised methods while processing video online. The authors note that late fusion of motion and appearance features did not yield considerable improvements and identify better feature fusion as future work.
- The proposed framework detects abnormal events without requiring training sequences and applies unmasking, previously unused in computer vision to the authors’ knowledge.
- Experiments on four data sets show better performance than the compared unsupervised method and some supervised methods.
- Unlike the compared unsupervised method, the framework processes video online without accuracy degradation.
- Late fusion of motion and appearance features produced no considerable improvement, motivating future work on improved fusion or unsupervised deep features.