Source-linked AI summary
Anomaly Detection in Video via Self-Supervised and Multi-Task Learning
Mariana-Iuliana Georgescu, Antonio Barbalau, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, Mubarak Shah
TL;DR
Video anomaly detection must identify context-dependent events despite the scarcity of anomalous training examples. The paper uses an object-centric 3D CNN with jointly learned self-supervised and knowledge-distillation tasks, achieving state-of-the-art results across three benchmarks. Its object-level analysis is constrained by detector failures, which late fusion with frame-level scores is intended to mitigate.
Problem
Context-dependent anomalies are difficult to collect for training, motivating video anomaly detection methods that learn without full supervision.
Method
A pre-trained object detector feeds an object-centric 3D CNN that jointly optimizes three self-supervised tasks and one knowledge-distillation task.
Results
The approach outperforms state-of-the-art methods on Avenue, ShanghaiTech, and UCSD Ped2, with 92.8% frame-level AUC on Avenue and 99.8% on UCSD Ped2.
Takeaways & Limitations
Jointly learning multiple proxy tasks outperforms single-task models, supporting multi-task learning for video anomaly detection.
Takeaways & Limitations
Object-level detection can produce false negatives when YOLOv3 misses objects because of limited categories or poor detection performance; late fusion with frame-level scores can recover some.
Abstract
from arXiv · showhide
Anomaly detection in video is a challenging computer vision problem. Due to the lack of anomalous events at training time, anomaly detection requires the design of learning methods without full supervision. In this paper, we approach anomalous event detection in video through self-supervised and multi-task learning at the object level. We first utilize a pre-trained detector to detect objects. Then, we train a 3D convolutional neural network to produce discriminative anomaly-specific information by jointly learning multiple proxy tasks: three self-supervised and one based on knowledge distillation. The self-supervised tasks are: (i) discrimination of forward/backward moving objects (arrow of time), (ii) discrimination of objects in consecutive/intermittent frames (motion irregularity) and (iii) reconstruction of object-specific appearance information. The knowledge distillation task takes into account both classification and detection information, generating large prediction discrepancies between teacher and student models when anomalies occur. To the best of our knowledge, we are the first to approach anomalous event detection in video as a multi-task learning problem, integrating multiple self-supervised and knowledge distillation proxy tasks in a single architecture. Our lightweight architecture outperforms the state-of-the-art methods on three benchmarks: Avenue, ShanghaiTech and UCSD Ped2. Additionally, we perform an ablation study demonstrating the importance of integrating self-supervised learning and normality-specific distillation in a multi-task learning setting.
1. Introduction
Video anomaly detection is difficult because anomalies are context-dependent and rarely available for training. The paper addresses this gap with an object-centric architecture that jointly learns multiple self-supervised and knowledge-distillation proxy tasks.
- Motivation: Context-dependent anomalies are difficult to collect for training, so detection is typically framed as outlier detection using normal training data.Events that deviate from a learned normality model are labeled anomalous.
- Motivation: Single proxy tasks can be poorly aligned with anomaly detection, such as reconstructing a stationary anomalous car without producing a large error.Future-frame prediction may reconstruct a stopped car easily even when its context makes it anomalous.
- Method: The proposed object-centric model jointly learns arrow-of-time, motion-irregularity, appearance-reconstruction, and knowledge-distillation tasks.A shared 3D CNN uses separate prediction heads for the four proxy tasks.
- Contribution: The framework integrates multiple self-supervised and knowledge-distillation proxy tasks into a single architecture for video anomaly detection.The authors identify this as their first multi-task learning formulation combining these task types in one object-centric model.
- Results: 92.8% frame-level AUC was achieved on Avenue, 90.2% on ShanghaiTech, and 99.8% on UCSD Ped2.The jointly optimized model also outperformed models trained on individual proxy tasks.
2. Related Work
Prior video anomaly-detection work includes reconstruction, metric-learning, global, local, and object-centric approaches. This paper follows the object-centric direction but distinguishes itself by using multiple self-supervised and distillation proxy tasks in a multi-task architecture.
- Existing approaches: Deep video anomaly-detection methods include learned appearance and motion representations, reconstruction errors, and metric learning between spatio-temporal patches.Examples include stacked denoising auto-encoders, auto-encoder reconstruction, and Siamese networks.
- Analysis level: Methods analyze anomalies globally at the frame level, locally in spatio-temporal cubes, or through multi-level frameworks.The literature is divided according to the level at which features are extracted and anomalies are evaluated.
- Object-centric methods: Object-centric methods detect objects before learning representations or classifying normality clusters.The cited object-centric approaches use single-shot detection followed by convolutional auto-encoders or related representations.
- This paper: Unlike earlier object-centric approaches, this paper analyzes detected objects through multiple self-supervised and model-distillation tasks.The shared feature is object detection; the proposed distinction is the multi-task proxy-task formulation.
- Problem formulation: The paper adopts the mainstream context-dependent formulation, where anomalous events are unfamiliar events in known contexts and are unavailable during training.Context-independent formulations instead use anomalous events for training and become equivalent to video action recognition.
3. Method
The framework reduces proxy-task misalignment by jointly training an object-centric 3D CNN on multiple self-supervised and knowledge-distillation tasks. It detects objects first, uses task-specific prediction heads, and combines their outputs into anomaly scores while supporting object- and frame-level processing.
- Inference: At inference, anomaly scores average the task-head scores, reassemble object scores into pixel maps, and derive frame scores through spatial and temporal filtering.A frame-level alternative removes YOLOv3, while late fusion can recover false negatives caused by object-detection failures.
- Proxy Tasks and Joint Learning: Arrow-of-time training distinguishes forward from reversed object-centric sequences using cross-entropy supervision.Each sequence is converted into forward and backward samples with one-hot labels.
- Proxy Tasks and Joint Learning: Motion-irregularity training distinguishes consecutive-frame sequences from intermittent-frame sequences using cross-entropy supervision.Intermittent sequences contain skipped frames, whereas consecutive sequences represent regular motion.
- Proxy Tasks and Joint Learning: Middle bounding box prediction reconstructs an object from preceding and succeeding crops, using an L1 loss over a 64 × 64 × 3 RGB output.The target is the object crop from the omitted middle frame.
- Motivation and Overview: The framework uses a shared 3D CNN with four prediction heads for arrow-of-time, motion-irregularity, appearance-reconstruction, and knowledge-distillation tasks.The distillation branch learns from ResNet-50 features and YOLOv3 class probabilities.
4. Experiments
Experiments evaluate the framework on Avenue, ShanghaiTech, and UCSD Ped2 using frame-level anomaly scores and localization criteria. Late fusion and joint proxy-task optimization achieve strong benchmark performance, while adding tasks and model capacity improves detection.
- Setup and evaluation: The evaluation uses Avenue, ShanghaiTech, and UCSD Ped2, with anomalous events reserved for testing.Frame-level AUC is the main metric; UCSD Ped2 also uses RBDC and TBDC because pixel-level AUC is considered flawed.
- Setup and evaluation: The reported deep+wide architecture compares object-level, frame-level, and late-fusion variants against state-of-the-art methods.Table 2 reports frame-level AUC scores for the three benchmarks and the three framework variants.
- Anomaly detection results: 92.8% frame-level AUC on Avenue follows late fusion of object- and frame-level anomaly scores, improving on the object-level result of 91.9%.The late-fusion result surpasses the cited state-of-the-art method by 1.5%.
- Anomaly detection results: 90.2% frame-level AUC on ShanghaiTech is reported for late fusion, exceeding the previous state of the art by 5.3%.The paper reports this as the first result above 90% on ShanghaiTech.
- Anomaly detection results: 99.8% frame-level AUC on UCSD Ped2 surpasses the previous state-of-the-art result of 99.2%.Under the newer localization protocol, the object-level framework also outperforms the compared methods across the reported metrics, except for a slightly lower RBDC score than one method.
- Ablation study: Ablations show increasingly better anomaly detection as proxy tasks are added, with further gains when the 3D CNN becomes deeper and wider.The architecture benefits especially from higher capacity when jointly optimizing three or four tasks.
5. Conclusion
The paper proposes an anomaly detection method based on self-supervised and multi-task learning, evaluated comprehensively on three benchmarks. Jointly learning multiple proxy tasks benefits video anomaly detection.
- The method is based on self-supervised and multi-task learning for video anomaly detection.
- The evaluation covers Avenue, ShanghaiTech and UCSD benchmarks.
- An ablation study shows benefits from jointly learning multiple proxy tasks.
6. Supplementary
The supplementary material reports qualitative detections, errors, frame-level examples, runtime, and discussion of object-detector dependence. Results span Avenue, ShanghaiTech and UCSD Ped2, with both object-centric and frame-level processing described.
- Qualitative Results: 92.8% frame-level AUC is reported on Avenue, including detections of unusual motion and object-related anomalies.False positives and detector-related false negatives are also illustrated.
- Qualitative Results: 98.97% and 98.51% frame-level AUC scores are reported for two ShanghaiTech videos.The examples concern a car anomaly and an individual running behind a group.
- Qualitative Results: 99.8% frame-level AUC is reported on UCSD Ped2, with missed objects and false positives shown as exceptions.Figures 9 and 10 are described as reaching perfect frame-level AUC scores.
- Running Time: 6 milliseconds is required for single-object anomaly scoring, while the complete object-level framework runs at 23 FPS with five objects per frame.Adding frame-level inference reduces speed from 23 FPS to 21 FPS.
- Supplementary Examples: The supplementary figures present true positives, false positives, false negatives, and frame-level score overlaps with ground-truth labels across three benchmarks.Figures 5–11 cover Avenue, ShanghaiTech and UCSD Ped2 examples.
- Discussion: Object-centric methods depend on detector quality, which can increase false negatives when objects are missed or truncated.The combined object-centric and frame-level streams are described as alleviating this limitation.