Source-linked AI summary
Abnormal Event Detection in Videos using Generative Adversarial Nets
Mahdyar Ravanbakhsh, Moin Nabi, Enver Sangineto, Lucio Marcenaro, Carlo Regazzoni, Nicu Sebe
TL;DR
Crowd abnormality detection is difficult because abnormal training samples are scarce and abnormality lacks an objective definition. The paper uses two conditional GANs trained only on normal frame–optical-flow pairs to reconstruct motion and appearance, then detects abnormalities from local reconstruction differences. Experiments on standard datasets show improved frame-level and pixel-level performance over the state of the art, with failures reported for small, occluded objects having normal motion.
Problem
Crowd abnormality detection remains challenging because abnormality datasets are small and abnormality lacks a clear, objective definition.
Method
Two conditional GANs learn bidirectional frame–optical-flow representations from paired data collected only from normal videos, and abnormality is detected from reconstruction differences.
Results
The approach outperforms the state of the art on both frame-level and pixel-level abnormality detection evaluations.
Takeaways & Limitations
Learning normal appearance and motion with GANs supports abnormal-area detection without abnormal-event training samples.
Takeaways & Limitations
A small, partially occluded object with motion like normal pedestrians can be missed.
Abstract
from arXiv · showhide
In this paper we address the abnormality detection problem in crowded scenes. We propose to use Generative Adversarial Nets (GANs), which are trained using normal frames and corresponding optical-flow images in order to learn an internal representation of the scene normality. Since our GANs are trained with only normal data, they are not able to generate abnormal events. At testing time the real data are compared with both the appearance and the motion representations reconstructed by our GANs and abnormal areas are detected by computing local differences. Experimental results on challenging abnormality detection datasets show the superiority of the proposed method compared to the state of the art in both frame-level and pixel-level abnormality detection tasks.
1 DITEN, University of Genova 2 DISI, University of Trento 3 Carlos III University of Madrid
The paper concerns video analysis, abnormal event detection, crowd behaviour analysis, and Generative Adversarial Networks.
- The paper addresses video analysis.
- Its application area is abnormal event detection and crowd behaviour analysis.
- Its methodological focus is Generative Adversarial Networks.
1. INTRODUCTION
Abnormality detection in crowds remains challenging because abnormal samples are scarce and abnormality lacks an objective definition. The paper addresses these constraints by learning normal crowd patterns with GANs using only normal data.
- Abnormality detection remains an open problem despite substantial prior research.
- Small abnormality datasets and the lack of an objective abnormality definition make detection challenging.Subjective definitions also make abnormal ground-truth collection harder.
- Generative approaches model only normal crowd patterns, then detect abnormalities by distance from the learned normal pattern.This paradigm requires only normal samples during training.
- The proposed GAN method learns normal crowd representations from normal samples and detects abnormal areas through differences between real and generated appearance and motion information.Experiments on challenging datasets report superiority over the state of the art.
2. BACKGROUND
Prior methods often rely on hand-crafted or externally pretrained representations. The paper instead uses adversarial training to learn deeper representations from small normality datasets without external classifiers.
- Earlier methods model normal activity with hand-crafted features or CNNs pretrained for other tasks.
- The closest prior work uses stacked denoising autoencoders, but its shallow networks require externally trained one-class SVM classifiers.
- The proposed adversarial training uses conditional GANs for representation learning, enabling deeper generative networks and eliminating external classifiers.The discriminator provides implicit data supervision and conditional GANs provide data augmentation, as described by the authors.
3. LEARNING THE NORMAL CROWD BEHAVIOUR
Two conditional GAN networks learn bidirectional mappings between normal frames and optical flow. At test time, reconstruction failures provide signals for abnormal-event detection.
- The method trains frame-to-optical-flow and optical-flow-to-frame conditional GAN networks.Each network contains a conditional generator and discriminator.
- The networks use conditional GAN and reconstruction losses on paired frame–optical-flow data.Optical flow is represented with horizontal, vertical, and magnitude channels.
- Training uses frames and optical flow collected exclusively from normal videos, without abnormal-event ground truth.This permits training on potentially large datasets without abnormality labels.
- At testing time, the generators reconstruct motion and appearance from each test frame and its corresponding optical flow.
- Because training contains only normal scenes, abnormal events are not reliably reconstructed and can be detected from reconstruction differences.The paper illustrates this with an unusually moving vehicle whose abnormal region is poorly reconstructed.
4. ABNORMALITY DETECTION
The method reconstructs optical flow and appearance from each test frame and compares these reconstructions with the real inputs to form a fused abnormality heatmap. Appearance differences are computed semantically, while motion differences use pixel-level optical-flow discrepancies.
- Motion and appearance reconstruction: The network reconstructs optical flow from each test frame, and the pixel-by-pixel difference with the real flow highlights local discrepancies.The reconstruction is pO = GF→O(F), compared with O to obtain ΔO.
- Motion and appearance reconstruction: The appearance network reconstructs pF from optical flow, but its difference from F is measured in AlexNet conv5 feature space rather than directly at pixels.The conv5 representation is selected as an abstract space that preserves geometric information.
- Difference-map fusion: The semantic difference map is upsampled before it is fused with the optical-flow difference map.The fusion combines ΔS and ΔO after upsampling ΔS.
- Difference-map fusion: Each difference map is normalized using the maximum difference value computed across all input frames of the test video.The optical-flow maximum mO is computed over all elements of ΔO; the semantic map uses an analogous mS over Δ′S.
- Difference-map fusion: The final abnormality heatmap is A = NS + λNO, with λ = 2 in all experiments.NS and NO are the normalized semantic and optical-flow difference maps.
5. EXPERIMENTAL RESULTS
The method is evaluated on UCSD and UMN using frame-level and pixel-level protocols and compared with state-of-the-art methods. The reported results show strong overall performance, while a small, occluded object with normal motion remains a failure case.
- Experimental setup: The evaluation uses UCSD and UMN datasets with both frame-level and pixel-level abnormality-detection protocols.GANs are trained on UCSD train sequences, with frames resized to 256 × 256 pixels and ten training epochs.
- Evaluation protocols: Frame-level detection labels an entire frame abnormal when at least one predicted abnormal pixel is present, with performance assessed across thresholds using ROC curves.The frame-level criterion is applied to both UCSD and UMN evaluations.
- Evaluation protocols: Pixel-level localization counts a prediction as true positive when it covers at least 40% of the ground-truth abnormal pixels.Otherwise, the frame is counted as a false positive.
- Results: The proposed approach sharply outperforms the compared methods in both frame-level and pixel-level evaluations on the reported datasets.Table 1 compares against state-of-the-art methods on UCSD, while Table 2 reports results on UMN.
- Results: Fusing appearance and motion refines detected areas and improves localization accuracy, while the learned spatial distribution of normal motion alleviates perspective issues.Qualitative examples show localization across different abnormality types.
- Failure case: A skateboard is missed because it is small, moves normally, and is partially occluded.This is identified as a failure case in the qualitative localization results.
6. CONCLUSIONS
The paper presents a two-conditional-GAN method trained only on normal data and detects abnormalities through differences between real and generated images. It reports state-of-the-art performance for both frame-level and pixel-level evaluation protocols.
- Conclusions: The proposed method uses two conditional GANs trained only on normal crowd data, so abnormal events cannot be generated.At test time, local differences between real and generated images identify possible abnormalities.
- Conclusions: The approach outperforms the state of the art on both frame-level and pixel-level abnormality detection protocols.The conclusion reports this result on standard datasets.