Source-linked AI summary
Training Adversarial Discriminators for Cross-channel Abnormal Event Detection in Crowds
Mahdyar Ravanbakhsh, Enver Sangineto, Moin Nabi, Nicu Sebe
TL;DR
Abnormal crowd-event detection is hindered by limited abnormal ground-truth data, making end-to-end deep-network training difficult. The paper trains GAN generators and discriminators only on normal videos, uses cross-channel transformations between appearance and motion, and applies the trained discriminators directly for detection. On standard benchmarks, the proposed approach sharply outperforms previous state-of-the-art methods.
Problem
Small abnormality ground-truth datasets make end-to-end training of large deep networks difficult for abnormal crowd-behaviour detection.
Method
The method jointly trains generators and discriminators on normal videos, using cross-channel transformations between raw pixels and optical flow, then uses the discriminators as final classifiers.
Results
The proposed approach sharply outperforms previous state-of-the-art methods on common abnormality-detection benchmarks.
Takeaways & Limitations
GAN mutual supervision enables end-to-end anomaly detectors trained with relatively small, weakly supervised normal-video sequences and without manually annotated abnormal data.
Takeaways & Limitations
The approach is evaluated with training data collected only from normal videos, so its training assumption excludes abnormal events.
Abstract
from arXiv · showhide
Abnormal crowd behaviour detection attracts a large interest due to its importance in video surveillance scenarios. However, the ambiguity and the lack of sufficient abnormal ground truth data makes end-to-end training of large deep networks hard in this domain. In this paper we propose to use Generative Adversarial Nets (GANs), which are trained to generate only the normal distribution of the data. During the adversarial GAN training, a discriminator (D) is used as a supervisor for the generator network (G) and vice versa. At testing time we use D to solve our discriminative task (abnormality detection), where D has been trained without the need of manually-annotated abnormal data. Moreover, in order to prevent G learn a trivial identity function, we use a cross-channel approach, forcing G to transform raw-pixel data in motion information and vice versa. The quantitative results on standard benchmarks show that our method outperforms previous state-of-the-art methods in both the frame-level and the pixel-level evaluation.
1. Introduction
Abnormal crowd behaviour detection remains difficult because abnormal ground-truth datasets are small, motivating methods that learn normality from weakly annotated data. This paper uses GAN training and cross-channel transformations so discriminators can detect abnormalities directly after end-to-end training.
- Motivation: Small abnormality ground-truth datasets make end-to-end training of large deep networks difficult.Existing methods therefore commonly learn only normal crowd patterns from videos containing normal behaviour.
- Approach: GAN training uses a generator and discriminator in an adversarial game, with each network providing indirect supervision to the other.The generator produces data while the discriminator distinguishes real training images from generated images.
- Approach: Training on only normal frames lets the generator model normal scenes while the discriminator learns to distinguish normal from abnormal outliers.The trained discriminator is used directly for abnormality detection, avoiding an additional one-class SVM or classifier.
- Approach: The method directly uses the discriminator after training for the final discriminative task rather than using the generator for data generation.This supports an end-to-end abnormality detector without manually annotated abnormal data.
- Novelty: Cross-channel transformations between raw pixels and optical flow prevent the generator from learning a trivial identity function.The two-channel representation combines appearance and motion information and encourages informative internal representations.
2. Related Work
Prior abnormality-detection methods often rely on handcrafted features, pretrained networks, shallow representations, or additional classifiers. The paper positions its GAN-based discriminator as an end-to-end alternative to reconstruction-based and feature-based approaches.
- Abnormality Detection: Most previous abnormality-detection methods model normal activity with handcrafted features such as optical flow and tracklets.The paper instead learns features from raw pixels using an end-to-end deep-learning protocol.
- Abnormality Detection: Several deep-learning approaches adapt pretrained CNNs rather than training the complete abnormality detector end to end.The cited methods may also require complex post-processing or precomputed feature codebooks.
- Abnormality Detection: Stacked Denoising Autoencoders learn motion and appearance features but are shallow and require multiple one-class SVMs for final classification.Their features may be suboptimal because they are not jointly optimized for the final classification task.
- GAN-based Methods: The Convolutional Autoencoder approach detects anomalies from frame reconstruction differences, while this paper directly exploits GAN discriminators.The paper also contrasts its discriminator version with reconstruction-based GAN variants that require an externally trained CNN and fusion strategy.
3. Cross-channel Generation Tasks
The proposed representation combines appearance and motion through two cross-channel generation tasks. Conditional generators transform frames into optical flow and optical flow into frames, while discriminators judge whether the paired inputs are real.
- Representation: The method uses appearance information from raw pixels and motion information from optical-flow images.Two cross-channel tasks are defined to connect the two modalities.
- Cross-channel Tasks: N F →O generates optical flow from a frame, whereas N O→F generates a frame from optical flow.The corresponding training pairs are (F_t, O_t) and (O_t, F_t), respectively.
- Generator: Each conditional generator takes an image x and noise vector z and outputs an image r with the same dimensions in the other channel.For N F →O, x is a frame and r reconstructs its corresponding optical-flow image.
- Discriminator: Each discriminator receives the input image and either the real target or generated output, then predicts the probability that the pair came from real data.The discriminator inputs concatenate six 2D components: three RGB channels and three optical-flow components.
- Architecture: The generators use U-Net encoder-decoder architectures with skip connections, while the discriminators use five convolutional layers and PatchGAN.All images are rescaled to 256 × 256.
4. Training
Both cross-channel networks are trained adversarially on frame–optical-flow pairs from normal videos, using conditional GAN and reconstruction losses. Because training excludes abnormal events, the generators fail to reconstruct anomalies and the discriminators learn plausibility boundaries for testing.
- Losses: Training uses both a conditional GAN loss LcGAN and an L1 reconstruction loss LL1.For N F →O, the training set consists of frame–optical-flow pairs X = {(F_t, O_t)}.
- Training Data: Both frame and optical-flow data are collected exclusively from normal training videos.This allows the discriminators to be trained without supervised abnormal-event data because the generator provides implicit supervision.
- Generator Behaviour: Generators trained only on normal scenes cannot accurately reconstruct abnormal events such as an unusually moving vehicle.The abnormal region is poorly reconstructed in the generated optical flow or appears as unstructured blobs in the generated frame.
- Testing: The discriminators learn to distinguish what is plausibly real in the observed scenario from what is not, and this capacity is used at testing time.The paper therefore uses discriminator outputs rather than relying only on reconstruction errors.
- Training Caveat: GAN training rarely produces fully realistic images in practice, with generated pedestrian details remaining smooth and blob-like.The paper notes that the generative and data distributions may therefore not completely overlap.
5. Abnormality Detection
At testing time, the method uses two patch-based discriminators to produce channel-specific abnormality maps, fuses them, normalizes the result, and filters it using motion information.
- Score-map construction: Two patch-based discriminators evaluate corresponding frame and optical-flow patches on a shared 30 × 30 grid, producing score maps S_O and S_F.The discriminators operate on the two learned channel-transformation tasks without requiring reconstruction images at test time.
- Score-map construction: Low discriminator scores indicate patches containing unusual objects or movements relative to the learned normal distribution.A threshold on the discriminator output defines the decision boundary, after channel-specific scoring.
- Score fusion: The channel-specific score maps are fused by equal-weight summation, S = S_O + S_F, before normalization.The resulting values are normalized using the maximum score observed across all input frames of each test video.
- Generator behavior: Figure 3 shows that abnormal vehicles are not reconstructed correctly by either cross-channel generator, unlike normal scene content.The examples include generated frames from optical flow and generated optical flow from frames.
- Final abnormality map: The normalized score map is upsampled to the original frame size and filtered to retain only areas with nonzero optical flow.This produces the final abnormality map while suppressing non-motion regions.
6. Experimental Results
The method is evaluated on UCSD and UMN benchmarks with frame-level and pixel-level protocols, using ROC-based metrics and comparisons with prior methods. Additional analysis examines computational cost, localization, and the roles of cross-channel and multi-channel design choices.
- Evaluation setup: The evaluation uses UCSD and UMN datasets with frame-level and pixel-level abnormality-detection protocols.The pixel-level protocol evaluates spatial localization against ground truth, while frame-level evaluation builds ROC curves from confidence thresholds.
- Implementation details: Training uses only normal UCSD sequences, with 10 epochs per network and no dataset-specific hyper-parameter tuning.Frames are resized to 256 × 256 pixels and optimization uses stochastic gradient descent with momentum 0.5 and batch size 1.
- Computational cost: A full network trains in less than half an hour, while one test frame takes 0.53 seconds on a Tesla K40 GPU.The reported testing time includes optical-flow computation and post-processing.
- Frame-level evaluation: Frame-level results are reported using Equal Error Rate (EER) and Area Under Curve (AUC) for the UCSD and UMN datasets.The proposed method is named Adversarial Discriminator in the reported comparisons.
- Pixel-level evaluation: Pixel-level evaluation counts a frame as a true positive when predicted abnormal pixels overlap the ground-truth area by at least 40%.Localization ROC curves and corresponding EER and AUC values are reported for the UCSD dataset.
7. Ablation Study
The ablation study evaluates discriminator-, generator-, cross-channel, and channel-fusion choices on UCSD Ped2, showing that cross-channel training and detector fusion improve the proposed approach while adversarial reconstruction remains highly competitive.
- Ablation results: Adversarial Generator achieves state-of-the-art accuracy against the methods in Table 1, excluding Adversarial Discriminator.It uses reconstruction errors from the two cross-channel generators as its abnormality signal.
- Ablation results: Same-Channel Discriminator drops significantly relative to Adversarial Discriminator, demonstrating the importance of cross-channel tasks.Despite this drop, it outperforms or closely approaches the best systems on the dataset.
- Ablation results: Adversarial Discriminator F performs worst and substantially below Adversarial Discriminator O, while fusing both detectors boosts the proposed method.The authors attribute the difference to the richer information in real frames than in optical-flow inputs.
- Comparison with prior work: The Convolutional Autoencoder reports 21.7% EER and 90% AUC on the same dataset, significantly worse than the GAN-based Adversarial Generator.Both approaches use reconstruction error for detection.
- Comparison with prior work: GAN-CNN is basically on par with Adversarial Discriminator, but Adversarial Discriminator uses a simpler and faster testing pipeline.GAN-CNN requires semantic and pixel-level error fusion, whereas Adversarial Discriminator does not use the generators or semantic network at test time.
8. Qualitative results
Qualitative results show that the generators reproduce normal scenes reasonably well but fail to reconstruct abnormal objects or fast movements, supporting their use for anomaly detection. Pixel-level visualizations also expose errors associated with small, occluded, or normally moving anomalies and with perspective-truncated pedestrians.
- Generator behavior: The generators produce normal scenes reasonably well but do not accurately reproduce high-resolution pedestrian structures.The imperfect overlap between data and generative distributions is consistent with prior GAN results.
- Generator behavior: Abnormal objects and fast movements are missing from reconstructions because the generators did not observe them during training.This behavior is exploited by Adversarial Generator and GAN-CNN for detection.
- Pixel-level detections: Pixel-level heat maps visualize Adversarial Discriminator detections on Ped1 and Ped2, while the final column marks prediction errors with red rectangles.The figure includes both detection examples and error examples.
- Detection errors: Missed detections commonly involve very small or partially occluded abnormal objects, or objects moving at normal pedestrian speeds.A false positive is also associated with severely truncated pedestrians whose visible body parts appear enlarged by perspective.
9. Conclusions
The paper presents a GAN-based abnormality detector trained with only weakly supervised normal video sequences. It directly uses the trained discriminators as classifiers and relies on cross-channel generative tasks, with results reported as sharply outperforming prior state of the art.
- The method trains end-to-end anomaly detectors using only relatively small, weakly supervised training video sequences.The discriminators are trained without manually annotated abnormal data.
- Unlike common GAN applications, the trained discriminators become the final classifiers while the generators are discarded.
- Two nontrivial cross-channel generative tasks are designed to make the GAN-based detection approach effective.
- The proposed approach sharply outperforms previous state-of-the-art methods on common abnormality-detection benchmarks.The paper also compares against reconstruction-based baselines and same-channel encoding/decoding tasks, reporting accuracy and computational advantages.