PaperlayerView exact revision on arXiv

Source-linked AI summary

A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in Video

Mariana-Iuliana Georgescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, Mubarak Shah

arXiv:2008.12328v5cs.CVeess.IV

TL;DR

Abnormal event detection is difficult because abnormality depends on context and representative abnormal training data are scarce. The paper proposes a background-agnostic object-centric framework using adversarially trained auto-encoders, pseudo-abnormal examples, and binary classifiers, reporting favorable performance across four datasets and releasing new annotations.

  • Problem

    Context-dependent abnormalities and scarce representative abnormal examples motivate normal-only outlier-detection methods.

  • Method

    The framework uses object detections, appearance and motion auto-encoders, adversarial pseudo-abnormal training, and binary classifiers.

  • Results

    The approach reports favorable performance on Avenue, ShanghaiTech, Subway, and UCSD Ped2, with consistently superior performance over the preliminary work.

  • Takeaways & Limitations

    Cross-database experiments support applying the framework across scenes with different backgrounds, while released annotations enable RBDC and TBDC reporting.

  • Takeaways & Limitations

    Region- and track-level results can be affected when the object detector misses abnormal objects, although frame-level anomaly scores may remain unaffected.

Abstract

from arXiv · show

Abnormal event detection in video is a complex computer vision problem that has attracted significant attention in recent years. The complexity of the task arises from the commonly-adopted definition of an abnormal event, that is, a rarely occurring event that typically depends on the surrounding context. Following the standard formulation of abnormal event detection as outlier detection, we propose a background-agnostic framework that learns from training videos containing only normal events. Our framework is composed of an object detector, a set of appearance and motion auto-encoders, and a set of classifiers. Since our framework only looks at object detections, it can be applied to different scenes, provided that normal events are defined identically across scenes and that the single main factor of variation is the background. To overcome the lack of abnormal data during training, we propose an adversarial learning strategy for the auto-encoders. We create a scene-agnostic set of out-of-domain pseudo-abnormal examples, which are correctly reconstructed by the auto-encoders before applying gradient ascent on the pseudo-abnormal examples. We further utilize the pseudo-abnormal examples to serve as abnormal examples when training appearance-based and motion-based binary classifiers to discriminate between normal and abnormal latent features and reconstructions. We compare our framework with the state-of-the-art methods on four benchmark data sets, using various evaluation metrics. Compared to existing methods, the empirical results indicate that our approach achieves favorable performance on all data sets. In addition, we provide region-based and track-based annotations for two large-scale abnormal event detection data sets from the literature, namely ShanghaiTech and Subway.

1 INTRODUCTION

The paper targets context-dependent abnormal events using normal-only learning and proposes a background-agnostic, adversarially trained object-centric framework. It reports favorable benchmark performance, cross-database background agnosticism, and new region- and track-level annotations.

  • Motivation: Context determines abnormality, making representative abnormal training data difficult to obtain and motivating predominantly outlier-detection approaches.The same event can be normal or abnormal depending on its surrounding scene.
  • Evaluation: Experiments on Avenue, ShanghaiTech, Subway, and UCSD Ped2 report favorable performance relative to state-of-the-art methods.The study also reports frame-level AUC together with region- and track-based evaluation measures.
  • Contributions: The redesigned model achieves consistently superior performance over the preliminary work, especially on RBDC and TBDC.
  • Contributions: The framework applies adversarial training to convolutional auto-encoders for video abnormal event detection.
  • Contributions: Object-centric processing makes the framework background agnostic, demonstrated through cross-database experiments.The approach analyzes detected objects rather than relying on scene backgrounds.
  • Contributions: Region-level and track-level annotations for ShanghaiTech and Subway enable future reporting with RBDC and TBDC.

2 RELATED WORK

Related work primarily treats abnormal event detection as outlier detection, while this paper combines object-centric auto-encoding with adversarial training and pseudo-abnormal supervision. Its approach differs from GAN-based methods and from methods trained directly on full frames.

  • Outlier detection: Most recent methods learn normality models from normal data and label divergent events as abnormal during inference.
  • Proposed positioning: This method uses supervised object detection and pseudo-abnormal examples while learning features in an otherwise unsupervised manner.
  • Auto-encoders: Unlike prior auto-encoder methods, it trains on detected objects to improve localization and background agnosticism, while adversarial training targets good reconstructions for normal objects only.
  • Adversarial methods: The approach does not employ GANs that train a generator and discriminator for realistic optical-flow generation.
  • Broader context: Auto-encoders and adversarial models have also been applied to time series, images, and network traffic, but video anomaly detection requires motion and appearance considerations.

3 METHOD

The method detects abnormality from object appearance and motion using three auto-encoder streams and binary classifiers trained with pseudo-abnormal examples. Segmentation and adversarial branches focus reconstruction on foreground objects and weaken reconstruction of pseudo-abnormal inputs.

  • Motivation: The design targets normal-only training and reuse across scenes with different backgrounds through object detection and pseudo-abnormal supervision.
  • Pipeline: The pipeline detects objects, computes forward and backward optical flow, reconstructs appearance and motion, and classifies reconstruction differences.
  • Pseudo-abnormal examples: A scene-agnostic pseudo-abnormal collection supplies abnormal training examples and encourages auto-encoders to retain normal-object reconstruction patterns.
  • Architecture: Three independent streams process appearance, backward motion, and forward motion, each using a convolutional auto-encoder followed by a binary classifier.
  • Architecture: Binary classifiers receive absolute input-reconstruction differences to distinguish normal from pseudo-abnormal examples.Using reconstructions directly performed considerably worse in preliminary experiments.
  • Training: The appearance auto-encoder jointly reconstructs the input and foreground segmentation map using pixel-wise mean squared error and logistic segmentation loss.
  • Training: Adversarial training applies gradient ascent to the adversarial decoder branch, with λ=0.2 used to weight the reversed gradient.
  • Inference: At inference, the anomaly score is one minus the mean of three classifier normality scores, and pseudo-abnormal classification corresponds to abnormality.

4 EXPERIMENTS

The evaluation covers four benchmark datasets and reports frame-level AUC alongside region- and track-based criteria. The protocol distinguishes micro- and macro-averaged frame AUC and uses dataset-specific detection thresholds.

  • Datasets: The experiments evaluate Avenue, ShanghaiTech, Subway, and UCSD Ped2, excluding UMN because it is considered small and saturated.
  • Datasets: ShanghaiTech contains 330 training videos, 107 test videos, and 316,154 frames at 480 × 856-pixel resolution.
  • Metrics: Frame-level AUC is computed from frame annotations, with both micro-averaged and macro-averaged versions reported.
  • Metrics: RBDC and TBDC complement frame-level AUC by evaluating abnormal regions and tracks.
  • Metrics: Pixel-level AUC is not reported independently because a post-processing step can make it equal to frame-level AUC.
  • Implementation: YOLOv3 detections use confidence thresholds of 0.8 for Avenue and ShanghaiTech and 0.5 for Subway and UCSD Ped2.

4.4 Preliminary Results

Adversarial training makes the auto-encoders reconstruct normal examples better than abnormal ones, supporting anomaly detection. On Avenue, the approach improves frame-level detection over the preliminary method and identifies an event that the earlier method misses.

  • Qualitative analysis: Without adversarial training, the auto-encoders reconstruct abnormal examples excessively well because they generalize to samples close to the training distribution.The authors therefore seek a larger reconstruction-quality difference between normal and abnormal samples.
  • Qualitative analysis: Adversarial training produces worse reconstructions for pseudo-abnormal and abnormal examples, which is the desired effect.The comparison includes appearance and motion auto-encoders trained with or without adversarial training.
  • Avenue comparison: 92.3% micro-averaged AUC makes the method the only approach above 90% on Avenue and nearly 5% higher than the preliminary method.The method surpasses all compared methods in frame-level AUC, while matching the preliminary method in macro-averaged AUC.
  • Avenue comparison: 23.85% improvement in RBDC over Ramachandra et al. is reported on Avenue.The authors attribute this to object detection producing higher overlaps between predicted and ground-truth regions.
  • Video-level example: 98.83% frame-level AUC versus 93.25% for the previous approach is reported on Avenue test video 03.The current method detects both abnormal events without false positives, whereas the earlier method detects only the second; adversarial training is credited with detecting the first.

4.6 Quantitative Results on ShanghaiTech

On ShanghaiTech, the current method improves over its preliminary version in frame-level, region-based, and track-based evaluation. A representative test video shows anomaly scores closely aligned with the ground-truth event.

  • Evaluation setup: Table 2 compares micro-averaged AUC, macro-averaged AUC, RBDC, and TBDC across methods on ShanghaiTech.Scores are reported in percent, with some values computed using official implementations when marked by an asterisk.
  • Frame-level evaluation: 4% micro-averaged AUC and 4.4% macro-averaged AUC improvements over the previous method are reported on ShanghaiTech.The previous method had the state-of-the-art frame-level AUC results, while other methods scored between 60.9% and 79.3%.
  • Region- and track-level evaluation: More than 20% RBDC improvement over the earlier version and a 34.25% TBDC improvement are reported on ShanghaiTech.The paper provides the region-level and track-level annotations needed for these evaluations.
  • Video-level example: 99.65% frame-level AUC versus 96.12% for the preliminary framework is reported on ShanghaiTech test video 05 0024.The current method's anomaly scores are described as strongly correlated with the ground-truth labels.

4.7 Quantitative Results on Subway

On Subway, the framework’s region- and track-based detection scores surpass Ionescu et al., while frame-level results depend on the label set and video.

  • Figure 5 compares frame-level anomaly scores with ground-truth events, showing our red scores against Ionescu et al.’s blue scores on Subway Exit.
  • Under the original labels for Subway Exit, Ionescu et al. achieve 95.1% frame-level AUC, while our method achieves 92.1%.
  • Under the new Subway Exit labels, our method outperforms Ionescu et al. by 0.9% in frame-level AUC.
  • On Subway Exit, our method exceeds Ionescu et al. by 24.1% in RBDC and 15.34% in TBDC.
  • On Subway Entrance, the new labels yield an improvement of over 10% against Ionescu et al. in frame-level AUC.
  • On Subway Entrance, our method surpasses Ionescu et al. by 41.27% in RBDC and more than 27% in TBDC.

4.9 Cross-Database Quantitative Results

Cross-database experiments evaluate whether the framework transfers across scenes, with performance generally remaining competitive despite domain differences.

  • Seven cross-database train-test pairs evaluate the framework’s scene-agnostic behavior, using in-domain results as upper bounds.
  • Training on ShanghaiTech instead of Avenue reduces micro-averaged AUC from 92.3% to 83.6%, yet remains above several target-trained methods.
  • The ShanghaiTech→Avenue experiment outperforms Ramachandra et al.’s in-domain method by 5.36% in RBDC, while TBDC drops by only 4.09%.
  • Training on ShanghaiTech and testing on UCSD Ped2 achieves a 90.6% micro-averaged frame-level AUC.
  • For Avenue→Subway Exit, the TBDC score drops from 67.96% in-domain to 63.93% cross-domain, while RBDC and TBDC remain above state-of-the-art in-domain scores.
  • Figure 7 presents true-positive, false-positive, and false-negative localizations across Avenue, ShanghaiTech, Subway Exit, and Subway Entrance.

4.11 Ablation Study

The ablation study progressively modifies the original object-centric framework, showing that binary classifiers and the final skip-connection design substantially improve performance, while some alternatives degrade it.

  • Ablation setup: The study progressively applies design changes to the original method until reaching the current framework.The evaluated metrics are micro-averaged AUC, macro-averaged AUC, RBDC, and TBDC.
  • Ablation results: Removing k-means clustering and the one-versus-rest SVM lowers AUC but improves some RBDC and TBDC scores.The simplified model computes anomaly scores using the mean absolute error between each CAE input and reconstruction.
  • Ablation results: Replacing SSD-FPN with YOLOv3 further reduces Avenue AUC and all ShanghaiTech metrics, but improves Avenue RBDC and TBDC.YOLOv3 detects four times more objects and has a lower false positive rate than SSD-FPN.
  • Ablation results: Integrating binary classifiers produces major improvements across all performance metrics.Replacing YOLOv3 with SSD-FPN after the other changes is ineffective, supporting YOLOv3 as the final detector choice.
  • Ablation results: Summed skip connections yield micro-averaged frame-level AUC scores of 92.3% on Avenue and 82.7% on ShanghaiTech, reported as state-of-the-art.Concatenated skip connections degrade performance before the summed variant is evaluated.

4.12 Running Time

The framework is evaluated for processing speed using its sequential pipeline and parallel execution across two threads.

  • Component speeds: YOLOv3 processes 64-frame mini-batches at 72 frames per second, while SelFlow runs at 20 FPS.The framework processes videos in mini-batches of 64 frames for optimal performance.
  • Overall speed: The complete sequential pipeline runs at 18 FPS with an average of five objects per frame.This single-thread speed is reported for fair comparison with other literature methods.
  • Overall speed: Parallel processing on two threads increases throughput to 24 FPS while using a single GPU.The authors state that this configuration can process video in real time.

4.13 Impact of Adversarial Training

Adversarial-training weight λ affects the four evaluation metrics differently across data sets, although adversarial training generally improves performance over λ = 0.

  • Experimental setup: The experiments vary λ from 0 to 1 in steps of 0.1, while the default configuration uses λ = 0.2.λ is the weight of the reversed gradient, and the default follows a recommendation from prior work.
  • Frame-level AUC: Adversarial training with λ ≥0.1 brings significant micro-averaged frame-level AUC gains on most data sets, but λ has almost no effect on Subway Exit.The optimal λ differs across data sets; ShanghaiTech reaches 83.1% at λ = 0.9, while UCSD Ped2 favors λ = 0.6.
  • Frame-level AUC: The top macro-averaged frame-level AUC is 89.9% on ShanghaiTech at λ = 0.9 and 91.3% on Avenue at λ = 0.1.Subway Exit has identical micro- and macro-averaged AUC because it contains one very long test video.
  • Region-based detection: The default λ = 0.2 produces optimal RBDC scores of 65.05% on Avenue and 41.34% on ShanghaiTech.UCSD Ped2 reaches 71.60% RBDC at λ = 0.4, while Subway Exit scores remain nearly constant.
  • Track-based detection: ShanghaiTech reaches its highest TBDC score of 79.45% at λ = 0.9, while λ = 0 produces significantly lower TBDC scores on three data sets.The reported lower scores occur on Avenue, ShanghaiTech, and UCSD Ped2.
  • Overall conclusion: The authors conclude that λ should preferably be tuned separately for each data set when a validation set is available, although adversarial training still improves results without optimal tuning.The conclusion is reported as consistent across the four evaluation metrics.

4.14 Testing Additional Design Choices

Additional design choices show that classifier weighting, reconstruction loss, input color, and difference representation affect performance unevenly across the four benchmarks.

  • Evaluation scope: The paper also reports additional design-choice results for Avenue, ShanghaiTech, Subway Exit, and UCSD Ped2, including figures for the four benchmarks.The corresponding results are presented in Table 7 and Figures 12–15.
  • Classifier weighting: Doubling the appearance-classifier weight improves Avenue RBDC and ShanghaiTech RBDC and TBDC, while other changes are minor.The original averaging of two motion classifiers and one appearance classifier implicitly gives motion greater weight.
  • Reconstruction loss: L1 reconstruction loss generally lowers performance on Avenue, ShanghaiTech, and Subway Exit, improving only UCSD Ped2.The authors consider L2 loss the better choice overall.
  • Input representation: Replacing grayscale with RGB input brings no improvement on Avenue or ShanghaiTech and is irrelevant for already-grayscale Subway and UCSD Ped2 videos.The appearance CAE uses grayscale input to retain the design of the earlier method.
  • Classifier input: Absolute differences generally outperform signed differences as binary-classifier inputs.The authors emphasize the magnitude of reconstruction error rather than whether reconstructed pixels are higher or lower.

4.15 Failure Cases

The framework has failure cases tied to object detection, frame-rate assumptions, and omitted location information. These constraints affect anomaly detection under certain data and scene conditions.

  • Object detection: Poor object detection on Street Scene’s small, atypically viewed objects prevents the framework from detecting some anomalies.The pre-trained YOLOv3 detector often misses objects of interest, including abnormal objects.
  • Object detection: Object-detector failures can cause false negatives for region-level and track-level metrics, although frame-level anomaly scores remain unaffected.On Avenue, papers and backpacks may be missed because of object-category limits or motion blur.
  • Frame rate: The pseudo-abnormal sampling approach assumes comparable training and testing frame rates; substantially lower test rates can falsely flag normal events and reduce cross-domain performance.The authors observed performance drops when UCSD Ped2 was the target dataset.
  • Location information: Because the benchmarks lacked location-based anomalies, the framework omitted location information from its representation.The authors suggest encoding bounding-box locations with pyramidal one-hot vectors as a possible remedy.

5 CONCLUSION

The paper revises an object-centric abnormal event detector with binary classifiers and adversarially enhanced auto-encoders, then evaluates it across four benchmarks. It reports improved region- and track-based performance, generally state-of-the-art results, background-agnostic behavior, and future work focused on computational speed.

  • Design changes: The revised framework replaces k-means and one-versus-rest SVMs with binary classifiers trained on normal and pseudo-abnormal examples.The auto-encoders also add adversarial and segmentation branches with skip connections.
  • Results: The design changes produce significant performance improvements on both region-based and track-based detection metrics.
  • Results: Experiments on Avenue, ShanghaiTech, Subway, and UCSD Ped2 generally attain state-of-the-art results, including when testing on scenes unseen during training.The paper also releases region-level and track-level annotations for ShanghaiTech and Subway.
  • Future work: Future work targets faster object detection and optical-flow estimation to reduce processing time.These components currently account for 95% of total processing time.
Loading 2008.12328v5…