PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Detection and Recounting of Abnormal Events by Learning Deep Generic Knowledge

Ryota Hinami, Tao Mei, Shin'ichi Satoh

arXiv:1709.09121v1cs.CV

TL;DR

The paper addresses joint detection and recounting of abnormal events, motivated by the need to explain surveillance alerts and the difficulty of applying CNNs when anomalies are environment-dependent. It integrates a generic multi-task CNN that learns visual concepts with environment-specific anomaly detectors. The framework outperforms prior methods on Avenue and UCSD Ped2 and produces promising abnormal-event recounting results.

  • Problem

    Joint abnormal-event detection and recounting is underexplored, while CNN-based detection is difficult because anomaly detection is environment-dependent and lacks positive samples.

  • Method

    A multi-task Fast R-CNN learns object, action, and attribute knowledge, whose semantic features and classification scores feed environment-specific anomaly detectors for detection and recounting.

  • Results

    The framework outperforms other methods on Avenue and UCSD Ped2 for abnormal-event detection and produces promising recounting results.

  • Takeaways & Limitations

    Generic visual knowledge can be integrated with environment-specific anomaly detection to support both abnormal-event detection and human-understandable recounting.

  • Takeaways & Limitations

    The approach is limited by appearance-only evidence and learned concepts, missing some directional, out-of-vocabulary, and object-interaction abnormalities.

Abstract

from arXiv · show

This paper addresses the problem of joint detection and recounting of abnormal events in videos. Recounting of abnormal events, i.e., explaining why they are judged to be abnormal, is an unexplored but critical task in video surveillance, because it helps human observers quickly judge if they are false alarms or not. To describe the events in the human-understandable form for event recounting, learning generic knowledge about visual concepts (e.g., object and action) is crucial. Although convolutional neural networks (CNNs) have achieved promising results in learning such concepts, it remains an open question as to how to effectively use CNNs for abnormal event detection, mainly due to the environment-dependent nature of the anomaly detection. In this paper, we tackle this problem by integrating a generic CNN model and environment-dependent anomaly detectors. Our approach first learns CNN with multiple visual tasks to exploit semantic information that is useful for detecting and recounting abnormal events. By appropriately plugging the model into anomaly detectors, we can detect and recount abnormal events while taking advantage of the discriminative power of CNNs. Our approach outperforms the state-of-the-art on Avenue and UCSD Ped2 benchmarks for abnormal event detection and also produces promising results of abnormal event recounting.

1. Introduction

The paper introduces joint abnormal-event detection and recounting for surveillance, using generic visual knowledge alongside environment-dependent anomaly detection. CNN-based semantic representations are integrated with anomaly detectors to identify abnormal events and explain their visual evidence.

  • Event recounting explains why a detected event is abnormal, helping observers quickly identify unimportant alerts without reviewing entire videos.
  • The paper targets joint abnormal-event detection and recounting, a task useful for surveillance applications and for understanding detection algorithms.
  • Generic knowledge about objects, actions, and attributes supports human-understandable recounting and aligns anomaly detection with recognizing visual concepts.
  • CNNs learn visual concepts effectively from positive examples, but anomaly detection lacks positive samples because abnormality depends on the environment.
  • The proposed framework integrates a supervised multi-task Fast R-CNN generic model with environment-specific anomaly detectors operating on CNN outputs.
  • The approach achieves superior benchmark performance and demonstrates the effectiveness of deep CNN features for abnormal-event detection.

2. Related Work

Prior video anomaly-detection methods model trajectories or local appearance and activity, often adding global consistency over local anomaly scores. The paper identifies CNN-based semantic anomaly detection as an underexplored direction because anomaly training lacks conventional supervision.

  • Object-trajectory modeling captures long-term semantics but can fail in crowded or cluttered scenes.
  • Local anomaly detection models normal features and assigns novelty scores using distance-based, reconstruction-based, domain-based, or probabilistic detectors.
  • Global inference methods impose temporal or spatial consistency on local anomaly scores using models such as HMMs and MRFs.
  • CNN application to abnormal-event detection remains underexplored because CNNs typically require rich supervision and abundant training data unavailable by definition.

3. Abnormal Event Detection and Recounting

The proposed pipeline learns generic visual concepts with multi-task Fast R-CNN, then learns environment-specific anomaly detectors from semantic features and concept scores. It uses these outputs both to detect abnormal regions and to recount which concepts provide anomalous evidence.

  • Multi-task Fast R-CNN extracts semantic features and visual-concept classification scores from local regions using large supervised datasets.
  • Environment-specific anomaly detectors model normal behavior from semantic features and concept scores, producing anomaly scores for test proposals.
  • The testing pipeline detects proposals, extracts CNN outputs, classifies normal or abnormal regions, and recounts concepts associated with abnormal events.
  • Generic knowledge is learned for objects, actions, and attributes so semantic information can support both detection and recounting.
  • The model jointly predicts object, action, and attribute concepts, using shared visual representations to capture different abnormality types.
  • Recounting combines predicted concept categories with anomaly scores for each concept to identify the evidence supporting an abnormal-event judgment.

4. Experiments

Experiments evaluate the framework on UCSD Ped2 and Avenue17 using object proposals, semantic features, and environment-specific anomaly detectors. Multi-task Fast R-CNN features outperform conventional appearance features and support strong detection and recounting, while qualitative results expose limitations in motion and interaction understanding.

  • Datasets: The evaluation uses UCSD Ped2 and Avenue17, with Avenue17 assessed using frame-level metrics because some pixel-level annotations are subjective.Ped2 was selected over Ped1 because its higher resolution better supports object proposal and CNN-based processing.
  • Implementation: The detection pipeline extracts object proposals, obtains Fast R-CNN semantic features and classification scores, and applies anomaly detectors to assign proposal-level anomaly scores.Proposals above a threshold are detected as abnormal events.
  • Appearance Features: FRCN features significantly outperform HOG and SDAE features across benchmarks because they discriminate visual concepts that the lower-level features cannot.FRCN action features perform slightly better than the other FRCN variants because challenging benchmark anomalies are often action-related.
  • Anomaly Detectors: FRCN features outperform HOG and SDAE with varied anomaly detectors, and performance is insensitive to the detector choice.The results indicate that FRCN features can replace conventional appearance features in abnormal-event-detection frameworks.
  • State-of-the-art Comparison: 89.2% AUC on Ped2 outperforms the state-of-the-art 66.5% AUC, while the method achieves best performance without motion features.The method also localizes abnormal objects and generalizes to untrained categories, but Ped1 performance is lower because of its low resolution.
  • Qualitative Evaluation: Qualitative recounting predicts abnormal concepts such as riding, truck, and bending while assigning lower anomaly scores to normal concepts such as person and black.Other examples reveal false positives, unknown concepts, and object interactions that the current approach cannot fully recount.

5. Evaluation with Artificial Datasets

The artificial-dataset evaluation addresses benchmark limitations by testing unseen visual concept detection and recounting under more varied, objectively evaluated conditions. Multi-task Fast R-CNN provides the strongest overall feature representation for detecting and recounting unseen concepts.

  • Evaluation settings: The benchmark is limited by small datasets, subjective abnormality definitions, and absent category labels for evaluating recounting.These issues complicate evaluation of both abnormal event detection and recounting.
  • Evaluation settings: Unseen concept detection was evaluated on artificially generated datasets based on COCO and PASCAL, with five training–test splits per dataset.Unseen categories were withheld from training, and randomly sampled distractor images balanced training and test sizes.
  • Unseen concept detection: Unseen categories were detected as regions with high nearest-neighbor anomaly scores, using ground-truth bounding boxes as input RoIs.The anomaly detector was trained separately for each training set.
  • Unseen concept detection: Fast R-CNN significantly outperformed HOG and SDAE, while multi-task Fast R-CNN achieved the second-highest mAP across all evaluated concept types.Single-task Fast R-CNN trained on the evaluation task performed best in every task; multi-task Fast R-CNN was significantly better than models trained on different tasks.
  • Unseen concept recounting: Multi-task Fast R-CNN achieved the best AUC for recounting all concept types, whereas HOG and SDAE could hardly recount unseen concepts.The result supports the role of deeply learned generic knowledge in concept-level abnormal-event recounting.

6. Conclusion

The paper unifies generic visual knowledge with environment-specific anomaly detection to jointly detect and recount abnormal events. Experiments report improved state-of-the-art abnormal-event detection and successful recounting examples, while the approach remains a first step toward more complex concepts.

  • Conclusion: The framework learns generic visual knowledge with multi-task Fast R-CNN and environment-specific knowledge with anomaly detectors trained on its outputs.The unified framework targets both abnormal event detection and recounting.
  • Conclusion: Experiments improved state-of-the-art performance on challenging abnormal-event detection benchmarks and produced successful recounting examples.The reported evaluation covers both abnormal event detection and recounting.
  • Conclusion: The study focuses on basic concepts such as actions, with extension to complex concepts such as object interactions identified as future work.The authors characterize this work as a first step toward higher-level abnormal event detection.
Loading 1709.09121v1…