PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-world Anomaly Detection in Surveillance Videos

Waqas Sultani, Chen Chen, Mubarak Shah

arXiv:1801.04264v3cs.CV

TL;DR

Real-world surveillance anomalies are diverse, while precise temporal annotations are costly and existing normal-only approaches can misfire under changing environments. The paper learns anomaly scores with weakly labeled multiple-instance ranking, using sparsity and temporal smoothness, and evaluates it on a 1900-video dataset. The proposed method performs significantly better than baseline anomaly-detection methods, while baseline activity recognition remains low on the challenging dataset.

  • Problem

    The paper addresses the need to detect diverse surveillance anomalies without relying on costly segment-level temporal annotations.

  • Method

    It uses deep multiple-instance ranking with video-level labels, treating videos as bags and temporal segments as instances.

  • Results

    The proposed anomaly-detection method performs significantly better than baseline methods on a new large-scale surveillance dataset.

  • Takeaways & Limitations

    The dataset provides a challenging benchmark spanning 1900 videos, 128 hours, and 13 realistic anomalous activities.

  • Takeaways & Limitations

    The method produces false alarms from occlusions and sudden normal group activity.

Abstract

from arXiv · show

Surveillance videos are able to capture a variety of realistic anomalies. In this paper, we propose to learn anomalies by exploiting both normal and anomalous videos. To avoid annotating the anomalous segments or clips in training videos, which is very time consuming, we propose to learn anomaly through the deep multiple instance ranking framework by leveraging weakly labeled training videos, i.e. the training labels (anomalous or normal) are at video-level instead of clip-level. In our approach, we consider normal and anomalous videos as bags and video segments as instances in multiple instance learning (MIL), and automatically learn a deep anomaly ranking model that predicts high anomaly scores for anomalous video segments. Furthermore, we introduce sparsity and temporal smoothness constraints in the ranking loss function to better localize anomaly during training. We also introduce a new large-scale first of its kind dataset of 128 hours of videos. It consists of 1900 long and untrimmed real-world surveillance videos, with 13 realistic anomalies such as fighting, road accident, burglary, robbery, etc. as well as normal activities. This dataset can be used for two tasks. First, general anomaly detection considering all anomalies in one group and all normal activities in another group. Second, for recognizing each of 13 anomalous activities. Our experimental results show that our MIL method for anomaly detection achieves significant improvement on anomaly detection performance as compared to the state-of-the-art approaches. We provide the results of several recent deep learning baselines on anomalous activity recognition. The low recognition performance of these baselines reveals that our dataset is very challenging and opens more opportunities for future work. The dataset is available at: https://webpages.uncc.edu/cchen62/dataset.html

1. Introduction

The paper addresses practical surveillance anomaly detection under rare, diverse, and ambiguously defined abnormal events. It proposes weakly supervised MIL ranking and introduces a large, challenging dataset, reporting superior anomaly-detection performance.

  • Motivation: Automatic anomaly detection is needed because surveillance cameras generate more monitoring demand than law-enforcement personnel can feasibly handle.A practical system should signal deviations promptly and identify the anomaly’s time window.
  • Motivation: Specific-event detectors have limited practical use because real-world anomalies are diverse and event-specific solutions do not generalize.The paper emphasizes that anomalous events are difficult to enumerate and may require minimal supervision.
  • Motivation: Sparse-coding approaches can produce high false-alarm rates when surveillance environments change across times of day and normal behaviors.These methods assume an initial video portion is representative of normal activity.
  • Contributions: The proposed method uses weakly labeled videos in a multiple-instance-learning ranking framework, with sparsity and smoothness constraints for segment-level anomaly scores.The training labels are video-level rather than temporal segment-level annotations.
  • Contributions: The dataset contains 1900 real-world surveillance videos spanning 128 hours and 13 anomalous events alongside normal activities.It is reported as more than 15 times larger in video count than existing anomaly datasets.
  • Results: Experimental results show superior anomaly-detection performance over state-of-the-art approaches, while baseline recognition results indicate a challenging activity-recognition benchmark.The recognition baselines include C3D and TCNN for 13 anomalous activities.

2. Related Work

Prior work models surveillance anomalies through event-specific detectors, normal-pattern deviations, sparse representations, autoencoders, and deep ranking. This paper instead formulates anomaly detection as weakly supervised multiple-instance ranking using both normal and anomalous videos.

  • Existing approaches: Earlier surveillance approaches targeted specific behaviors such as violence or aggression, limiting coverage beyond those events.Other methods modeled normal motion or global patterns without requiring person tracking.
  • Existing approaches: Sparse-representation methods detect anomalies through large reconstruction errors, but video-action learning also faces difficult and laborious annotation requirements.Deep autoencoder methods learn normal behavior and use reconstruction loss for anomaly detection.
  • Ranking: Deep ranking networks have been applied across computer-vision tasks, including feature learning, retrieval, recognition, and verification.The related work positions ranking as a framework for improving relative item scores.
  • Proposed formulation: The paper formulates anomaly detection as regression in a ranking framework using normal and anomalous data.This distinguishes the approach from methods that model only normal behavior.
  • Proposed formulation: Multiple-instance learning avoids precise temporal annotations by using video-level normal or abnormal labels to learn segment-level anomaly detection.Video-level labels are easier to obtain than temporal annotations of anomalous parts.

3. Proposed Anomaly Detection Method

The method frames anomaly detection as weakly supervised multiple instance learning, using video-level labels while learning segment-level anomaly scores. A ranking loss focuses on the highest-scored segments and adds sparsity and temporal smoothness to better reflect anomaly duration and continuity.

  • 3.1. Multiple Instance Learning: Videos become bags of temporal segments, allowing MIL to use video-level positive or negative labels without precise temporal annotations.Positive bags contain at least one anomalous instance, whereas negative bags contain none.
  • 3.2. Deep MIL Ranking Model: The model treats anomaly detection as regression, seeking higher scores for anomalous segments than for normal segments.A conventional segment-level ranking loss would require annotations unavailable in the weakly supervised setting.
  • 3.2. Deep MIL Ranking Model: Ranking compares the highest-scored instance in each positive and negative bag, using the positive maximum as the likely anomaly and the negative maximum as a hard normal instance.This focuses learning on the segments most relevant to distinguishing anomalous from normal videos.
  • 3.2. Deep MIL Ranking Model: Sparsity encourages only a few anomalous segments to receive high scores, while temporal smoothness reduces score differences between adjacent segments.These constraints address short-duration anomalies and the sequential structure of videos.
  • 3.2. Deep MIL Ranking Model: Training uses fixed non-overlapping temporal segments represented by 3D convolutional features and optimizes the complete objective over model weights.The extracted features capture appearance and motion dynamics while remaining computationally efficient.

4. Dataset

The paper introduces a large-scale dataset of long, untrimmed surveillance videos covering diverse real-world anomalies. Its construction and annotation process supports weakly supervised training and temporal evaluation across all 13 anomaly types.

  • 4.2. Our dataset: The dataset addresses limitations of earlier small, location-specific, or staged benchmarks by covering 13 real-world anomaly categories.Categories include abuse, arrest, arson, assault, accident, burglary, explosion, fighting, robbery, shooting, stealing, shoplifting, and vandalism.
  • 4.2. Our dataset: Videos were collected by trained annotators through multilingual searches on YouTube and LiveLeak using varied queries for each anomaly.The collection process was designed to retrieve diverse examples and ensure dataset quality.
  • 4.2. Our dataset: Training requires only video-level labels, while testing uses temporal annotations marking the start and end frames of anomalies.Multiple annotators label each testing anomaly, and the final temporal annotations average their judgments.
  • 4.2. Our dataset: The dataset is split into 800 normal and 810 anomalous training videos, plus 150 normal and 140 anomalous testing videos.Both splits contain all 13 anomalies at varied temporal locations, and some videos include multiple anomalies.

5. Experiments

Experiments evaluate the proposed anomaly detector against established approaches, assess its false-alarm behavior and qualitative localization, and test the dataset for anomalous activity recognition. The method outperforms existing anomaly-detection approaches but remains challenged by difficult activity-recognition and failure-case scenarios.

  • Comparison with the State-of-the-art: The method significantly outperforms existing approaches, achieving much higher true positive rates at false positive rates of 0.1–0.3.Comparisons include a binary classifier, Lu et al., Hasan et al., and the proposed method with and without constraints.
  • Comparison with the State-of-the-art: Training with anomalous and normal videos is indispensable for robust anomaly detection because competing methods miss anomalies or generate high scores for new normal patterns.The binary classifier produces low anomaly scores, the dictionary approach reconstructs anomalous patterns well, and Hasan et al. produces false alarms on new normal patterns.
  • Qualitative Results: The method detects several anomalous events with high scores and produces scores close to 0 throughout two normal videos, but fails in dark scenes and during unusual group activity.The qualitative evaluation includes animal abuse, explosion, road accident, shooting, burglary, and normal videos.
  • False Alarm Rate: The method has a much lower false alarm rate than other approaches on normal testing videos at a 50% threshold.The authors interpret this as evidence that joint training on anomalous and normal videos helps the model learn more general normal patterns.
  • Anomalous Activity Recognition Experiments: C3D and TCNN perform poorly on the dataset’s anomalous activity recognition task, which uses event labels unavailable to the anomaly-detection method.The activity-recognition evaluation uses 50 videos per event, 75/25 training and testing splits, and 4-fold cross-validation.

6. Conclusions

The paper proposes deep multiple instance ranking with weakly labeled surveillance videos to detect realistic anomalies without temporal segment annotations. It introduces a large-scale anomaly dataset, and reports significantly better anomaly detection than baseline methods while supporting anomalous activity recognition.

  • The proposed anomaly detector uses deep multiple instance ranking with weakly labeled normal and anomalous videos.Temporal annotations of anomalous segments are avoided by learning from video-level labels.
  • The paper introduces a new large-scale anomaly dataset containing varied real-world surveillance anomalies.The dataset also supports anomalous activity recognition as a second task.
  • The proposed anomaly detection approach performs significantly better than baseline methods on the introduced dataset.
Loading 1801.04264v3…