PaperlayerView exact revision on arXiv

Source-linked AI summary

RWF-2000: An Open Large Scale Video Database for Violence Detection

Ming Cheng, Kunjing Cai, Ming Li

arXiv:1911.05913v3cs.CV

TL;DR

Manually monitoring large volumes of surveillance video is expensive, motivating automatic violence detection. The paper introduces the 2,000-clip RWF-2000 surveillance dataset and a Flow Gated Network combining appearance and temporal features, achieving 87.25% accuracy.

  • Problem

    Automatically recognizing violent behavior is needed because manually monitoring large volumes of surveillance video is expensive and challenging.

  • Method

    The paper collects the 2,000-clip RWF-2000 surveillance dataset and proposes a self-learned pooling mechanism combining appearance and temporal features through optical flow.

  • Results

    87.25% accuracy is achieved by the fused version of the proposed method.

  • Takeaways & Limitations

    RWF-2000 provides a large surveillance video dataset for violence detection in realistic scenes, while the method uses optical flow for temporal feature pooling.

Abstract

from arXiv · show

In recent years, surveillance cameras are widely deployed in public places, and the general crime rate has been reduced significantly due to these ubiquitous devices. Usually, these cameras provide cues and evidence after crimes are conducted, while they are rarely used to prevent or stop criminal activities in time. It is both time and labor consuming to manually monitor a large amount of video data from surveillance cameras. Therefore, automatically recognizing violent behaviors from video signals becomes essential. This paper summarizes several existing video datasets for violence detection and proposes the RWF-2000 database with 2,000 videos captured by surveillance cameras in real-world scenes. Also, we present a new method that utilizes both the merits of 3D-CNNs and optical flow, namely Flow Gated Network. The proposed approach obtains an accuracy of 87.25% on the test set of our proposed database. The database and source codes are currently open to access.

I. INTRODUCTION

Violence detection in surveillance video is important but difficult because manual monitoring is costly and video requires effective spatial-temporal modeling. The paper addresses limitations in existing datasets and introduces a larger, realistic dataset with a model combining appearance and temporal features.

  • Manual real-time monitoring of extensive surveillance video is expensive, motivating automatic recognition of violent or criminal scenes.
  • Video violence detection must model both spatial content and temporal motion across consecutive, highly correlated frames.
  • Earlier datasets suffer from small scale, limited diversity, low resolution, or movie-based footage unlike real-world scenes.
  • RWF-2000 contains 2,000 surveillance-video clips collected from real-world scenes and is freely released to researchers.
  • The proposed model uses a self-learned pooling mechanism to combine appearance and temporal features.

II. RELATED WORK

Prior violence datasets vary in annotation, scale, capture devices, and realism, leaving gaps in large, diverse surveillance footage. RWF-2000 is constructed as 2,000 trimmed surveillance clips from realistic scenes to address these limitations.

  • Violence datasets may be trimmed with video-level labels or untrimmed with frame-level start and end annotations.
  • Movies Fight has 200 clips, while Hockey Fight has 1,000 clips from a single hockey setting and therefore limited diversity.
  • Crowd Violence contains 246 short videos focused on crowded scenes but has low image quality.
  • Other datasets use actor-performed interactions, mobile or car-mounted cameras, or edited Hollywood movies rather than consistently realistic surveillance footage.
  • Existing datasets are limited by insufficient data, crude annotations, or video sources that are not close enough to realistic violence.
  • RWF-2000 consists of 2,000 trimmed clips captured by surveillance cameras in real-world scenes to address these dataset limitations.

B. Previous Methods

Violence-recognition methods range from hand-crafted features with shallow classifiers to deep networks using multiple video modalities. The paper emphasizes RGB and optical flow with self-learned temporal pooling while noting complexity and over-fitting concerns in prior approaches.

  • Traditional approaches extract hand-crafted motion or space-time features and classify them with machine-learning models.
  • Prior methods use optical-flow descriptors, motion vectors, improved trajectories, or Fisher encoding to represent violent activity.
  • Deep-learning approaches such as TSN, ECO, and C3D have become mainstream for video action recognition and violence detection.
  • Multi-stream models combine RGB, optical flow, and acceleration maps, sometimes followed by LSTM-based temporal modeling.
  • ConvLSTM models extract frame-level spatial features with CNNs and model temporal relationships using LSTMs.
  • Multimodal FightNet uses RGB, optical flow, and acceleration images but is vulnerable to over-fitting under insufficient violence data.
  • The Flow-Gated architecture uses RGB and optical flow to self-learn temporal pooling and reduce temporal input channels.

A. Data Collection

RWF-2000 is a 2,000-clip surveillance-video dataset collected from real-world scenes, with varied resolutions and nearly uniform violence proportions across scales.

  • A. Data Collection: RWF-2000 contains 2,000 video clips captured by surveillance cameras in real-world scenes.The dataset is collected from YouTube and uses unmodified surveillance footage.
  • A. Data Collection: Collection searches used violence-related keywords in multiple languages to increase dataset diversity.The process included English, Chinese, German, French, Japanese, Russian, and other languages.
  • A. Data Collection: The dataset is split into 80% training and 20% test clips, with violent and non-violent videos evenly represented.Half of the videos contain violent behaviors, while the remainder are non-violent.
  • A. Data Collection: The resolution distribution forms clusters around 240P, 320P, 720P, and 1080P.Random jitter was added to overlapping resolution points for visualization.
  • A. Data Collection: The proportion of violent videos is almost uniform across different resolution scales.

B. Flow Gated Network

The Flow Gated Network combines RGB and optical-flow processing with a self-learned temporal pooling mechanism that uses motion information to control feature retention.

  • B. Flow Gated Network: The architecture uses self-learned temporal pooling instead of relying solely on human-designed pooling strategies.The design aims to preserve useful motion information through network learning.
  • B. Flow Gated Network: The network has RGB and optical-flow channels, a merging block, and fully connected layers.The RGB and optical-flow branches use consistent cascaded 3D CNN structures before fusion.
  • B. Flow Gated Network: The model incorporates depth-wise separable convolutions from MobileNet and pseudo-3D residual networks.
  • B. Flow Gated Network: An optical-flow branch acts as a gate that scales RGB features before temporal max-pooling.Sigmoid outputs between 0 and 1 determine which RGB information is more likely to be retained or dropped.

A. Basic Settings

The implementation reduces video input through motion-guided cropping and uniform frame sampling, producing fixed-size clips with RGB and optical-flow channels.

  • A. Basic Settings: Dense optical flow identifies the region with the greatest motion intensity for cropping.Vector norms form motion-intensity heat maps, which are summed into a final motion map.
  • A. Basic Settings: Uniform frame sampling generates fixed-length video clips while cropping reduces the spatial input region.Together, these operations substantially decrease the amount of input data.
  • A. Basic Settings: Table II specifies the model architecture parameters, with T denoting the number of repeats.
  • A. Basic Settings: The processed input has shape 64 × 224 × 224 × 5.The five channels comprise three RGB channels and two optical-flow components.
  • A. Basic Settings: Training uses SGD with momentum 0.9 and learning-rate decay 1e-6.Brightness transformation and random rotation are used as augmentation for varied lighting and camera angles.

B. Ablation Experiments

Ablation experiments show that combining RGB and optical flow performs best, while depth-wise separable 3D convolutions improve accuracy and reduce parameter count relative to traditional 3D convolutions.

  • B. Ablation Experiments: The ablation study evaluates four different Flow Gated Network cases on RWF-2000.
  • B. Ablation Experiments: The RGB Only case removes the optical-flow channel, whereas OPT Only removes the RGB channel.
  • B. Ablation Experiments: 87.25% accuracy is achieved by the fused RGB-and-optical-flow P3D configuration.This is the best accuracy among the evaluated cases.
  • B. Ablation Experiments: 85.75% accuracy is achieved by the fused C3D configuration using traditional 3D convolutions.This configuration has nearly double the model parameters of the depth-wise separable alternative.

C. Comparisons

Comparisons show that dataset characteristics strongly affect violence-recognition performance, particularly for deep learning and optical-flow-based methods. On RWF-2000, the proposed method achieves accurate violence recognition with fewer trainable parameters.

  • Movies Fight causes over-fitting across deep learning methods and is no longer suitable for current evaluation.
  • Optical-flow methods perform poorly on Crowd Violence because its camera movement, unstable lighting, and low image quality violate optical-flow assumptions.
  • RWF-2000 benchmarks evaluate models without pre-training and report accurate violence recognition with fewer trainable parameters for the proposed method.

V. CONCLUSION

The paper concludes by presenting RWF-2000 and its surveillance-video violence-detection method as its main contributions. It also identifies future work on avoiding explicit optical flow and expanding the dataset.

  • RWF-2000 is presented as the largest surveillance-video violence dataset for realistic scenes.
  • The method uses a unique optical-flow pooling mechanism for temporal feature pooling instead of human-designed strategies.
  • Future work will explore methods without explicit optical flow and expand RWF-2000.
Loading 1911.05913v3…