Source-linked AI summary
Learning to Detect Violent Videos using Convolutional Long Short-Term Memory
Swathikiran Sudhakaran, Oswald Lanz
TL;DR
Manual surveillance-video inspection makes automatic violence recognition valuable, while existing approaches rely largely on hand-crafted features. The paper proposes an end-to-end CNN–ConvLSTM model using adjacent frame differences, and reports improved performance over state-of-the-art methods across three datasets, with ConvLSTM using fewer parameters than LSTM.
Problem
Manual surveillance-video inspection is impractical, and few deep-learning methods address violence detection despite the limitations of hand-crafted features.
Method
An end-to-end network extracts frame features with a CNN, aggregates localized spatio-temporal changes with ConvLSTM, and uses adjacent frame differences as input.
Results
The proposed method improves performance over state-of-the-art methods on three datasets, while ConvLSTM produces better representations than LSTM with fewer parameters.
Takeaways & Limitations
Frame-difference inputs and convolutional recurrent aggregation are presented as effective components for violent-video recognition.
Takeaways & Limitations
Violence is operationalized as aggressive human behavior rather than the presence of blood or fire, creating a definition boundary for the task.
Abstract
from arXiv · showhide
Developing a technique for the automatic analysis of surveillance videos in order to identify the presence of violence is of broad interest. In this work, we propose a deep neural network for the purpose of recognizing violent videos. A convolutional neural network is used to extract frame level features from a video. The frame level features are then aggregated using a variant of the long short term memory that uses convolutional gates. The convolutional neural network along with the convolutional long short term memory is capable of capturing localized spatio-temporal features which enables the analysis of local motion taking place in the video. We also propose to use adjacent frame differences as the input to the model thereby forcing it to encode the changes occurring in the video. The performance of the proposed feature extraction pipeline is evaluated on three standard benchmark datasets in terms of recognition accuracy. Comparison of the results obtained with the state of the art techniques revealed the promising capability of the proposed method in recognizing violent videos.
1. Introduction
The paper addresses the impracticality of manually inspecting surveillance videos for violence and develops a deep-learning approach for automatic violent-video classification. It focuses on aggressive human behavior and evaluates the approach on three benchmarks.
- Manual inspection of surveillance videos for violence is practically infeasible and inefficient.
- Deep learning is proposed because few existing violence-detection methods use it, while most rely on hand-crafted visual features.The paper highlights deep learning's generalization and ability to process raw pixels with limited preprocessing.
- The study develops an end-to-end trainable deep neural network for violent-video classification.
- Localized spatio-temporal recurrent encoding is presented as a more parameter-efficient representation for detecting violence.
- A model trained on adjacent frame differences performs better than one trained on raw frames.
- The method is experimentally validated on three widely used benchmarks for violent-video classification.
2. Related Works
Prior violence-detection work primarily uses visual cues, audio, or both, with visual-only methods relying heavily on hand-crafted motion and appearance descriptors. ConvLSTM extends sequence modeling by replacing fully connected LSTM gates with convolutional layers that preserve spatio-temporal structure.
- Violence-detection methods use visual content, audio content, or both, although this review concentrates on visual cues because surveillance audio is generally unavailable.
- Earlier visual methods characterize violence using motion trajectories, limb orientations, spatio-temporal interest points, optical flow, blobs, and improved trajectories.
- Hand-crafted approaches aggregate frame-level features with methods including bag of words, histograms, and improved Fisher encoding.
- LSTM recurrent networks provide sequence modeling for tasks such as translation, speech recognition, caption generation, and video action recognition.
- ConvLSTM replaces fully connected LSTM gates with convolutional layers, enabling its memory cell to encode spatio-temporal information.
3. Proposed method
The proposed network extracts frame features with convolutional layers and aggregates them over time using ConvLSTM before classification. It models localized changes through convolutional recurrence and uses adjacent frame differences instead of raw frames as input.
- The model classifies videos as violent or non-violent using convolutional feature extraction followed by ConvLSTM-based temporal encoding.
- Network Architecture: The architecture uses alternating convolutional, normalization, and pooling layers, followed by ConvLSTM and fully connected classification layers.
- ConvLSTM: CNNs represent individual frames, while ConvLSTM encodes spatial and temporal changes through convolutional gates.
- Network Architecture: The ConvLSTM hidden state at the final time step represents the input video and is passed to fully connected layers for classification.
- Implementation: The implementation uses a pre-trained AlexNet CNN for extracting frame-level features.
- Frame-difference input: Adjacent frame differences replace raw frames as inputs, forcing the network to model changes between frames.The difference image is described as a crude approximation of optical-flow images.
4. Experiments and Results
The proposed method is evaluated on three benchmark datasets using cross-validation, frame differences, and pretrained CNN features. It outperforms existing methods on two datasets, while performance is limited on crowd violence videos and convLSTM uses fewer parameters than LSTM.
- Datasets: The method is evaluated on the Hockey Fight, Movies, and Violent-Flows Crowd Violence datasets.The datasets include hockey footage, movie fight sequences, and crowd violence videos.
- Evaluation protocol: Five-fold cross-validation is used, and model architecture selection is performed on the Hockey Fight Dataset.The study follows the evaluation technique used in existing literature.
- Model selection: Pretrained ImageNet features outperform randomly initialized features, leading the authors to use frame differences with a pretrained network.The comparison is reported for video frames versus frame differences as inputs.
- Benchmark results: The proposed method outperforms existing techniques on the Hockey Fight and Movies datasets.The comparison includes 10 state-of-the-art techniques.
- Benchmark results: On Violent-Flows, the method ranks second because aggressive behavior often involves only a small part of a large crowd.The majority of people are spectators, which challenges the network’s classification.
- ConvLSTM comparison: ConvLSTM requires 9.6M parameters versus 77.5M for LSTM and is reported to generalize better without overfitting on limited data.The proposed model processes 31 frames per second on an NVIDIA K40 GPU.
5. Conclusions
The paper presents an end-to-end CNN–convLSTM model for violence detection that captures frame-level and temporal information. Across three datasets, it improves over state-of-the-art methods, and frame-difference inputs outperform raw frames.
- Conclusions: The model uses a CNN for frame-level feature extraction and convLSTM for temporal feature aggregation.This architecture is end-to-end trainable.
- Conclusions: The proposed method improves performance compared with state-of-the-art methods across three evaluated datasets.The conclusion reports improved performance without specifying dataset-level values here.
- Conclusions: A network trained on frame differences performs better than a network trained using frames as inputs.The frame-difference representation is intended to model changes between frames.
- Conclusions: ConvLSTM generates a better video representation than fully connected LSTM with fewer parameters, thereby avoiding overfitting.The comparison is reported as part of the study’s conclusions.