Source-linked AI summary
Abnormal Event Detection in Videos using Spatiotemporal Autoencoder
Yong Shean Chong, Yong Haur Tay
TL;DR
Video anomaly detection must identify rare, contextual events despite limited abnormal examples and difficult crowded-scene modeling. The paper proposes an unsupervised spatiotemporal autoencoder that learns spatial representations and regular temporal patterns, achieving accuracy comparable to state-of-the-art methods at speeds up to 140 fps. Its scope is constrained by limitations in related approaches, including unavailable abnormal footage, collapsed temporal information, and memory-intensive ConvLSTM processing.
Problem
Video anomalies are rare, contextual, and difficult to label, while high dimensionality, noise, ambiguity, and crowded scenes complicate learning.
Method
A spatiotemporal autoencoder learns spatial video representations with convolutional autoencoders and regular temporal patterns with convolutional temporal autoencoders.
Results
The method achieves detection accuracy comparable to state-of-the-art methods while running at speeds up to 140 fps.
Takeaways & Limitations
The framework is domain free, requires no additional human effort, and can be applied across different scenes using normal video training data.
Takeaways & Limitations
Related methods face practical limits from unavailable abnormal footage, collapsed temporal information, and memory-intensive ConvLSTM layers.
Abstract
from arXiv · showhide
We present an efficient method for detecting anomalies in videos. Recent applications of convolutional neural networks have shown promises of convolutional layers for object detection and recognition, especially in images. However, convolutional neural networks are supervised and require labels as learning signals. We propose a spatiotemporal architecture for anomaly detection in videos including crowded scenes. Our architecture includes two main components, one for spatial feature representation, and one for learning the temporal evolution of the spatial features. Experimental results on Avenue, Subway and UCSD benchmarks confirm that the detection accuracy of our method is comparable to state-of-the-art methods at a considerable speed of up to 140 fps.
1 Introduction
Video anomaly detection is needed for rare, context-dependent events that are costly to label and difficult to generalize across scenes. The paper addresses this with unsupervised hierarchical feature learning and temporal modeling, reporting broad applicability and efficient performance.
- Motivation: Rare events in long videos make manual anomaly detection labor-intensive and motivate automated detection and segmentation.Existing systems also require substantial per-stream configuration and predefined heuristics, limiting generalization across surveillance scenes.
- Challenges: Video anomalies are difficult to model because video is high-dimensional and noisy, while anomalousness depends on context and can be ambiguous.For example, running may be anomalous in a restaurant but normal in a park.
- Prior limitations: Supervised action-recognition and binary-classification approaches require labeled abnormal events, which are costly to obtain and may not cover rare activities or crowded scenes.Unsupervised approaches instead learn from unlabelled footage containing little or no abnormal activity.
- Approach: The proposed framework learns hierarchical video representations with stacked convolutional autoencoders and models regular temporal patterns using convolutional temporal autoencoders.Spatial features are learned from frames before being grouped into a video representation for temporal learning.
- Reported contribution: The method is domain free, requires no additional human effort, applies across scenes, and consistently outperforms similar methods while maintaining short running time.The paper evaluates these claims on real-world datasets.
- Contribution: The authors aim to replace labor-intensive handcrafted feature engineering with representative hierarchical features learned directly from data.Autoencoders learn features rather than requiring researchers to design suitable representations from domain knowledge.
2 Related Work
Prior anomaly-detection methods either depend on unavailable abnormal examples, fragile tracking, handcrafted local features, or architectures with limited temporal modeling and high computational cost. These limitations motivate learned spatiotemporal representations.
- Unsupervised setting: Anomaly detection can learn normal visual patterns from mostly normal training videos and flag events that deviate from those patterns.This setting reflects the greater availability of normal footage than abnormal footage.
- Trajectory methods: Trajectory-based methods rely heavily on precise tracking and become impractical in crowded or complex scenes.They are more suitable for scenes containing relatively few objects.
- Non-tracking methods: Non-tracking approaches use handcrafted local features such as HOG, histogram of oriented flows, and optical flow, often grouped into bags of visual words.Their appeal includes low computational cost and attention to abnormal behavior.
- BOV limitations: Classical bag-of-visual-words methods destroy compositional information when grouping visual volumes and require trial-and-error selection of cluster counts.Codebook search during testing can also make real-time anomaly detection impractical.
- Deep-learning background: The paper positions deep learning as a way to learn useful hierarchical features directly from data with minimal preprocessing.Convolutional autoencoders replace fully connected layers and classifiers with mirrored convolutional layers.
- Deep-learning limitations: 3D ConvNets are impractical when abnormal training segments are unavailable, while 2D convolutional autoencoders collapse temporal information after the first layer.Convolutional LSTM methods may also require small mini-batches, slowing training and testing.
3 Methodology
The method treats anomaly detection as spatiotemporal sequence outlier detection. It combines a spatial feature extractor with a temporal encoder-decoder trained on normal video volumes to minimize reconstruction error.
- Methodology: The model assumes abnormal events make recent video frames significantly different from older frames.This assumption motivates learning temporal regularities from normal sequences.
- Methodology: The architecture combines a spatial feature extractor with a temporal encoder-decoder that learns temporal patterns in input frame volumes.Both components are trained end to end on video volumes containing only normal scenes.
- Pipeline: The approach is organized into three main stages.The supplied methodology passage identifies the staged design without specifying all stage operations.
3.1 Preprocessing
Preprocessing converts raw videos into normalized frame inputs and constructs training volumes from consecutive frames sampled with multiple temporal strides.
- Preprocessing: Each extracted frame is resized to 227 × 227, scaled between 0 and 1, and mean-normalized using the training dataset’s global mean image.This produces aligned inputs on a common scale.
- Temporal sampling: Each model input is a video volume of 10 consecutive frames, with temporal augmentation using stride-1, stride-2, and stride-3 sequences.The augmentation increases the size of the training dataset.
3.2 Feature Learning
The proposed feature-learning architecture separates spatial representation from temporal modeling, using convolutional autoencoding and ConvLSTM layers to reconstruct video volumes.
- Architecture: The model uses a spatial autoencoder for frame-level structures and a temporal encoder-decoder for patterns in encoded spatial features.The spatial encoder and decoder each contain two convolutional and deconvolutional layers, while the temporal encoder uses three ConvLSTM layers.
- Autoencoding: An autoencoder learns features by encoding inputs and decoding them to minimize reconstruction error in an unsupervised manner.Nonlinear activation allows the autoencoder to extract features beyond common linear transformations.
- Spatiotemporal processing: The spatial encoder processes frames individually, concatenates the encoded features after T = 10 frames, and sends them to the temporal encoder for motion encoding.The decoders mirror the encoders to reconstruct the video volume.
- Spatial convolution: Convolution preserves spatial relationships by applying learned filters to local input regions, with an m × m filter producing an (n−m+1)×(n−m+1) output.Increasing the number of filters can extract more image features but also increases computation time and memory use.
- Long Short Term Memory: LSTM uses recurrent gates and cell states to model sequence dependencies, prevent vanishing or exploding backpropagated errors, and capture higher-level information when stacked.The equations describe forgetting, adding new information, combining old and new information, and producing outputs for the next timestep.
- Convolutional LSTM: ConvLSTM replaces fully connected matrix operations with convolutions, requiring fewer weights while preserving spatial feature maps through time.Its convolutional input-to-hidden and hidden-to-hidden connections allow spatial characteristics to propagate temporally through each state.
3.3 Regularity Score
The regularity-score pipeline converts reconstruction error into a normalized abnormality score, then derives regularity as its complement.
- Reconstruction error: The frame reconstruction error is the Euclidean distance between the input frame and its reconstructed frame.The reconstruction uses the learned weights of the spatiotemporal model.
3.4 Anomaly Detection
Frames are classified using reconstruction-error thresholds, while temporal persistence groups nearby score minima into shared abnormal events.
- Thresholding: A frame is classified as anomalous according to its reconstruction error relative to a chosen threshold.Lower thresholds make detection more sensitive and trigger more alarms, affecting the true-positive and false-positive rates.
- Temporal grouping: Persistence1D groups local regularity-score minima within a fixed 50-frame temporal window as one abnormal event.The paper assumes this window is reasonable because meaningful abnormal events should last at least 2–3 seconds at 24–25 fps.
4 Experiments
Experiments evaluate the method on five fixed-camera benchmarks containing normal-only training footage and mixed normal/abnormal testing footage. Results cover detection accuracy, event counts, qualitative regularity scores, and runtime.
- Datasets: The evaluation uses Avenue, UCSD Ped1, UCSD Ped2, Subway Entrance, and Subway Exit, with normal-only training videos and mixed testing videos.Avenue includes camera shakes and sparse normal patterns; Subway datasets contain multiple unusual-event types.
- Quantitative analysis: The method outperforms all considered methods in frame-level EER across the five datasets.Table 1 compares area under the ROC curve (AUC) and Equal Error Rate (EER), where higher AUC and lower EER are better.
- Quantitative analysis: For Subway Entrance, the method detects the same number of anomalies as ConvAE with fewer false alarms, while Subway Exit gains more detections at a higher false-alarm rate.These comparisons are reported in the event-count analysis against method.
- Quantitative analysis: Throwing, loitering, and irregular-interaction events are well captured, while crowded scenes, distant objects, wrong-direction events, and motion resembling normal walking cause missed detections.Some Subway Entrance no-payment events are difficult to detect because their motion resembles people walking through the barrier.
- Runtime analysis: Testing takes less than a quarter second per frame on both CPU and GPU, with GPU execution recommended because it is nearly 30 times faster.The runtime comparison uses an Intel Xeon E5-2620 CPU and an NVIDIA Maxwell Titan X GPU.
- Qualitative analysis: The system detects anomalies in illustrated crowded-scene examples, and anomalies generally appear as downward spikes indicating low regularity scores.A difficult-to-detect skateboarding activity also produces a strong downward spike.
5 Conclusion
The paper formulates video anomaly detection as spatiotemporal sequence outlier detection using an end-to-end encoding-decoding model. It requires only a long fixed-view segment containing normal events, but scene complexity can increase false alarms.
- The model combines a spatial feature extractor with a ConvLSTM temporal sequencer for video anomaly detection.Convolutional feature extraction is incorporated in both spatial and temporal encoding-decoding stages.
- The end-to-end model uses convolutional feature extractors in both spatial and temporal spaces.
- Training requires only a long video segment containing normal events from a fixed view.The paper characterizes this training setup as semi-supervised.
- The model is described as robust to noise while detecting abnormal events.
- More complex activity scenes may produce more false alarms than other methods.The paper identifies active learning with human feedback as future work for improving detection and reducing false alarms.