Source-linked AI summary
Anomaly Detection in Video Using Predictive Convolutional Long Short-Term Memory Networks
Jefferson Ryan Medel, Andreas Savakis
TL;DR
The paper tackles anomaly detection in long videos when anomalous events are vaguely defined and supervision is limited. It develops composite Conv-LSTM generative models for reconstruction and future prediction, then uses regularity scores from reconstruction errors to identify anomalies. The models achieve competitive results across anomaly-detection datasets and effectively model and predict video sequences.
Problem
Detecting rare video anomalies is difficult because anomalous events are vaguely defined and training data contains little or no anomalous activity.
Method
Composite Conv-LSTM encoder-decoder networks reconstruct input sequences, predict near-term future frames, and use reconstruction and prediction errors to compute regularity scores.
Results
The model performs competitively with state-of-the-art anomaly-detection methods across multiple datasets and accurately models familiar movements without adapting to unusual ones.
Takeaways & Limitations
Conv-LSTM networks can support anomaly recognition when training data is loosely supervised to contain mostly normal events.
Takeaways & Limitations
Performance is constrained when anomalous events occur in the background or when training data poorly represents normal activity and ground-truth labels omit events.
Abstract
from arXiv · showhide
Automating the detection of anomalous events within long video sequences is challenging due to the ambiguity of how such events are defined. We approach the problem by learning generative models that can identify anomalies in videos using limited supervision. We propose end-to-end trainable composite Convolutional Long Short-Term Memory (Conv-LSTM) networks that are able to predict the evolution of a video sequence from a small number of input frames. Regularity scores are derived from the reconstruction errors of a set of predictions with abnormal video sequences yielding lower regularity scores as they diverge further from the actual sequence over time. The models utilize a composite structure and examine the effects of conditioning in learning more meaningful representations. The best model is chosen based on the reconstruction and prediction accuracy. The Conv-LSTM models are evaluated both qualitatively and quantitatively, demonstrating competitive results on anomaly detection datasets. Conv-LSTM units are shown to be an effective tool for modeling and predicting video sequences.
1. Introduction
The paper addresses the difficulty of detecting vaguely defined, rare anomalous events in long videos with limited supervision. It proposes composite Conv-LSTM generative models that reconstruct inputs, predict near-term futures, and support regularity-based anomaly detection.
- Video anomalies are unusual events whose rarity makes manual detection in long sequences labor-intensive.
- Unlike clearly defined actions, anomalies may span a wide range of activities and are therefore vaguely defined.
- The proposed composite Conv-LSTM model encodes, reconstructs, and predicts video sequences while learning spatiotemporal representations.It includes both unconditioned and conditioned variants, with the conditioned model using the previous output to condition the next timestep.
- The model detects anomalous segments by applying a regularity evaluation algorithm to reconstruction and prediction outputs.
- The approach is evaluated on UCSD Pedestrian 1, UCSD Pedestrian 2, Subway, and Avenue datasets.
2. Related Works
Prior anomaly-detection methods use limited supervision, reconstruction errors, sparse coding, handcrafted features, and CNNs, while Conv-LSTM architectures retain spatial information across time. Composite encoder-decoder models combine reconstruction and future prediction to learn more meaningful video representations.
- Anomaly detection is often formulated as semi-supervised or unsupervised classification because anomalous labels are uncommon and training data contains few anomalies.
- Sparse-coding methods detect anomalies through reconstruction errors, but dictionaries may contain unused or noisy elements.
- Handcrafted-feature methods estimate video normality from dynamic textures, spatial maps, or low-level observation likelihoods, but may not adapt to unexpected features.
- CNN-based temporal regularity modeling can detect anomalous segments, whereas Conv-LSTMs are designed to incorporate temporal features while preserving spatial characteristics.
- Convolutional LSTM Units: Conv-LSTM units replace fully connected LSTM weights with convolutional filters, allowing spatial characteristics to propagate temporally through hidden states.
- Future Video Prediction: Composite encoder-decoder models jointly reconstruct current or past frames and predict future frames, encouraging encodings that retain more meaningful information.Reconstruction can memorize inputs, while future prediction may emphasize recent frames and lose older information.
- Future Video Prediction: Prior Conv-LSTM encoder-decoder work retained spatiotemporal information and outperformed LSTM models on future prediction for the synthetic Bouncing MNIST dataset.
3. Anomaly Detection with Conv-LSTMs
The proposed composite Conv-LSTM Encoder-Decoder reconstructs input sequences, predicts their near-term future, and derives anomaly signals from reconstruction errors. Its conditioned future decoder and regularity scoring help distinguish abnormal segments from normal video.
- Composite Conv-LSTM Encoder-Decoder: Non-overlapping frame patches reduce detail but constrain the model to learn more significant video characteristics and meaningful encodings.The encoder uses the final timestep outputs of each Conv-LSTM layer as the encoding.
- Composite Conv-LSTM Encoder-Decoder: The end-to-end architecture uses an encoder and two decoders to reconstruct the past input sequence and predict future frames.The encoder processes chronological stacks of non-overlapping frame patches, while decoder outputs are combined through 1x1 convolution for reconstruction.
- Composite Conv-LSTM Encoder-Decoder: The future decoder can be conditioned on the previous predicted frame, whereas the past decoder remains unconditioned because the past has one possible outcome.Conditioning feeds each timestep’s summed output into the first layer of the subsequent timestep and may limit future variation.
- Anomaly Evaluation Algorithm: Reconstruction error is measured with mean squared error over output and target pixel values across frames.The MSE uses output pixel value θ̂, target value θ, total pixels per frame p, and number of frames n.
- Anomaly Evaluation Algorithm: Regularity scores normalize reconstruction errors within each video: normal sequences score higher, while abnormal sequences score lower.Distinct local minima identify likely anomalous frames, and local maxima help shorten proposed anomalous regions and reduce false positives.
4. Results
The proposed Conv-LSTM models reconstruct past frames and predict future frames, using reconstruction and prediction accuracy to select models and regularity-based errors for anomaly detection across multiple video datasets.
- The experiments used UCSD Pedestrian 1, UCSD Pedestrian 2, Avenue, and Subway datasets, analyzing visual reconstructions, future predictions, average loss, and detection rates.
- Parameter Selection: Longer-horizon predictions became progressively worse, while ten input frames reduced MSE per frame but required 1.5 times longer training.
- Parameter Selection: The composite models achieved lower MSE per frame than the baseline, with the unconditioned model performing slightly better.
- Predicting the Future of a Video Sequence: The model reconstructed normal pedestrian motion accurately, but anomalous vehicle details deteriorated across future timesteps and increasingly resembled learned pedestrian motion.
- Anomalous Event Detection: On UCSD Pedestrian 2, the model achieved 100% recall with one false positive.
- Anomalous Event Detection: On Subway Entrance, the approach detected 62 of 66 anomalies with 93.9% recall and outperformed other methods.
- Anomalous Event Detection: Subway Exit performance depended on annotation quality because the original ground truth listed 19 anomalous events while closer inspection found 30.
- Anomalous Event Detection: The model precisely differentiated normal and anomalous activity on UCSD Pedestrian 1 but missed background jogging when smaller deviations produced less significant regularity changes.
5. Conclusions
The composite Conv-LSTM models reconstruct video, predict future frames, and identify anomalies through prediction differences and regularity scores. They perform competitively across several anomaly-detection datasets.
- The composite Conv-LSTM architecture models video sequences, reconstructs inputs, and predicts future frames.
- Comparing normal and anomalous events shows accurate modeling of familiar movements but poor adaptation to unusual movements.
- Regularity scores derived from reconstruction and prediction errors help locate anomalous events in video sequences.
- Quantitative analysis finds competitive performance with state-of-the-art anomaly-detection methods across multiple datasets.
1. Video Input Reconstruction and Future Prediction
The model reconstructs observed video accurately and predicts plausible future frames for normal activity, while anomalous movements progressively distort or disappear in its outputs.
- The visualization layout compares ground truth, input reconstruction, and future prediction across successive timesteps.
- UCSD Pedestrian: Normal pedestrian sequences are reconstructed nearly identically, while future predictions remain recognizable despite increasing blur over time.
- Subway and Avenue: Normal subway and crowd sequences are reconstructed accurately, although future predictions show slight fading or progressively greater blur.
- Anomalous sequences: Bicycle anomalies become distorted, merge with the rider, and disappear by T+4 in future predictions.
- Anomalous sequences: Unusual walking directions produce distorted reconstructions and blurred predictions, while familiar background pedestrians remain correctly modeled.
2. Anomaly Detection Evaluation
Anomaly evaluation uses regularity-score graphs and accompanying frames to connect score changes with anomalous events. The approach generally aligns predicted anomalous segments with target regions, but training-data coverage limits performance in Subway Exit.
- Dips in regularity scores correspond well with target anomalous regions across the evaluated datasets.
- Regularity-score graphs plot scores by input-sequence starting frame, with shaded regions marking anomalous ground truth and predictions.
- Accompanying video frames identify the objects or events responsible for each regularity score, with yellow boxes marking difficult-to-find objects.
- Windowing: A 100-frame anomaly window is retained despite generally longer events because the videos also contain short anomalies.