Source-linked AI summary
Future Frame Prediction for Anomaly Detection -- A New Baseline
Wen Liu, Weixin Luo, Dongze Lian, Shenghua Gao
TL;DR
Video anomaly detection lacks a reliable guarantee that reconstruction errors will be larger for abnormal events. The paper instead predicts future frames from observed video, adds optical-flow consistency to appearance constraints, and reports robustness to normal-event uncertainty and sensitivity to abnormal events.
Problem
Existing reconstruction-based anomaly detection methods cannot guarantee larger reconstruction errors for abnormal events.
Method
The method predicts a future frame from historical observations using appearance constraints and an optical-flow motion constraint, then compares prediction with ground truth.
Results
Experiments on a toy dataset and publicly available datasets validate robustness to uncertainty in normal events and sensitivity to abnormal events.
Takeaways & Limitations
Future-frame prediction provides a video anomaly detection framework in which normal events are predictable and abnormal events produce larger prediction differences.
Takeaways & Limitations
The approach assumes that normal events can be well predicted and uses prediction–ground-truth differences for anomaly scoring.
Abstract
from arXiv · showhide
Anomaly detection in videos refers to the identification of events that do not conform to expected behavior. However, almost all existing methods tackle the problem by minimizing the reconstruction errors of training data, which cannot guarantee a larger reconstruction error for an abnormal event. In this paper, we propose to tackle the anomaly detection problem within a video prediction framework. To the best of our knowledge, this is the first work that leverages the difference between a predicted future frame and its ground truth to detect an abnormal event. To predict a future frame with higher quality for normal events, other than the commonly used appearance (spatial) constraints on intensity and gradient, we also introduce a motion (temporal) constraint in video prediction by enforcing the optical flow between predicted frames and ground truth frames to be consistent, and this is the first work that introduces a temporal constraint into the video prediction task. Such spatial and motion constraints facilitate the future frame prediction for normal events, and consequently facilitate to identify those abnormal events that do not conform the expectation. Extensive experiments on both a toy dataset and some publicly available datasets validate the effectiveness of our method in terms of robustness to the uncertainty in normal events and the sensitivity to abnormal events.
1. Introduction
The paper reframes video anomaly detection as future-frame prediction because reconstruction errors do not reliably become larger for abnormal events. It introduces spatial and motion constraints to improve predictions of normal events and distinguish abnormal ones.
- Video anomaly detection is challenging because abnormal events are unbounded, making exhaustive classification impractical.
- Reconstruction-based methods cannot guarantee larger errors for abnormal events because dictionaries may be overcomplete and deep networks have high capacity.
- The prediction network combines appearance constraints on intensity and gradient with motion constraints based on optical flow, alongside adversarial training.
- The proposed framework predicts a future frame from historical observations and compares it with ground truth to identify anomalies.
- Toy and real-dataset experiments support robustness to uncertainty in normal events and report performance above existing methods.
2. Related Work
Related work includes hand-crafted-feature methods that model regular patterns and deep-learning methods that learn representations or reconstruct normal frames. Video prediction research provides a separate foundation for forecasting future frames.
- Hand-crafted approaches extract trajectory or spatial-temporal features and model regular patterns to identify isolated clusters or outliers.
- Deep-learning anomaly detection methods use learned features and auto-encoders, including 3D convolutional auto-encoders, to model regular frames.
- Video prediction research has explored ConvLSTM forecasting, adversarial multi-scale generation, and predictive neural networks for video representation.
3. Future Frame Prediction Based Anomaly Detection Method
The method detects anomalies by predicting a future frame from preceding frames and comparing it with ground truth, while enforcing appearance, motion, and adversarial constraints. It scores test frames using normalized PSNR, with lower scores indicating greater irregularity.
- Future frame prediction compares a predicted frame with its ground truth to identify events as normal or abnormal.The network predicts I_t+1 from consecutive preceding frames.
- A U-Net prediction network uses same-resolution shortcut connections between corresponding encoder and decoder layers.The architecture is intended to avoid gradient vanishing and information imbalance.
- Optical-flow consistency complements intensity and gradient constraints to preserve motion information in predicted normal events.A fixed pretrained Flownet estimates optical flow for the temporal loss.
- Adversarial training adds a discriminator-generator module, with least-square losses used while alternately fixing the other network’s weights.The discriminator classifies ground-truth frames as genuine and generated frames as fake; the generator aims for genuine classification.
- The objective combines appearance, motion, and adversarial constraints, and training uses normalized 256 × 256 frames with five-frame clips.The implementation uses t = 4, batch size 4, and Adam-based stochastic gradient descent.
- PSNR measures prediction quality, is normalized within each testing video, and supports thresholding frames by regular score S(t).Higher PSNR indicates that a frame is more likely to be normal.
4. Experiments
The experiments evaluate the proposed method on three public anomaly-detection datasets and a toy dataset. The public datasets cover varied anomaly types and settings, while the toy dataset tests robustness to uncertainty in normal events.
- Experiments use CUHK Avenue, UCSD Pedestrian, and ShanghaiTech datasets, plus a toy dataset for uncertainty in normal events.The authors state that the toy dataset tests whether normal and abnormal events remain correctly classified under normal-event uncertainty.
- CUHK Avenue contains 16 training and 21 testing videos with 47 abnormal events, including throwing objects, loitering, and running.People’s apparent size may vary with camera position and angle.
- UCSD Pedestrian 1 contains 34 training and 36 testing videos with 40 vehicle-related irregular events, while Pedestrian 2 has 16 training and 12 testing videos with 12 abnormal events.The two UCSD parts are evaluated separately.
- Table 1 reports AUC for different methods on Avenue, Ped1, Ped2, and ShanghaiTech.
- ShanghaiTech contains 330 training and 107 testing videos, 130 abnormal events, 13 scenes, and varied anomaly types.Following the cited setting, the model is trained on all scenes.
4.2. Evaluation Metric
The evaluation uses frame-level area under the ROC curve to measure anomaly-detection performance across threshold settings. Higher AUC indicates better performance.
- Frame-level AUC is computed from the ROC curve as the regular-score threshold changes.The ROC is obtained by gradually changing the threshold of regular scores.
- Higher AUC indicates better anomaly-detection performance.
4.3. Comparison with Existing Methods
The U-Net prediction network outperforms Beyond-MSE on anomaly-discrimination measures, and the full method improves the score gap across datasets and loss configurations.
- Prediction-network comparison: U-Net achieves larger score gaps and higher AUC than Beyond-MSE on Ped1 and Ped2, even without the motion constraint.These results motivate selecting U-Net as the prediction architecture; the authors state that adding motion constraint further boosts AUC.
- Loss-component comparison: Figure 5 compares loss-function variants using average normal and abnormal scores and their gap on Avenue.The gap is computed by subtracting the abnormal score from the normal score.
- Comparison with Conv-AE: Figure 6 uses Δs to compare discrimination across Ped1, Ped2, and Avenue, with larger gaps corresponding to lower false-alarm rates and higher detection rates.The comparison is between the proposed method and Conv-AE.
- Comparison with Conv-AE: The reported results show that the proposed method consistently achieves a larger normal–abnormal score gap than Conv-AE.The figure caption describes this gap as the difference between average normal-frame and abnormal-frame scores.
4.5. Impact of Constraint on Motion.
Removing the motion constraint provides a baseline for testing its contribution to prediction and anomaly detection. The constraint improves optical-flow accuracy and anomaly-detection AUC.
- Ablation design: The ablation removes the motion term from the training objective and compares the resulting baseline with the full method.This experiment evaluates the constraint's importance for both video-frame generation and anomaly detection.
- Motion prediction: Optical-flow MSE is 7.51 with motion constraint versus 8.26 without it.The constrained model's optical flow is also described as more consistent with ground truth.
- Anomaly detection: The motion-constrained model consistently achieves higher AUC than the unconstrained model on Ped1 and Ped2.The authors interpret this as evidence that explicit motion consistency is necessary for anomaly detection.
4.7. Comparison of Prediction Network and AutoEncoder Networks for Anomaly Detection
The paper compares video-prediction and auto-encoder anomaly detection, finding that prediction yields stronger separation between normal and abnormal events. Visualizations illustrate how motion and unseen events affect predictions.
- Method comparison: Conv-AE may reconstruct both normal and abnormal testing frames, weakening reconstruction-based discrimination.The paper uses this consideration to motivate comparison with a video-prediction network.
- Motion visualization: Figure 7 compares optical-flow and predicted-image visualizations with and without motion constraint on Ped1.The red boxes mark differences, and the constrained prediction is closer to ground-truth optical flow.
- Anomaly visualization: Figure 8 visualizes vehicle-intrusion and fighting anomalies, where the unseen truck and fighting motion are poorly predicted.Orange circles indicate uncertain normal events, while red circles indicate abnormal events.
- Quantitative comparison: The proposed video-prediction solution achieves higher score gaps than Conv-AE across Ped1, Ped2, and Avenue.The gap is used to measure separation between normal and abnormal scores.
4.8. Evaluation with A Toy Dataset
A toy pedestrian dataset tests whether future-frame prediction remains robust to uncertain normal motion while exposing abnormal events. The method reports strong anomaly-detection performance on these cases.
- Dataset design: The toy dataset trains on pedestrians walking and turning at crossroads, then tests vehicle intrusions and human fighting as abnormal cases.The dataset is designed to evaluate performance when abnormal events differ from the training scenarios.
- Dataset design: The training data contains 210 frames, while the testing data contains 1242 frames.
- Results: Normal motion uncertainty causes only a temporary PSNR drop before prediction improves after the pedestrian chooses a direction.By contrast, an abnormal truck intrusion produces continuously lower PSNR, and the toy-dataset AUC is 98.9%.
4.9. Running Time
The framework runs at about 25 fps, including both future-frame generation and anomaly prediction.
- 25 fps is the framework’s average running time, including video frame generation and anomaly prediction.Reported comparisons include 20 fps, 150 fps, and 0.5 fps for other methods.
5. Conclusion
The method detects anomalies through future-frame prediction: normal events are modeled in appearance and motion, while larger prediction errors indicate abnormal events. Experiments on three datasets report a large-margin improvement over existing methods.
- The method uses a future-frame prediction network because normal events are predictable while abnormal events do not conform to expectations.A U-Net serves as the basic prediction network.
- The network combines adversarial training, appearance constraints, and an optical-flow loss to make predicted normal frames realistic in appearance and motion.The temporal loss encourages predicted-frame optical flow to match ground-truth optical flow.
- Events with larger differences between predicted and ground-truth frames are classified as anomalies.
- The method outperforms existing methods by a large margin on three datasets.The conclusion presents this as evidence of the method’s effectiveness for anomaly detection.