Source-linked AI summary
Anomaly Detection in Video Sequence with Appearance-Motion Correspondence
Trong Nguyen Nguyen, Jean Meunier
TL;DR
The paper addresses costly anomaly detection in surveillance videos, where diverse abnormal events make manual checking difficult. It uses shared-encoder appearance reconstruction and motion translation, with patch-based scoring, and reports competitive performance on six benchmark datasets. The authors also identify unstable decisions and annotation ambiguity in some entrance-gate events.
Problem
Surveillance-video anomaly detection is challenging because abnormal events are diverse, rare, and costly to identify through manual checking.
Method
The model combines a convolutional autoencoder and U-Net that share an encoder to learn correspondence between regular appearances and their associated motions, using patch-based normality scoring.
Results
Experiments on 6 benchmark datasets demonstrate competitive performance compared with state-of-the-art methods.
Takeaways & Limitations
Patch-based scoring reduces the effect of noise in model outputs, while combining appearance and motion scores improves detection ability compared with individual measures.
Takeaways & Limitations
Entrance-gate decisions were unstable for movement stopping and loitering, partly because of score localization and ambiguous or inconsistent ground-truth annotations.
Abstract
from arXiv · showhide
Anomaly detection in surveillance videos is currently a challenge because of the diversity of possible events. We propose a deep convolutional neural network (CNN) that addresses this problem by learning a correspondence between common object appearances (e.g. pedestrian, background, tree, etc.) and their associated motions. Our model is designed as a combination of a reconstruction network and an image translation model that share the same encoder. The former sub-network determines the most significant structures that appear in video frames and the latter one attempts to associate motion templates to such structures. The training stage is performed using only videos of normal events and the model is then capable to estimate frame-level scores for an unknown input. The experiments on 6 benchmark datasets demonstrate the competitive performance of the proposed approach with respect to state-of-the-art methods.
1. Introduction
The paper targets costly surveillance-video anomaly detection with a CNN that associates regular appearances and motions. It combines shared-encoder reconstruction and translation streams, adds patch-based scoring, and reports competitive results on six benchmark datasets.
- Anomaly detection in surveillance videos is costly because abnormal events are rare and manual checking requires substantial resources.
- The appearance stream processes a single frame to learn regular structures, while the other stream considers temporal information by translating frames to optical flow.
- The paper argues that replacing a second U-Net with a shared-encoder convolutional autoencoder avoids redundant same-structure CNNs and improves the model design.
- The model combines a convolutional autoencoder and U-Net that share an encoder, with each stream contributing to anomalous-frame detection.The model is trained end-to-end.
- An Inception module is inserted after the input layer to reduce the effect of network depth, which requires careful hyperparameter selection.
- A patch-based frame-level normality score reduces noise in model outputs, and experiments on 6 benchmark datasets show competitive performance against state-of-the-art methods.
2. Related work
Video anomaly-detection research spans trajectory-based, sparse-coding, and CNN reconstruction or translation approaches. The proposed architecture combines appearance reconstruction with motion translation to model normal video structure and motion.
- Trajectory-based methods: Trajectory-based methods use object detection, tracking, trajectory features, and classification, but cluttered backgrounds can degrade them.They also do not cover the full spectrum of surveillance-video anomalies.
- Sparse-coding methods: Sparse-coding methods reconstruct spatiotemporal patches from normal training samples, but finding combination coefficients is computationally expensive.Variants reduce complexity through modified learning algorithms or data structures.
- CNN-based methods: CNN-based methods commonly use image reconstruction or translation and compare outputs with ground truth to obtain frame-level anomaly scores.Some approaches instead use pretrained classification networks to extract video features.
- Proposed architecture: The proposed network combines a Conv-AE and U-Net sharing an encoder, with appearance reconstruction and optical-flow prediction as separate streams.The Conv-AE learns common appearance structures, while the U-Net associates input patterns with motion.
- Inception module: An Inception module uses parallel 1 × 1, 3 × 3, 5 × 5, and 7 × 7 convolutions to select suitable receptive fields near the input.The module is placed after the input to address differing object scales in fixed-position surveillance videos.
- Scoring and adversarial objective: The model uses feature-level adversarial classification and patch-based anomaly scoring to retain localized evidence rather than relying only on whole-frame differences.Each activated discriminator feature map contributes a classification result, while patch scores combine appearance and motion errors.
4. Experiments
Experiments across six benchmark datasets evaluate frame- and event-level anomaly detection using normal-event training data. The model generally performs competitively, while false alarms, annotation ambiguity, and optical-flow errors expose important evaluation and deployment boundaries.
- Datasets: Training uses only normal events across CUHK Avenue, UCSD Ped2, Subway Entrance and Exit, Traffic-Belleview, and Traffic-Train.Avenue and Ped2 provide frame-level ground truth; Subway evaluation uses event-level assessment because frame-level labels are unavailable.
- CUHK Avenue and UCSD Ped2: Ped2 is selected because FlowNet2 performs poorly on very small, distant pedestrians and because some training events have inconsistent normality labels.Ped2 contains 16 training and 12 testing clips; Avenue provides examples of people moving toward and away from the camera.
- CUHK Avenue and UCSD Ped2: The model outperforms recent methods on frame-level anomaly detection for CUHK Avenue and UCSD Ped2.Table 1 reports frame-level AUC comparisons, while Figure 4 visualizes reconstructed frames, predicted optical flow, and motion-error maps.
- Subway Entrance and Exit gates: On Subway datasets, the model detects most anomalous events but produces more false alarms than other recent studies.The evaluation treats local maxima in score sequences as anomalous events and merges nearby maxima; Table 2 reports true positives and false alarms.
- Subway Entrance and Exit gates: Movement stopping and loitering yield unstable decisions, with both missed detections and false alarms linked to score localization and ambiguous annotations.The training set contains loitering labeled as normal, while another loitering case is not labeled anomalous in testing.
- Traffic-Belleview and Traffic-Train: Traffic datasets expose sensitivity to lighting variation, camera jitter, and optical-flow estimation errors; SSIM-based appearance scoring improves results, especially on Traffic-Train.The authors suggest another optical-flow estimator or tuning FlowNet2 with a more appropriate dataset to address these effects.
5. Conclusion
The paper combines appearance reconstruction and motion prediction through shared encoding, and adds patch-based anomaly scoring to reduce output noise. Experiments covered six benchmark datasets, with further discussion aimed at guiding future improvements.
- The model combines an autoencoder-based appearance reconstruction stream with a U-Net motion-prediction stream that shares the same encoder.The shared encoder is intended to make the model learn correspondence between pattern appearances and their associated motions.
- A patch-based anomaly-score estimation scheme reduces the effect of noise in model outputs.
- Experiments on 6 benchmark datasets demonstrated the potential of the proposed method.
- The paper presents detailed discussions that provide improvement suggestions for further works.
Anomaly Detection in Video Sequence with Appearance-Motion Correspondence —– Supplementary Material —–
The supplementary material identifies the paper and its authors and affiliation, and records the arXiv version and date.
- The paper is authored by Trong-Nguyen Nguyen and Jean Meunier.
- The authors are affiliated with DIRO, University of Montreal.
- The listed contact addresses are {nguyetn, meunier}@iro.umontreal.ca.
- The document is identified as arXiv:1908.06351v1, dated 17 Aug 2019.
Abstract
The supplementary material organizes additional evaluations and visualizations of the proposed anomaly-detection streams and scoring scheme. It includes curves, stream comparisons, integration analyses, feature maps, and training visualizations.
- The supplement provides ROC curves for CUHK Avenue and UCSD Ped2 and Precision-Recall curves for the traffic datasets.
- It reports experimental results using either the appearance reconstruction stream or the motion prediction stream for score estimation.
- It examines the impact of integrating the motion stream and patch-based score estimation.
- It visualizes feature maps from different blocks obtained in the experiments.
- It shows reconstructed frames and predicted motions after some training epochs.
1. Flow field color coding
The paper uses a color coding to visualize optical flow, where color represents motion direction and saturation represents pixel displacement. The supplementary figures also present ROC, PR, and AUC/AP visualizations across benchmark datasets.
- Flow field color coding: Optical-flow visualizations use color coding in which color indicates motion direction and saturation corresponds to pixel displacement.
- Flow field color coding: Figure 2 places ROC curves for Ped2 and Avenue on top and PR curves for Belleview and Train below.
- Flow field color coding: Figure 2 also provides the corresponding Area Under Curve (AUC) and Average Precision (AP).
2. Evaluation curves on 4 datasets
Figure 2 compares the proposed frame-level scores with several state-of-the-art methods using ROC and PR curves.
- ROC and PR curves visualize the proposed frame-level scores alongside eight state-of-the-art methods.
3. Experimental results on single streams
The frame-level score combines appearance and motion partial scores, whose individual and combined evaluation is compared using AUC and AP. Combining them improved detection metrics over either individual score.
- The frame-level score is a weighted combination of partial scores from the motion and appearance streams.The motion and appearance weights come from training data, while λS controls their relative contribution.
- The evaluation compares using each partial score alone as the frame-level indicator.
- AUC and AP are reported for comparison, but their values are not directly comparable and depend on class balance.
- Combining the two partial scores increased AUC and AP relative to individual measures.On the Subway datasets, combination reduced false detections but slightly decreased detected anomalous events at Subway Entrance.
4. Impact of motion stream and patch-based score estimation for anomaly detection
Experiments assess patch-based normality scores with and without motion, using SSIM on the appearance stream and dataset-specific evaluation measures. Motion improved the first four datasets, while appearance-only performance remained effective and SSIM slightly declined on two datasets.
- Table 1 evaluates six datasets using patch-based normality assessment and SSIM on the appearance stream.
- The motion stream and motion-oriented discriminator are removed to assess motion impact.Without motion, the model becomes a single-frame reconstruction autoencoder.
- The reported measures include True Positive / False Alarm for Entrance and Exit, AUROC, and AP.
- Without motion, results on the first five datasets still demonstrated the efficiency of the patch-based normality score.
- Using motion significantly improved results on the first four datasets.
- 0.830 vs. 0.832 for Belleview and 0.798 vs. 0.808 for Traffic-Train were the reported SSIM-on-appearance values with and without motion.
5. Feature maps
Feature-map visualizations show different representations in the shared encoder and the two decoders, while training outputs reveal appearance–motion correspondence and optimization difficulty under challenging conditions.
- Feature maps: Figure 4 uses four image rows to show input and shared-encoder maps, followed by motion- and appearance-decoder maps.Two feature maps are shown for each layer block, except the Inception module, which shows four filter-scale maps.
- Feature maps: The motion stream emphasizes image edges for smooth optical flow, whereas the appearance stream reconstructs textures.
- Feature maps: The Inception module’s 7 × 7 filters extracted informative details only on CUHK Avenue, Subway Entrance, and Traffic-Belleview.
- Training outputs: Training outputs show that sharper reconstructed frames correspond to motion with more details as epochs increase.The UCSD Ped2 examples compare epochs 2 and 4.
- Training outputs: Traffic-Train was difficult to optimize because of sudden lighting changes and camera jitter.The overall scene structure remained preserved, and SSIM improved anomaly-detection results.