Source-linked AI summary
Unsupervised Video Anomaly Detection via Normalizing Flows with Implicit Latent Features
MyeongAh Cho, Taeoh Kim, Woo Jin Kim, Suhwan Cho, Sangyoun Lee
TL;DR
Rare and varied anomalies often leave surveillance training sets with only normal videos, while anomalies can involve appearance, motion, or both. The paper combines an implicit two-path autoencoder with normalizing-flow density estimation of learned features. It reports competitive performance on six benchmarks without external datasets or models.
Problem
Rare and unbounded anomalies often occur only at test time, leaving surveillance training sets with normal videos; anomalies may differ in appearance, motion, or both.
Source: §1 · p. 2Method
ITAE combines static- and dynamic-focused encoders with a shared decoder, while normalizing flows estimate normality from the learned features.
Source: §1 · p. 2Results
Competitive performance is reported on six surveillance anomaly detection benchmarks using unsupervised learning without external datasets or models.
Source: §1 · p. 2Takeaways & Limitations
The approach demonstrates high effectiveness on scenes with abnormal motion and models normal appearance and motion patterns using ITAE features.
Source: §6 · p. 12The method can miss loitering when a pedestrian's appearance and walking motion both resemble normal patterns.
Source: §5.5 · p. 12
Abstract
from arXiv · showhide
In contemporary society, surveillance anomaly detection, i.e., spotting anomalous events such as crimes or accidents in surveillance videos, is a critical task. As anomalies occur rarely, most training data consists of unlabeled videos without anomalous events, which makes the task challenging. Most existing methods use an autoencoder (AE) to learn to reconstruct normal videos; they then detect anomalies based on their failure to reconstruct the appearance of abnormal scenes. However, because anomalies are distinguished by appearance as well as motion, many previous approaches have explicitly separated appearance and motion information-for example, using a pre-trained optical flow model. This explicit separation restricts reciprocal representation capabilities between two types of information. In contrast, we propose an implicit two-path AE (ITAE), a structure in which two encoders implicitly model appearance and motion features, along with a single decoder that combines them to learn normal video patterns. For the complex distribution of normal scenes, we suggest normal density estimation of ITAE features through normalizing flow (NF)-based generative models to learn the tractable likelihoods and identify anomalies using out of distribution detection. NF models intensify ITAE performance by learning normality through implicitly learned features. Finally, we demonstrate the effectiveness of ITAE and its feature distribution modeling on six benchmarks, including databases that contain various anomalies in real-world scenarios.
1. Introduction
Surveillance anomaly detection is difficult because abnormal events are rare, varied, and often absent from training data, while anomalies can differ in appearance, motion, or both. The paper proposes ITAE to model both kinds of features implicitly and uses normalizing flows to estimate their normality, reporting competitive performance on six benchmarks.
1. Introduction
- Because anomalies are rare and unbounded, most surveillance training sets contain only normal videos, making general supervised training and exhaustive abnormal-event labeling difficult.Source: §1 · p. 2
- Anomalies may differ in appearance, motion, or both, while explicitly separating those cues with pre-trained extractors can constrain end-to-end spatio-temporal representations.Source: §1 · p. 2
- ITAE uses two encoders for relatively static and dynamic features and one decoder that combines them to reconstruct the input frames.The paths use different temporal and channel sizes to focus on appearance or motion.Source: §1 · p. 2
- Normalizing-flow models estimate normality from ITAE latent features using tractable likelihoods, enabling out-of-distribution detection in complex scenes.The paper reports especially satisfactory results on the diverse ST database.Source: §1 · p. 2
- Competitive performance is reported on six surveillance anomaly detection benchmarks using unsupervised learning without external datasets or models.Source: §1 · p. 2
2. Related Works
Prior video methods include convolutional, recurrent, and prediction-based anomaly detectors, as well as models that estimate normal-feature density. This paper places ITAE among shallow two-path reconstruction models and uses normalizing flows to estimate likelihoods from its learned static and dynamic features.
2.1. Convolutional Networks for Video.
- Video representation work includes 3D convolutional and transformer models, explicit two-stream motion methods, and slow/fast pathways; ITAE instead uses two shallow encoders and one decoder to reconstruct frames.Source: §2.1 · p. 3
2.2. Video Anomaly Detection.
- Anomaly detectors use convolutional AEs, recurrent or 3D-convolutional models, memory, sparse coding, and frame prediction; other approaches model compact normal clusters.Prediction methods generally require heavier structures or optical flow.Source: §2.2 · p. 3
2.3. Generative Models.
- Generative approaches include implicit models without defined likelihoods, approximate-likelihood VAEs, EM, and autoregressive estimators; the latter can be inefficient for high-dimensional data.Source: §2.3 · p. 3
- Normalizing flows provide tractable density estimation, and this paper applies them to ITAE's learned static and dynamic features, including on the complex ShanghaiTech dataset without external pre-trained models.Source: §2.3 · pp. 3–4, Figure 2 · p. 4
3. Proposed Method
The method combines implicit static and dynamic feature learning with normalizing-flow density estimation to detect anomalous surveillance video patterns. Its anomaly score uses both reconstruction error and feature likelihood.
3.1. Overview
- The framework trains ITAE to reconstruct normal frames, then freezes it while separate normalizing flows model its static and dynamic embeddings.At test time, reconstruction error and the estimated likelihood contribute to the abnormality score.Source: §3.1 · p. 4
3.2. Implicit two-path AE (ITAE)
- ITAE uses two encoders to capture static and dynamic information implicitly, transfers features between paths, and fuses them through one decoder.The paths use different sampling rates and temporal and channel sizes; the decoder omits encoder-decoder skip connections.Source: §3.2 · pp. 4–5
3.3. Learning Normality Distribution
- Normalizing flows estimate the density of ITAE’s static and dynamic latent features, using tractable likelihoods to identify features unlike those of normal scenes.Pooled path features feed separate flow models, and resized input intensity is also concatenated to the static-flow input.Source: §3.3 · pp. 5–6
3.4. Training and Testing
- Reconstruction training sums L2, multi-scale SSIM, and gradient losses, while flow training minimizes the negative log-likelihoods of static and dynamic embeddings.At testing, the frame score combines patch-based reconstruction error with normalized static and dynamic negative log-likelihoods.Source: §3.4 · p. 6
4. Experiments
The experiments evaluate anomaly detection using frame-level annotations and ROC-based metrics, including a micro-AUC for fair comparison.
4. Experiments
- AUC and Equal Error Rate are computed from ROC curves by varying the anomaly-score threshold on frame-level annotated data.The ROC curves are reported in supplementary material.Source: §4 · p. 6
- Micro-AUC is reported for fair comparison, including with [14] [22] [13].Source: §4 · p. 6
- The evaluation uses anomaly scores and frame-level annotations to assess detection across thresholds.Source: §4 · p. 6
4.1. Databases
The experiments span walkway, campus, and real-world surveillance settings, with datasets varying in scene diversity, video scale, and anomaly types.
4.1. Databases
- UCSD experiments use Ped2 because Ped1 has low resolution and inconsistent testing protocols.Ped2 includes walkway anomalies such as cars and skaters, as well as varying pedestrian density.Source: §4.1 · p. 6
- CUHK Avenue contains normal training videos with some outliers, while its test set includes unusual directions, actions, motion, and foreground scale variation.Shanghai Tech spans 13 scenes and diverse events, with complex camera angles and lighting.Source: §4.1 · p. 7
- UCF-Crimes, Live-Videos, and UBI-Fights extend evaluation to real-world events and varied recording conditions.UCF-Crimes has 1,900 videos totaling 128 hours; Live-Videos covers indoor and outdoor settings, and UBI-Fights includes 216 fight videos and 784 normal videos.Source: §4.1 · p. 7
4.2. Implementations
Implementation details specify training configurations for the three principal datasets and apply the ShanghaiTech settings to the real-world datasets.
4.2. Implementations
- Training configurations differ across UCSD, CUHK, and ShanghaiTech in input size, batch size, learning rate, and reconstruction-to-likelihood scaling.The input sequence length is T = 16; UCSD uses its original 240×360 frame size, while the other listed inputs are resized to 256×256.Source: §4.2 · p. 7
- Glow models use K = 32 and L = 3, except UCSD, which uses L = 1.Source: §4.2 · p. 7
- UCF-Crimes, Live-Videos, and UBI-Fights use the same settings as ShanghaiTech; detailed experimental information is relegated to supplementary material.Source: §4.2 · p. 7
4.3. Ablation Studies
The ablations favor ITAE’s shallow static-and-dynamic encoder design, show that normalizing flows model learned ITAE features effectively, and find benefits from dynamic features and NF modeling across datasets. The best scaling factor varies by dataset, with λL = 0.7 associated with the highest improvement on the diverse ST dataset.
4.3.1 Structure of ITAE
- ITAE’s shallow, non-residual network performs better than the deeper SlowFast AE variants, and combining static and dynamic paths improves performance.The comparison indicates that greater model complexity alone does not yield anomaly-detection gains.Source: §4.3.1 · p. 7
4.3.2 Distribution modeling with ITAE features
- NF models using learned ITAE features achieve superior or comparable results to raw-frame NF models, and perform well on diverse ST scenes even without an ITAE reconstruction score.The ST dataset has more scenes than the other two benchmarks.Source: §4.3.2 · pp. 7–8
4.3.4 Ablation studies of λL
- For UCSD, CUHK, and ST, the selected λL values are 0.3, 0.1, and 0.7, respectively; ST has its highest performance improvement at λL = 0.7.Figure 3 reports AUC scores for three datasets across the tested λL values.Source: Figure 3 · p. 8, §4.3.4 · pp. 8–9
4.3.3 Ablation studies of proposed framework
- Adding the dynamic encoder improves results on all three databases; the largest reported gain is 1.7% on ST.On CUHK, dynamic feature modeling is more effective than static feature modeling for anomalies associated primarily with motion; adding NFs to ITAE-flow improves performance most on ST.Source: §4.3.3 · p. 8, Table 4 · p. 8
4.4. Comparison
The comparison reports competitive or superior results on three benchmarks, including strong UCSD Ped2 AUC scores without external models or datasets. The paper also reports gains on the complex, multi-scene ST database over memory-based approaches.
4.4. Comparison
- On UCSD Ped2, ITAE and ITAE+NFs achieve 98.7% and 99.2% AUC, respectively, without external models or datasets.The paper reports superior or competitive performance against state-of-the-art methods on three benchmarks.Source: §4.4 · p. 9
- ITAE+NFs outperforms several object-detector methods on Ped2, while the comparison notes that detector-based approaches cannot detect anomalies from object classes absent during pretraining.The passage reports Ped2 performance differences of 0.5%, 4.9%, and 1.7% relative to some methods using object detectors.Source: §4.4 · p. 9
- On ST, ITAE achieves 74.8% and ITAE+NFs 76.3%, with the latter showing the largest improvement among the three benchmarks for multi-scene anomaly detection.The passage contrasts this with memory-based approaches that perform poorly on ST’s complex, multiple scenes.Source: §4.4 · p. 9
5. Discussions
The experiments examine qualitative anomaly cues, cross-domain generalization, scene diversity, real-world benchmarks, error cases, and computational cost. They report complementary reconstruction and likelihood scores, with performance varying by dataset and setting.
5.1. Qualitative Results
- In qualitative examples, ITAE’s reconstruction errors respond to abnormal motion or appearance, while static and dynamic NF likelihoods are low in abnormal frames; combining scores works across varied pedestrian density and scale.The score combination is described as satisfactory even when pedestrian density is high or low and foreground scale is small or large.Source: §5.1 · pp. 9–10, Figure 6 · p. 10, Figure 8 · p. 11
5.2. Cross-domain Testing Results
- Cross-domain performance drops less than BG-Augnostic and Fewshot in reported comparisons; CUHK→Ped2 degradation is 4.9% for the proposed method, versus 11.7% for BG-Augnostic and 14.2% for Fewshot.ST→Ped2 degradation is 2.4% for the proposed method and 8.1% for BG-Augnostic; the passage reports the largest degradation for CUHK→Ped2, where source and target differ in color domain and other dataset characteristics.Source: Table 6 · p. 10, §5.2 · p. 10
5.3. Analysis According to the Number of Scenes
- In the 13-scene ST subset, NF models improve performance by 1.5%, while ITAE performs better with fewer scenes and has limitations as scene diversity increases.The passage reports that NF performance is higher on multiple-scene than single-scene subsets, whereas ITAE’s performance declines as scenes become more diverse.Source: §5.3 · p. 11, Table 7 · p. 11
5.5. False Positive and Negative Samples
- Figure 9’s examples include a long-stick false alarm prevented by adding the NF score and a loitering false negative that both ITAE and NFs miss.The loitering pedestrian’s appearance and walking motion both resemble normal patterns.Source: §5.5 · pp. 11–12, Figure 9 · p. 11
5.4. Experimental Result and Analysis on Real-world Scenario Benchmarks
- ITAE with NFs scores 70.9% on UCF-Crimes, 77.5% on LV, and 57.8% on UBI-Fights, with results compared against unsupervised methods.For UBI-Fights, ITAE and NFs alone score 57.7% and 53.3% AUC, respectively; the passage attributes ITAE’s stronger result to fighting’s large frame-level appearance and motion changes.Source: §5.4 · p. 11, Table 8 · p. 12
5.6. Running Time and Model Complexity
- On UCSD Ped2 at 240 × 360 resolution, the total framework takes 52 ms per frame and runs at 19 FPS on an NVIDIA GeForce RTX 3090.ITAE inference takes 9.1 ms per frame; static and dynamic NFs take 22.2 ms and 20.7 ms, respectively, while anomaly-score calculation takes 0.5 ms.Source: §5.6 · p. 12
6. Conclusion
The conclusion presents an unsupervised video anomaly detector that combines ITAE with NF-based modeling of normal feature distributions. It reports effectiveness on scenes with abnormal motion and especially on extensive, diverse databases.
6. Conclusion
- The proposed unsupervised method combines ITAE with NF-based distribution modeling of normal features, using static and dynamic information without a pre-trained network.The NF models estimate appearance and motion normality from ITAE latent features through tractable likelihoods.Source: §6 · p. 12
- ITAE is effective in scenes with abnormal motion, and modeling ITAE feature normality achieves superior results especially on extensive databases composed of diverse scenes.The conclusion describes these findings as results from experiments on standard benchmarks.Source: §6 · p. 12
- The authors expect unsupervised learning from varied real-world surveillance scenarios to support general normal-distribution modeling and practical anomaly-detection applications.The conclusion also notes that surveillance anomaly detection may help quickly detect accidents and crimes, while acknowledging privacy concerns associated with CCTV use.Source: §6 · p. 12