Source-linked AI summary
Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events
Guang Yu, Siqi Wang, Zhiping Cai, En Zhu, Chuanfu Xu, Jianping Yin, Marius Kloft
TL;DR
Video anomaly detection must model rare, ambiguous deviations from normal activity while existing DNN methods struggle with precise localization and richer semantics and temporal context. VEC extracts normalized spatio-temporal video events using appearance and motion cues, completes erased appearance and optical-flow patches through visual cloze tests, and ensembles the resulting signals. It consistently outperforms existing state-of-the-art DNN-based VAD methods on three benchmarks, with at least 3% AUROC gains on Avenue and at least 1% on ShanghaiTech.
Problem
VAD is challenging because anomalies are scarce and ambiguous, while existing DNN methods lack precise, comprehensive localization and sufficient high-level semantic and temporal modeling.
Method
VEC extracts normalized spatio-temporal video events using appearance and motion cues, then trains DNNs to complete erased appearance and optical-flow patches and ensembles event types and modalities.
Results
VEC consistently outperforms existing state-of-the-art DNN-based VAD methods on three benchmarks, with at least 3% AUROC gains on Avenue and at least 1% on ShanghaiTech.
Takeaways & Limitations
VEC provides a promising alternative to reconstruction- and frame-prediction-based DNN approaches for video anomaly detection.
Takeaways & Limitations
The experiments found p = 2 works well for the patch-generation loss, while other p values or adversarial training remain possible alternatives.
Abstract
from arXiv · showhide
As a vital topic in media content interpretation, video anomaly detection (VAD) has made fruitful progress via deep neural network (DNN). However, existing methods usually follow a reconstruction or frame prediction routine. They suffer from two gaps: (1) They cannot localize video activities in a both precise and comprehensive manner. (2) They lack sufficient abilities to utilize high-level semantics and temporal context information. Inspired by frequently-used cloze test in language study, we propose a brand-new VAD solution named Video Event Completion (VEC) to bridge gaps above: First, we propose a novel pipeline to achieve both precise and comprehensive enclosure of video activities. Appearance and motion are exploited as mutually complimentary cues to localize regions of interest (RoIs). A normalized spatio-temporal cube (STC) is built from each RoI as a video event, which lays the foundation of VEC and serves as a basic processing unit. Second, we encourage DNN to capture high-level semantics by solving a visual cloze test. To build such a visual cloze test, a certain patch of STC is erased to yield an incomplete event (IE). The DNN learns to restore the original video event from the IE by inferring the missing patch. Third, to incorporate richer motion dynamics, another DNN is trained to infer erased patches' optical flow. Finally, two ensemble strategies using different types of IE and modalities are proposed to boost VAD performance, so as to fully exploit the temporal context and modality information for VAD. VEC can consistently outperform state-of-the-art methods by a notable margin (typically 1.5%-5% AUROC) on commonly-used VAD benchmarks. Our codes and results can be verified at github.com/yuguangnudt/VEC_VAD.
1 INTRODUCTION
DNN-based VAD has advanced, but existing approaches struggle with precise, comprehensive activity localization and richer semantic and temporal modeling. VEC addresses these gaps by extracting video events with appearance and motion cues, then completing erased patches through visual cloze tests and complementary ensembles.
- Motivation: VAD commonly uses normal-only training because anomalies are scarce and semantically ambiguous.Normal activities are more accessible for training, while deviations from the learned normality model are treated as anomalies.
- Research gaps: Existing DNN-based VAD methods cannot localize video activities both precisely and comprehensively.Sliding windows are imprecise, whole-frame learning suffers from scale variation and foreground-background imbalance, and object detectors introduce a closed-world problem.
- VEC: VEC combines appearance and motion cues to localize activities and extract video events.The method uses normalized spatio-temporal cubes as the basis for subsequent event modeling.
- VEC: VEC trains DNNs to complete erased patches of incomplete video events as visual cloze tests.Different erased positions create different incomplete-event types, replacing conventional reconstruction or frame-prediction routines.
- VEC: VEC completes erased optical-flow patches and ensembles incomplete-event types and modalities to exploit motion dynamics and temporal context.These strategies are designed to further boost VAD performance.
2 RELATED WORK
Classic VAD relies on hand-crafted features and traditional machine-learning detectors, whereas DNN-based VAD learns features automatically from raw inputs. Existing DNN methods mainly reconstruct normal inputs or predict frames.
- Classic VAD: Classic VAD extracts hand-crafted descriptors before applying traditional machine-learning methods for anomaly detection.Descriptors include motion trajectories, dynamic textures, optical-flow histograms, spatio-temporal gradients, and 3D SIFT.
- Classic VAD: Classic VAD feature engineering is labor-intensive, often weakly discriminative, and difficult to transfer across scenes.The passage identifies feature extraction as the major bottleneck of classic VAD.
- DNN-Based VAD: DNN-based VAD learns features automatically from raw inputs and feeds them into classic models or end-to-end DNN systems.This distinguishes DNN-based methods from classic pipelines centered on manually designed descriptors.
- DNN-Based VAD: With only normal videos for training, existing DNN-based VAD methods mainly follow reconstruction or frame-prediction routines.Reconstruction methods interpret large reconstruction errors as anomalies, often using autoencoder variants.
3 VIDEO EVENT COMPLETION (VEC)
VEC extracts normalized spatio-temporal cubes as video events using complementary appearance and motion cues, then trains visual cloze tests to complete missing appearance and motion patches.
- Video Event Extraction: VEC combines appearance and motion cues to localize video activities precisely and comprehensively, addressing complementary weaknesses of existing strategies.Appearance cues target detected subjects, while motion cues cover activities outside the detector’s closed world.
- Video Event Extraction: Appearance-based RoIs come from a pre-trained object detector and heuristic filtering based on confidence, area, and overlap thresholds.The filtering removes overly small, nested, or significantly overlapping regions.
- Video Event Extraction: Motion-based RoIs are obtained from thresholded temporal gradients after subtracting appearance RoIs, followed by contour detection and heuristic filtering.Subtracting appearance regions helps reveal objects missed by appearance cues and reduces computation.
- Video Event Extraction: Each RoI is converted into an STC by extracting corresponding patches from the current and previous D −1 frames, resizing them, and stacking them.The resulting STC represents a video event and incorporates temporal context as the basic processing unit.
- Visual Cloze Tests: VEC creates a type-i incomplete event by erasing one STC patch and trains a DNN to reconstruct the missing appearance patch from the remaining context.Erasing different temporal positions creates multiple cloze tests that expose different temporal-context combinations.
- Visual Cloze Tests: A separate motion-completion DNN infers the optical-flow patch corresponding to the erased STC patch.Optical flow supplies a complementary motion representation for completion.
- Visual Cloze Tests: VEC ensembles scores across incomplete-event types to exploit temporal context during inference.Different erased patches produce different incomplete events and temporal-context combinations.
4 EVALUATION
VEC is evaluated on three standard VAD benchmarks against state-of-the-art methods, with experiments covering architecture, localization, ensemble strategies, and completion behavior. Results show consistent gains from VEC, including strong benchmark performance and benefits from comprehensive localization, temporal-context ensemble, and modality ensemble.
- Experimental setup: Evaluation uses UCSDped2, Avenue, and ShanghaiTech, with Cascade R-CNN for event extraction and U-Net architectures for generative DNNs.The experiments also specify dataset-specific thresholds, cube dimensions H = W = 32 and D = 5, and training settings.
- Comparison with state-of-the-art methods: VEC-A and VEC-AM consistently outperform existing state-of-the-art DNN-based VAD methods on all three benchmarks.VEC achieves at least 3% AUROC improvement on Avenue and at least 1% on ShanghaiTech against all state-of-the-art methods.
- Comparison with state-of-the-art methods: VEC-A exceeds 90% frame-level AUROC on Avenue, reported as the best performance achieved on that dataset to the authors’ knowledge.The comparison is based on frame-level AUROC and excludes a method using a different evaluation metric.
- Ablation studies: Appearance-and-motion localization improves AUROC over appearance-only localization by 1.8%, 2.5%, and 1.2% on UCSDped2, Avenue, and ShanghaiTech, respectively.Imprecise sliding-window localization performs worse than no localization on UCSDped2 and Avenue.
- Ablation studies: IE-type ensemble contributes 1.3%, 2.1%, and 0.4% AUROC on UCSDped2, Avenue, and ShanghaiTech, while modality ensemble yields an 8% gain on UCSDped2.Modality ensemble also improves Avenue and ShanghaiTech by over 1%; the paper attributes the larger UCSDped2 gain to the importance of motion clues in low-resolution grayscale frames.
- Detailed analysis: Replacing visual cloze tests with plain reconstruction causes a 3% to 7% AUROC loss, while visualizations show larger completion errors in semantically meaningful abnormal regions.Normal events are completed with minor errors, whereas abnormal events produce prominent errors in both raw pixels and optical flow.
5 CONCLUSION
VEC extracts precise, comprehensive video events and applies visual cloze completion to appearance and optical-flow patches, with ensemble strategies exploiting temporal context and motion dynamics.
- VEC extracts normalized spatio-temporal cubes using appearance and motion cues, enabling precise and comprehensive video event extraction.Each STC serves as the basic processing unit for VEC.
- VEC trains DNNs to infer deliberately erased patches from incomplete video events, using visual cloze tests instead of reconstruction or frame prediction.
- Figure 7 visualizes erased patches and optical flow targets, VEC outputs, and completion errors, with brighter colors indicating larger errors.
- A separate DNN infers erased patches’ optical flow to incorporate motion modality and learn richer motion dynamics.
- Two ensemble strategies combine different incomplete-event types and modalities to exploit temporal context and motion dynamics for enhanced VAD performance.