PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models

Yang Liu, Dingkang Yang, Yan Wang, Jing Liu, Jun Liu, Azzedine Boukerche, Peng Sun, Liang Song

arXiv:2302.05087v3cs.CVcs.MM

TL;DR

The survey addresses limited systematic organization across emerging video anomaly detection pathways and the constrained performance of unsupervised methods on complex datasets. It unifies deep-learning routes into a generalized framework, assembles research resources, and compares model performance while identifying challenges and future directions.

  • Problem

    Emerging weakly-supervised anomaly detection is described as lacking systematic organization, while unsupervised methods perform much worse on complex than simple datasets.

  • Method

    The survey integrates deep-learning technical routes based on different assumptions and learning frameworks into a unified generalized video anomaly event detection framework.

  • Results

    The survey provides a unified taxonomy and quantitatively compares model strengths and weaknesses across video anomaly event detection methods.

  • Takeaways & Limitations

    The framework organizes diverse video anomaly detection methods and supports comparison of their performance, resources, and research directions.

  • Takeaways & Limitations

    Unsupervised video anomaly detection is reported to be valid only for simple scenarios, with inferior performance on complex datasets and false alarms on unseen normal events.

Abstract

from arXiv · show

Video Anomaly Detection (VAD) serves as a pivotal technology in the intelligent surveillance systems, enabling the temporal or spatial identification of anomalous events within videos. While existing reviews predominantly concentrate on conventional unsupervised methods, they often overlook the emergence of weakly-supervised and fully-unsupervised approaches. To address this gap, this survey extends the conventional scope of VAD beyond unsupervised methods, encompassing a broader spectrum termed Generalized Video Anomaly Event Detection (GVAED). By skillfully incorporating recent advancements rooted in diverse assumptions and learning frameworks, this survey introduces an intuitive taxonomy that seamlessly navigates through unsupervised, weakly-supervised, supervised and fully-unsupervised VAD methodologies, elucidating the distinctions and interconnections within these research trajectories. In addition, this survey facilitates prospective researchers by assembling a compilation of research resources, including public datasets, available codebases, programming tools, and pertinent literature. Furthermore, this survey quantitatively assesses model performance, delves into research challenges and directions, and outlines potential avenues for future exploration.

1 INTRODUCTION

Video anomaly detection has expanded beyond narrow unsupervised methods, motivating a generalized framework that organizes multiple supervision settings and learning approaches. This survey compares these methods, assembles research resources, and analyzes challenges and future directions.

  • VAD automatically analyzes video patterns and locates abnormal events, but manual inspection is laborious and difficult to scale to vast video streams.
  • UVAD learns normality from normal videos, yet diverse normal events and incomplete coverage make anomaly modeling difficult.
  • UVAD models can either falsely flag unseen normal events or reconstruct anomalous events, and experiments find them effective mainly in simple scenarios.
  • WAED uses Multiple Instance Learning to compare normal and abnormal spatial-temporal features, but its setting resembles binary classification with imbalance and multiple abnormal subcategories.
  • Fully-unsupervised VAD removes label and training-data prerequisites and is designated FVAD to distinguish it from UVAD's normal-only training assumption.
  • The survey introduces GVAED as a unified taxonomy spanning UVAD, WAED, FVAD, and SVAD, compares methods, provides public resources, and analyzes research challenges and trends.

2.1 Definition of the Anomaly

The survey distinguishes anomaly definitions by whether anomalies are deviations from normal training data or subjectively defined abnormal events. These assumptions determine labeling requirements and practical applicability.

  • UVAD: UVAD treats events absent from training data as anomalies, although this can label unfamiliar normal events as anomalous.The survey notes that collecting all possible normal events is costly and that normal and abnormal frames often share appearance and motion patterns.
  • WAED: WAED defines anomalies as events humans perceive as abnormal and trains on both normal and abnormal videos with video-level labels.Video-level labels indicate whether a video contains an abnormal event without identifying its exact temporal location.
  • Comparison: In practical settings, WAED offers more reliable supervision than UVAD because it incorporates abnormal examples and human-oriented anomaly definitions.The survey presents this as a contrast between subjective anomaly definitions and deviations from a normal-only training set.
  • WAED: WAED requires predefined abnormal events, so it can distinguish specified abnormal categories rather than every possible anomaly.This requirement reflects the difficulty of collecting anomalous events across the full range of real-world categories.

2.2 Problem Formation

The survey formulates VAD as learning normality or discriminating abnormality under different supervision regimes. UVAD separates proxy-task normality learning from downstream scoring, whereas WAED and FVAD learn anomaly scores more directly.

  • UVAD: UVAD trains on normal events, learns a normality model, and uses test-sample deviation from that model as an anomaly signal.The deviation may be measured with prediction error, feature-space distance, or related distances, then converted into a relative score.
  • UVAD: UVAD’s proxy-task performance does not directly determine anomaly-detection performance because proxy learning and downstream detection are separate.This two-stage structure is illustrated as normality learning followed by test-phase anomaly detection.
  • UVAD: UVAD methods typically normalize deviation into anomaly scores in [0, 1], but their offline formulation can produce false alarms on normal test videos.The supplied passages state that these methods assume normal training data and that test videos must include anomalous events.
  • WAED: WAED uses MIL ranking: videos become bags of instances, and the maximum score of an anomalous bag is trained to exceed that of a normal bag.WAED computes anomaly scores without an additional self-supervised proxy task, while smoothness and sparsity constraints may supplement ranking.
  • WAED: WAED performance partly depends on pretrained feature extraction, making its feature-processing and scoring pipeline computationally costly.The regression module receives extracted feature representations before assigning instance-level anomaly scores.
  • FVAD: FVAD learns from mixed normal and abnormal data without labels, either modeling normality directly or jointly learning representations and anomaly detection.Because anomalies are infrequent, direct optimization can still be dominated by normal frames and produce large errors on anomalous events.
  • FVAD: In the FVAD formulation, a backbone maps frames to features, an anomaly representation learner transforms them, and a regression layer predicts anomaly scores using focal loss.The jointly optimized parameters include the backbone, anomaly representation learner, and score-regression layer.

2.3 Benchmark Datasets

GVAED benchmarks span early single-scene datasets, weakly supervised and multimodal collections, and newer supervised synthetic benchmarks. Their evolution reflects movement toward larger, cross-scene, multimodal, and more realistic evaluation settings.

  • Dataset landscape: Table 2 compares representative GVAED datasets by dataset properties and annotation signals, distinguishing WAED and multimodal benchmarks.The dataset progression includes changes in supervision, modality, and scenario complexity.
  • Early datasets: Subway contains entrance and exit videos with ticket evasion, atypical cleaning, and wrong-direction travel, but unclear anomaly definitions limit evaluation use.The review omits quantitative comparisons because labeling is cursory and anomalous events are insufficiently defined.
  • Early datasets: UMN simulates crowd evacuation in researcher-designed scenes, while its artificial anomalies overlook real-world diversity and rarity.The dataset was later abandoned alongside Subway because of limitations including absent spatial annotation.
  • Single-scene benchmarks: UCSD Ped1 and Ped2 use simple campus-road scenes, with pedestrian walking normal and biking, skateboarding, and driving abnormal; frame-level AUC reaches 99%.The simple scenes have saturated performance and constrain further GVAED development.
  • Single-scene benchmarks: CUHK Avenue provides frame- and pixel-level annotations, but UCSD and Avenue remain single-scene datasets that do not represent cross-scene pattern shifts.ShanghaiTech introduced 13 scenes as a larger UVAD benchmark, although its FFP treated scenarios together rather than explicitly modeling cross-scene challenges.
  • Multimodal benchmarks: XD-Violence extends GVAED to audio-video violence detection, using both real surveillance and movie footage and broadening modalities beyond video.Its events include abuse, explosions, car accidents, struggles, shootings, and riots, while the dataset also motivates sound, text, and other modalities.
  • Synthetic and open-set benchmarks: UBnormal is a supervised synthetic benchmark with pixel-level labels, unseen anomalous events in the test set, and a validation set for model tuning.It targets the open-set nature of anomalies that WAED overlooks and makes fine-grained labeling possible through virtual scenes.

2.4 Performance Evaluation

GVAED evaluation combines detection accuracy with operational cost, using threshold-based confusion matrices and metrics suited to discrimination, imbalance, localization, and deployment efficiency.

  • Evaluation dimensions: GVAED evaluation considers detection accuracy and operational cost, including parameter size, inference speed, and FLOPs on the same platform.Accuracy measures anomaly discrimination, while operational cost reflects deployment potential on resource-limited devices.
  • Localization criteria: Detection criteria are organized into temporal, object, and spatial-localization levels, covering event boundaries, anomaly objects or tracks, and abnormal pixels.This taxonomy separates temporal detection from increasingly fine-grained spatial localization.
  • Thresholding and confusion matrices: Continuous anomaly scores require a threshold before comparison with binary ground-truth labels, producing TP, FN, FP, and TN confusion-matrix counts.Scores below the threshold are normal and scores at or above it are abnormal.
  • Thresholding and confusion matrices: TPR, FPR, TNR, and FNR are computed from confusion-matrix counts and used to derive AUROC and Average Precision.The supplied definitions associate these rates with correct and incorrect abnormal or normal classifications.
  • Accuracy metrics: AUROC integrates ROC curves formed from FPR and TPR across thresholds, ranges from 0 to 1, and higher values indicate better performance.The survey uses AUROC to assess generalization and help select an alarm threshold.
  • Accuracy metrics: Average Precision is suited to highly imbalanced GVAED data because the precision-recall curve is less dominated by abundant true negatives.AP has become a principal metric for multimodal models and is widely used on XD-Violence.

3 UNSUPERVISED VIDEO ANOMALY DETECTION

The survey organizes deep UVAD methods by input structure and model architecture, moving from frame-level processing toward patch- and object-level representations that better address complex scenes.

  • Taxonomy and motivation: Existing UVAD reviews commonly classify methods by deviation calculation, but this survey emphasizes a deep-learning taxonomy for evolving models.Earlier approaches relied on manually engineered features with limited representational power, whereas newer methods use end-to-end deep representations.
  • Deep model families: Deep UVAD methods use autoencoders, GANs, and vision transformers to learn task-specific spatiotemporal representations.Representative mechanisms include reconstruction, future-frame prediction, adversarial learning, memory modules, sparse coding, and predictive coding.
  • Input-structure taxonomy: The input-structure taxonomy divides UVAD into frame-level, patch-level, and object-level methods.Frame methods model whole-frame context, patch methods focus on selected regions, and object methods analyze foreground objects and their behavior.
  • Input-structure taxonomy: Patch-level methods discard redundant regions and interactions, providing advantages in computational efficiency and inference speed.They extract features from designated regions of interest rather than processing the entire frame.
  • Frame-level models: Frame-level UVAD models learn normality from RGB frames, sequences, or optical flow using single-stream or multi-stream architectures.Single-stream models commonly reconstruct inputs or predict future frames, while multi-stream models separate appearance and motion.
  • Representative methods: Residual connections improved R-STAE frame-level AUC by 3%, 2%, and 5% on CUHK Avenue, LV, and UCSD Ped2, respectively.The reported gains are attributed to the residual design addressing gradient disappearance during normality learning.
  • Object-level models: Object-level methods model specific objects, their attributes, and object-background relationships, showing stronger performance on multi-scene datasets such as ShanghaiTech.The survey also identifies them as feasible for scene-adaptive GVAED models.

4 WEAKLY-SUPERVISED ABNORMAL EVENT DETECTION

Weakly supervised anomaly detection uses video-level labels to learn clip-level anomaly scores, and its models are organized by unimodal or multimodal inputs and by temporal, spatial, and label-noise handling.

  • Weak supervision: Weakly supervised GVAED uses video-level labels, with multiple clips forming a bag that shares one label under multiple-instance learning.The MIL ranking framework predicts high scores for anomalous clips while applying sparsity and smoothness constraints.
  • Datasets and setting: UCF-Crime introduced 13 classes of real-world criminal behavior with video-level training labels, supporting the development of WAED benchmarks.Subsequent benchmarks moved some anomalous test videos into training and included reorganized UCSD Ped2 and ShanghaiTech Weakly.
  • Input modalities: Unimodal WAED models primarily use RGB frames, whereas multimodal models combine video, audio, text, and optical flow for anomaly-related cues.The taxonomy separates these approaches by input modality and supports the transition from video processing toward broader event understanding.
  • Input modalities: Multimodal models are described as more robust and versatile for capturing complex abnormal events, and their integration broadens GVAED toward comprehensive event understanding.XD-Violence is identified as the first multimodal GVAED dataset, while later work explores multimodal fusion and lightweight dual-stream designs.
  • Temporal modeling: WAED methods address temporal structure through attention, temporal encoding, self-attention, temporal relationships, and fusion of spatial and temporal contexts.Examples include attention-enhanced MIL, DTED, RTFM, WSTR, and WSAL, which target temporal evolution, correlations, or localization.
  • Learning under weak labels: Label-noise and weak-label challenges are addressed with graph-based correction, clustering, pseudo-labels, self-training, and ranking losses.These methods seek cleaner supervision or more task-specific representations from video-level annotations.

5 SUPERVISED VIDEO ANOMALY DETECTION

Supervised video anomaly detection uses labeled frames or pixels, but its practical formulation remains open-set recognition because unseen anomalous events must be detected. Fine-grained labeling is costly and discrete labels poorly represent anomaly severity and temporal continuity.

  • Supervised video anomaly detection requires frame-level or pixel-level labels to distinguish normal from anomalous content.
  • Fine-grained anomalous-event labeling is time-consuming, while gradual anomalies and relative severity are inadequately represented by discrete 0/1 labels.
  • Synthetic game-engine data can provide personalized frame-level and pixel-level annotations for supervised GVAED training, but domain adaptation remains necessary because synthetic and real data differ.
  • Supervised models must generalize from partially labeled anomalies to unseen anomalous events in real-world scenarios.
  • Supervised video anomaly detection is an open-set recognition task rather than supervised binary classification.

6 FULLY-UNSUPERVISED VIDEO ANOMALY DETECTION

Fully-unsupervised video anomaly detection learns anomaly detectors from raw, unannotated data. Existing approaches use pseudo-labeling, self-training, cooperative learning, rarity assumptions, masked temporal representations, or iterative unmasking.

  • Fully-unsupervised video anomaly detection learns anomaly detectors from random raw data without restricting training-data composition or requiring annotation.
  • Distribution clustering can identify likely anomalous samples, after which clustered normal subsets repeatedly train an encoder that describes normality.
  • SDOR initializes pseudo-normal and abnormal sets, computes scores with pretrained ResNet-50 and FCN, and iteratively optimizes representations and scoring.
  • Causal analysis of SDOR pseudo-label generation targets confounding effects and noisy pseudo-labels, while CIL reportedly improves significantly.
  • GCL uses mutually supervised generator and discriminator modules, with negative learning to distort anomaly representations and generate pseudo-labels.
  • Other FVAD methods exploit anomaly rarity, temporal masking of foreground cubes, or iterative removal of discriminative features from consecutive sequences.

7 PERFORMANCE COMPARISON

The survey compares GVAED methods across public benchmarks and reports patterns involving object-level processing, multi-stream modeling, feature extractors, and computational overhead. It also notes that benchmark and inference-speed comparisons remain difficult to standardize.

  • Object-level methods outperform other frame-level and patch-level models on the cross-scene ShanghaiTech dataset.
  • 90.2% frame-level AUC is reported for the Multi-task model on ShanghaiTech.
  • Using foreground object detectors can reduce background interference in cross-scene GVAED tasks, while multi-stream models generally outperform single-stream models.
  • UCSD Ped2, CUHK Avenue, and ShanghaiTech are the prevailing benchmarks for UVAD evaluation, with future work encouraged to compare methods across all three.
  • I3D features outperform C3D features for the same WAED model because I3D separately considers temporal-information variation.
  • Inference-speed figures often lack direct comparability because experimental environments and computational platforms vary, and preprocessing overhead may be excluded.

8 CHALLENGES AND TRENDS

GVAED faces gaps between simulated and real anomalies, single-scene and multi-scene deployment, unimodal and multimodal data, offline and online operation, and limited benchmark realism. The survey identifies larger, more realistic, multimodal, synthetic, and deployment-aware research as continuing directions.

  • Mock anomalies vs. Real anomalies: Mock anomalies are simpler and less diverse than real anomalies, causing models trained on simulated datasets to perform poorly in real-world scenarios.
  • Single-scene vs. Multi-scenes: Most UVAD methods learn normality directly from video without modeling scene differences, although deployment commonly requires cross-scene models.
  • Real data vs. Synthetic data: Synthetic data can provide diverse anomalous samples and precise frame-level or pixel-level annotations, but covariate shifts may prevent models from working in real scenes.
  • Unimodal vs. Multimodal: Multimodal GVAED remains limited by scarce datasets, uncertain model validity, and the need for heterogeneous-data alignment, fusion, and domain-offset bridging.
  • Single-view vs. Multi-view: Existing GVAED datasets are mostly single-view, and combining views through shared training and voting is costly while ignoring view differences and complementarities.
  • Online detection: Existing GVAED models primarily target public datasets rather than real-time streams, making online detection on resource-limited devices a deployment challenge.
  • Trends: GVAED datasets are moving toward larger and more realistic scenarios, with multimodal data demonstrating positive impact and synthetic data gaining importance.

9 CONCLUSION

This survey unifies deep learning-driven GVAED routes under a taxonomy organized by supervision, input data, and network structure. It also consolidates benchmark resources, compares methods, and analyzes challenges and future trends for research and deployment.

  • The survey integrates deep learning-driven technical routes based on different assumptions and learning frameworks into a unified GVAED framework.
  • Its hierarchical taxonomy organizes existing literature by supervision, input data, and network structure, including weakly-supervised, fully-unsupervised, and multimodal methods.
  • The survey collects benchmark datasets and available codes, sorts out method development lines, and performs performance comparisons and strengths analysis.
  • It analyzes connections among deep GVAED routes, research challenges, future trends, and deployment problems to guide researchers and engineers.
Loading 2302.05087v3…