PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Single-Scene Video Anomaly Detection

Bharathkumar Ramachandra, Michael J. Jones, Ranga Raju Vatsavai

arXiv:2004.05993v2cs.CV

TL;DR

Single-scene video anomaly detection is difficult because anomalies depend on scene-specific appearance, motion, location, and time, while training data generally contains only normal activity. This survey organizes formulations, datasets, anomaly types, evaluation criteria, and prior methods, then compares algorithms across standard benchmarks. Its comparison finds no universally best method, and performance on small datasets can differ markedly from performance on the larger, more complex Street Scene dataset.

  • Problem

    Video anomalies are scene-dependent, and prior work often failed to distinguish single-scene from multi-scene detection, despite location-dependent anomalies being central to the former.

  • Method

    The survey categorizes formulations and methods, reviews datasets and evaluation criteria, and conducts comparative evaluations of algorithms on standard single-scene benchmarks.

  • Results

    The comparison finds no single best method across datasets and criteria, while methods strong on small datasets can perform poorly on the larger, more complex Street Scene dataset.

  • Takeaways & Limitations

    Algorithm performance should be interpreted across datasets and evaluation criteria rather than treated as universally transferable from small benchmarks.

  • Takeaways & Limitations

    Current single-scene datasets are predominantly static-camera datasets, and no benchmark datasets or algorithms address the moving-camera version.

Abstract

from arXiv · show

This survey article summarizes research trends on the topic of anomaly detection in video feeds of a single scene. We discuss the various problem formulations, publicly available datasets and evaluation criteria. We categorize and situate past research into an intuitive taxonomy and provide a comprehensive comparison of the accuracy of many algorithms on standard test sets. Finally, we also provide best practices and suggest some possible directions for future research.

1 INTRODUCTION

Single-scene video anomaly detection localizes unusual activities in a particular scene, where normality is scene-dependent and training commonly uses only normal video. The survey distinguishes this formulation from multi-scene detection and outlines a model-building phase followed by anomaly scoring and thresholding.

  • Video anomalies are unusual appearance or motion attributes, or usual attributes occurring in unusual locations or times.
  • Normal video is needed to represent the variety of activities that may occur in a particular scene, while anomalous examples are typically unavailable for training.
  • Single-scene detection matters in surveillance because people struggle to monitor mostly uneventful camera feeds for long periods.
  • Unlike multi-scene detection, single-scene detection can model location-dependent anomalies and does not require normal activity to be consistent across unrelated scenes.
  • The survey focuses on single-scene video anomaly detection because of its practical importance in real-world applications.
  • Available single-scene datasets are predominantly static-camera datasets, although static cameras are not required by the formulation.
  • Typical algorithms learn normal activity from anomaly-free videos, score voxels in new video from the same scene, and threshold scores into spatio-temporal masks.

1.1 Other formulations of the problem

Video anomaly detection includes alternative formulations and several anomaly types, each requiring different contextual information. The survey emphasizes clear formulation boundaries because inconsistent definitions make datasets, comparisons, and reported performance harder to interpret.

  • 1.1 Other formulations of the problem: Multi-view video anomaly detection is an alternate formulation discussed separately from the single-scene problem.
  • 1.1 Other formulations of the problem: Training-free detection uses no normal training video and instead detects testing-video changes or the most unusual segments as anomaly proxies.
  • 1.1 Other formulations of the problem: Unclear problem formulations create ambiguity about which datasets to test, which methods to compare, and how performance should be understood.
  • 1.2.1 Appearance-only Anomalies: Appearance-only anomalies involve unusual object appearance and can be detected from a local region of a single frame.
  • 1.2.2 Short-term Motion-only Anomalies: Short-term motion-only anomalies involve unusual object motion and require inspecting a local video region over a short period.
  • 1.2.3 Long-term Trajectory Anomalies: Long-term trajectory anomalies involve unusual object paths and require analysis of longer video segments.
  • 1.2.4 Group Anomalies: Group anomalies involve unusual interactions among two or more video regions, such as people moving in an unusual formation.
  • 1.2.5 Time-of-Day Anomalies: Time-of-day anomalies are activities that become unusual because of when they occur and may require separate normality models for different times.

A note on the types of anomalies

Anomalousness is continuous and context-dependent rather than strictly binary, while useful detection generally requires spatial as well as temporal localization. The survey also distinguishes its broad coverage from narrower prior surveys.

  • Video anomaly detection is application-dependent, and public datasets mainly contain appearance-only and short-term motion-only anomalies.
  • Anomalousness is fluid: activities receive higher scores as their appearance, motion, or context becomes less similar to normal examples.
  • The field uses competing labels such as unsupervised, supervised, semi-supervised, and weakly-supervised for settings with only normal training data or threshold supervision.
  • Spatial localization is important because temporal-only detection does not identify which parts of a busy frame are anomalous.
  • Earlier surveys variously emphasize crowded-scene analysis, high-level surveillance coverage, video modeling, or reconstruction rather than comprehensive anomaly-detection comparisons.

2.1 Datasets

The survey reviews publicly available datasets and recommends evaluating them by annotation quality, size, and utility. It highlights strong dataset differences, important annotation and evaluation caveats, and the exclusion of multi-scene collections from the single-scene formulation.

  • Benchmark datasets define problem scope and enable fair algorithm comparisons; the survey summarizes recommended datasets using their characteristics and example frames.
  • Subway contains two indoor-scene videos and 85 temporally labeled anomalous events, but unclear extraction, training, testing, and spatial annotation make it unsuitable for formal evaluation.
  • UMN contains 11 clips from three cameras with one staged crowd-dispersal event per clip, but ambiguous extraction and split specifications and temporal-only labels limit evaluation.
  • UCSD Ped1 and Ped2 are widely used static-camera pedestrian-walkway datasets with spatial and temporal annotations, though Ped1 originally had partial pixel-wise ground truth.
  • CUHK Avenue contains 16 training and 21 testing videos at 640×360 resolution, with temporal and pixel-level bounding-box annotations.
  • Reported Avenue pixel-level AUC results from buggy evaluation code can exceed frame-level AUC, which is impossible because frame-level AUC upper-bounds pixel-level AUC.
  • Street Scene is the largest recent dataset, with 46 training and 35 testing sequences, 56,847 training frames, 146,410 testing frames, and 205 naturally occurring anomalies.
  • Multi-scene datasets combine unrelated scenes for one normality model and therefore do not apply to the single-scene formulation.

2.2 Evaluation Protocol

The survey reviews frame- and pixel-level evaluation criteria, their counting rules, and newer region- and track-based alternatives. It emphasizes that conventional measures can misrepresent localization quality and should be complemented by qualitative inspection.

  • Traditional criteria: Frame-level evaluation labels a frame anomalous when at least one pixel exceeds the threshold, then computes TPR, FPR, AUC, and EER from frame annotations.It does not evaluate spatial localization.
  • Traditional criteria: Pixel-level evaluation counts a true positive when predicted anomalous pixels cover over 40% of annotated anomalous pixels, while any prediction in an anomaly-free frame counts as a false positive.Despite using spatial masks, positives and false positives remain counted at the frame level.
  • Traditional criteria: The pixel-level criterion permits one positive per frame, ignores incorrect regions within anomalous frames, and does not penalize masks expanded to the entire frame.These properties can make crude postprocessing improve apparent performance.
  • Traditional criteria: Earlier pixel-level evaluation was incompletely specified, and a later clarification reduced false-positive counts; some earlier reported AUCs may therefore use the incorrect interpretation.The survey identifies this as a possible source of much lower reported pixel-level AUCs.
  • Recent criteria: Region- and track-based criteria use IOU, region-level counting, and partial track detection to better accommodate annotation ambiguity and multiple detections.They require bounding-box annotations with unique anomaly and track IDs for supported datasets.
  • Evaluation considerations: AUC summarizes only a narrow view of performance, so qualitative visualizations should assess false-positive plausibility when numerical scores are comparable.The survey notes that correctly used traditional criteria are already saturated on smaller datasets and have serious flaws.

2.3 A Taxonomy of Video Anomaly Detection Approaches

The survey organizes single-scene video anomaly detection by modeling approach, representation, processing unit, and handling of spatial context. It highlights recurring shortcomings in spatial localization and per-video score normalization.

  • Approach taxonomy: Past approaches are broadly categorized as distance-based, probabilistic, or reconstruction-based, although these categories can overlap.The taxonomy covers works evaluated on standard video anomaly detection benchmark datasets.
  • Representations and processing units: Representations include hand-crafted features and CNN-derived deep features, while algorithms process image patches, video patches, full frames, or short video snippets.The processing unit ranges from local patches to temporal sequences of complete frames.
  • Modeling themes: Modeling themes include one-class SVMs, nearest neighbors, Hidden Markov Models, probabilistic graphical models, and adversarial deep learning.These themes describe recurring model families across prior work.
  • Limitations: Methods focused on temporal localization often fail to provide adequate spatial localization, especially when video snippets are the processing unit.The model objective can prioritize frame-level detection over spatial precision.
  • Limitations: Full-frame and snippet methods may overlook location-dependent anomalies, such as distinguishing loitering outside an embassy from loitering in a nearby public park.Such methods use a looser anomaly definition than scene-specific localization requires.
  • Limitations: Per-video min-max normalization assumes every test sequence contains normal and anomalous frames, making scores across videos incomparable.This practice also relies on information from each test sequence.

2.4 Other Less Common Settings and Related Modeling

The survey contrasts less common anomaly-detection settings with standard single-scene assumptions, emphasizing limitations of object tracking and supervised learning. Supervised methods require anomalous examples, while object-based methods depend on reliable trajectories.

  • Object detection and tracking approach: Object-detection and tracking methods require complete object trajectories, but occlusions and diverse natural scenes make this suboptimal for general video anomaly detection.The limitation is especially pronounced when scenes contain objects beyond reliably detectable categories such as humans or cars.
  • Supervised anomaly detection: Supervised approaches cannot include every possible future anomaly, which is rare and difficult to annotate in natural scenes.This assumption conflicts with the goal of detecting deviations from normality rather than only known anomalous classes.
  • Supervised anomaly detection: If all anomalous activities were available for supervision, anomaly detection would reduce to binary video classification with a known anomalous class.The authors distinguish this from detecting any deviation from normality.
  • Weakly supervised anomaly detection: Weakly supervised methods use video-level labels, but their formulation is considered overly specific because obtaining videos of all possible anomalies remains a concern.Multiple Instance Learning treats videos as bags of snippets, with positive bags containing at least one anomalous snippet.
  • Weakly supervised anomaly detection: A later approach converts weak supervision into noisy fully supervised learning, making label-noise cleaning primary and anomaly detection secondary to binary action recognition.The method uses a graph convolutional network with alternating optimization to clean noisy labels.

3 DISTANCE-BASED APPROACHES

Distance-based approaches model normality from training data and score anomalies by measuring deviations from that model. The survey covers diverse features, distances, learned representations, and location-aware or fast variants, including simple methods that outperform prior state of the art on some datasets.

  • Overview: Distance-based approaches create a model of normality from training data and measure deviations from it to assign anomaly scores.The survey describes them as a general form encompassing probabilistic and reconstruction-based approaches.
  • Modeling choices: One-class SVMs commonly define decision boundaries around normal feature vectors, but updating them with new normal data requires rerunning learning on all data.This makes incremental model updates expensive.
  • Features and representations: Deep and hand-crafted representations include CNN features, optical flow, trajectories, foreground masks, motion descriptors, STIP features, and multi-stream appearance-motion inputs.Recent approaches increasingly use deep-network features, which generally have higher accuracy than earlier hand-crafted representations.
  • Representative methods: Location-dependent one-class SVMs, Gaussian cascades, clustering, geometric convex-hull tests, and adaptive Growing Neural Gas provide varied mechanisms for scoring deviations from normality.These methods operate on motion, appearance, trajectory, or spatio-temporal features and sometimes explicitly model changing scenes.
  • Reported results: Simple normalized L1 or L2 voxel-wise distances with concise exemplar models outperform some previous state-of-the-art methods on other datasets.The authors suggest this may indicate dataset-specific algorithmic biases.

4 PROBABILISTIC APPROACHES

Probabilistic approaches measure anomaly distance within probability spaces, using traditional probabilistic models and motion-oriented features. The surveyed methods include local and global likelihood models, graphical models, mixtures, and location- or trajectory-based formulations.

  • Overview: Probabilistic approaches compute distance under a probability-space model, often using graphical models or high-dimensional mixtures.Many earlier methods use spatio-temporal gradients, optical flow, or STIP features with traditional probabilistic models.
  • Local likelihood models: Fixed-location monitors model optical-flow buffers with histograms or kernel density estimation and flag low-likelihood test observations as anomalies.The model is tied to the corresponding monitor’s location in the camera frame.
  • Motion models: Social-force methods estimate interactions from differences between a pixel’s flow and neighborhood-average flow, then detect low-likelihood frames using a bag-of-words model.The representation treats deviations from neighboring motion as evidence of particle interactions.
  • Graphical models: Markov Random Fields model co-occurrence statistics of background-subtraction motion labels in spatio-temporal neighborhoods and use likelihood-ratio testing for detection.The co-occurrence matrix serves as the MRF potential function.
  • Temporal and spatial correlations: Coupled Hidden Markov Models combine spatial and temporal correlations between patches whose spatio-temporal-gradient distributions are modeled with Gaussian mixtures.The method uses location-specific mixtures for video-patch distributions.
  • Additional probabilistic formulations: Other formulations model trajectories with chaotic invariants, normality with MoPPCA-based spatio-temporal MRFs, or anomalies through low-likelihood interaction templates and exemplar explanations.These approaches vary in whether they emphasize trajectories, neighboring patches, global interactions, or explaining normal foreground structure.

5 RECONSTRUCTION-BASED APPROACHES

Reconstruction-based approaches learn compact representations from normal video and reconstruct inputs, using reconstruction difficulty as an anomaly proxy. The survey covers auto-encoders, GANs, prediction, memory, and sparse reconstruction, while noting retraining, localization, and training-coverage limitations.

  • Overview: Reconstruction approaches learn a compact representation from normal video and reconstruct inputs using only that representation.They assume anomalies are harder to reconstruct than in-distribution normal data, so reconstruction error serves as an anomaly score.
  • General limitations: Most reconstruction methods use convolutional auto-encoders or GANs, but many require retraining for new normal video and omit spatial localization despite pixelwise errors.The survey suggests low spatial localization accuracy may explain the omission.
  • Architectures: Convolutional auto-encoders, convolutional LSTMs, GANs, RBMs, and future-frame prediction networks reconstruct or predict video using intensity, gradient, flow, or multimodal losses.Several methods combine prediction and reconstruction or use multiple levels and modalities to form anomaly maps.
  • Evaluation caveats: A generalized auto-encoder trained across several datasets performs about as well as dataset-specific training, which the survey interprets as evidence of shared motion bias.The authors attribute the pattern to anomalous activities often involving faster-moving objects in the evaluated datasets.
  • Memory-augmented methods: A learned memory module is used to address CNN reconstruction networks that may reconstruct anomalous events too well, combining reconstruction quality with distance to stored normal patterns.One method uses PSNR and nearest-memory distance; another uses attention-based addressing and hard shrinkage for sparse reconstructions.
  • Sparse reconstruction approaches: Sparse reconstruction methods are fast and easier to update online, but can over-rely on memorized salient normal features and poorly represent continuously varying anomalousness.Their effectiveness therefore depends heavily on an exhaustive normal training set.
  • Sparse reconstruction approaches: 1000 FPS is reported for an online sparse-combination solver, up from 150 FPS, while preserving sparse reconstruction for dynamic and large-scale data.The approach extends earlier sparse combination learning with a birth-and-death online solver.

6 A COMPARATIVE STUDY OF METHODS

The survey compares methods across multiple datasets and evaluation criteria, finding no universally best method and exposing gaps between traditional benchmarks and more realistic testing.

  • Method taxonomy: Table 2 groups prior methods by representation, modeling strategy, normal-activity model, and processing unit.The survey distinguishes representations such as gradients, optical flow, deep features, and video snippets from modeling choices including nearest neighbors, HMMs, adversarial methods, and auto-encoders.
  • Evaluation scope: The comparison covers frame-, pixel-, track-, and region-based criteria across UCSD Ped1, UCSD Ped2, CUHK Avenue, and Street Scene.Tables 3–5 compile reported results under complementary evaluation schemes and benchmark datasets.
  • Comparative results: Methods performing very well on traditional criteria for small datasets can perform very poorly on the larger and more complex Street Scene dataset.This pattern appears across track-, region-, pixel-, and frame-level criteria.
  • Comparative results: There is currently no single best method: the strongest method varies across datasets and evaluation criteria.Methods leading on UCSD Ped1 with the pixel-level criterion can have only middle-of-the-pack accuracy on UCSD Ped2 with that criterion.
  • Practical limitations: Progress is difficult to translate into high-performing deployed systems because realistic datasets and evaluation criteria remain limited.The survey connects this limitation with the field’s infrequent reporting of running times.
  • Runtime: Probabilistic approaches can detect anomalies in a principled framework but struggle to perform detection in real time.The runtime comparison reports inference times where available and notes that frame resolution affects processing time for most methods.

7 DISCUSSION

The discussion synthesizes the survey’s taxonomy and comparisons into recommendations for reliable evaluation and reproducibility. It also identifies representation trends and unresolved anomaly types that limit practical coverage.

  • Best Practices Going Forward: The survey recommends reliable datasets, a new evaluation protocol, reproducible research, runtime analyses, qualitative false-positive evaluation, and testing across multiple datasets.CUHK Avenue and Street Scene are identified as useful supplements because they contain more varied anomalous activity.
  • Best Practices Going Forward: Methods evaluated only on UCSD Ped1, UCSD Ped2, and UMN can be biased toward anomalies involving larger motion magnitudes.The survey recommends multiple datasets to address this limitation.
  • Representation Trends: Input representations progressed from fixed-size image patches to video patches and increasingly multimodal video patches combining raw frames with estimated optical flow.The shift toward temporal and multimodal representations required greater compute power.
  • Modeling Trends: Modeling shifted from simple hand-crafted features and assumptions toward data-learned representations and optimization schemes as datasets and compute increased.The survey expects data-driven modeling to continue as larger, more complex datasets become available.
  • Looking Ahead: Existing methods often miss loitering anomalies because they rely heavily on motion cues, while rare but normal activities can produce false positives.The discussion cites multi-timescale modeling as one attempt to address loitering and uses tying a shoe as an example of rare normal activity.
  • Looking Ahead: Group, trajectory, and time-of-day anomalies remain largely unaddressed because benchmark datasets containing them do not yet exist.The survey calls for datasets with these properties.
Loading 2004.05993v2…