PaperlayerView exact revision on arXiv

Source-linked AI summary

An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos

B Ravi Kiran, Dilip Mathew Thomas, Ranjith Parakkal

arXiv:1801.03149v2cs.CV

TL;DR

Video anomaly detection must learn from surveillance videos that contain little annotation for rare anomalous events. The paper reviews deep learning methods, studies their approaches, and examines evaluation criteria for spatio-temporal detection. It reports model-performance patterns across PCA, VAEs, ConvLSTM variants, and 3D convolutional autoencoders, while identifying sensitivity to image size and costly optical-flow estimation as limitations.

  • Problem

    Rare anomalies and scarce labels make it difficult to train deep models to distinguish abnormal video appearance and motion from normal behavior.

  • Method

    The paper reviews and categorizes unsupervised and semi-supervised deep video anomaly-detection models by representation, prior information, and detection criteria.

  • Results

    VAEs perform as well as or better than PCA on optical flow, while ConvLSTM variants and 3D convolutional autoencoders achieve comparable performance in the reviewed studies.

  • Takeaways & Limitations

    Temporal coherence, transformation sensitivity, and evaluation with metrics suited to rare positives are important criteria for video anomaly-detection representations.

  • Takeaways & Limitations

    Performance can change substantially with image resizing, and optical-flow estimation remains a costly algorithmic step.

Abstract

from arXiv · show

Videos represent the primary source of information for surveillance applications and are available in large amounts but in most cases contain little or no annotation for supervised learning. This article reviews the state-of-the-art deep learning based methods for video anomaly detection and categorizes them based on the type of model and criteria of detection. We also perform simple studies to understand the different approaches and provide the criteria of evaluation for spatio-temporal anomaly detection.

1 INTRODUCTION

Video anomaly detection learns normal appearance and motion from mostly unlabeled surveillance data, then identifies rare deviations using reconstruction, probability, or prediction errors. The review focuses on deep representations and generative or predictive models for this task.

  • Video anomaly detection is difficult because videos combine high-dimensional spatial structure with nonlocal temporal variation.
  • The review examines convolutional feature learning together with predictive and generative models, including VAEs, GANs, LSTMs, and related architectures.
  • The task typically uses training videos containing normal behavior without anomalies and seeks representations that capture normal motion and appearance patterns.
  • Anomaly scores can measure reconstruction deviation, low model probability, or prediction error for future samples conditioned on past observations.
  • Detection declares a test sample anomalous when its anomaly score exceeds a threshold, while probabilistic models identify low-density regions.

2 REPRESENTATION LEARNING FOR VIDEO ANOMALY DETECTION(VAD)

The review frames video anomaly detection as learning representations that reconstruct or predict normal spatio-temporal patterns and score deviations in test videos. It organizes approaches by representation and prior information while emphasizing spatial localization, temporal scoring, and evaluation under scarce anomaly labels.

  • Taxonomy: The survey categorizes deep video anomaly-detection methods according to the prior information used to construct their representations.
  • Taxonomy: Representation-learning methods model normal appearance or motion through reconstruction, while predictive methods model temporal patterns and conditional future-frame distributions.
  • Context of the review: Evaluation considers spatial and temporal anomaly behavior, with examples including unusual objects, actions, locations, and directions in surveillance datasets.
  • Scope: The review focuses on deep convolutional networks and deep generative models for end-to-end spatio-temporal representation learning.
  • Problem setup: The task assigns each test frame a temporal anomaly score and a spatial score that localizes anomalous regions.
  • Context of the review: The study treats anomaly detection as semi-supervised when normal samples are available but labels for the rare positive class are absent.

3 RECONSTRUCTION MODELS

Reconstruction models learn representations of normal appearance or motion and detect anomalies through reconstruction deviations. The section progresses from PCA and autoencoders to convolutional and spatio-temporal variants with regularization.

  • 3.1 Principal Component Analysis: PCA models spatial pixel correlations through maximal-variance directions and uses reconstruction deviation as an anomaly signal.The reconstruction can be scored with Mahalanobis distance or variance-scaled reconstruction error.
  • 3.1 Principal Component Analysis: Optical-flow reconstruction provides a motion baseline, but optical-flow estimation is a costly algorithmic step.Fast approximations trade accuracy for evaluation speed.
  • 3.2 Autoencoders: Autoencoders provide nonlinear dimensionality reduction by mapping inputs into latent representations and reconstructing them with differentiable point-wise transformations.Typical nonlinearities include ReLU and sigmoid functions.
  • 3.3 Convolutional Autoencoders: Convolutional autoencoders preserve spatial structure while reducing parameter redundancy through weights shared across input locations.Traditional PCA and autoencoders ignore pixel location, whereas convolutional weights exploit local spatial organization.
  • 3.4 Spatio-Temporal Stacked Frame Autoencoder: The STSAE reconstructs a stack of p frames by treating each temporal slice as a separate input channel and regularizing model weights with an L2 penalty.Its compressed convolutional representation is intended to enforce temporal regularity.
  • 3.5 Other Autoencoder Variants: Spatio-temporal autoencoders compare original and reconstructed frames or tensors, using reconstruction error or normalized variants as anomaly scores.Video-patch, sparse, robust, and contractive autoencoders extend reconstruction with parameter, sparsity, or invariance constraints.

4 PREDICTIVE MODELING

Predictive models learn temporal dependencies by forecasting current or future video states from past frames. The reviewed approaches include recurrent, convolutional recurrent, composite reconstruction-prediction, three-dimensional, and slow-feature models.

  • 4.1 Predictive Models: Predictive models estimate the current frame from preceding frames, treating video as a temporal sequence or autoregressive process.LSTMs add gating to recurrent networks to address vanishing gradients during backpropagation through time.
  • 4.2 Convolutional LSTM: The composite LSTM combines an autoencoder with a predictive LSTM, separating reconstruction from temporal prediction.The design addresses the tension between memorizing inputs for reconstruction and retaining temporal history for forecasting.
  • 4.2 Convolutional LSTM: ConvLSTM replaces fully connected layers with convolutions to model spatio-temporal correlations and forecast future states on a spatial grid.An encoder compresses sequences into hidden-state tensors, while a forecasting network unfolds them into predictions.
  • 4.2 Convolutional LSTM: Composite ConvLSTM systems use separate reconstruction and prediction branches for video anomaly detection, with reported promising results.Related formulations also deconvolve ConvLSTM outputs back to the original resolution.
  • 4.3 3D Convolutional Models: The 3D autoencoder stacks frames and uses three-dimensional kernels to learn representations invariant to spatio-temporal movement.Its reconstruction and prediction branches jointly model appearance and local temporal coherence.
  • 4.4 Slow Feature Analysis (SFA): Slow Feature Analysis extracts slowly varying representations from rapidly varying high-dimensional inputs under zero-mean, unit-covariance, and decorrelation constraints.Incremental and localized variants support online slow-feature extraction and subsequent anomaly detection.

5 DEEP GENERATIVE MODELS

Deep generative models estimate distributions over high-dimensional inputs using explicit likelihoods or latent-variable approximations. The section reviews VAEs, adversarial autoencoders, and GAN-based approaches for anomaly detection.

  • 5.1 Deep Generative Models: Generative models estimate distributions over high-dimensional images or spatio-temporal tensors, differing in how they represent or approximate likelihood.Explicit models evaluate the model density directly and maximize likelihood.
  • 5.1 Deep Generative Models: The review covers variational and adversarial autoencoders alongside generative adversarial networks for anomaly detection.These models are discussed as stochastic autoencoder and GAN approaches within deep generative modeling.
  • 5.2 Variational Autoencoders (VAEs): VAEs approximate the data distribution with probabilistic encoders and decoders that represent uncertainty through a latent-variable model.The encoder produces a Gaussian latent distribution, while the decoder produces a distribution over inputs.
  • 5.2 Variational Autoencoders (VAEs): VAE training combines expected negative reconstruction error with KL-divergence regularization to optimize a variational lower bound on marginal log-likelihood.The latent prior and approximate posterior are modeled as Gaussian distributions.
  • 5.2 Variational Autoencoders (VAEs): Reparameterization rewrites latent sampling using a differentiable deterministic function of an auxiliary random variable, enabling gradient-based optimization.The unit-Gaussian case expresses samples using a mean, standard deviation, and independent noise.

5.3 Anomaly detection using VAE

VAE anomaly detection accounts for variability in the latent space when reconstructing test samples. This distinguishes its reconstruction probability from deterministic autoencoder reconstruction.

  • 5.3 Anomaly detection using VAE: VAE reconstruction probability for a test sample incorporates multiple samples drawn from its latent-variable distribution.This sampling accounts for latent-space variability during reconstruction.
  • 5.3 Anomaly detection using VAE: Unlike a standard autoencoder, a VAE represents latent variables through probabilistic distributions rather than deterministic mappings.The distinction is central to interpreting its reconstruction probability.

5.4 Generative Adversarial Networks (GANs)

GAN-based anomaly detection models learn the distribution of normal data through adversarial generator–discriminator training. For anomaly scoring, some methods map test images back to the latent space and compare them with generated manifold samples.

  • GAN fundamentals: A GAN uses a generator to map latent noise z to images and a discriminator to distinguish generated samples from real training data.The generator learns a data distribution through G(z), while the discriminator assigns real or fake probabilities.
  • GAN fundamentals: Adversarial training improves the generator’s realism while advancing the discriminator’s ability to identify generated images.The generator minimizes its adversarial objective to fool the discriminator, whereas the discriminator learns to classify real and generated samples.
  • GANs for anomaly detection: For anomaly detection, GANs model the density of normal training vectors and use negative log-likelihood to score test samples.The method evaluates how well test points fit the probability density learned from anomaly-free training data.
  • GANs for anomaly detection: Because GANs do not directly provide a likelihood mapping from images to latent space, test images are matched to generator outputs by optimizing a latent vector.The optimized G(z) is the visually most similar point on the learned image manifold, and similarity reflects conformity to the training distribution.

5.6 Adversarial Discriminators using Cross-channel prediction

Cross-channel prediction uses paired RGB frames and optical flow to learn mutually predictive models of normal video. Anomalies are detected when cross-channel prediction becomes poor and the discriminators assign low scores.

  • Cross-channel formulation: The method formulates video anomaly detection as predicting raw-image values from optical flow and optical flow from raw-image frames.The two channels are RGB frames F_t and optical-flow vectors O_t associated with consecutive video frames.
  • Cross-channel formulation: Two conditional GANs learn the mappings N_O→F and N_F→O between optical flow and RGB frames.Each generator predicts one channel from the other, extending cross-channel autoencoder ideas with conditional adversarial learning.
  • Architecture: The generators and discriminators use U-net architectures with skip connections to exploit shared spatial structure between the channels.The generators map frames and optical flow to their cross-channel counterparts, while the discriminator receives a 6-D tensor.
  • Anomaly scoring: Anomalous test frames are intended to produce poor cross-channel predictions, causing the trained discriminators to assign low probability scores.This reverses the normal training relationship in which the two channels are mutually predictable.
  • Anomaly scoring: The final normalized anomaly score is S = S_O + S_F, combining patch-discriminator scores without reconstructing the channels.The discriminators produce grid-based scores at a resolution smaller than the image, and the combined score is normalized to [0,1] per frame.

5.7 Adversarial Autoencoders (AAEs)

Adversarial autoencoders combine reconstruction with adversarial regularization of the latent representation. In the reviewed anomaly-detection application, the latent prior is Gaussian, but the method does not model temporal structure in video sequences.

  • AAE formulation: Adversarial autoencoders use GANs to approximate the aggregated posterior of an autoencoder’s latent representation under an arbitrary prior.The adversarial component regularizes the latent distribution rather than replacing the reconstruction path.
  • AAE formulation: In an AAE, x denotes training inputs, q(z|x) the encoder posterior, and p(z) the user-imposed prior over latent vectors.The latent-space distribution is shaped by matching the encoder’s aggregated output to the imposed prior.
  • AAE training: The encoder acts as the adversarial generator, improving reconstruction while fooling the discriminator into treating q(z) as drawn from p(z).Joint training updates the encoder through both the autoencoder reconstruction objective and the adversarial latent-space objective.
  • Application and scope: The reviewed method uses a 256-dimensional Gaussian prior and dropout probability 0.5, achieving close to state-of-the-art performance.The authors explicitly note that the AAE does not take temporal structure in video sequences into account.

5.8 Controlling reconstruction for anomaly detection

Several methods address autoencoders’ tendency to reconstruct anomalous samples well by introducing objectives that separate normal and anomalous reconstruction. These approaches use negative examples to preserve good reconstruction for normal data while degrading it for anomalous data.

  • Motivation: Deep autoencoders can achieve low reconstruction errors on anomalous events when trained on sufficiently many anomaly-free video samples.Their reconstruction capability may generalize broadly enough to approximate test samples that should be anomalous.
  • Negative learning: Negative-learning GAN objectives reconstruct normal samples while explicitly enforcing poor reconstruction for anomalous examples without requiring labels.The resulting setup is weakly supervised because it uses negative examples but does not require labeled classes.
  • Negative learning: The objective assumes strong class imbalance, with very few anomalous samples Y compared with normal samples X.Training minimizes reconstruction error for positive samples and maximizes it for negative samples.
  • Discriminative autoencoders: Discriminative autoencoders learn low-dimensional representations that reconstruct positive data better than negative data.Negative examples are pushed away from the learned manifold while positive reconstruction error is minimized.
  • Discriminative autoencoders: The hinge-loss formulation uses t(x) ∈ {−1,+1} as the sample label and e(x) = ∥x−x̂∥ as distance to the manifold.Minimizing this loss produces representations with comparatively better reconstruction for positive data.

6 EXPERIMENTS

The experiments compare reconstruction, predictive, and generative approaches for video anomaly detection using raw frames, frame stacks, and optical flow. Results show model performance depends on the input representation, temporal structure, anomaly frequency, and evaluation metric.

  • Experimental setup: The experiments evaluate single-frame and spatio-temporal reconstruction on raw image values or optical flow, alongside predictive and generative models.The study uses UCSD and CUHK-Avenue datasets, with PCA and VAR serving as baselines.
  • Observations and issues: Deep autoencoders can reconstruct previously unseen anomalous objects because of their high capacity, weakening anomaly detection based on poor reconstruction.The issue is reduced, but not eliminated, when reconstructing spatio-temporal frame blocks; negative examples are suggested as a possible control.
  • Evaluation measures: AU-PR scores are more sensitive than AU-ROC to rare anomalous classes, so lower CUHK-Avenue AU-PR scores do not necessarily indicate worse detection performance.UCSD anomalies last several hundred frames on average, whereas Avenue anomalies last only a few tens of frames, making the positive class rarer in Avenue.
  • Observations and issues: Changing the input image size can substantially change performance, while convolutional LSTM training requires many epochs, more video data, and application-specific augmentation choices.The reviewed experiments standardize images to 200 × 200, but translations and rotations may need to remain detectable depending on the surveillance application.
  • Model performance: VAEs perform consistently as well as or better than PCA on optical flow, while convolutional LSTMs and 3D convolutional autoencoders provide competitive alternatives.Convolutional LSTMs on raw images perform about as well as PCA in some cases and poorer in others; 3D convolutional autoencoders model local motion patterns.
  • Model performance: PCA performs well on random projections of individual frames for Avenue but poorly on UCSD, where motion patterns are important for detecting anomalies.The observation suggests that the discriminative representation differs across datasets and anomaly types.

7 CONCLUSION

The review categorizes video anomaly-detection methods into reconstruction-based, spatio-temporal predictive, and generative models, then identifies representation sensitivity and threshold selection as key evaluation concerns. It also notes emerging challenges from labeled anomalies, class imbalance, temporal-scale variation, and uncovered optimization components.

  • Model categorization: The review organizes unsupervised video anomaly-detection models into reconstruction-based, spatio-temporal predictive, and generative classes.These classes differ in whether they minimize reconstruction error, prediction error over spatio-temporal sequences, or distances involving generated and training distributions.
  • Evaluation criteria: Temporal coherence and transformation sensitivity are important criteria for evaluating learned video representations.The review highlights sensitivity to time warping and viewpoint changes, alongside invariances associated with convolutional or predictive structures.
  • Evaluation criteria: Anomaly-score threshold selection remains a design component not covered by the review, although ROC plots evaluate performance across thresholds.Thresholding determines detection decisions, while ROC evaluation summarizes performance over threshold values.
  • Emerging challenges: As surveillance data acquire labeled anomalies, methods must incorporate anomalous-class information and handle class imbalance.The no-labeled-anomalies assumption progressively becomes false, and some regular anomalous patterns can be estimated as a second class.
  • Emerging challenges: Variation in motion time scales across videos creates a practical need for representations invariant to temporal warping.This challenge arises even when videos share similar background and foreground.
  • Scope boundary: Batch normalization and dropout regularization were outside the review and warrant systematic study for video anomaly detection.The passage identifies both components as important in regularizing deep learning architectures.
Loading 1801.03149v2…