PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencoder

Yaxiang Fan, Gongjian Wen, Deren Li, Shaohua Qiu, Martin D. Levine

arXiv:1805.11223v1cs.CV

TL;DR

The paper addresses video anomaly detection and localization when training data consist only of normal samples. It combines a Gaussian-mixture variational autoencoder with fully convolutional and two-stream modeling, and reports superior performance to state-of-the-art methods on UCSD and Avenue.

  • Problem

    Video anomalies are difficult to define, motivating detection methods that learn normal behavior from normal samples alone.

  • Method

    The method uses two GMFC-VAE streams to learn Gaussian-mixture latent representations from RGB frames and dynamic flows, then scores patches with sample energy.

  • Results

    The method reports superiority over state-of-the-art methods on the UCSD and Avenue datasets.

  • Takeaways & Limitations

    The study demonstrates a normal-only, end-to-end framework for detecting and locating anomalous events in surveillance videos.

Abstract

from arXiv · show

We present a novel end-to-end partially supervised deep learning approach for video anomaly detection and localization using only normal samples. The insight that motivates this study is that the normal samples can be associated with at least one Gaussian component of a Gaussian Mixture Model (GMM), while anomalies either do not belong to any Gaussian component. The method is based on Gaussian Mixture Variational Autoencoder, which can learn feature representations of the normal samples as a Gaussian Mixture Model trained using deep learning. A Fully Convolutional Network (FCN) that does not contain a fully-connected layer is employed for the encoder-decoder structure to preserve relative spatial coordinates between the input image and the output feature map. Based on the joint probabilities of each of the Gaussian mixture components, we introduce a sample energy based method to score the anomaly of image test patches. A two-stream network framework is employed to combine the appearance and motion anomalies, using RGB frames for the former and dynamic flow images, for the latter. We test our approach on two popular benchmarks (UCSD Dataset and Avenue Dataset). The experimental results verify the superiority of our method compared to the state of the arts.

I. INTRODUCTION

The paper proposes an end-to-end, normal-only framework for video anomaly detection and localization using Gaussian-mixture latent representations, fully convolutional encoding, and two-stream appearance and motion modeling. Experiments on UCSD and Avenue report superiority over state-of-the-art methods.

  • I. INTRODUCTION: The method trains exclusively on normal samples and declares a test sample anomalous when it cannot be associated with any GMM component.This hypothesis underlies the Gaussian Mixture Fully Convolutional Variational Autoencoder (GMFC-VAE).
  • I. INTRODUCTION: GMFC-VAE jointly learns low-dimensional feature representations and their Gaussian Mixture Model through an encoder-decoder architecture.The encoder-decoder and Gaussian-mixture constraint address feature extraction and model construction together.
  • I. INTRODUCTION: A fully convolutional encoder-decoder preserves relative spatial coordinates between input images and output feature maps.The network omits fully connected layers.
  • I. INTRODUCTION: A two-stream framework uses RGB frames for appearance anomalies and dynamic flows for motion anomalies.Dynamic flows are generated using a Ranking SVM formulation instead of conventional optical flow.
  • I. INTRODUCTION: A sample-energy method scores image patches using the joint probabilities of all Gaussian-mixture components.The approach computes anomaly scores for the learned representations of test patches.
  • I. INTRODUCTION: Experiments on the UCSD and Avenue public datasets report superiority over state-of-the-art methods.The evaluation includes qualitative and quantitative comparisons.

II. RELATED WORK

Prior anomaly-detection methods use trajectories, local cuboids, or deep networks, but handcrafted features adapt poorly to varied anomalies and trajectory methods struggle in crowded scenes. The paper introduces VAE-based modeling as motivation for its Gaussian-mixture approach.

  • Anomaly detection methods are broadly divided into handcrafted-feature and deep-learning approaches.
  • Handcrafted Features: Trajectory methods represent object movement through tracked coordinates and flag trajectories that differ from learned normal patterns.
  • Handcrafted Features: Trajectory-based methods perform well for speed and direction anomalies but depend on object detection and tracking that are not robust in crowded scenarios.
  • Deep Learning Methods: The paper’s closest prior work combines three autoencoder streams with one-class SVM scoring, whereas this approach learns a Gaussian mixture end to end and uses dynamic flows.
  • Variational Autoencoder: A VAE maps inputs to latent representations through encoder and decoder networks while imposing a Gaussian latent prior for generative modeling.
  • Variational Autoencoder: VAE-based anomaly detection assumes normal latent representations follow one Gaussian cluster, although normal samples may form multiple clusters.

III. METHOD

The method represents motion with dynamic flows that aggregate sequential optical-flow information, then uses a two-stream GMFC-VAE framework to model appearance and motion from normal samples. Dynamic flows retain longer-term temporal information and make moving objects more salient than conventional optical flow.

  • A. Obtaining Dynamic Flow: Dynamic flow amalgamates sequential optical-flow frames into a two-channel representation of video motion.
  • A. Obtaining Dynamic Flow: Dynamic flow captures long-term temporal information rather than only motion between consecutive frames.
  • A. Obtaining Dynamic Flow: Each dynamic flow uses horizontal and vertical flow channels to represent motion across the video.
  • A. Obtaining Dynamic Flow: RankSVM solves the ranking formulation that converts multiple sequential optical-flow frames into dynamic flow.
  • A. Obtaining Dynamic Flow: Dynamic-flow colors encode motion onset, evolution, and movement intensity, while background removal makes moving objects more salient than in optical flow.

B. Learning Appearance and Motion Anomaly Detection

The appearance and motion models are separate GMFC-VAEs trained on RGB and dynamic-flow patches. Their fully convolutional encoder-decoder preserves spatial information while a Gaussian-mixture latent prior models normal-sample distributions.

  • Separate GMFC-VAE models are trained on densely sampled RGB patches and dynamic-flow patches for appearance and motion cues.
  • The latent prior is a Mixture-of-Gaussians, allowing the GM-VAE to describe normal samples distributed across multiple Gaussian components.
  • The generative process selects a mixture component, samples a latent vector, and reconstructs the input from that representation.
  • The GM-VAE objective combines reconstruction cost with a KL-divergence term between the mixture prior and variational posterior.
  • The fully convolutional encoder-decoder replaces spatially lossy fully connected and pooling operations to preserve spatial information.

C. Prediction

At prediction time, each test patch is encoded and scored by how well its latent representation matches the learned Gaussian components. Appearance and motion energy scores are combined, and patches exceeding a threshold are labeled anomalous.

  • A test patch is encoded into a latent representation, from which its conditional probabilities under the Gaussian components are computed.
  • Normal patches should associate with at least one Gaussian component, producing relatively high conditional probability for that component.
  • Anomalous latent representations are expected to have low conditional probabilities across the Gaussian components.
  • The sample-energy anomaly score is computed from log-likelihood, with higher scores indicating anomalies.
  • Overall anomaly energy combines appearance and motion scores using importance factors α and β.
  • A test patch is identified as anomalous when its overall energy exceeds threshold θ.

IV. EXPERIMENTS

The experiments evaluate the proposed algorithm qualitatively and quantitatively against state-of-the-art methods on the UCSD and Avenue datasets.

  • Experiments compare the proposed algorithm with state-of-the-art anomaly-detection methods.
  • The evaluation uses two public benchmarks: the UCSD Dataset and the Avenue Dataset.

A. Datasets

The study evaluates on UCSD and Avenue, fixed-camera video datasets containing normal and abnormal clips with different ground-truth annotations.

  • The UCSD dataset comprises fixed-camera Ped1 and Ped2 subsets recorded in different scenes, with normal and abnormal video clips.Ped1 contains 34 normal and 36 abnormal clips; Ped2 contains 16 normal and 14 abnormal clips.
  • Pixel-level ground truth is provided for 10 Ped1 test clips and 12 Ped2 test clips.
  • The Avenue Dataset contains 15 normal and 21 abnormal fixed-camera clips recorded in front of school corridors.Anomalies include running, throwing objects, and loitering; object-level rectangular ground truth is provided.

B. Evaluation Criteria

The evaluation uses frame-level and pixel-level detection criteria, with ROC-based AUC and EER as quantitative indexes.

  • Frame-level detection counts a frame as a true positive when it contains at least one anomalous pixel.
  • Pixel-level detection requires more than 40% overlap with a ground-truth anomaly region and evaluates localization capability.
  • ROC curves plot true positive rate against false positive rate to measure detection accuracy.TPR denotes correctly labeled frames, whereas FPR denotes incorrectly labeled frames.
  • The quantitative indexes are Area Under Curve and Equal Error Rate.EER is the misclassified-frame ratio when FPR equals the miss rate.

C. Implementation Details

The implementation combines RGB and dynamic-flow GMFC-VAEs operating on image patches, producing spatial anomaly score maps from a convolutional encoder-decoder architecture.

  • C. Implementation Details: Frames are resized to 420 × 280, and dynamic flow is computed from consecutive-frame optical flow.
  • C. Implementation Details: Dynamic-flow channels are transformed to [0, 255], stacked, and supplemented with flow magnitude F_m = √(F_u^2 + F_v^2).The flow-generation window is set to Δt = 20.
  • C. Implementation Details: Two GMFC-VAEs separately process RGB frames and dynamic-flow images to detect appearance and motion anomalies.
  • C. Implementation Details: Training uses 28 × 28 patches with stride 17, removes patches without moving pixels, and randomly samples 960K training patches.
  • C. Implementation Details: Testing uses 28 × 28 sliding windows with stride 28, producing a 15 × 10 score map that divides each frame into 150 square samples.
  • C. Implementation Details: Pretraining uses a stacked auto-encoder, and optimization uses Adam with learning rate 0.0001, momentum 0.9, weight decay 0.0005, and batch size 100.
  • C. Implementation Details: The encoder has four convolutional layers, while the decoder uses the reverse architecture and reconstructs image patches from sampled latent representations.Parallel fully connected layers produce component means and covariances before latent sampling.

D. Experimental results

The method is evaluated on UCSD Ped1, UCSD Ped2, and Avenue using ROC-based comparisons, quantitative metrics, and qualitative detection examples. It achieves strong performance across these benchmarks, with results varying by dataset and evaluation level.

  • UCSD datasets: 94.9% frame-level AUC and 11.3% frame-level EER on UCSD Ped1 outperform all comparison methods.
  • UCSD datasets: 91.4% pixel-level AUC and 36.3% pixel-level EER on UCSD Ped1 exceed other methods except Statistical Hypothesis Detector.
  • UCSD datasets: On UCSD Ped2, the method achieves 12.6% frame-level EER and 19.2% pixel-level EER, while outperforming all methods on both AUC measures.
  • Qualitative results: Qualitative examples show detections of bicycling, skateboarding, cars, and wheelchair events on the UCSD dataset, and waving hands, throwing papers, camera blocking, and running on Avenue.
  • Avenue dataset: The Avenue evaluation uses frame-level AUC and EER because its rectangular labels include background and foreground pixels.
  • Avenue dataset: 2.5% improvement in frame-level AUC over the best state-of-the-art result is reported on Avenue.

E. Analysis

The analysis examines mixture-component count and the contributions of appearance and motion streams. It identifies K = 20 as a performance–computation trade-off and finds late fusion beneficial for combining complementary cues.

  • Number of mixture components: Performance on UCSD Ped1 and Ped2 increases with K below 20, then remains steady as the number of mixture components grows.
  • Number of mixture components: K = 20 is selected as a reasonable trade-off because larger K increases computational cost without clear additional performance gains.
  • Spatial and temporal streams: Both spatial-only and temporal-only settings perform worse than late fusion on UCSD Ped1.
  • Spatial and temporal streams: The temporal stream detects some appearance anomalies but misses skaters standing on skates, bikers on bikes, and people walking across the walkway or grass.
  • Spatial and temporal streams: The temporal stream identifies patches missed by the spatial stream, including the skater and biker examples.
  • Spatial and temporal streams: Late fusion combines motion and appearance cues to mitigate spatial-stream misses and temporal-stream false detections caused by complex motion and occlusion.

V. CONCLUSION

The paper presents a partially supervised deep learning methodology for detecting and locating anomalous events in surveillance videos using a two-stream framework.

  • The methodology uses RGB frames and dynamic flows in a two-stream framework for surveillance-video anomaly detection and localization.
Loading 1805.11223v1…