PaperlayerView exact revision on arXiv

Source-linked AI summary

Future Frame Prediction Using Convolutional VRNN for Anomaly Detection

Yiwei Lu, Mahesh Kumar Krishna Reddy, Seyed shahabeddin Nabavi, Yang Wang

arXiv:1909.02168v2cs.CV

TL;DR

Video anomaly detection faces broad definitions of abnormality and a shortage of labeled examples. The paper combines ConvLSTM and VAE for future-frame prediction, and reports stronger performance than existing state-of-the-art methods on three benchmark datasets.

  • Problem

    Anomalies are broadly defined, and collecting labeled video data covering all possible cases is infeasible.

    Source: §1 · p. 1

  • Method

    The paper combines ConvLSTM with VAE in a sequential generative model that predicts a future frame for anomaly detection.

    Source: §1 · pp. 1–2

  • Results

    On three benchmark datasets, the proposed approach outperforms existing state-of-the-art methods.

    Source: §6 · p. 6

  • Takeaways & Limitations

    The authors conclude that combining ConvLSTM and VAE can effectively capture temporal information for future-frame prediction.

    Source: §6 · p. 6

    The prior GAN-based method [15] directly concatenates observed frames and uses optical flow only as a training constraint, not during testing.

    Source: §1 · p. 1

Abstract

from arXiv · show

Anomaly detection in videos aims at reporting anything that does not conform the normal behaviour or distribution. However, due to the sparsity of abnormal video clips in real life, collecting annotated data for supervised learning is exceptionally cumbersome. Inspired by the practicability of generative models for semi-supervised learning, we propose a novel sequential generative model based on variational autoencoder (VAE) for future frame prediction with convolutional LSTM (ConvLSTM). To the best of our knowledge, this is the first work that considers temporal information in future frame prediction based anomaly detection framework from the model perspective. Our experiments demonstrate that our approach is superior to the state-of-the-art methods on three benchmark datasets.

1. Introduction

Video anomaly detection is difficult because anomalies are broad and labeled examples are hard to collect. The paper combines sequential modeling with generative prediction to detect anomalies from future-frame prediction errors.

  • Anomalies are broadly defined, and collecting labels that cover all possible cases is infeasible, motivating unsupervised approaches.Source: §1 · p. 1
  • A future-frame prediction error can flag an anomaly when the predicted frame differs from the actual frame.Source: Figure 1 · p. 1, §1 · p. 2
  • Unlike prior GAN-based prediction [15], the proposed approach combines ConvLSTM and VAE in an end-to-end sequential generative model.Source: §1 · pp. 1–2
  • The proposed model outperforms existing state-of-the-art approaches, even without optical flow features.Source: §1 · p. 2

2. Related Work

Prior video-anomaly research includes hand-crafted and learned features, reconstruction and prediction models, and generative approaches. The paper situates its work at the intersection of sequential modeling and generative models for video.

  • Hand-crafted approaches use trajectories, low-level features, dictionaries, and statistical models, with reported limits in crowded scenes and representing normal events.Source: §2 · p. 2
  • Deep-learning methods often detect anomalies through frame-reconstruction error, while other approaches combine appearance and motion information.Source: §2 · p. 2
  • Prior future-frame models use ConvLSTM or autoencoder-ConvLSTM, while deterministic prediction is reported to struggle with highly structured moving objects [22].Source: §2 · p. 2
  • Sequential generative models had been used for speech, handwriting, or music, but their potential in computer vision had not been explored [3] [23].Source: §2 · p. 2

3. Background

The background introduces VAE as a latent-variable generative model and VRNN as its recurrent extension for sequential data. The proposed model overview organizes processing into four stages at each time step.

3.1. Variational Autoencoder

  • VAE encodes an input into a latent variable and decodes from that variable; its objective includes a prior-posterior KL divergence term.Source: §3.1 · pp. 2–3
  • At each time step, the model overview lists prior calculation, posterior and latent-variable encoding, recurrence, and prediction decoding.Source: Figure 2 · p. 3

3.2. Variational Recurrent Neural Network

  • VRNN extends VAE with a recurrent neural network to model sequential data, which vanilla VAE cannot directly handle.Source: §3.2 · p. 3

4. Approach

The approach predicts a future video frame from observed frames and treats a large prediction error as evidence of anomaly. Conv-VRNN combines a VAE with ConvLSTM to model temporal information, then converts prediction losses into frame-level anomaly scores.

4. Approach

  • Given a sequence of T observed frames, the model predicts frame x(T + 1), and a sufficiently large prediction difference marks that frame as anomalous.The paper sets T to 4 in its formulation.Source: §4 · p. 3

4.1. Conv-VRNN for Future Frame Prediction

  • Conv-VRNN uses a previous hidden state to form a VAE prior, encodes the current frame and hidden state into a latent variable, and combines latent and image features in ConvLSTM before decoding the next frame.The latent variable is resized to match the feature-map dimensions used by the recurrent module.Source: §4.1 · pp. 3–4
  • Training combines L1, multi-scale SSIM, and gradient-difference losses for frame quality with a KL-divergence term for the latent distributions.The frame-prediction objective minimizes the difference between predicted and actual future frames.Source: §4 · p. 3, §4.1 · p. 4

4.2. Anomaly Detection

  • At test time, the model predicts the next frame, compares it with the ground truth, and normalizes prediction losses to [0, 1] as anomaly scores.A higher score indicates greater likelihood that the frame is anomalous.Source: §4.2 · p. 4

5. Experiments

The experiments evaluate Conv-VRNN on three benchmark datasets and report quantitative and qualitative results, followed by ablations of temporal modeling, loss terms, and optical-flow features.

5.1. Experimental Setup

  • The evaluation uses UCSD Ped1, UCSD Ped2, and CUHK Avenue, with frame-level AUC as the metric.A higher AUC represents a more accurate anomaly detection system.Source: §5.1 · p. 5

5.2. Experimental Results

  • Conv-VRNN outperforms existing methods on all three datasets.Table 1 compares methods by AUC on UCSD Ped1, UCSD Ped2, and CUHK Avenue.Source: Table 1 · p. 5, §5.2 · p. 5
  • Normal-frame predictions tend to be close to the actual future frame, while anomalous-frame predictions tend to be blurry or distorted.Bounding boxes in the examples are for visualization and are not model predictions.Source: §5.2 · p. 5, Figure 4 · p. 6

5.3. Ablation Study

  • The paper includes additional ablation studies to gain further insight into the proposed methods.Source: §5.3 · p. 5

5.3.1 Conv-VAE vs Conv-VRNN

  • Conv-VRNN outperforms Conv-VAE, the variant without an RNN, in the temporal-information ablation.Conv-VAE was tested with either one input frame or four frames; Table 2 compares the variants by AUC across three datasets.Source: §5.3.1 · p. 5, Table 2 · p. 5

5.3.2 Analysis on Losses

  • On Ped1, the combination of intensity, structure, and gradient losses gives the best performance among the tested loss combinations.The tested objectives use intensity alone, intensity plus structure, or all three constraints; more constraints better capture appearance information.Source: Table 3 · p. 6, §5.3.2 · p. 6

5.3.3 Sequential Model vs Optical Flow

  • Conv-VRNN outperforms Conv-VAE with optical flow, despite not using optical-flow features.The optical-flow comparison uses a pretrained FlowNet loss as a motion constraint during training.Source: Table 4 · p. 6, §5.3.3 · p. 6

6. Conclusion

The paper proposes a sequential generative network for anomaly detection that combines ConvLSTM with VAE in a future-frame-prediction framework. It reports that the approach outperforms state-of-the-art methods on three benchmark datasets.

  • The proposed sequential generative network uses convolutional VRNN for anomaly detection within a future-frame-prediction framework.Source: §6 · p. 6
  • Combining ConvLSTM with VAE is presented as a way to capture temporal information important for future-frame prediction.Source: §6 · p. 6
  • On three benchmark datasets, the approach outperforms existing state-of-the-art methods.Source: §6 · p. 6
Loading 1909.02168v2…