PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction

Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, Guiqing Li

arXiv:2108.06852v1cs.CV

TL;DR

Video anomaly detection must identify rare, unbounded abnormal events from normal-data training. HF2-VAD combines multi-level memory-based flow reconstruction with CVAE future-frame prediction conditioned on reconstructed flows, and reports superior benchmark performance, with a stated failure for distant objects.

  • Problem

    Video anomaly detection is difficult because abnormal events are rare and their forms are unbounded, making comprehensive abnormal-data collection impractical.

  • Method

    HF2-VAD reconstructs optical flows with ML-MemAE-SC, then uses those flows and previous frames in a CVAE to predict the next frame.

  • Results

    HF2-VAD outperforms previous reconstruction-only, prediction-only, and hybrid methods; its best hybrid ablation reaches 99.31% AUROC on UCSD Ped2.

  • Takeaways & Limitations

    Reconstructed-flow quality provides a mechanism for amplifying abnormality signals through both flow-reconstruction and future-frame-prediction errors.

  • Takeaways & Limitations

    On ShanghaiTech, HF2-VAD fails to detect a running person reliably when the person is far from the camera; the authors identify scene depth as a needed variable.

Abstract

from arXiv · show

In this paper, we propose $\text{HF}^2$-VAD, a Hybrid framework that integrates Flow reconstruction and Frame prediction seamlessly to handle Video Anomaly Detection. Firstly, we design the network of ML-MemAE-SC (Multi-Level Memory modules in an Autoencoder with Skip Connections) to memorize normal patterns for optical flow reconstruction so that abnormal events can be sensitively identified with larger flow reconstruction errors. More importantly, conditioned on the reconstructed flows, we then employ a Conditional Variational Autoencoder (CVAE), which captures the high correlation between video frame and optical flow, to predict the next frame given several previous frames. By CVAE, the quality of flow reconstruction essentially influences that of frame prediction. Therefore, poorly reconstructed optical flows of abnormal events further deteriorate the quality of the final predicted future frame, making the anomalies more detectable. Experimental results demonstrate the effectiveness of the proposed method. Code is available at \href{https://github.com/LiUzHiAn/hf2vad}{https://github.com/LiUzHiAn/hf2vad}.

1. Introduction

HF2-VAD addresses unsupervised video anomaly detection by combining optical-flow reconstruction with reconstructed-flow-guided future-frame prediction. Its memory-based flow reconstruction and CVAE pipeline amplifies anomaly signals, and experiments report improved detection performance.

  • Motivation: Video anomaly detection identifies events that deviate from expected behavior, typically by training unsupervised models on normal data.Abnormal events are rare and unbounded, making comprehensive abnormal-data collection impractical.
  • Proposed framework: HF2-VAD combines optical-flow reconstruction and flow-guided future-frame prediction in a unified anomaly-detection framework.The CVAE receives previous video frames and reconstructed optical flows rather than original flows.
  • Flow reconstruction: ML-MemAE-SC uses multi-level memory modules and skip connections to memorize normal flow patterns while increasing reconstruction errors for abnormal inputs.The memories operate at different feature levels, while skip connections compensate for information compression.
  • Flow-guided prediction: The CVAE predicts future frames from previous frames and reconstructed flows, so poorer abnormal-flow reconstruction enlarges the prediction error.Both flow reconstruction and frame prediction errors are used as anomaly cues.
  • Experiments: Extensive experiments on three public datasets show that HF2-VAD achieves better anomaly-detection performance than state-of-the-art methods.The supplied evidence reports the cross-dataset conclusion without specifying the three dataset names or aggregate metric.

2. Related work

Prior VAD work centers on reconstruction and prediction paradigms, while memory modules and conditional variational models provide foundations for HF2-VAD. The proposed method extends bottleneck memory toward multi-level flow modeling and uses reconstructed flows for prediction.

  • Video anomaly detection: Deep VAD methods mainly follow reconstruction-based or future-prediction-based paradigms, with hybrid approaches combining both but remaining imperfect.The related-work discussion distinguishes these two mainstream approaches from newer hybrid methods.
  • Memory networks: Memory-augmented autoencoders store normal patterns to reduce the reconstruction quality of anomalous inputs.Earlier memory modules were placed at the bottleneck, whereas HF2-VAD extends memory across feature levels with skip connections for optical-flow reconstruction.
  • Connection to HF2-VAD: HF2-VAD combines multi-level memory-based flow reconstruction with CVAE frame prediction, extending the two related-work foundations into one pipeline.The method reconstructs optical flows before using them in prediction.
  • VAE and CVAE: A CVAE models an output distribution conditioned on observations through recognition, conditional-prior, and generation networks.Its ELBO objective provides the variational training framework for conditional generation.

3. Methodology

HF2-VAD reconstructs optical flow with a multi-level memory autoencoder, then uses the reconstructed flows to guide CVAE-based future-frame prediction. It detects anomalies by combining flow reconstruction and frame prediction errors.

  • Framework overview: HF2-VAD trains ML-MemAE-SC for flow reconstruction and CVAE for future-frame prediction on normal data.At test time, both reconstruction and prediction errors contribute to anomaly detection.
  • Flow reconstruction: ML-MemAE-SC places memory modules across multiple autoencoder levels and adds skip connections to preserve information lost through memory filtering.The design uses three encoding-decoding levels plus a bottleneck, with memory modules in decoder levels.
  • Flow reconstruction: The outermost skip connection is omitted because it could let high-level encoding-decoding bypass the lower-level memories.Such bypassing would make the other encoding, decoding, and memory blocks ineffective.
  • Flow reconstruction: Each memory module represents input features as weighted combinations of similar memory slots, encouraging memorization of normal patterns.Training reconstructs normal inputs with an ℓ2 loss and adds an entropy loss on memory-slot matching probabilities.
  • Future-frame prediction: The CVAE models future frames conditioned on previous optical flows and frames, using flow-derived priors and frame-flow posteriors to generate the next frame.Its training objective combines the variational loss with a gradient loss; deterministic testing uses the posterior mean for z.
  • Anomaly scoring: The final anomaly score fuses normalized flow reconstruction and future-frame prediction errors with weighted coefficients.The two components are respectively based on ||ŷ1:t − y1:t||2 and ||x̂t+1 − xt+1||2.

4. Experiments

Experiments evaluate HF2-VAD on MNIST and three video anomaly detection benchmarks using AUROC, qualitative comparisons, ablations, and runtime analysis. The results show stronger anomaly discrimination from the hybrid design, while a far-away running object exposes a scene-depth limitation.

  • 4.1. Toy experiments on MNIST: On MNIST, ML-MemAE-SC progressively improves normality learning over MemAE, with paired memories and skip connections reconstructing normal digits clearly while mapping abnormal digits toward “2”.Adding memories alone can over-filter information, whereas matched memory–skip-connection pairs preserve normal detail and better suppress abnormal inputs.
  • 4.2. Video Anomaly Detection Experiments: HF2-VAD is evaluated on UCSD Ped2, CUHK Avenue, and ShanghaiTech using frame-level AUROC, where higher AUROC indicates better VAD accuracy.The experiments compare against reconstruction, prediction, and hybrid methods, including variants without frame prediction or flow reconstruction.
  • 4.2. Video Anomaly Detection Experiments: 91.1% AUROC on CUHK Avenue exceeds the best previous hybrid result, 86.9% by AMC, while HF2-VAD achieves the best results across all three benchmarks.The full model combines complementary strengths: flow reconstruction is relatively more useful on Ped2, while frame prediction is relatively more useful on Avenue and ShanghaiTech.
  • 4.3. Results: In qualitative anomaly curves, HF2-VAD produces lower, more stable scores in normal sections and better recognizes abnormal events than MNAD-R, MNAD-P, and VEC.Figure 5 compares sampled frames, abnormal ground truth, and method-specific anomaly curves; larger curve values indicate greater anomaly likelihood.
  • 4.3. Results: HF2-VAD yields smaller frame-prediction differences for normal images and larger differences for abnormal images than the compared methods.For a running person, reconstructed abnormal flow becomes inconsistent with input images, producing pixel shifting and color confusion in the predicted future frame.
  • 4.4. Discussion: 99.31% AUROC is achieved by the hybrid ML-MemAE-SC-with-three-memories plus CVAE variant, exceeding reconstruction-only and prediction-only variants in the ablation study.Reconstruction-only scores are 98.81%, 97.75%, and 96.27%; CVAE improves prediction-only performance from 89.96% to 94.48%.
  • 4.4. Discussion: HF2-VAD runs at about 10fps, but its anomaly score decreases for a running object farther from the camera because absolute optical-flow values resemble those of nearby normal objects.The authors conjecture that scene depth should be incorporated to improve ShanghaiTech VAD accuracy.

5. Conclusion

HF2-VAD combines flow reconstruction with flow-guided future-frame prediction, with reconstruction influencing prediction quality rather than merely preceding it. Experiments report superiority over reconstruction-only, prediction-only, and prior hybrid methods.

  • HF2-VAD integrates reconstruction and prediction into a hybrid video anomaly detection framework.
  • The method uses ML-MemAE-SC to preprocess optical flows before CVAE-based future-frame prediction.
  • Reconstructed flows influence prediction quality, creating entanglement between the two stages rather than a simple combination.
  • Experiments show the method outperforms previous reconstruction-only and prediction-only methods and is strongest among hybrid approaches.

A. Detailed Network Design

The detailed design uses a four-level ML-MemAE-SC flow-reconstruction network and a CVAE prediction network with dual encoders, one decoder, and skip connections. Table 3 specifies how original and reconstructed flows are arranged for comparison.

  • ML-MemAE-SC: ML-MemAE-SC contains four levels, with 3×3 kernels throughout its convolutional layers.
  • CVAE: The CVAE uses two similarly structured encoders, Eθ and Fφ, together with one decoder, Dψ.
  • CVAE: Skip connections between Fφ and Dψ help generate the future frame x_t+1.
  • Flow inputs: Table 3 organizes variants by which temporal flows are original and which are reconstructed before entering the CVAE.

B. Sampling strategies during test time

The study compares stochastic posterior sampling with deterministic sampling using the posterior mean during CVAE testing. On UCSD Ped2, deterministic sampling achieves the reported AUROC of 99.3078%.

  • The CVAE is tested with random posterior sampling and deterministic sampling using the posterior mean.
  • 99.3078% AUROC is obtained on UCSD Ped2 with deterministic posterior-mean sampling.
  • Because CVAE generation depends on latent codes, different testing sampling strategies can produce different outputs.

C. Number of reconstructed flows to CVAE

The experiment varies how many of the four temporal flows are reconstructed before CVAE prediction. Using all four reconstructed flows achieves the best reported VAD performance on Ped2.

  • The method uses four previous frames and four corresponding optical flows, with t = 4 in the experiments.
  • Four variants compare different mixtures of original and reconstructed flows supplied to the CVAE.
  • Using all four reconstructed flows yields the best VAD performance among the tested variants on Ped2.
  • Figure 9 presents the detailed CVAE architecture used for flow-guided future-frame prediction.

D. Evaluation on UCF Crime

The paper evaluates its method on surveillance videos with static backgrounds and foreground-object anomalies, while also testing whole-video processing on UCF-Crime.

  • The three main VAD datasets contain static-background surveillance videos where anomalies arise from dynamic foreground objects.
  • The method extracts spatiotemporal cubes and processes each foreground object separately on these datasets.
  • The method can also process entire video frames, which the paper evaluates using UCF-Crime.
  • The UCF-Crime experiment uses 10 videos for training and 6 videos for testing.

E. Anomaly Detecting Cases

The paper presents additional anomaly-detection visualizations for UCSD Ped2, CUHK Avenue, and ShanghaiTech using framewise anomaly curves and AUROC comparisons.

  • Additional examples cover UCSD Ped2, CUHK Avenue, and ShanghaiTech anomaly-detection videos.
  • Red boxes in the video frames mark ground-truth abnormal objects.
  • Each example plots the anomaly score of every frame over time.
  • AUROC is calculated under different model settings, with higher AUROC indicating better anomaly-detection accuracy.

F. More Qualitative Examples

Qualitative results visualize flow reconstruction and future-frame prediction across three datasets, showing how abnormal motion and frames produce larger errors.

  • ML-MemAE-SC reconstructs normal flow patches well but fails to reconstruct abnormal patches accurately.
  • Conditioning prediction on reconstructed motion makes abnormal future frames significantly different from the actual future frames.
  • The examples include bicycle riding, wrong-direction movement, and throwing a backpack in Ped2, Avenue, and ShanghaiTech videos.
  • Figure 10 uses time on the horizontal axis and anomaly score on the vertical axis, where higher values indicate more likely abnormal events.
  • For each dataset, the visualizations present ground-truth flow, reconstructed flow, ground-truth future frame, predicted future frame, and prediction-error map.
  • In Figure 11 difference maps, lighter colors denote larger prediction errors, with normal samples above abnormal samples.
Loading 2108.06852v1…