PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning

Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W. Verjans, Gustavo Carneiro

arXiv:2101.10030v3cs.CV

TL;DR

Weakly supervised video anomaly detection must identify rare abnormal snippets despite dominant normal snippets, subtle differences, and neglected temporal dependencies. RTFM learns robust temporal feature magnitudes and combines dilated convolutions with self-attention, improving MIL recognition. Across four benchmarks, it outperforms state-of-the-art methods and improves subtle-anomaly discriminability and sample efficiency.

  • Problem

    Weakly supervised MIL can misidentify rare abnormal snippets because abnormal videos contain many normal snippets, subtle anomalies resemble normal events, and temporal dependencies may be ignored.

  • Method

    RTFM learns a temporal feature-magnitude mapping and uses pyramid dilated convolutions with temporal self-attention to model snippet dependencies within an MIL framework.

  • Results

    RTFM outperforms current state-of-the-art methods by a large margin on ShanghaiTech, UCF-Crime, XD-Violence, and UCSD-Peds, with improved subtle-anomaly discriminability and sample efficiency.

  • Takeaways & Limitations

    Feature magnitude learning improves positive-instance recognition, discriminative feature learning for subtle anomalies, and exploitation of abnormal training data in MIL.

  • Takeaways & Limitations

    RTFM assumes that abnormal snippets have mean feature magnitude at least as large as normal snippets.

Abstract

from arXiv · show

Anomaly detection with weakly supervised video-level labels is typically formulated as a multiple instance learning (MIL) problem, in which we aim to identify snippets containing abnormal events, with each video represented as a bag of video snippets. Although current methods show effective detection performance, their recognition of the positive instances, i.e., rare abnormal snippets in the abnormal videos, is largely biased by the dominant negative instances, especially when the abnormal events are subtle anomalies that exhibit only small differences compared with normal events. This issue is exacerbated in many methods that ignore important video temporal dependencies. To address this issue, we introduce a novel and theoretically sound method, named Robust Temporal Feature Magnitude learning (RTFM), which trains a feature magnitude learning function to effectively recognise the positive instances, substantially improving the robustness of the MIL approach to the negative instances from abnormal videos. RTFM also adapts dilated convolutions and self-attention mechanisms to capture long- and short-range temporal dependencies to learn the feature magnitude more faithfully. Extensive experiments show that the RTFM-enabled MIL model (i) outperforms several state-of-the-art methods by a large margin on four benchmark data sets (ShanghaiTech, UCF-Crime, XD-Violence and UCSD-Peds) and (ii) achieves significantly improved subtle anomaly discriminability and sample efficiency. Code is available at https://github.com/tianyu0207/RTFM.

1. Introduction

Weakly supervised video anomaly detection uses video-level labels to identify abnormal snippets, but rare anomalies can be overwhelmed by normal snippets and difficult to separate from them. RTFM addresses these MIL weaknesses by learning temporal feature magnitudes and modeling temporal dependencies, improving detection and benchmark performance.

  • Weakly supervised detection uses video-level normal or abnormal labels, reducing annotation effort compared with one-class approaches.
  • Abnormal videos contain mostly normal snippets, while subtle anomalies may closely resemble normal events, complicating positive-instance identification.
  • MIL balances bags by selecting top-scoring abnormal-video snippets and random normal-video snippets, but top selections can be normal and introduce label noise.
  • RTFM learns snippet feature magnitudes, assuming abnormal snippets have larger mean magnitudes, to improve abnormal-instance selection and address MIL weaknesses.It also supports selecting more abnormal snippets and harder negatives for classifier training.
  • RTFM combines pyramid dilated convolutions and temporal self-attention to capture long- and short-range dependencies, then outperforms current methods across four benchmarks.The evaluated datasets are ShanghaiTech, UCF-Crime, XD-Violence, and UCSD-Peds, using C3D and I3D features.

2. Related Work

Video anomaly detection has shifted from unsupervised normal-only training toward weak supervision with cheaper video-level annotations. MIL-based methods dominate this setting but can suffer from noisy positive bags when normal snippets are selected as abnormal.

  • Unsupervised anomaly detection trains on normal data using one-class classification, deep features, compact normality representations, or generative reconstruction.
  • Weakly supervised methods use labelled abnormal samples and video-level annotations because frame-level labels are expensive to obtain.
  • UCF-Crime introduced a large-scale weakly supervised video anomaly detection dataset based on video-level labels.
  • Most weakly supervised methods use MIL, but normal snippets mistakenly selected as top abnormal events create label noise in positive bags.

3. The Proposed Method: RTFM

RTFM learns snippet representations whose feature magnitudes separate abnormal from normal videos, using top-k instances for robust MIL classifier training. Its multi-scale temporal network combines dilated convolutions, self-attention, and skip connections to model temporal dependencies.

  • 3.1. Theoretical Motivation of RTFM: RTFM uses temporal feature magnitude, with lower-magnitude snippets representing normal events and higher-magnitude snippets representing abnormal events.The approach assumes the mean magnitude of abnormal snippets is at least that of normal snippets.
  • 3. The Proposed Method: RTFM: The RTFM pipeline maps a T × D snippet-feature matrix to temporal features, then optimizes feature separability and trains a classifier using top-k abnormal and normal snippets.The temporal extractor and snippet classifier are jointly optimized in the model formulation.
  • 3.1. Theoretical Motivation of RTFM: For k ≤ µ, increasing the number of selected abnormal-video snippets increases expected separability even when some selected snippets are normal.The theorem also states that selecting more than µ top instances makes abnormal and normal video scores indistinguishable.
  • 3.2. Multi-scale Temporal Feature Learning: The multi-scale temporal network captures local and global dependencies using a pyramid of dilated temporal convolutions and temporal self-attention.The dilations are 1, 2, and 4; concatenated module outputs are combined with the original features through a skip connection.
  • 3.3. Feature Magnitude Learning: The method maximizes separability between abnormal and normal videos by enforcing a margin between their top-k largest-magnitude snippet features.The resulting top-k features are used to train the snippet classifier.
  • 3.4. RTFM-enabled Snippet Classifier Learning: RTFM selects the k snippets with largest ℓ2-norm features and trains a binary cross-entropy snippet classifier on them.The classifier loss is accompanied by temporal smoothness and sparsity regularization.

4. Experiments

Experiments evaluate RTFM on four weakly supervised video anomaly detection benchmarks using AUC or AP, with comparisons, implementation details, and analyses of temporal modeling, subtle anomalies, and sample efficiency.

  • Datasets and evaluation: RTFM is evaluated on ShanghaiTech, UCF-Crime, XD-Violence, and UCSD-Peds using frame-level AUC, plus AP on XD-Violence.Larger AUC and AP values indicate better performance.
  • Results on ShanghaiTech: RTFM achieves 97.21% AUC with I3D-RGB features on ShanghaiTech, outperforming current SOTA MIL methods by 10% to 14%.It also exceeds one method by more than 5% despite that method using I3D-RGB and I3D Flow.
  • Results on ShanghaiTech: RTFM surpasses the GCN-based method by 11.7% and reaches 91.51% AUC with C3D-RGB features on ShanghaiTech.The authors associate the comparison with the effectiveness of the MTN module for temporal dependencies.
  • Results on UCF-Crime: On UCF-Crime, RTFM exceeds same-feature MIL baselines by 1.59% to 8.62% with I3D-RGB and by 2.95% to 7.87% with C3D features.It also outperforms the alternating-training method with 82.12% AUC by 1.91%.
  • Results on XD-Violence: On XD-Violence, RTFM improves over same-feature weakly supervised methods by 2.13% to 2.4% and achieves 75.89% AUC with C3D features.It surpasses unsupervised approaches by at least 27.03% in AP.
  • Results on UCSD-Peds: On UCSD-Ped2, RTFM surpasses the previous SOTA by 3.2% with TSN-Gray and achieves 98.6% mean AUC with I3D features.The I3D result exceeds Sultani et al. by 6.3%.
  • Sample Efficiency Analysis: RTFM still outperforms Sultani et al. when using 60% fewer labelled abnormal training videos.The authors attribute this to more effective recognition of positive instances in abnormal videos.
  • Analysis: The feature-magnitude module produces small magnitudes for normal snippets and large magnitudes for abnormal snippets while maintaining large anomaly-score margins.Figure 3 covers test videos from UCF-Crime and ShanghaiTech, with pink regions marking manually labelled abnormal events.

5. Conclusion

RTFM enables top-k MIL for weakly supervised video anomaly detection by learning temporal feature magnitudes that identify rare abnormal snippets and separate them from normal snippets. The resulting model improves subtle-anomaly discrimination, sample efficiency, and benchmark performance.

  • RTFM enables top-k MIL approaches for weakly supervised video anomaly detection.
  • RTFM maps snippet features to magnitudes that identify rare abnormal snippets and create a large margin from normal snippets.
  • The learned features improve discrimination of subtle anomalies from hard negative examples.
  • RTFM improves abnormal-data exploitation and sample efficiency compared with current state-of-the-art MIL methods.
  • These capabilities drive state-of-the-art performance on all three large benchmarks.

A.1. Theoretical Motivation of RTFM

The theoretical motivation assumes abnormal snippets have at least as large expected feature magnitude as normal snippets. Under this assumption, magnitude-based separability improves for suitable top-k values but vanishes as k grows without bound, motivating RTFM to enlarge abnormal-feature magnitudes.

  • The theorem assumes abnormal samples have expected feature magnitude at least as large as normal samples.
  • For 0 < k < µ, the expected magnitude-based separability score between abnormal and normal videos grows.
  • As k approaches infinity, the magnitude-based separability score reduces to zero.
  • Because separability depends on abnormal features having larger magnitudes, RTFM is used to enforce that property.

A.2. Multi-scale Temporal Feature Learning

The multi-scale temporal network captures both local multi-resolution dependencies and global dependencies between video snippets.

  • MTN captures multi-resolution local temporal dependencies and global temporal dependencies between video snippets.

A.3. Computational Efficiency

The method is evaluated for real-time operation and processes a 16-frame clip in 0.76 seconds, including I3D feature extraction. Its temporal network combines dilated convolutions with self-attention, whose outputs are concatenated.

  • 0.76 seconds is required to process a 16-frame clip during inference on an Nvidia 2080Ti, including I3D extraction.
  • The reported processing time indicates real-time detection capability in real-world applications.
  • Pyramid dilated convolutions capture local consecutive-snippet dependencies across temporal scales.
  • A self-attention network computes global temporal correlations, and its features are concatenated with convolutional features to produce the MTN output.

A.4. Temporal Dependency

Prior anomaly-detection methods model temporal dependencies using handcrafted motion trajectories, stacked RNNs, future-frame prediction consistency, and convolutional LSTMs. These approaches have been used to capture temporal consistency at different stages of video modeling.

  • Traditional methods convert consecutive frames into handcrafted motion trajectories to capture local consistency between neighbouring frames.
  • Deep anomaly-detection methods have explored stacked RNNs, temporal consistency in future-frame prediction, and convolutional LSTMs for temporal dependency modeling.
  • The surveyed temporal-dependency approaches include both handcrafted motion modeling and recurrent or predictive deep-learning mechanisms.

A.5. Ablations for k and m

On ShanghaiTech, the model’s AUC depends on top-k and margin m: performance peaks at sufficiently large k, flattens near k ≈µ, and remains stable across a broad margin range.

  • Performance peaks at a sufficiently large top-k, flattens around k ≈µ, and then drops as k increases further.
  • Figure S2 plots AUC against top-k on the left and margin m on the right.
  • AUC remains stable at [93%, 96%] for margin m values spanning m ∈[50, 1200].
Loading 2101.10030v3…