PaperlayerView exact revision on arXiv

Source-linked AI summary

MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection

Jia-Chang Feng, Fa-Ting Hong, Wei-Shi Zheng

arXiv:2104.01633v1cs.CV

TL;DR

WS-VAD needs discriminative representations, but existing methods can be hindered by noisy clip pseudo labels and limited task-specific feature refinement. MIST uses a multiple instance pseudo label generator, self-guided attention boosted encoder, and two-stage self-training to refine representations from video-level annotations. It reports significant improvements on two public datasets, including a frame-level AUC of 94.83% on ShanghaiTech.

  • Problem

    Existing WS-VAD methods can begin with seriously noisy clip pseudo labels, while task-specific feature encoder training remains insufficiently efficient.

  • Method

    MIST generates clip-level pseudo labels with multiple instance learning and sparse continuous sampling, then uses self-guided attention and two-stage self-training to refine ESGA.

  • Results

    MIST makes significant improvements on two public datasets and achieves a frame-level AUC of 94.83% on ShanghaiTech.

  • Takeaways & Limitations

    The resulting task-specific encoder provides discriminative representations that can boost other existing methods.

Abstract

from arXiv · show

Weakly supervised video anomaly detection (WS-VAD) is to distinguish anomalies from normal events based on discriminative representations. Most existing works are limited in insufficient video representations. In this work, we develop a multiple instance self-training framework (MIST)to efficiently refine task-specific discriminative representations with only video-level annotations. In particular, MIST is composed of 1) a multiple instance pseudo label generator, which adapts a sparse continuous sampling strategy to produce more reliable clip-level pseudo labels, and 2) a self-guided attention boosted feature encoder that aims to automatically focus on anomalous regions in frames while extracting task-specific representations. Moreover, we adopt a self-training scheme to optimize both components and finally obtain a task-specific feature encoder. Extensive experiments on two public datasets demonstrate the efficacy of our method, and our method performs comparably to or even better than existing supervised and weakly supervised methods, specifically obtaining a frame-level AUC 94.83% on ShanghaiTech.

1. Introduction

MIST addresses weakly supervised video anomaly detection by efficiently learning task-specific discriminative representations from video-level labels. It combines reliable clip-level pseudo labels, self-guided attention, and two-stage self-training, with experiments showing effectiveness on two datasets.

  • Motivation: WS-VAD uses realistic video-level labels, but assigning them to clips creates serious pseudo-label noise that slows representation refinement.Existing encoder-based methods use iterative noise cleaning, yet early noisy pseudo labels can mist each model.
  • Approach: MIST introduces a two-stage self-training procedure with a multiple instance pseudo label generator and a self-guided attention boosted feature encoder.The generator produces clip-level pseudo labels, which are then used to refine ESGA with abnormal and normal videos.
  • Approach: Sparse continuous sampling is designed to produce more reliable pseudo labels while emphasizing context around the most anomalous part.The strategy addresses coarse sampling that can bury abnormal patterns and fine sampling that can ignore anomaly context.
  • Approach: ESGA targets surveillance-video domain differences by focusing on anomalous regions and learning task-specific discriminative representations.The method is motivated by the mismatch between action-recognition features and surveillance videos.
  • Evaluation: MIST is evaluated with C3D and I3D on UCF-Crime and ShanghaiTech, including ablations of sparse sampling and self-guided attention.The experiments also include comparisons with encoder-agnostic methods and visualized results.

2. Related Works

Related work frames WS-VAD around encoder-agnostic and encoder-based methods, while MIST uses multiple instance pseudo labels and two-stage self-training to refine its feature encoder. Its design also connects to broader self-training and weakly supervised video tasks.

  • Weakly supervised video anomaly detection: Weakly supervised VAD uses video-level labels to distinguish abnormal and normal events and includes encoder-agnostic and encoder-based approaches.Encoder-agnostic methods train classifiers over vanilla features, whereas encoder-based methods train feature encoders and classifiers.
  • Novelty relative to prior work: MIST uses learned clip-level pseudo labels to optimize ESGA instead of directly treating video-level labels as clip pseudo labels.Its two-stage scheme optimizes the pseudo label generator and feature encoder rather than relying on iterative optimization.
  • Multiple instance learning: MIST’s multiple instance generator treats videos as bags of clips and uses video-level labels for indirect instance-level supervision.The framework adopts sparse continuous sampling to emphasize context around the most anomalous part.
  • Self-training: Self-training augments labeled data with pseudo labels and can support representation learning and classifier refinement.MIST applies this principle to fine-tune a self-guided attention boosted feature encoder using information from all videos.

3. Approach

MIST uses two-stage self-training to generate clip-level pseudo labels from video-level annotations, then fine-tune a self-guided attention feature encoder for task-specific anomaly representations.

  • Multiple Instance Pseudo Labels: Each video is treated as a bag of clips, with negative bags containing no anomalous instances and positive bags containing at least one.This multiple instance formulation uses video-level labels while modeling clip-level anomaly evidence.
  • Overview: MIST first extracts vanilla-encoder features and trains a multiple instance pseudo label generator to predict clip-level labels for abnormal videos.The generator is optimized with a deep MIL ranking loss.
  • Multiple Instance Pseudo Labels: Sparse continuous sampling uniformly selects L sub-bags of T consecutive clips, balancing fine-grained anomaly localization with temporal context.T also represents the assumed minimum anomaly duration.
  • Self-Training: The trained generator smooths and normalizes clip scores into soft pseudo labels, which are combined with clip-level normal-video labels to fine-tune ESGA.Temporal smoothing uses a moving average filter to reduce score jitter.
  • Self-Guided Attention: ESGA adds self-guided attention to a vanilla encoder so pseudo-label supervision guides attention-map generation toward anomalous regions and task-specific representations.The attention module uses three convolutional encoding units, F1, F2, and F3; its outputs support weighted classification and guided classification heads.
  • Optimization: Cross-entropy losses train ESGA's weighted and guided classification heads using pseudo labels for abnormal videos and clip-level annotations for normal videos.The deep MIL ranking loss also includes a sparsity term so only a few sub-bags may contain anomalies.

4. Experiments

Experiments on UCF-Crime and ShanghaiTech evaluate MIST across datasets, encoders, metrics, comparisons, ablations, and visualizations. MIST achieves strong anomaly-detection performance, improves pretrained features, and benefits from pseudo-labeling, attention, and sparse continuous sampling.

  • Datasets and Metrics: MIST is evaluated on UCF-Crime and ShanghaiTech using C3D or I3D feature encoders.UCF-Crime contains 1,900 long untrimmed surveillance videos; ShanghaiTech uses 238 training and 199 testing videos in the weakly supervised setting.
  • Datasets and Metrics: AUC measures frame-level anomaly discrimination, while FAR evaluates robustness to false alarms.The experiments also report comparisons under different supervision levels, prediction fineness, and test-time augmentation settings.
  • Comparisons with Related Methods: 81.08% versus 81.40% AUC and 2.2% versus 2.19% FAR are reported for MIST versus Zhong et al. using 10-crop augmentation.MIST slightly exceeds Zhong et al. in the stated comparison while avoiding the additional 10-crop computation.
  • Comparisons with Related Methods: 94.83% frame-level AUC and 0.05% FAR are achieved on ShanghaiTech, surpassing other RGB-based methods and AR-Net’s multi-model result.MIST exceeds AR-Net on AUC by more than 4%.
  • Feature Evaluation: MIST fine-tuned features improve encoder-agnostic methods, including AR-Net from 85.38% to 92.27% on UCF-Crime and a 6.89% ShanghaiTech improvement.These results indicate reduced domain gap and more powerful task-specific representations.
  • Ablation Study: Generated pseudo labels, self-guided attention, and sparse continuous sampling each contribute to performance improvements.Self-guided attention raises AUC by 2% and score gap by 5%; sparse continuous sampling gains 2.93% and 6.05% over uniform sampling on ShanghaiTech with two feature types.

5. Conclusions

MIST efficiently fine-tunes a task-specific feature encoder through multiple-instance self-training, producing discriminative representations for video anomaly detection.

  • MIST uses sparse continuous sampling to produce more reliable pseudo labels in its multiple instance pseudo label generator.
  • The self-guided attention module directs the feature encoder toward the most probable anomalous regions in frames.
  • After two-stage self-training, MIST trains a task-specific encoder with discriminative representations that can boost existing methods.
  • MIST reports significant improvements on two public datasets.

A. Comparisons of Action Recognition Datasets and Anomaly Detection Datasets

MIST addresses a domain gap between actor-centered Kinetics-400 features and surveillance videos whose anomalies are often small and low-resolution.

  • Kinetics-400 samples are actor-centered, whereas UCF-Crime samples are not.
  • Anomalies in surveillance-video frames are usually small and low-resolution.
  • MIST minimizes this domain gap by jointly training the feature encoder and classifier in a two-stage self-training scheme.

B.1. Feature Extraction and Sampling

MIST extracts features with pretrained C3D or I3D encoders, applies sparse continuous sampling to form feature bags, and evaluates sampling-related variations.

  • MIST uses C3D pretrained on Sport-1M or I3D pretrained on Kinetics-400 for generator-training feature extraction.
  • The method densely samples 16 frames per clip, except 12 frames per clip for I3D on UCF-Crime.
  • Sparse continuous sampling then selects L · T clips to form bags of features for generator optimization.
  • Figure 10 examines T with 32 sub-bags on ShanghaiTech using I3DRGB features.
  • Figure 11 reports AUC across multiple-detector K values on UCF-Crime with C3D, including training without self-guided attention.
  • A temporal moving-average filter with kernel size k = 5 and min-max normalization refine anomaly scores into normalized outputs.

C.1. Implementation of Self-Guided Attention Module

The self-guided attention module uses convolutional encoding units to generate attention maps and guided classification predictions from intermediate feature maps.

  • The module contains three convolutional encoding units, F1, F2, and F3.
  • F1 applies strided 3D convolutions with ReLU activations, while F2 uses a Sigmoid-activated convolution and F3 uses a 3D convolution.
  • The attention map A is calculated from an intermediate feature map through F1 and F2.
  • The guided classification prediction ˆp is aggregated from M using spatiotemporal and class-specific channel-wise average pooling.
  • The guided prediction is optimized with L2 to guide class-wise discriminative feature-map optimization and indirectly strengthen attention-map generation.

C.2. Implementation of ESGA Finetuning

ESGA finetuning uses class-reweighted cross-entropy to address clip-level class imbalance, with K selected to balance performance and efficiency. After finetuning, ESGA focuses on anomalous frame regions and outperforms Zhong et al.’s encoder-based method.

  • Finetuning objective: Class-reweighted cross-entropy addresses imbalance because only a few clips in abnormal videos are anomalous.The method introduces class weights for abnormal and normal classes.
  • Finetuning objective: The framework uses w0 = 1.2 and w1 = 0.8 on UCF-Crime, and w0 = 0.8 and w1 = 0.65 on ShanghaiTech.
  • Hyperparameter selection: K = 8 is selected because performance peaks at K = 8 or 16, while larger values appear to overfit and reduce efficiency.The choice reflects a trade-off between efficiency and effectiveness.
  • Finetuning outcome: After finetuning, ESGA outperforms Zhong et al.’s state-of-the-art encoder-based method.
  • Finetuning outcome: ESGA’s self-guided attention focuses on anomalous frame regions, while failures occur when anomaly regions are too small.

D.1. Speed and Computational Complexity

MIST is evaluated against Zhong et al. in speed, computational complexity, quantitative performance, spatial localization, and label-noise refinement. The reported comparisons favor MIST’s efficiency and performance, while very small or vague anomaly regions remain a failure setting.

  • Speed and computational complexity: MIST is compared with Zhong et al. on speed and computational complexity in Table 6.
  • Speed and computational complexity: C3D-based MIST has 85 M parameters, 39.26 G FLOPs, and 197.10 FPS; I3D-based MIST has 31 M, 45.68 G, and 324.46 FPS.Compared with Zhong et al.’s 10-crop testing augmentation, MIST is reported as faster and less computationally complex.
  • Quantitative comparisons: MIST shows huge improvement on ShanghaiTech and performs better on UCF-Crime under fair comparison without 10-crop testing.It also performs better at iteration 1 because it does not require iterative optimization.
  • Spatial anomaly localization: MIST’s spatial visualizations perform better than versions without SGA or Hg, but extremely small and vague anomaly regions can fail detection.Figure 13 marks ground-truth spatial annotations with red boxes and separates successful from failure cases.
  • Label-noise refinement: Min-max normalization, temporal smoothing, and larger batches aided label-noise reduction in the reported refinement experiments.The refinement comparison is reported on ShanghaiTech using anomaly-video AUC scores.
  • Feature encoders: RGB-only TSN experiments failed, whereas C3D and I3D spatiotemporal encoders verified MIST’s capacity to represent temporal information.The RGB branch operates on a single frame and therefore fails to capture motion.
  • Application setting: MIST targets online fine-grained anomaly detection, unlike prior coarse-grained approaches that do not meet precise and rapid streaming-surveillance requirements.
Loading 2104.01633v1…