PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection

Chen Zhang, Guorong Li, Yuankai Qi, Shuhui Wang, Laiyun Qing, Qingming Huang, Ming-Hsuan Yang

arXiv:2212.04090v1cs.CV

TL;DR

Weakly supervised video anomaly detection must predict frame-level anomaly scores from video-level labels, and existing pseudo-label methods face completeness and uncertainty challenges. The paper combines a multi-head diversity-based generator with iterative uncertainty-aware refinement. It reports favorable performance on UCF-Crime, TAD, and XD-Violence.

  • Problem

    Weakly supervised video anomaly detection must predict frame-level anomaly scores from video-level labels, while MIL pseudo-label generation can miss abnormal clips and noisy labels can hamper final-classifier training.

    Source: §1 · p. 2

  • Method

    The method generates pseudo labels with a multi-head classifier and diversity loss, then iteratively uses uncertainty estimates to select reliable clips and refine labels.

    Source: §1 · p. 2

  • Results

    Experiments on UCF-Crime, TAD, and XD-Violence demonstrate favorable performance compared with several state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The conclusion presents completeness-aware generation and uncertainty-aware refinement as a two-stage approach for improving pseudo-label quality in weakly supervised video anomaly detection.

    Source: §5 · p. 9

Abstract

from arXiv · show

Weakly supervised video anomaly detection aims to identify abnormal events in videos using only video-level labels. Recently, two-stage self-training methods have achieved significant improvements by self-generating pseudo labels and self-refining anomaly scores with these labels. As the pseudo labels play a crucial role, we propose an enhancement framework by exploiting completeness and uncertainty properties for effective self-training. Specifically, we first design a multi-head classification module (each head serves as a classifier) with a diversity loss to maximize the distribution differences of predicted pseudo labels across heads. This encourages the generated pseudo labels to cover as many abnormal events as possible. We then devise an iterative uncertainty pseudo label refinement strategy, which improves not only the initial pseudo labels but also the updated ones obtained by the desired classifier in the second stage. Extensive experimental results demonstrate the proposed method performs favorably against state-of-the-art approaches on the UCF-Crime, TAD, and XD-Violence benchmark datasets.

1. Introduction

Weakly supervised video anomaly detection must locate abnormal events at frame level using only video-level labels, while existing methods can miss abnormal clips beyond the most anomalous one. The paper addresses this with more complete pseudo labels and iterative uncertainty-aware refinement, reporting favorable performance on three benchmarks.

  • Weak supervision provides only video-level labels, while testing requires frame-level anomaly scores to locate sparse abnormal events.Source: §1 · p. 1
  • MIL-based ranking can focus on the most likely abnormal clip in a positive video, leaving other abnormal clips unrepresented in pseudo labels.The paper identifies this as a limitation of pseudo-label completeness.Source: Figure 1 · p. 1, §1 · p. 2
  • A multi-head classifier with a diversity loss encourages different heads to detect different abnormal events and cover more events in pseudo labels.Source: §1 · p. 2
  • Iterative uncertainty-aware training selects lower-uncertainty clips to refine initial pseudo labels and later outputs of the desired classifier.The paper measures uncertainty using Monte Carlo Dropout.Source: §1 · p. 2
  • Experiments on UCF-Crime, TAD, and XD-Violence demonstrate favorable performance compared with several state-of-the-art methods.Source: §1 · p. 2

2. Related Work

Related work spans semi-supervised approaches and weakly supervised methods, especially MIL and two-stage self-training. Prior self-training methods refine features or anomaly scores, while the paper identifies their lack of uncertainty handling as a concern.

  • Semi-supervised video anomaly detection uses normal-only training data and includes one-class classification and reconstruction-based approaches.Reconstruction methods use reconstruction error to discriminate abnormality; some model normal-video dictionaries or sequences.Source: §2 · p. 2
  • Weakly supervised methods train with normal and abnormal videos but lack frame-level annotations, and many use MIL-based ranking losses.Prior work also adds inner-bag loss, motion-aware features, top-k MIL, context encoding, or diversified MIL losses.Source: §2 · p. 2
  • Two-stage self-training methods generate pseudo labels before refining features or anomaly scores, including MIST [5] and MSL.Source: §2 · pp. 2–3
  • Prior self-training mechanisms do not account for pseudo-label uncertainty, which the paper says can lead to a deviating process guided by noisy labels.The proposed uncertainty-aware strategy is described as reducing the effect of unreliable pseudo labels.Source: §2 · p. 3

3. Method

The method first generates more complete segment-level pseudo labels with a multi-head classifier, then iteratively refines them using uncertainty-based clip selection to train the final classifier.

3.1. Notations and Preliminaries

  • The training setup uses video-level labels and represents each video with segment features formed by averaging consecutive clip features.Input clips are non-overlapping 16-frame clips with features extracted by a pretrained 3D convolutional network.Source: §3.1 · p. 4

3.2. Completeness of Pseudo Labels

  • Parallel classifier heads produce distinct anomaly-score distributions through a diversity loss, with norm regularization balancing the heads before their scores form segment labels.The heads operate on video-segment features; their average scores are passed through a sigmoid.Source: §3.2 · p. 4
  • A MIL ranking loss constrains the highest anomaly score in abnormal videos to exceed the highest score in normal videos.Source: §3.2 · p. 4

3.3. Uncertainty of Pseudo Labels

  • MC Dropout estimates clip-label uncertainty, and the process repeatedly selects each video's least-uncertain R% of clips to train and refine the clip classifier.The multi-head model supplies the first iteration; the clip classifier supplies later iterations.Source: §3.3 · pp. 4–5

3.4. Model Training and Inference

  • The two-stage training process progressively refines clip-level labels, and the trained clip classifier directly predicts anomaly scores for test videos.Source: §3.4 · pp. 5–6

4. Experimental Results

The experiments evaluate the method on UCF-Crime, TAD, and XD-Violence, using frame-level AUC on the first two datasets and AP on the third. The method outperforms or compares favorably with prior approaches across the three benchmarks, while ablations, hyperparameter analyses, and qualitative examples examine its components and predictions.

4.1. Datasets and Evaluation Metrics

  • The evaluation uses UCF-Crime, TAD, and XD-Violence; frame-level AUC is reported for the first two and AP for XD-Violence.The datasets provide video-level training labels and frame-level test labels; XD-Violence also provides audio-visual signals.Source: §4.1 · p. 6, §4 · p. 6

4.2. Implementation Details

  • The implementation extracts I3D features from 16 consecutive frames and divides each video into 32 segments; XD-Violence also uses VGGish audio features.The multi-head classifier uses K = 2 heads, and initial pseudo-label generation uses M = 50 stochastic forward passes; refinement uses M = 5.Source: §4.2 · p. 6

4.3. Comparisons with Prior Work

  • On UCF-Crime and TAD, the method outperforms prior approaches; on XD-Violence it reaches 78.74% AP and improves 2.79% over Wu et al. [34] with fused features.With the same I3D features, the UCF-Crime AUC gain is 0.92% over the best previous two-stage method, and the TAD AUC gains are 2.40% over MIST [5] and 2.02% over RTFM [29].Source: Table 1 · p. 6, §4.3 · pp. 6–7, Table 2 · p. 7

4.4. Ablation Study

  • 2.03% AUC improvement accompanies the completeness-focused pseudo-label generator, and iterative uncertainty-aware refinement adds 1.83% AUC over the baseline on UCF-Crime.Across iterations, uncertainty-aware self-training improves rapidly and then stabilizes, while standard self-training yields a much smaller improvement.Source: §4.4 · pp. 7–8, Table 4 · p. 7, Figure 3 · p. 7

4.5. Hyperparameter Analysis

  • On UCF-Crime, K = 2 gives the best reported head-count performance, while the AUC rises as α increases to 10 and declines at 100.Performance generally converges after 3–5 self-training iterations; the chosen counts are 3 for UCF-Crime and XD-Violence and 5 for TAD. The sample ratio R = 0.5 is optimal on UCF-Crime, with larger ratios for larger training sets and smaller ones for smaller sets.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6. Qualitative Results

  • Qualitative examples show relatively accurate scores for multi-segment UCF-Crime and long-term TAD anomalies, two detected XD-Violence abnormal intervals, and near-zero scores for a normal video.Figure 5 marks abnormal intervals with pink squares and plots predicted frame anomaly scores in blue.Source: Figure 5 · p. 8, §4.6 · pp. 8–9

5. Conclusions

The method enhances pseudo-label quality through diversity-driven coverage and iterative uncertainty-aware refinement, with favorable results on three datasets.

5. Conclusions

  • Experiments on UCF-Crime, TAD, and XD-Violence show favorable performance, while ablations support exploiting pseudo-label completeness and uncertainty.The conclusion reports no specific metrics or baselines.Source: §5 · p. 9
  • A multi-head classification module constrained by diversity loss generates pseudo labels intended to cover as many anomalous events as possible.The heads are used to encourage different event coverage.Source: §5 · p. 9
  • An iterative uncertainty-aware strategy selects reliable samples to train the clip classifier and gradually refines its output into pseudo labels for the final classifier.Source: §5 · p. 9
Loading 2212.04090v1…