PaperlayerView exact revision on arXiv

Source-linked AI summary

Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detection

Hui Lv, Zhongqi Yue, Qianru Sun, Bin Luo, Zhen Cui, Hanwang Zhang

arXiv:2303.12369v1cs.CV

TL;DR

WSVAD must infer snippet-level anomalies from video-level labels, while MIL can become biased toward confident context shortcuts and mishandle ambiguous snippets. UMIL separates confident and ambiguous snippets, clusters the ambiguous ones, and learns an invariant detector with jointly fine-tuned features. It improves performance on UCF-Crime and TAD, including reported gains of 1.4% AUC and 3.3% AUC over current state-of-the-art methods.

  • Problem

    WSVAD requires snippet-level anomaly detection from video-level labels, but MIL can learn context-biased predictions that fail on ambiguous snippets.

  • Method

    UMIL divides snippets into confident and ambiguous sets, clusters the ambiguous snippets, and jointly trains the detector and feature representation to learn invariant anomaly features.

  • Results

    UMIL improves over current state-of-the-art methods by 1.4% AUC on UCF-Crime and 3.3% AUC on TAD.

  • Takeaways & Limitations

    UMIL learns an unbiased anomaly classifier and tailored representation that achieve state-of-the-art performance on standard WSVAD benchmarks.

  • Takeaways & Limitations

    The detector outputs abnormal probability directly, with normal probability computed as 1 −p.

Abstract

from arXiv · show

Weakly Supervised Video Anomaly Detection (WSVAD) is challenging because the binary anomaly label is only given on the video level, but the output requires snippet-level predictions. So, Multiple Instance Learning (MIL) is prevailing in WSVAD. However, MIL is notoriously known to suffer from many false alarms because the snippet-level detector is easily biased towards the abnormal snippets with simple context, confused by the normality with the same bias, and missing the anomaly with a different pattern. To this end, we propose a new MIL framework: Unbiased MIL (UMIL), to learn unbiased anomaly features that improve WSVAD. At each MIL training iteration, we use the current detector to divide the samples into two groups with different context biases: the most confident abnormal/normal snippets and the rest ambiguous ones. Then, by seeking the invariant features across the two sample groups, we can remove the variant context biases. Extensive experiments on benchmarks UCF-Crime and TAD demonstrate the effectiveness of our UMIL. Our code is provided at https://github.com/ktr-hubrt/UMIL.

1. Introduction

WSVAD must learn snippet-level anomaly detection from video-level labels, but MIL can learn context-biased detectors that trigger false alarms and miss anomalies with different patterns. UMIL addresses this by combining confident and ambiguous snippets to learn invariant features, with fine-grained partitioning and improved benchmark performance.

  • Problem: WSVAD trains a snippet-level anomaly detector using only video-level normal-versus-abnormal labels.Videos are partitioned into snippets; normal videos contain only normal snippets, while abnormal videos contain at least one anomalous snippet.
  • Problem: MIL can focus on simple context shortcuts, producing false alarms on normal snippets and missing anomalies expressed through different contexts.Observed failures include smoke without an explosion, drastic motion without the subtle vandalism behavior, and ambiguous snippets with differing context biases.
  • Method: UMIL divides snippets into confident and ambiguous sets using the current detector, then seeks invariant distinctions across both groups.The ambiguous snippets are clustered into two unsupervised groups so the detector can use information beyond the confident samples that dominate standard MIL.
  • Method: UMIL combines feature fine-tuning and detector learning end to end, yielding a more tailored representation for video anomaly detection.The method also uses fine-grained video partitioning to preserve subtle anomaly information in snippets.
  • Results: UMIL improves over current state-of-the-art methods by 1.4% AUC on UCF-Crime and 3.3% AUC on TAD.It also gains more than 2% AUC over the MIL baseline on both datasets.

2. Related Work

Video anomaly detection research includes unsupervised and weakly supervised settings. Unsupervised methods learn from unlabeled or normal-only data but can produce false alarms for unseen normal patterns, motivating weak supervision with normal and abnormal videos.

  • Research settings: Video anomaly detection research is organized into unsupervised and weakly supervised settings.Unsupervised approaches include unlabeled-data methods, direct train-and-test methods, and one-class classification using normal data only.
  • Unsupervised methods: Unsupervised methods can trigger false alarms for unseen normal patterns because one dataset cannot contain every form of normality.Recent approaches include handcrafted or learned features, self-supervised temporal prediction, and anomaly classifiers trained on normal data.

3. Method

UMIL addresses MIL’s biased sample selection by using confident and ambiguous snippets together. It clusters ambiguous snippets and adds pair-wise supervision so the anomaly detector can reduce context bias while retaining confident-label supervision.

  • Problem setup: WSVAD trains a snippet-level anomaly classifier from video-level labels, with MIL selecting the most anomalous snippet from each video for supervision.The backbone extracts snippet features, while f predicts each snippet’s abnormality probability.
  • MIL limitation: MIL’s max-based training increasingly favors confident snippets, discards ambiguous ones, and produces context-biased anomaly predictions.This selection scheme can bias the detector toward simple contextual cues and miss anomalies with different patterns.
  • Step 1: Divide snippets: UMIL divides snippets into a labeled confident set C and an unlabeled ambiguous set A using prediction history and prediction variance.The N snippets with the least variance form C; the remaining M snippets form A.
  • Step 2: Cluster ambiguous snippets: UMIL clusters A into two groups with a cluster head g trained from pairwise feature similarity, then assigns each ambiguous feature the highest-probability cluster label.Pairs with cosine similarity above threshold τ are treated as similar, and g is trained with BCE on pairwise cluster relations.
  • Step 3: Overall objective: The anomaly head f receives sample-wise supervision from C and pair-wise supervision from A, matching predictions within clusters and separating predictions across clusters.UMIL optimizes the combined objective using BCE(C), BCE(Ag), and BCE(Af), with α and β controlling the auxiliary terms.
  • Training and testing: UMIL initializes from MIL pretraining, jointly optimizes the backbone, anomaly head, and cluster head, and predicts frame-level anomalies using sliding-window snippets.Unlike MIL, the objective also fine-tunes the backbone to obtain a representation tailored to video anomaly detection.

4. Experiments

Experiments on UCF-Crime and TAD evaluate UMIL using frame-level AUC, ablations, ROC curves, and qualitative anomaly-localization cases. UMIL improves overall and abnormal-video performance, while ambiguous-snippet training contributes to gains over MIL.

  • Datasets and Evaluation Metrics: Experiments use UCF-Crime and TAD, with frame-level ROC AUC reported for overall videos (AUCO) and abnormal videos alone (AUCA).Training videos have video-level labels, while test videos have frame-level labels.
  • Main Results: UMIL achieves the best AUCO and AUCA on UCF-Crime, improving by +1.37% and +1.3%, respectively, and improves TAD by +3.3% AUCO and +4.2% AUCA.The reported gains are against the compared methods in Table 1 and Table 2.
  • Main Results: UMIL improves AUCA over MIL by +4.2% on TAD, indicating stronger performance on anomalous snippets beyond gains from easy normal videos.The authors associate this improvement with using ambiguous snippets to learn an invariant classifier.
  • Ablations: Self-training raises AUCO from 80.67% to 82.01% on UCF-Crime and from 89.10% to 90.80% on TAD.UMIL further outperforms the self-trained RTFM* baseline by +3.3% on UCF-Crime and +1.7% on TAD.
  • Ablations: A 10–50% threshold is reasonable for dividing confident and ambiguous snippets, with 30% performing best; α and β = 0.1 are suitable across both datasets.The threshold controls selection by prediction-history variance, while α and β weight ambiguous-set supervision and clustering.
  • Qualitative Analysis: Qualitative curves show UMIL more consistently assigns high anomaly scores within ground-truth abnormal regions, although RTFM is more consistent in one displayed video.The visualizations compare baseline, RTFM, and UMIL predictions against ground-truth regions.

5. Conclusion

The conclusion presents UMIL as an unbiased MIL scheme for WSVAD that addresses context bias by using both confident and ambiguous snippets. Experiments and ablations support its performance on standard benchmarks, while future work will strengthen ambiguous-snippet discovery and representation learning.

  • Conclusion: UMIL learns an anomaly classifier and tailored representation by seeking invariance across confident snippets and unsupervised clusters of ambiguous snippets.This replaces MIL training focused only on apparent normal and abnormal snippets.
  • Conclusion: The authors report state-of-the-art performance and extensive ablation validation on standard WSVAD benchmarks.The conclusion attributes the result to removing context bias that fails among ambiguous snippets.
  • Conclusion: Future work will add prior knowledge for distinguishing ambiguous snippets and explore representation learning such as disentanglement.These directions aim to highlight anomaly features more systematically.

7. Appendix

The appendix provides implementation details, additional experimental analyses, expanded benchmark results, and supplementary ROC curves for UMIL.

  • Training objectives: Section 8 details the training objectives and FixMatch-based self-training implementation.
  • Feature clustering: Section 9 explains how feature clustering boosts UMIL during training.
  • Supplementary experiments: Sections 10 and 11 provide additional benchmark comparisons, standard deviations, anomaly-event statistics, and full ROC curves.

8. Loss Objectives

UMIL combines MIL with a FixMatch-based self-training objective that generates pseudo-labels from original snippets and aligns predictions on augmented versions.

  • Combined objective: The overall MIL pre-training objective combines binary cross-entropy with a weighted self-training loss.The balance weight is denoted by λ.
  • Self-training objective: The self-training objective generates pseudo-labels from original snippets and minimizes prediction entropy between augmented and original data.The method uses feature pairs x and x′ from original and randomly augmented data.
  • Confidence filtering: The self-training loss is activated only when the detector prediction exceeds the confident threshold δ.The indicator function selects confident predictions, and δ is tuned by grid search.

9. Discussion on clustering

UMIL’s clustering component depends on feature-space separation between normal and abnormal snippets, with pretrained representations supplying the prior knowledge used to support that separation.

  • Clustering risk: Wrong clustering during UMIL training can introduce risks.
  • Anomaly-event statistics: Figure 9 reports anomaly-event lengths and their ratios to coarse snippet lengths.
  • Prior knowledge: Pretrained backbones such as CLIP are assumed to express intrinsic normal–abnormal differences sufficiently for clustering.This supports separating normal and abnormal snippets into different clusters.

10. Additional Experiments

Additional experiments analyze fine-grained processing, self-training, clustering, confident-sample selection, and benchmark performance, supporting UMIL’s design choices across UCF-Crime and TAD.

  • Video sequence partition: Average pooling over coarse snippets can dilute subtle anomaly features, motivating one-second fine-grained snippets in UMIL training.The average anomaly-event length is about 698 frames, versus about 200 frames for coarse snippets.
  • Feature fine-tuning: 86.75% versus 83.44% on UCF and 92.93% versus 90.71% on TAD shows fine-tuning outperforming a frozen backbone.The comparisons are with fine-tuning versus frozen pre-trained Kinetics400 weights.
  • Self-training ablation: 83.66% on UCF and 91.74% on TAD after removing self-training, decreases of 3.09% and 1.19%, respectively, support the self-training component.
  • Confidence threshold: A confident threshold of 0.8 performs well on both datasets, while a threshold of 1 selects too few samples for effective self-training.
  • Clustering ablation: Removing A-supervision reduces UMIL toward existing MIL because the classifier is then trained only on confident snippets.
  • Confident sample selection: Historical Variance outperforms Max Confidence, reaching 86.8% versus 85.9% on UCF-Crime and 93.0% versus 92.2% on TAD.The results support using score-variance history to distinguish confident and ambiguous samples.
  • Additional benchmark: On ShanghaiTech, UMIL achieves 96.78% AUC, compared with 95.20% for the baseline and 97.21% for RTFM.The reported comparison also includes GCN at 84.44% AUC.

11. Visualization of ROC Curves

Figure 10 presents the full ROC curves for UCF and TAD, enabling inspection of detector performance on both benchmarks.

  • Figure 10 shows the full ROC curves for the evaluation benchmarks.
  • The UCF ROC curve appears on the left side of the figure.
  • The TAD ROC curve appears on the right side of the figure.
Loading 2303.12369v1…