Generative Cooperative Learning for Unsupervised Video Anomaly Detection

Muhammad Zaigham Zaheer; Arif Mahmood; Muhammad Haris Khan; Mattia Segu; Fisher Yu; Seung-Ik Lee

Source-linked AI summary

Generative Cooperative Learning for Unsupervised Video Anomaly Detection

Muhammad Zaigham Zaheer, Arif Mahmood, Muhammad Haris Khan, Mattia Segu, Fisher Yu, Seung-Ik Lee

arXiv:2203.03962v1Mar 8, 2022cs.CV

TL;DR

Unsupervised video anomaly detection remains sparse because anomalies are rare and ill-defined, despite its potential to eliminate annotation costs. GCL addresses this with cooperative generator–discriminator training on unlabeled videos and reports excellent performance across two public benchmark datasets and varying supervision levels.

Problem
Unsupervised video anomaly detection is sparse because anomalies are rare and ill-defined, while obtaining sufficient anomaly examples and exhaustive annotations is cumbersome.
Method
GCL trains a generator and discriminator cooperatively on unlabeled videos, using generator-derived pseudo-labels and negative learning to support frame-level anomaly scoring.
Results
GCL shows excellent performance on two public benchmark datasets across no-supervision, one-class, and weak-supervision settings.
Takeaways & Limitations
The approach can be deployed without manual annotations and supports anomaly detection training without normal-class or binary annotations.
Takeaways & Limitations
The unsupervised setting assumes anomalies are present in the training dataset and less frequent than normal happenings.

Abstract

from arXiv · show

Video anomaly detection is well investigated in weakly-supervised and one-class classification (OCC) settings. However, unsupervised video anomaly detection methods are quite sparse, likely because anomalies are less frequent in occurrence and usually not well-defined, which when coupled with the absence of ground truth supervision, could adversely affect the performance of the learning algorithms. This problem is challenging yet rewarding as it can completely eradicate the costs of obtaining laborious annotations and enable such systems to be deployed without human intervention. To this end, we propose a novel unsupervised Generative Cooperative Learning (GCL) approach for video anomaly detection that exploits the low frequency of anomalies towards building a cross-supervision between a generator and a discriminator. In essence, both networks get trained in a cooperative fashion, thereby allowing unsupervised learning. We conduct extensive experiments on two large-scale video anomaly detection datasets, UCF crime, and ShanghaiTech. Consistent improvement over the existing state-of-the-art unsupervised and OCC methods corroborate the effectiveness of our approach.

1. Introduction

Video anomaly detection is difficult because anomalies are rare, unconstrained, and costly to annotate. The paper therefore targets fully unsupervised learning, using domain knowledge about video content and anomaly frequency to train GCL.

1. Introduction: Anomaly detection is difficult because anomalous events are rare and unconstrained, making sufficient training examples difficult to collect.An exhaustive anomaly set required for fully supervised training is unlikely to be collected.
1. Introduction: Unsupervised video anomaly detection requires no training-data annotations and is more challenging than fully, weakly, or one-class supervision.The paper distinguishes this setting from literature that calls OCC unsupervised despite assuming all training data are normal.
1. Introduction: OCC methods learn dominant representations from normal data but may miss normal variations and produce false alarms for unseen normal activities.This limitation is especially relevant to surveillance videos with diverse classes and dynamic situations.
1. Introduction: Weakly supervised methods reduce annotation costs with video-level labels, but still require inspecting full videos and remain impractical in many applications.Unlabelled raw footage could instead be used without annotation cost.
1. Introduction: GCL uses unlabelled videos and alternates pseudo-label exchange between a generator and discriminator to predict frame-level anomaly scores.The generator reconstructs normal representations and distorts high-confidence anomalous ones, while the discriminator estimates anomaly probability.

2. Related Work

Prior work spans OCC and generative approaches that model normal data, while cooperative learning has addressed related multi-model settings. GCL instead applies mutual pseudo-label supervision to two models solving the same anomaly-detection task in one domain.

2. Related Work: OCC methods are widely used in anomaly detection and include approaches based on handcrafted or deep features and generative models of normal data.Generative variants aim to learn representations corresponding to normal data.
2. Related Work: OCC methods can reconstruct anomalous inputs because training only on normal data may produce a boundary that insufficiently excludes anomalies.This is identified as a challenge for OCC-based approaches.
2. Related Work: GCL introduces mutual learning from pseudo-labels generated by the participating models using unlabelled training data.The related-work discussion characterizes this cohort-style unsupervised mutual learning as unexplored.
2. Related Work: Earlier cooperative learning jointly trains models for the same task across different domains, whereas GCL uses two models for the same task in the same domain.The contrast is illustrated by RGB and depth models communicating domain-invariant object attributes.

3. Method

GCL is an annotation-free anomaly-detection system composed of a feature extractor, an autoencoding generator, a discriminator, and two pseudo-label generators.

3. Method: GCL comprises a feature extractor, generator network, discriminator network, and two pseudo-label generators.The architecture is presented as the overall Generative Cooperative Learning approach for anomaly detection.

3.1. Training Data Organization

The method converts videos into compact segment-level features, randomly batches them for training, and uses a fixed segment size across each dataset's training and test videos.

3.1. Training Data Organization: A deep feature extractor converts segmented videos into compact features to reduce computational complexity and training time.The extracted features are randomly arranged into batches for iteration.
3.1. Training Data Organization: Each of n training videos is partitioned into non-overlapping segments S(i,j) containing p frames.The segment size p is kept identical across training and test videos within a dataset.
3.1. Training Data Organization: For every segment S(i,j), the feature extractor E(·) computes a d-dimensional feature vector f(i,j).The paper denotes this operation as f(i,j)=E(S(i,j)).
3.1. Training Data Organization: Unlike approaches that preserve temporal order at batch or video level, the proposed organization uses randomly input temporally consistent feature batches.The stated purpose of randomization is to minimize correlation between consecutive batches.

3.2. Generative Cooperative Learning

GCL alternates pseudo-label supervision between an autoencoder generator and a fully connected discriminator, without annotations. Generator-based labels train the discriminator, while discriminator-based labels guide negative reconstruction learning in the generator.

GCL uses an autoencoder generator G and fully connected discriminator D, trained cooperatively without normal-class or binary annotations.
Pseudo Labels from Generator: G assigns higher reconstruction-loss features anomalous and lower-loss features normal, using a threshold selected from a fixed percentage of maximum-error samples.Error histograms can also guide threshold selection because class boundaries often occur in low-density regions.
Discriminator Network: D is trained with binary cross-entropy using pseudo annotations from G, with G-generated labels supervising D's anomaly predictions.The discriminator output is the predicted anomaly probability for each feature vector.
Pseudo Labels from Discriminator: D-generated pseudo-labels identify anomalous features for the next iteration, and these annotations fine-tune G to improve reconstruction discrimination.Features with higher discriminator probabilities are treated as anomalous through thresholding.
Negative Learning of Generator Network: Negative learning makes G reconstruct normal pseudo-labeled samples normally while poorly reconstructing anomalous samples through pseudo reconstruction targets.For anomalous inputs, the reconstruction target is replaced by a different feature vector rather than simply making the loss negative.

3.3. Self-Supervised Pre-training

GCL uses self-supervised pre-training to initialize its cooperative learning loop. Temporal differences filter likely anomalous features for generator pre-training, after which noisy generator labels pre-train the discriminator.

Pre-training both G and D is used to improve convergence and stability in the unconstrained unsupervised learning problem.The authors empirically observe that pre-trained G improves overall system performance.
Temporal differences between consecutive feature vectors filter the pre-training data because anomalous frames are usually more eventful than normal frames.
A feature vector is retained for G pre-training when ||f_t+1_i,j||_2 ≤ D_th, although this filtering does not guarantee complete anomaly removal.The cleaned data provides an effective initialization rather than a fully anomaly-free training set.
After G pre-training, its noisy pseudo-labels pre-train D; G acts as a lousy teacher while D learns to discriminate normal and anomalous features.The pre-trained models then enter the collaborative learning loop.

3.4. Anomaly Scoring

At test time, anomaly scores can come from G's reconstruction error or D's prediction scores. The reported results use D predictions by default because D remains efficient while G remains relatively lousy across training iterations.

Test-time anomaly scoring can use either G's reconstruction error or D's prediction scores.
D predictions are used by default because D remains efficient while G remains relatively lousy across consecutive training iterations.

4. Experiments

Experiments on UCF-Crime and ShanghaiTech evaluate GCL under unsupervised, OCC-assisted, and weakly supervised settings, including component ablations and qualitative analyses. GCL and its pre-trained variant outperform relevant baselines, while negative learning, discriminator-based pseudo-labeling, and unsupervised pre-training improve performance.

Comparisons with State-Of-The-Art (SOTA): UCF-Crime GCLB achieved 68.17% AUC, 11.85% above AEAllData trained on complete unlabeled training data.The reconstruction-score distributions indicate that AEAllData is not sufficiently discriminative, while the discriminator separates normal and anomalous examples better than the generator.
Comparisons with State-Of-The-Art (SOTA): UCF-Crime GCLP_T reached 71.04% AUC, improving 2.87% over GCLB and outperforming existing OCC methods without labeled supervision.Across ten random seeds, GCLP_T averaged 70.13 ± 0.52 AUC versus 67.09 ± 0.65 for GCLB, with lower variation.
Comparisons with State-Of-The-Art (SOTA): On ShanghaiTech, GCLB obtained 72.41% AUC and GCLP_T obtained 78.93%, a 6.5% improvement over GCLB.The unsupervised GCLP_T variant also outperformed existing OCC methods on this dataset.
Ablation Study and Analysis: On UCF-Crime, negative learning improved GCLB by 3.94%, while unsupervised pre-training raised the complete GCLP_T system to 71.04% AUC.Adding one-class supervision further increased performance to 74.20% AUC in GCLOCC.
Ablation Study and Analysis: The ‘ones’ pseudo target performed best among tested negative-learning targets, supporting a fixed target for more consistent discrimination.tSNE reconstructions likewise showed anomalous features forming a distinct cluster with negative learning and ‘ones’ targets.
Ablation Study and Analysis: Training the generator with its own pseudo-labels produced 62.28% AUC, below GCLP_T’s 71.04%, indicating discriminator-based pseudo-labeling is critical.The authors attribute this difference to the discriminator’s robustness to noisy labels and its ability to clean generator-produced pseudo-labels.

5. Conclusion

GCL is an unsupervised video anomaly detection approach trained on unlabeled videos, requiring no manual annotations. It performs well across no-supervision, one-class, and weak-supervision settings, while assuming anomalies occur in training data.

GCL uses unlabeled training videos and can be deployed without manual annotations.
GCL shows excellent performance on two public benchmark datasets across no-supervision, one-class, and weak-supervision settings.
The unsupervised setting assumes anomalies are present in the training dataset, which the authors consider more realistic than OCC methods.