PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection

Jia-Xing Zhong, Nannan Li, Weijie Kong, Shan Liu, Thomas H. Li, Ge Li

arXiv:1903.07256v1cs.CV

TL;DR

Anomaly-labeled videos can contain normal snippets, producing noisy labels that complicate supervised action classification. The paper uses a graph convolutional network to clean labels and reports state-of-the-art performance across three datasets and two classifier types, including 82.12% frame-level AUC on UCF-Crime.

  • Problem

    Anomaly-labeled videos can contain normal snippets, producing incorrect snippet labels that must be cleaned before fully supervised action classifiers can be trained.

    Source: §1 · p. 2

  • Method

    A graph convolutional network alternates with an action classifier, correcting low-confidence predictions using supervisory signals from high-confidence snippets.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Across three differently scaled datasets and two classifier types, the method reports state-of-the-art performance, with 82.12% frame-level AUC on UCF-Crime.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    At test time, the trained classifier produces snippet-wise predictions without post-processing, integrating feature extraction and abnormality decisions in one model.

    Source: §1 · p. 2

    The classifier may miss test anomalies in scenes unlike those represented in the training data.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

Video anomaly detection under weak labels is formulated as a typical multiple-instance learning problem in previous works. In this paper, we provide a new perspective, i.e., a supervised learning task under noisy labels. In such a viewpoint, as long as cleaning away label noise, we can directly apply fully supervised action classifiers to weakly supervised anomaly detection, and take maximum advantage of these well-developed classifiers. For this purpose, we devise a graph convolutional network to correct noisy labels. Based upon feature similarity and temporal consistency, our network propagates supervisory signals from high-confidence snippets to low-confidence ones. In this manner, the network is capable of providing cleaned supervision for action classifiers. During the test phase, we only need to obtain snippet-wise predictions from the action classifier without any extra post-processing. Extensive experiments on 3 datasets at different scales with 2 types of action classifiers demonstrate the efficacy of our method. Remarkably, we obtain the frame-level AUC score of 82.12% on UCF-Crime.

1. Introduction

The paper reframes weakly supervised video anomaly detection as supervised action classification with noisy snippet labels, and proposes alternating graph-based label cleaning and classifier training.

1. Introduction

  • Weakly supervised anomaly detection trains with only video-level anomaly labels, without trimmed anomalous segments or temporal annotations; prior work models it as MIL, treating videos or snippet sets as bags and learning instance-level labels from bag-level annotations [58].Source: §1 · p. 1
  • The paper formulates detection as supervised learning with noisy labels, enabling fully supervised action classifiers once normal snippets within anomalous videos are relabeled.The proposed GCN is described as the first application of a GCN to label-noise correction in video analytics.Source: §1 · p. 2
  • Using an action classifier throughout training preserves its designed architecture and pretrained weights, while test-time prediction needs no post-processing.Feature extraction and anomaly decisions are integrated into one model.Source: §1 · p. 2
  • The method alternates between GCN-based label cleaning and classifier retraining until convergence, propagating anomaly information from high-confidence to low-confidence snippets.At test time, the cleaner is unnecessary: the trained classifier directly produces snippet-wise anomaly results.Source: Figure 1 · p. 1, §1 · p. 2
  • Experiments use C3D [59] and TSN [62] on UCF-Crime [58], ShanghaiTech, and UCSD-Peds, and the paper reports state-of-the-art performance.The introduction presents this evaluation as evidence for the approach’s effectiveness.Source: §1 · p. 2

2. Related Work · 3. Problem Statement

The paper reframes weakly supervised video anomaly detection as learning with one-sided label noise, so correcting noisy labels can enable action classifiers to detect anomalies. It contrasts this approach with prior MIL-based anomaly detection and motivates video-specific label cleaning.

2. Related Work

  • Prior weakly supervised anomaly methods use MIL, including Sultani et al.’s anomaly-ranking classifier [58], whereas this paper uses noisy-label supervision and alternate training to promote action-classifier discrimination.Source: §2 · p. 2, §3 · p. 3
  • Action classifiers range from two-stream networks and C3D to variants, while temporal action localization differs from anomaly detection’s frame-level performance across discrimination thresholds.The paper aims to leverage action classifiers for anomaly detection in a simple, feasible way.Source: §2 · pp. 2–3
  • Existing noisy-label methods address noise reduction or loss correction, but the paper’s GCN is designed for videos and exploits video-based characteristics.Source: §2 · p. 3
  • The label cleaner treats a whole video as a graph and classifies snippet nodes under supervision from high-confidence annotations.The related work situates graph convolution among approaches to graph-structured data, including spectral methods.Source: §2 · p. 3

3. Problem Statement

  • Training provides only a video-level binary label Y, with no temporal annotation; detection must pinpoint abnormality locations in test videos.Source: §3 · p. 3
  • Under MIL, clips are unlabeled instances in positive or negative bags: positive videos contain at least one anomalous clip, while negative videos contain only normal snippets.Source: §3 · p. 3
  • Weak-label anomaly detection is reframed from MIL as learning with one-sided label noise, enabling well-developed action classifiers once positive-video label noise is handled.Negative videos are noiseless because all snippets are normal; positive videos may contain both normal and anomalous clips.Source: §3 · p. 3

4. Graph Convolutional Label Noise Cleaner

The cleaner alternates with an action classifier, using feature similarity and temporal consistency to propagate high-confidence supervisory signals to low-confidence snippets. Its outputs supervise standard classifier training, so the final classifier can predict anomalies directly.

4. Graph Convolutional Label Noise Cleaner

  • The cleaner alternates with the action classifier, correcting low-confidence snippet anomaly scores using high-confidence predictions and video-specific feature similarity and temporal consistency.The classifier supplies rough snippet-wise anomaly probabilities to the cleaner at each training step.Source: §4 · p. 3

4.1. Feature Similarity Graph Module

  • Compressed classifier features define a snippet-similarity graph whose weighted connections propagate label assignments; differentiable graph-convolution layers support end-to-end training.The graph models similarity among entire snippets or frames, rather than long-term dependencies among correlated objects or regions.Source: §4.1 · pp. 3–4

4.2. Temporal Consistency Graph Module

  • The temporal graph connects snippets according to their positions, using an exponential kernel to distinguish temporal distances and connect nearby snippets.Its graph-convolution layers can be stacked and included in neural networks.Source: §4.2 · p. 4

4.3. Loss Function

  • The cleaner fuses both graph outputs for snippet-level probabilities, trained with direct cross-entropy supervision on high-confidence snippets and indirect temporal ensembling across training steps.Classifier prediction variance is used to identify confidence, with lower variance indicating higher confidence.Source: §4.3 · p. 5

4.4. Alternate Optimization

  • The classifier is trained as in fully supervised learning using the cleaner’s snippet-wise predictions, and the final trained classifier directly predicts anomaly results.The two models are alternately optimized several times, with almost no changes required to the classifier during training or testing.Source: §4.4 · p. 5

5. Experiments

Experiments evaluate the label-noise cleaner with action classifiers on three datasets and report a UCF-Crime AUC of up to 82.12%.

5.1. Datasets and Evaluation Metric

  • Experiments use UCF-Crime, ShanghaiTech, and UCSD-Peds; ShanghaiTech is reorganized to include anomalies in training, while UCSD-Peds2 uses 10 repeated random splits.ShanghaiTech training and testing sets both cover all 13 scenes. Each UCSD-Peds2 training split includes 6 anomaly and 4 normal videos, with remaining videos used for testing; performance is averaged over 10 runs. Frame-level ROC AUC is the evaluation metric.Source: §5.1 · p. 5

5.2. Implementation Details

  • The experiments test general applicability with C3D and two-stream TSN action classifiers using features from their respective pretrained backbones.C3D uses Sports-1M-pretrained fc7 features, while TSN uses a Kinetics-400-pretrained BN-Inception backbone and global-pool features; both classifiers use Caffe with shared video-sampling and augmentation settings.Source: §5.2 · pp. 5–6

5.3. Experiments on UCF-Crime

  • 82.12% AUC is the method’s best UCF-Crime result; TSN-RGB also reaches a 0.1% false-alarm rate, nearly 1/20 of the best-so-far result.The classifiers’ ROC curves almost completely enclose competing curves, indicating consistently stronger performance across thresholds; C3D is slightly inferior to Sultani et al. on false-alarm rate.Source: Table 2 · p. 6, §5.3 · p. 7, Figure 4 · p. 7
  • UCF-Crime ablations show that indirect supervision, temporal consistency, and feature-similarity graph convolution each contribute to detection performance.Without indirect supervision, AUC falls from 74.60% to 73.79%; temporal consistency alone reaches 72.93%, and feature similarity alone raises AUC from 70.87% to 72.44%, while damaging its graph lowers AUC to 67.23%.Source: §5.3 · p. 6, Table 1 · p. 6

5.4. Experiments on ShanghaiTech

  • Alternate training improves every ShanghaiTech classifier, while excessive Step-3 iterations cause only a slight AUC drop; combining temporal and feature modules yields more precise labels.Temporal consistency smooths high-confidence predictions but can miss ground-truth intervals with insufficient high-confidence inputs; feature similarity can propagate labels to similar actions beyond the anomaly interval.Source: §5.4 · p. 7, Table 3 · p. 7

5.5. Experiments on UCSD-Peds

  • Across 10 UCSD-Peds2 runs, the method improves detection and stabilizes predictions, outperforming comparison methods with both input modalities.Experiments use TSN because some ground truths span only 4 frames while C3D predictions span 16 frames; grayscale frames are duplicated across three channels to match the RGB input dimension.Source: §5.5 · pp. 7–8, Table 4 · p. 7

6. Conclusion

The paper reframes weakly supervised anomaly detection as supervised learning with noisy labels, using a GCN to clean supervision for action classifiers. It reports effectiveness and versatility across three datasets of different scales and two types of action-classification networks.

6. Conclusion

  • The approach casts weakly supervised anomaly detection as supervised learning under noisy labels, rather than a multiple-instance learning problem.It inherits the strengths of well-developed action classifiers and uses an integral end-to-end model for convenient anomaly detection.Source: §6 · p. 8
  • The proposed model shows effectiveness and versatility across three datasets of different scales and two types of action-classification networks.The conclusion describes its performance as superior but provides no benchmark values here.Source: §6 · p. 8
  • During alternate optimization, a GCN cleans training labels by propagating anomaly information from high-confidence predictions to low-confidence ones.The cleaned supervision is used to train an action classifier.Source: §6 · p. 8

1. Processing Speed in the Test Phase

The action classifiers retain real-time or faster test-phase performance for anomaly detection even when preprocessing costs are included.

1. Processing Speed in the Test Phase

  • The three action classifiers achieve real-time or super-real-time testing performance despite time-consuming preprocessing.FPS measurements were taken on a Titan-XP GPU and include operations such as resizing, 10-crop oversampling, and zero-centering.Source: Table 1 · p. 11, §1 · p. 11

2. Implementation of Label Noise Cleaner

The label cleaner iteratively expands a high-confidence snippet set, while normal videos and specified classifier and graph settings support training. The implementation limits graph sampling and uses three alternate-optimization iterations in most cases.

2. Implementation of Label Noise Cleaner

  • The cleaner selects the 30% and 60% highest-confidence snippets for two-stream and C3D networks, respectively, then increases the set by 30% at each step.Normal videos are also included in training to learn an unbiased model.Source: §2 · p. 11
  • Classifier label assignments concentrate output probability into one anomaly category using min-max normalization, while the graph modules use two convolutional layers.The first two fully connected layers have 512 and 128 outputs with 60% dropout; each graph module has a 32-unit ReLU hidden layer and a 1-unit output layer.Source: §2 · p. 11
  • The implementation samples at most 1,600 high-confidence snippets and limits each to 8 neighbors per video, using three alternate-optimization iterations in most cases.It uses a base learning rate of 0.0001, momentum of 0.9, and weight decay of 0.0005, and compares the last—not always best—results with other methods.Source: §2 · p. 11

3. More Comparisons on UCSD-Peds

The section addresses comparison fairness on UCSD-Peds: several 2018 unary-classification works use different data splits, so their default results are not directly comparable. Open-source works are reproduced on the split in, with original-paper results shown in brackets for reference.

3. More Comparisons on UCSD-Peds

  • Open-source comparisons are reproduced on the data split in, while original-paper results appear in brackets for reference.The passage identifies Table 2 as reporting the original results.Source: §3 · p. 11
  • Several unary-classification works from 2018 also report UCSD-Peds experiments, but their default implementations use different splits and are not directly comparable.Source: §3 · p. 11
  • For UCF-Crime, apart from the official reference and its comparisons, the authors report finding no public results or source code after its June 10, 2018 release.They state that they hope their work can fill this gap.Source: §3 · p. 11

4. Vectorized Feature Similarity Module

The module builds a feature-similarity graph over video snippets and vectorizes graph operations to propagate anomaly labels. Its differentiable layers support end-to-end training and stacking with action-classification networks.

4. Vectorized Feature Similarity Module

  • The module represents a video as a feature-similarity graph F=(V,E,X), where vertices are snippets, edges describe their similarity, and X contains d-dimensional features for N snippets.Source: §4 · p. 11
  • Adjacency entries A(i,j) measure feature similarity between snippets, and Equation 1 is rewritten in an equivalent vectorized form.Source: §4 · p. 12
  • A graph-Laplacian operation approximated with a renormalization trick drives nearby vertices toward the same anomaly label, using self-loop adjacency A+I and its corresponding degree matrix.The vectorized Equation 4 implementation uses vectorized summation and PyTorch broadcasting diagonal functions.Source: §4 · p. 12
  • A feature-similarity graph layer computes output H using a trainable matrix W and activation function σ.Source: §4 · p. 12
  • Because the procedure is differentiable, the module can be trained end to end and used singly or in stacked modules; temporal similarity can likewise be implemented in vectorized form.Source: §4 · p. 12

5. Details of Indirectly Supervised Loss Term

The indirectly supervised loss term is treated as a temporal ensembling strategy and uses already-obtained rough classifier predictions.

5. Details of Indirectly Supervised Loss Term

  • The indirectly supervised loss term follows temporal ensembling, uses γ = 0.5 in all experiments, and omits cool-start initialization and bias correction because rough classifier predictions are available.Source: §5 · p. 12

6. Reorganization of ShanghaiTech · 7. Discuss the Formulation

The paper reorganizes ShanghaiTech into training and testing subsets and clarifies how its noisy-label formulation differs from MIL in both concept and seed-example selection. It also distinguishes its EM-like optimization from EM-based methods.

6. Reorganization of ShanghaiTech

  • ShanghaiTech’s 437 videos are split into 238 training videos and 199 testing videos.The paper also depicts the normal and anomaly video counts by scene for each subset.Source: §6 · p. 12, Table 4 · p. 12

7. Discuss the Formulation

  • The paper expands its discussion of the noisy-label formulation and its EM-like optimization mechanism in response to a reviewer suggestion.Source: §7 · p. 13

7.1. Concept: MIL vs Noisy-labeled Learning

  • For a positive bag, MIL emphasizes positive instances, while noisy-labeled training focuses on noisy negative labels and the remaining positive labels; the conceptions are complementary and transformationally related.Source: §7.1 · p. 13

7.2. Practice: EM-like MIL vs Ours

  • For seed selection, EM-like MIL seeks likely positive instances, whereas noisy-labeled optimization selects predictions judged most reliable.Source: §7.2 · p. 13
  • With anomalous-video predictions of 0.2 across 10 crops, He et al., Hou et al., and Zhang et al. exclude or neglect the snippet, while this method selects it because its predictive variance is 0.Source: §7.2 · p. 13

7.3. Terminology: EM-like vs EM-based

  • The optimization is EM-like because it alternates update-and-fix steps, but it is not EM-based because it does not explicitly estimate mathematical expectations during training.Source: §7.3 · p. 13

Related papers

Loading 1903.07256v1…