PaperlayerView exact revision on arXiv

Source-linked AI summary

Localizing Anomalies from Weakly-Labeled Videos

Hui Lv, Chuanwei Zhou, Chunyan Xu, Zhen Cui, Jian Yang

arXiv:2008.08944v3cs.CV

TL;DR

Video-level supervision has advanced anomaly detection but leaves temporal anomaly localization insufficiently addressed. The paper proposes WSAL, which combines high-order temporal context, semantic and dynamic cues, and supervision enhancement, and reports state-of-the-art performance on UCF-Crime and TAD.

  • Problem

    Video-level labels support anomaly detection, but previous methods often fail to accurately localize anomalous events temporally within videos.

  • Method

    WSAL encodes adjacent-segment context to extract semantic representations and dynamic variations, aggregates both cues, and augments training with noise and pseudo-location data.

  • Results

    The method achieves new state-of-the-art performance on the UCF-Crime and TAD datasets.

  • Takeaways & Limitations

    The TAD dataset broadens anomaly-detection evaluation with realistic traffic-scene videos containing varied appearance and motion patterns.

Abstract

from arXiv · show

Video anomaly detection under video-level labels is currently a challenging task. Previous works have made progresses on discriminating whether a video sequencecontains anomalies. However, most of them fail to accurately localize the anomalous events within videos in the temporal domain. In this paper, we propose a Weakly Supervised Anomaly Localization (WSAL) method focusing on temporally localizing anomalous segments within anomalous videos. Inspired by the appearance difference in anomalous videos, the evolution of adjacent temporal segments is evaluated for the localization of anomalous segments. To this end, a high-order context encoding model is proposed to not only extract semantic representations but also measure the dynamic variations so that the temporal context could be effectively utilized. In addition, in order to fully utilize the spatial context information, the immediate semantics are directly derived from the segment representations. The dynamic variations as well as the immediate semantics, are efficiently aggregated to obtain the final anomaly scores. An enhancement strategy is further proposed to deal with noise interference and the absence of localization guidance in anomaly detection. Moreover, to facilitate the diversity requirement for anomaly detection benchmarks, we also collect a new traffic anomaly (TAD) dataset which specifies in the traffic conditions, differing greatly from the current popular anomaly detection evaluation benchmarks.Extensive experiments are conducted to verify the effectiveness of different components, and our proposed method achieves new state-of-the-art performance on the UCF-Crime and TAD datasets.

I. INTRODUCTION

The paper addresses the limited temporal localization of anomalies under video-level supervision by proposing WSAL and introducing the TAD traffic benchmark. Its method combines temporal variation modeling, semantic cues, and supervision enhancement to improve localization.

  • Motivation: Video-level anomaly detectors can identify anomalous videos but still lack an efficient way to temporally localize anomalous frames.Overall test performance can conceal poor localization within anomalous videos because normal-video accuracy dominates evaluation.
  • Proposed Method: The enhancement strategy stimulates video noise and constructs virtual indicative locations to suppress false-anomaly signals and provide localization guidance.The resulting augmented weak labels include pseudo-location signals as auxiliary supervision.
  • Dataset and Evaluation: The paper introduces the large-scale traffic anomaly detection dataset with realistic road anomalies and varied appearance and motion patterns.The dataset is intended to increase benchmark diversity beyond existing small-scale or constrained scenarios.
  • Proposed Method: WSAL uses high-order context encoding to represent temporal variations and high-level semantic information for weakly supervised anomaly localization.The method evaluates adjacent temporal segments because sudden dynamic changes can reveal anomalies.
  • Dataset and Evaluation: The proposed method reports new state-of-the-art results on both UCF-Crime and the TAD dataset.The paper compares multiple anomaly detection approaches on the two benchmarks.

II. RELATED WORK

Prior anomaly detection methods use unsupervised, semi-supervised, or weakly supervised settings, but weakly supervised approaches still struggle to predict anomaly frames because temporal context is neglected.

  • Unsupervised Methods: Unsupervised methods model normal behavior from normal samples and detect testing samples that deviate from the learned pattern.Trajectory-based approaches can be fast but are vulnerable to tracking failures in crowded or cluttered scenes.
  • Weakly Supervised Methods: Weakly supervised methods use normal and abnormal data with video-level annotations to build anomaly detection models.This direction was encouraged by the release of the UCF-Crime dataset.
  • Weakly Supervised Methods: MIL and attention-based methods improve anomaly detection or localization, but the absence of anomaly positions during training limits frame prediction.Iterative pseudo-position refinement was introduced to construct supervised signals for anomaly locations.
  • Research Gap: Existing methods focus on predicting segment labels while neglecting hidden temporal context information.This gap motivates the paper’s explicit modeling of temporal context for weakly supervised anomaly localization.

A. Formulation

The formulation divides videos into equal-length segments and learns a weakly supervised margin objective that compares anomaly-score differences across temporally contextualized segments. Training is further augmented with noise and pseudo-location samples.

  • Formulation: A video is divided into equal-length segments, and the model estimates each segment’s anomaly status from spatial semantics and dynamic variations.Segmenting reduces computation from nearly repetitive video content.
  • Formulation: The score function applies a high-order context encoder to an anchored segment and its adjacent 2k segments, then measures anomaly-score margins between segment positions.The encoder models local temporal context while the margin metric compares predicted scores.
  • Formulation: The maximum pairwise score distance represents the video anomaly score, with normal videos expected to have smaller distances than anomalous videos.Anomalies embedded in normal backgrounds create larger score margins.
  • Formulation: The margin loss depends only on video-level labels, so the learning process remains weakly supervised.The formulation uses separate normal and anomaly sample counts in the loss.
  • Augmented Training: Training adds noise data and pseudo-location data, combining their loss with the original weak-label loss through a balance factor λ.Noise data reduces misjudgments on noisy normal videos, while pseudo-location data directly guides anomalous-frame localization.

B. High-order Context Encoding

The High-order Context Encoding model combines immediate semantic cues with temporal dynamic variations to generate anomaly scores from segment-level video features.

  • HCE encodes high-level semantic features and temporal variations from feature vectors extracted from consecutive video segments.The input consists of segment features (x1, · · · , xm), and the regression process models their temporal context.
  • Each high-order feature aggregates information from the anchored segment and its 2k neighboring segments, supporting spatial semantic mining and local dynamic-variation modeling.The neighbor size k controls the temporal context, while one-layer regression performs well in practice.
  • Immediate semantics are converted into anomaly scores through a fully connected layer with Sigmoid activation.The resulting semantics score is denoted ψsem(ext).
  • Adjacent-segment cosine distance measures temporal variation, with larger values indicating more dramatic changes.The distance is 1−cos(ext−1, ext), and a second-order discrepancy of local variations is used as an anomaly indicator.
  • The framework aggregates immediate semantics and dynamic variations into anomaly scores supervised by a MIL Margin objective using video-level labels.The two cues provide complementary signals for weakly supervised temporal localization.

C. Enhanced Weak Supervision

The Enhanced Weak Supervision strategy augments training with simulated noise and hand-crafted pseudo-location data to reduce interference and provide localization guidance.

  • Noise simulation fuses raw videos with blur, picture interruption, and lens jitter to mitigate false alarms caused by noisy artifacts.The strategy targets interference in untrimmed videos whose quality cannot be guaranteed.
  • The enhancement objective applies supervised constraints to predicted anomaly states on simulated-noise data and pseudo-location segments.The overall objective combines the augmentation techniques with the base training objective.
  • Hand-crafted anomalies create pseudo anomalous sequences by replacing selected normal segments with segments from abnormal videos.The fused segments provide explicit location instructions for anomaly localization.
  • Abnormal features are fused with normal ones using randomly generated coefficients ranging from 0.2 to 0.5 to reduce abrupt transitions.The method avoids assigning a fixed anomaly score and instead pushes pseudo-location segments above the others.

D. Traffic Anomaly Detection (TAD) Dataset

The TAD dataset addresses limited diversity in anomaly benchmarks by providing long, realistic traffic-surveillance videos for weakly supervised anomaly detection.

  • Existing anomaly datasets are often small, scene-constrained, or designed for unsupervised settings, while UCF-Crime is the main public weak-supervision benchmark described here.UCF-Crime contains 1,900 long surveillance videos covering 13 real-world anomalies, with video-level training labels and frame-level test annotations.
  • TAD targets traffic scenes because prior datasets do not cover many road anomalies and traffic conditions increase detection difficulty.High vehicle speeds and varied road conditions are identified as contributing challenges.
  • TAD contains long untrimmed videos covering seven real-world road anomalies, including vehicle accidents, illegal turns, and retrograde motion.The dataset also includes illegal occupations, pedestrians on roads, road spills, and a residual “Else” category.
  • The dataset is presented as a benchmark with abnormal scenarios different from current benchmarks for evaluating anomaly detection algorithms from another perspective.The authors intend it to promote development toward real traffic applications.

A. Implementation Details

Implementation uses BN-Inception TSN features and divides each video into 32 nonoverlapping segments, producing a 32 × 1024 feature matrix.

  • The backbone is the BN-Inception version of Temporal Segment Network, used as an action feature extractor.Features are extracted from the global average pooling layer.
  • Each video is divided into 32 nonoverlapping segments, yielding a 32 × 1024 feature matrix for the model.UCF-Crime uses weights fine-tuned on that dataset, whereas TAD uses weights pretrained on Kinetics-400.

B. Evaluation Metrics

The paper evaluates anomaly detection with ROC curves and frame-level AUC, while visualizations compare predicted anomaly scores against ground-truth temporal anomalies.

  • ROC curves are generated by varying the threshold on predicted anomaly scores, with AUC summarizing detection performance.Higher AUC indicates better anomaly detection performance.
  • Figure 4 plots anomaly scores against video frame number for two prior methods and the proposed model.Green curves show predictions, while light orange regions mark ground-truth anomalies.
  • Figure 5 compares ROC curves for anomaly detection methods on UCF-Crime and TAD.

C. Comparison with SOTA Methods

WSAL is compared with unsupervised and weakly supervised methods on UCF-Crime and TAD. It improves both overall detection and anomaly-subset localization performance.

  • UCF-Crime: 85.38% overall AUC is achieved on UCF-Crime, up from 82.12% for the best prior performance.On the anomaly subset, WSAL exceeds by 9% and by 13%.
  • UCF-Crime: Weakly supervised methods outperform unsupervised methods on UCF-Crime because video-level labels provide useful anomaly information.
  • Qualitative comparison: Figure 6 illustrates immediate-semantics, dynamic-variations, and consensus anomaly-score curves on a UCF-Crime case.The consensus curve is described as more robust and smooth.
  • TAD: WSAL achieves a 6% AUC gain over previous SOTA on the TAD dataset.The authors report advances on both large-scale benchmarks for detecting and localizing anomalies.
  • TAD: TAD contains varied traffic scenes and anomaly types, creating a challenging setting for unsupervised methods.

D. Ablation Studies

Ablations show that dual context cues, high-order temporal encoding, enhancement strategies, and feature-source choices each affect WSAL performance and robustness.

  • Dual context ensemble: 81.44% anomaly detection accuracy and 61.13% anomaly localization accuracy are obtained using immediate semantics alone.Adding dynamic variations improves these results by 1.12% and 1.25%, respectively.
  • Dual context ensemble: Dynamic variations capture sudden anomaly occurrence, while combining them with immediate semantics yields more robust detection and localization.The two cues represent complementary aspects of anomalies.
  • HCE model: Increasing HCE window size from 0 to 1 improves video-level prediction accuracy by 1.6%, after which gains slow as the window grows.The selected window size is k = 2 to balance model size and performance.
  • Feature sources: HCE raises anomaly localization accuracy from 51.23% to 55.04% with I3D inputs and from 47.96% to 51.25% with R(2+1)D inputs.With TSN features, overall AUC increases from 82.95% to 84.44%.
  • Enhanced weak supervision: Noise augmentation raises anomaly detection AUC from 84.44% to 84.86% and further improves localization accuracy by 1.33%.
  • Enhanced weak supervision: Combining noise suppression and hand-crafted anomalies yields gains of 0.51% and 1.60% over training without noise augmentation.The authors associate the combined strategy with suppressing environmental or hardware interference and generating pseudo-anomaly signals.
  • Speed analysis: The full model runs at 44 FPS on a single RTX 2080Ti GPU, while anomaly-score prediction alone runs at 550 FPS.

V. CONCLUSION

The paper proposes a weakly supervised network for surveillance-video anomaly localization and reports improved performance on UCF-Crime and TAD. It combines temporal dynamics, semantic information, and enhanced supervision while introducing a traffic anomaly dataset.

  • The proposed network explores temporal context in consecutive video segments for weakly supervised anomaly localization.
  • It encodes temporal dynamic variations and high-level semantic information for anomaly detection and localization.
  • Video noise augmentation and pseudo-location data provide enhanced supervision for improving anomaly localization accuracy.
  • The authors collect a traffic anomaly detection dataset to evaluate methods under realistic road scenarios.
  • Experiments on UCF-Crime and TAD show significantly better performance than previous methods.
Loading 2008.08944v3…