Source-linked AI summary
Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detection
Guansong Pang, Cheng Yan, Chunhua Shen, Anton van den Hengel, Xiao Bai
TL;DR
Video anomaly detection must identify rare events without manually labeled normal/abnormal training data, despite the limitations of existing two-step methods. The paper uses self-trained ordinal regression in an end-to-end anomaly learner that jointly optimizes representations and scores. Experiments and analyses report improved detection, anomaly localization, and human-in-the-loop use, with stated scope boundaries around data composition and feature coverage.
Problem
Video anomaly detection is important for focusing human attention on rare events, but existing approaches often require manually labeled normal data and separate feature learning from anomaly scoring.
Method
The method uses self-trained two-class ordinal regression with an end-to-end learner that jointly optimizes feature representations and anomaly scores from iteratively updated pseudo labels.
Results
The method outperforms current state-of-the-art approaches by a significant margin and provides accurate anomaly localization and effective human-in-the-loop detection.
Takeaways & Limitations
Self-training ordinal regression enables anomaly detection without manually labeled training data while supporting end-to-end score optimization, user feedback, and frame-level localization.
Takeaways & Limitations
The reported dataset setting merges training and test sets because the test sets contain unusually large anomaly proportions, and the authors identify motion features as future work.
Abstract
from arXiv · showhide
Video anomaly detection is of critical practical importance to a variety of real applications because it allows human attention to be focused on events that are likely to be of interest, in spite of an otherwise overwhelming volume of video. We show that applying self-trained deep ordinal regression to video anomaly detection overcomes two key limitations of existing methods, namely, 1) being highly dependent on manually labeled normal training data; and 2) sub-optimal feature learning. By formulating a surrogate two-class ordinal regression task we devise an end-to-end trainable video anomaly detection approach that enables joint representation learning and anomaly scoring without manually labeled normal/abnormal data. Experiments on eight real-world video scenes show that our proposed method outperforms state-of-the-art methods that require no labeled training data by a substantial margin, and enables easy and accurate localization of the identified anomalies. Furthermore, we demonstrate that our method offers effective human-in-the-loop anomaly detection which can be critical in applications where anomalies are rare and the false-negative cost is high.
1. Introduction
Video anomaly detection matters for directing attention toward rare, significant events without manually labeled normal/abnormal training data. The paper addresses limitations of two-step methods with self-trained ordinal regression and end-to-end learning, while adding human-in-the-loop detection and anomaly localization.
- Video anomaly detection identifies frames depicting events that significantly deviate from the norm, supporting timely responses amid overwhelming video volumes.
- Existing methods often depend on labeled normal videos, limiting use where normal behavior is diverse, changing, or costly to annotate.
- Two-step methods separate feature learning from anomaly scoring, producing inflexible and sub-optimal anomaly scoring.
- The proposed approach uses self-trained ordinal regression to enable weakly supervised, end-to-end anomaly detection without manually labeled normal/abnormal data.It initializes pseudo labels using generic anomaly detectors and relevant auxiliary labeled data.
- The end-to-end neural network jointly learns representations and anomaly scores, and supports human-in-the-loop updates and frame-level anomaly localization.
2. Problem Formulation
The paper formulates unlabeled video anomaly detection as self-training two-class ordinal regression. Anomaly candidates and normal candidates receive ordered pseudo labels, while an end-to-end learner iteratively refines scoring and candidate membership.
- The goal is to learn an end-to-end scoring function that assigns higher scores to anomalous frames than to normal frames without class labels.
- A surrogate two-class ordinal task assigns larger scalar label c1 to anomaly candidates and smaller label c2 to normal candidates, with c1 > c2.
- Optimizing ordinal regression brings suspicious abnormal and normal samples toward c1 and c2, respectively, yielding an anomaly ranking.
- The scoring function composes a feature mapping ψ with parameters Θr and an anomaly scoring function η with parameters Θs for joint optimization.
- Candidate sets are initialized from existing unsupervised detectors and iteratively updated as newly learned scores generate more accurate sets.
- The initial anomaly candidate set may omit anomalies or include normal samples, so self-training continually refines candidate membership.
3. The Proposed Method
The method initializes pseudo-normal and pseudo-anomalous frame sets, then jointly learns representations and anomaly scores through iterative self-training. It repeatedly updates pseudo-label memberships and aggregates sequential models for stable final scoring.
- 3.3. Iterative Learning via Self-training: The framework recomputes all-frame anomaly scores, updates A and N, repeats the process, and averages the sequentially trained models for stable final detection.The iterative procedure produces a set of optimized models whose scores are aggregated.
- 3.1. Initial Anomaly Detection: Initial anomaly detection combines Sp and iForest scores to select the most likely anomalous frames for A and normal frames for N.The two detectors provide complementary anomaly estimates before end-to-end learning begins.
- 3.2. End-to-end Anomaly Score Learner: The end-to-end score learner stacks a feature representation learner with an anomaly score learner, directly mapping raw frames to scalar anomaly scores.The feature learner can use ResNet-50, while the scoring learner is a fully connected network.
- 3.2. End-to-end Anomaly Score Learner: Two-class ordinal regression assigns ordinal targets c1 and c2 to pseudo-anomalous and pseudo-normal frames, respectively, and optimizes their anomaly scores.Absolute loss is used to reduce the negative effects of false pseudo-labels while encouraging larger scores for anomalous frames.
- 3.3. Iterative Learning via Self-training: Each self-training iteration replaces the previous pseudo-label sets with newly obtained A and N sets before retraining the end-to-end anomaly learner.Discarding previous sets avoids combining unsupervised pseudo-labels that may be inconsistent, and the authors report that the strategy worked well across datasets.
4. Experiments
Experiments evaluate the self-trained ordinal-regression detector across real-world scenes, compare it with unsupervised and labeled-data methods, and test its human-in-the-loop, localization, architecture, and self-training behavior. The method consistently improves anomaly scoring over unsupervised baselines, while performance gains plateau after several self-training iterations.
- Datasets and evaluation: The experiments use UCSD, Subway, and UMN video data, merging training and test sets because the original test sets can contain unusually high anomaly rates.Evaluation uses ground truth only for evaluation, preserving the intended rare-anomaly setting.
- Datasets and evaluation: AUC is the primary evaluation metric, calculated from frame-level anomaly scores and ground truth; EER is omitted because it can mislead when anomalies are rare.
- Effectiveness in real-world datasets: 2%-15% AUC improvement is achieved over Sp + iForest, while 5%-25% AUC improvement is achieved over both Del Giorno et al. variants across the datasets.The comparisons use the same evaluation protocol for the unsupervised methods and identical feature inputs for the second Del Giorno et al. variant.
- Effectiveness in real-world datasets: Compared with unmasking, the method improves by 3%, 17%, 7%, and 12% on UCSD-Ped1, Subway-Entrance, Subway-Exit, and UMN-Scene2, respectively.It performs comparably better on the other datasets and retains similar improvement over the two-sample-test variant except on UCSD-Ped2 and Subway-Exit.
- Human-in-the-loop anomaly detection: More than 6% AUC improvement is obtained on both UCSD-Ped1 and Subway-Exit after five human-machine interactions, as limited feedback reduces false positives.The interaction uses l = ⌊0.1N⌋ and k = 5 on these two representative datasets.
- Anomaly localization: CAM-based saliency maps highlight anomalous regions across four scenes, although normal patches can also attract activation in some cases.
- Ablation studies: The method works with different popular backbones, and self-training improves performance during the first few iterations before stabilizing around the fourth or fifth.Five iterations are empirically often sufficient, while end-to-end score learning outperforms a two-step method using the same feature inputs.
5. Conclusions
The paper concludes that self-training deep ordinal regression addresses key limitations of video anomaly detection through end-to-end learning, while providing human-in-the-loop detection and accurate localization. The method also has a scope boundary: incorporating motion features remains future work for identifying other anomaly types.
- Self-training deep ordinal regression overcomes key limitations of existing video anomaly detection approaches.The conclusion frames the approach as addressing limitations through self-training and end-to-end learning.
- End-to-end learning produces better optimized anomaly scores than the two-step approach.The feature learner and anomaly scoring process are jointly optimized rather than separated.
- Self-training ordinal regression can iteratively enhance detection performance through the end-to-end anomaly score learner.
- The method offers human-in-the-loop anomaly detection and accurate anomaly localization.
- Incorporating motion features remains future work for identifying other types of anomalies.