Source-linked AI summary
UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection
Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mubarak Shah
TL;DR
Video anomaly detection lacks abnormal training data and often evaluates either normal-only open-set detection or closed-set action categories. UBnormal introduces a supervised open-set benchmark with pixel-level abnormal annotations and disjoint anomaly types, and UBnormal data improves a state-of-the-art framework on Avenue and ShanghaiTech. Its main scope boundary is that the benchmark uses virtual characters and simulated actions rather than natural scenes.
Problem
Video anomaly detection is challenged by context-dependent anomalies, scarce abnormal training data, and closed-set evaluations that do not test detection of new anomaly types.
Method
UBnormal provides multiple virtual scenes with pixel-level abnormal training annotations and disjoint anomaly categories between training and test videos.
Results
UBnormal data enhances a state-of-the-art multi-task anomaly-detection framework on both Avenue and ShanghaiTech, including without CycleGAN-based distribution-gap closure.
Takeaways & Limitations
The benchmark enables fair head-to-head comparison between one-class open-set and supervised closed-set models while providing abnormal data useful for training anomaly detectors.
Takeaways & Limitations
UBnormal uses virtual characters and simulated actions, so its scenes belong to a different distribution from natural scenes.
Abstract
from arXiv · showhide
Detecting abnormal events in video is commonly framed as a one-class classification task, where training videos contain only normal events, while test videos encompass both normal and abnormal events. In this scenario, anomaly detection is an open-set problem. However, some studies assimilate anomaly detection to action recognition. This is a closed-set scenario that fails to test the capability of systems at detecting new anomaly types. To this end, we propose UBnormal, a new supervised open-set benchmark composed of multiple virtual scenes for video anomaly detection. Unlike existing data sets, we introduce abnormal events annotated at the pixel level at training time, for the first time enabling the use of fully-supervised learning methods for abnormal event detection. To preserve the typical open-set formulation, we make sure to include disjoint sets of anomaly types in our training and test collections of videos. To our knowledge, UBnormal is the first video anomaly detection benchmark to allow a fair head-to-head comparison between one-class open-set models and supervised closed-set models, as shown in our experiments. Moreover, we provide empirical evidence showing that UBnormal can enhance the performance of a state-of-the-art anomaly detection framework on two prominent data sets, Avenue and ShanghaiTech. Our benchmark is freely available at https://github.com/lilygeorgescu/UBnormal.
1. Introduction
Video anomaly detection remains difficult because anomalies depend on context and abnormal training data is scarce. UBnormal addresses this by defining supervised open-set detection with virtual scenes, disjoint anomaly types, and pixel-level training annotations.
- Motivation: Context-dependent anomalies create an effectively unbounded set of possible anomaly types, while collecting abnormal examples can be unethical or impractical.A truck on a street may be normal but the same truck in a pedestrian area may be abnormal.
- Existing formulation: The conventional formulation trains on normal events and tests on both normal and abnormal events, treating anomaly detection as one-class classification.This formulation learns normality from familiar events.
- Proposed formulation: The proposed supervised open-set formulation provides normal and abnormal training events while reserving distinct anomaly categories for inference.This preserves evaluation on unexpected anomaly types while allowing supervised learning.
- UBnormal: UBnormal uses virtual scenes to provide abnormal training data and supports fully supervised models, evaluation on unexpected anomalies, and fair comparison with one-class methods.The benchmark is generated with virtual animated characters and objects placed in real-world backgrounds.
- Results: UBnormal data enhances a state-of-the-art multi-task framework on Avenue and ShanghaiTech, even without closing the simulated-to-real distribution gap with CycleGAN.The authors report that these gains can directly improve state-of-the-art models on real-world data.
- Contributions: UBnormal contains 29 virtual scenes and 236,902 video frames, and abnormal training data helps state-of-the-art models for abnormal event detection.The study also evaluates whether UBnormal data improves a recent method on Avenue and ShanghaiTech.
2. Related Work
Prior video anomaly benchmarks and methods largely use either normal-only training or weakly supervised closed-set anomaly categories. UBnormal fills a benchmark gap by combining pixel-level abnormal annotations with disjoint training and test anomaly types.
- Existing formulations: Weakly supervised action-recognition approaches train with normal and abnormal videos, but test anomalies belong to the same categories as training anomalies, creating a closed-set problem.Their abnormal annotations are provided at the video level.
- Dataset comparison: Table 1 compares benchmark statistics across datasets, emphasizing abnormal-event count, anomaly-category breadth, and scene coverage.The caption states that UBnormal exceeds other open-set benchmarks on these dimensions.
- Methods: Existing methods mainly use normal-only training or abnormal videos with video-level annotations, leaving no off-the-shelf method that fully exploits supervised open-set data.The authors modify considered baselines to leverage abnormal samples available in UBnormal.
- Datasets: Open-set datasets include single-scene and multiple-scene benchmarks, with the latter testing whether models learn normality that generalizes across scenes.ShanghaiTech is described as the largest multiple-scene open-set dataset, but it has 158 anomalies across 11 types.
- Datasets: UCF-Crime contains 13 anomaly categories shared between training and test videos and lacks pixel-level anomaly annotations, so it does not follow the open-set paradigm.The benchmark uses videos retrieved from YouTube and LiveLeak through text queries.
- UBnormal: UBnormal introduces pixel-level training annotations and disjoint anomaly types between training and testing, establishing a supervised open-set benchmark.The benchmark is presented as the first of its kind to the authors’ knowledge.
3. UBnormal Benchmark
UBnormal is generated from 29 natural-image backgrounds representing varied environments and animated 3D characters. Character and appearance diversity is used to broaden the variety of anomalous events.
- Scenes: UBnormal uses Cinema4D to combine 2D background images with 3D animations in scenes such as streets, train stations, and offices.The benchmark selects 29 natural images representing these environments and others.
- Characters: The benchmark animates 19 different characters and varies clothing and hair colors to increase character diversity.This differs from datasets where anomalous events are performed by fewer recurring characters.
4. Methods
The benchmark evaluates multiple anomaly-detection paradigms, including one-class, supervised closed-set, action-recognition, and self-supervised multi-task models. Its methods include pseudoanomaly augmentation, spatiotemporal transformer classification, and an additional abnormality proxy task.
- One-class open-set model: The one-class baseline uses three auto-encoders and three classifiers, with adversarial learning and scene-agnostic pseudoanomalies compensating for missing abnormal training samples.The benchmark first evaluates the framework unchanged, then augments its pseudo-abnormal pool with abnormal samples from the training set.
- Supervised closed-set model: The supervised closed-set baseline trains on video features so the highest score in an anomalous bag exceeds the highest score in a normal bag.
- Action recognition framework: The TimeSformer action-recognition baseline learns separate spatial and temporal attention maps from frame-patch sequences to classify normal versus abnormal actions.It predicts the probability that a frame sequence is abnormal.
- Self-supervised multi-task model: The self-supervised multi-task model extends a shared 3D CNN with a fifth proxy task that distinguishes normal and abnormal UBnormal objects.The model can either train directly on UBnormal examples or process objects seen only during training through the added task.
- Self-supervised multi-task model: The joint training objective combines the first five task losses as Ltotal = LT1 + LT2 + LT3 + λ · LT4 + LT5.
5. Experiments
Experiments evaluate UBnormal baselines and test whether UBnormal samples improve anomaly detection and localization on Avenue and ShanghaiTech. Results show strong benchmark performance and gains on real-world datasets, with CycleGAN-based adaptation further improving outcomes.
- UBnormal evaluation: 58.5% validation and 59.3% test micro-averaged frame-level AUC are obtained by Georgescu et al. [18], whose framework provides the strongest localization capability among the baselines.The framework produces higher RBDC and TBDC scores than the other two approaches because those methods only support anomaly detection.
- UBnormal evaluation: Experiments evaluate UBnormal on frame-level detection and region- and track-based localization metrics using three baseline methods.Table 2 reports micro- and macro-averaged frame-level AUC, RBDC, and TBDC; only Georgescu et al. [18] performs anomaly localization.
- Qualitative results: Qualitative examples show anomaly scores and localization for UBnormal, Avenue, and ShanghaiTech test videos, including precise or accurate detection and localization in the reported examples.The Avenue example contains two anomalies, while the ShanghaiTech example concerns video 06 0153.
- UBnormal evaluation: 68.5% micro-averaged frame-level AUC is achieved by TimeSformer on the UBnormal test set using a 1/32 video sample rate.Smaller sample rates of 1/8 and 1/4 did not improve performance.
- Avenue: 93.0% micro-averaged and 93.2% macro-averaged frame-level AUC are obtained on Avenue after CycleGAN processing, surpassing the original method by at least 1.3% on all four metrics.Without CycleGAN, UBnormal samples still improve micro-averaged AUC by 0.4% and macro-averaged AUC by 0.5%.
- ShanghaiTech: 83.7% micro-averaged and 90.5% macro-averaged frame-level AUC are obtained on ShanghaiTech after CycleGAN processing.The approach also surpasses the object-level baseline by 4.35% in RBDC and 2.25% in TBDC; gains occur with or without domain adaptation.
6. Conclusion
UBnormal is presented as a challenging supervised open-set benchmark whose virtual construction enables comparisons between open-set and closed-set models, while limiting direct realism. Its usefulness for closed-set datasets may be narrower.
- UBnormal is described as the first and only benchmark for supervised open-set anomaly detection.
- The benchmark enables fair head-to-head comparison between open-set and closed-set models.
- UBnormal is limited by its use of virtual characters and simulated actions.
- Three state-of-the-art models indicate that UBnormal remains a very challenging benchmark.
- The authors expect limited gains from adding UBnormal to UCF-Crime because its abnormal classes appear in both training and test data.
7. Supplementary
The supplementary material describes UBnormal’s virtual-scene diversity through varied environments, characters, objects, and abnormal actions, alongside dataset statistics. The benchmark contains 29 scenes and 22 anomaly categories spanning object-related and non-object events.
- UBnormal contains 29 scenes, including 22 outdoor and 7 indoor scenes with varied weather and lighting conditions.
- The dataset uses 19 unique characters whose hair and clothing colors are varied to increase diversity.
- UBnormal includes five object categories besides people, with object colors varied to increase diversity.
- The benchmark contains 22 anomaly types: 20 related to objects and two non-object types, fire and smoke.
- UBnormal comprises 543 videos distributed across 268 training, 64 validation, and 211 test videos.
- The dataset contains 660 anomalies, 89,015 abnormal frames, and 142,107 abnormal regions.