Source-linked AI summary
Street Scene: A new dataset and evaluation protocol for video anomaly detection
Bharathkumar Ramachandra, Michael Jones
TL;DR
Video anomaly detection is hindered by limited datasets and evaluation criteria that do not realistically assess practical performance. This paper introduces the Street Scene dataset, new evaluation criteria, and nearest-neighbor baseline variants; the baselines outperform two previously published algorithms on Street Scene.
Problem
Video anomaly detection research is limited by small, low-variety datasets and evaluation criteria that do not properly assess spatial localization or false positives.
Method
The paper introduces the Street Scene dataset, two practical evaluation criteria, and two nearest-neighbor-based baseline algorithm variations.
Results
The two nearest-neighbor-based baseline algorithms outperform two previously published algorithms on Street Scene.
Takeaways & Limitations
Street Scene and the new criteria provide a more realistic dataset and evaluation baseline for future video anomaly detection work.
Takeaways & Limitations
The existing pixel-level criterion can be equivalent to the frame-level criterion and therefore does not properly assess localization tightness or multiple false-positive regions.
Abstract
from arXiv · showhide
Progress in video anomaly detection research is currently slowed by small datasets that lack a wide variety of activities as well as flawed evaluation criteria. This paper aims to help move this research effort forward by introducing a large and varied new dataset called Street Scene, as well as two new evaluation criteria that provide a better estimate of how an algorithm will perform in practice. In addition to the new dataset and evaluation criteria, we present two variations of a novel baseline video anomaly detection algorithm and show they are much more accurate on Street Scene than two state-of-the-art algorithms from the literature.
1. Introduction
Video anomaly detection aims to identify unusual, spatially and temporally localized events from normal training video in a particular scene. The paper argues that progress is limited by deficient datasets and evaluation criteria, and introduces Street Scene plus new criteria and baselines.
- Motivation: Video anomaly detection flags unusual video segments for human inspection because continuously monitoring ubiquitous surveillance cameras is impractical.The task focuses attention on the small fraction of surveillance footage considered unusual.
- Problem formulation: The standard single-scene formulation learns normal activity from training videos of one scene and detects significantly different localized events in test video.Spatial and temporal localization are part of the stated anomaly definition.
- Problem formulation: Using normal data from multiple scenes restricts detection of anomalies that depend on particular spatial regions, such as walking in a scene-specific restricted area.The paper explains that other scenes cannot establish which locations are restricted in the target scene.
- Problem formulation: For busy scenes, roughly localizing anomalies in both space and time helps assess whether detections have the right causes and supports human inspection.The paper contrasts this with formulations that define anomalies temporally but not spatially.
- Dataset and evaluation gap: Existing datasets often have simple scenes, few or narrow anomaly types, low resolution, staged events, inconsistent annotations, or missing spatial ground truth.These shortcomings motivate a more realistic benchmark for the single-scene setting.
- Paper contributions: Street Scene provides a high-resolution, activity-rich urban street dataset with more labeled and varied anomalies, alongside criteria intended to estimate practical performance more accurately.The paper also presents two novel algorithm variations that outperform two state-of-the-art algorithms on Street Scene.
2. Existing Datasets and Evaluation Criteria
Prior single-scene datasets are often small, staged, ambiguously split, temporally labeled only, or otherwise limited in anomaly variety and spatial ground truth. The widely used frame- and pixel-level criteria also fail to measure localization quality and false-positive regions realistically.
- Existing datasets: UCSD Pedestrian uses two static-camera walkway datasets with five anomaly types, but is modest in frames, total anomalies, and anomaly diversity.Its anomalies can also be detected from a single frame at a time.
- Existing datasets: Subway contains two long videos with ambiguities about frame extraction, train/test splitting, and anomaly labels, and lacks spatial ground truth.The supplied passage truncates the final statement but explicitly identifies these ambiguities and the spatial-ground-truth issue.
- Existing datasets: CUHK Avenue mainly contains staged anomalies in short clips from one outdoor camera, while UMN uses staged clips with one evacuation-like event per clip and only temporal labels.UMN also lacks a clear train/test split specification.
- Existing datasets: ShanghaiTech and UCF-Crime target multi-scene or activity-detection formulations rather than the single-scene setting, with UCF-Crime providing only temporal test labels for spatial evaluation.These datasets therefore do not directly match the paper’s target formulation.
- Evaluation criteria: The standard frame-level criterion counts frames containing any detected anomalous pixels, while the pixel-level criterion uses a 40% overlap threshold for anomalous pixels.The pixel-level rule ignores non-overlapping detections on frames containing ground-truth anomalies.
- Evaluation criteria: The pixel-level criterion can be reduced to the frame-level criterion by labeling every pixel anomalous whenever any pixel is detected.This fails to reward tight localization, penalize loose localization, or count multiple false-positive regions properly.
3. Description of Street Scene
Street Scene is a high-resolution, single-scene dataset designed to capture varied, natural activity and anomalous events in an urban street environment. It includes extensive training and testing video with tracked anomaly annotations.
- Dataset composition: Street Scene contains 46 training and 35 testing sequences from a static camera overlooking a two-lane street with bike lanes and sidewalks.Videos were collected during daytime across two consecutive summers.
- Dataset composition: 203,257 color frames are provided at 1280 x 720 resolution and 15 frames per second.The dataset contains 56,847 training frames and 146,410 testing frames.
- Scene content: The scene contains varied activity, including cars, pedestrians, and bikers, along with changing shadows and moving background elements.Examples include driving, turning, parking, walking, jogging, stroller-pushing, and riding in bike lanes.
- Normal activity: Training sequences define normal activity using lawful, direction-consistent behavior by pedestrians, cars, and bikers.These conditions exclude examples such as jaywalking, loitering, illegal parking, and bikers outside bike lanes from training.
- Anomaly annotations: The test set contains 205 anomalous events spanning 17 anomaly types, with bounding-box tracks supplied for each testing-video anomaly.A frame may contain more than one labeled anomaly.
4. New Evaluation Criteria
The paper replaces frame-oriented evaluation with region- and track-based criteria that better reflect localization quality, multiple anomalies, and practical detection over time. Both criteria measure detection against false-positive regions per frame, using IOU-based matching.
- Motivation: Existing criteria inadequately handle multiple anomalies, mixed true and false detections, and multiple false-positive regions within a frame.The proposed alternative measures detected-anomaly coverage and false-positive regions per frame.
- Shared matching rule: An anomalous region is detected when its IOU with a detected region reaches β, using β = 0.1 in the experiments.The low threshold permits imprecise boxes, merged nearby anomalies, and shadow-inflated detections.
- Track-Based Detection Criterion: The track-based detection criterion measures track-based detection rate against false-positive regions per frame.A ground-truth track is detected when at least a fraction α of its regions are detected; the experiments use α = 0.1.
- Region-Based Detection Criterion: The region-based detection criterion measures the fraction of anomalous regions detected across all test frames against false-positive regions per frame.False positives are counted per detected region that lacks sufficient IOU with every ground-truth region in its frame.
5. Baseline Algorithms
The baseline divides video into spatio-temporal patches, represents normal activity with region-specific exemplars, and scores test patches by nearest-neighbor distance. Two variations differ in their patch features: blurred foreground masks or optical flow fields.
- Model structure: The algorithm divides each video into spatio-temporal regions called video patches and models normal activity separately for each spatial region.Training patches are represented by feature vectors and selected as exemplars for each region.
- Model structure: The two variations differ only in the feature vector used for each video patch and the distance function used to compare vectors.The foreground-mask variation uses L2 distance, while the flow-based variation uses normalized L1 distance.
- Feature variations: The foreground-mask variation computes blurred masks from an updating background model, concatenates masks across frames, and vectorizes them.The experimental background model is a per-pixel mean color value, and Gaussian blurring makes L2 comparison more robust.
- Feature variations: The flow-based variation replaces foreground masks with optical-flow fields, concatenating and vectorizing them into features containing dx and dy components.Its feature vector is twice as long as the foreground-mask feature vector because it includes both flow components.
- Anomaly scoring: For each test patch, the distance to the nearest exemplar in the corresponding region becomes its anomaly score, which is averaged across overlapping patches per pixel.Scores are assigned to the middle frame of each temporal patch; thresholded anomalous pixels are converted into connected anomalous regions for evaluation.
6. Experiments
On Street Scene, the proposed baseline methods outperform the dictionary and auto-encoder methods under the new track-based and region-based criteria, while performance depends on anomaly type and criterion. Traditional frame-level evaluation can mask poor spatial localization, motivating the authors’ emphasis on the newer criteria.
- New evaluation criteria: The baseline methods perform much better than the dictionary and auto-encoder methods on Street Scene under the track-based and region-based criteria, though substantial improvement remains possible.The figure reports areas under the ROC curve for false-positive rates from 0 to 1.
- Baseline comparison: Flow-based detection is strongest for track-based evaluation at low false-positive rates, whereas foreground-based detection performs better under the region-based criterion.The flow field is more useful for most anomalies, while foreground masks better capture some static anomalies.
- Anomaly types: Jaywalking, illegal U-turns, and vehicles outside their lanes are comparatively well detected because their motions differ from typical activity in the corresponding regions.These anomalies contain distinctive motion relative to their scene context.
- Anomaly types: Loitering and other largely static anomalies are hardest for the baselines because they contain little motion after the initial transition.The flow method cannot detect loitering, while the foreground-based method can detect some cases before the background model absorbs the motionless person.
- Baseline comparison: The flow method detects a greater fraction of anomalous tracks at low false-positive rates but a smaller fraction of anomalous regions.Static anomalies form a disproportionate share of anomalous regions because many persist across numerous frames.
- Evaluation analysis: Traditional frame-level results can make inaccurate methods appear competitive because they ignore spatial localization and may count detections far from the ground-truth anomaly as correct.The authors present these results to illustrate deficiencies in the traditional criteria rather than recommend them for future Street Scene comparisons.
7. Conclusions
The paper contributes the Street Scene dataset, revised evaluation criteria, and two baseline algorithm variations for future video anomaly detection research. The baselines outperform two previously published algorithms on Street Scene.
- Street Scene is a large-scale dataset with a more complex scene and almost as many anomalous events as all currently available datasets combined.
- The new evaluation criteria address problems in typical criteria and aim to estimate practical algorithm performance more realistically.
- The paper presents two straightforward variations of a new video anomaly detection algorithm as baselines for future comparisons.
- The new baseline algorithms outperform two previously published algorithms that perform well on previous datasets but not on Street Scene.
8. Supplemental Material
Supplemental results illustrate detections from the flow-based method across several anomalous activities and false-positive cases. Additional experiments clarify the baseline’s intended role and show comparable performance on UCSD datasets.
- Baseline scope: The baseline method is presented to provide reasonable Street Scene results for future comparison, not as a claim of superiority on existing datasets.
- Existing-dataset results: On UCSD Ped1 and Ped2, the foreground baseline’s traditional frame-level and pixel-level results are comparable to many recent results, especially under the pixel-level criterion.
- Detection examples: The supplemental examples include a jaywalker detection with a false positive and a false positive caused by a car shadow.
9. Detailed List of Anomalies in Street Scene
Tables 4 and 5 enumerate every annotated anomaly across all 35 Street Scene testing videos. The lists are informative and are distributed with the dataset, while anomaly types are excluded from evaluation.
- Tables 4 and 5 list every annotated anomaly in each of the 35 Street Scene testing videos.
- The anomaly lists and ground-truth bounding boxes for all frames are intended for inclusion with the publicly released dataset.
- The lists provide an informative view of the dataset contents, but anomaly types are not used in the evaluation criteria.