Source-linked AI summary
Motion-Aware Feature for Improved Video Anomaly Detection
Yi Zhu, Shawn Newsam
TL;DR
Video anomaly detection must localize rare events despite massive recordings and mostly video-level labels. This paper learns motion-aware features and adds temporal attention to MIL, achieving new state-of-the-art results on anomaly detection and anomalous action recognition in UCF Crime.
Problem
Video anomaly detection needs weakly supervised localization because precise frame-level annotations are costly, while datasets typically provide only video-level labels.
Method
A temporal augmented network learns motion-aware features from optical flow, while an attention-based temporal MIL ranking model incorporates context.
Results
The approach achieves new state-of-the-art results on both anomaly detection and anomalous action recognition tasks in the UCF Crime dataset.
Takeaways & Limitations
Motion-aware features are complementary to existing features, and learned attention weights better differentiate anomalous from normal video segments.
Takeaways & Limitations
The original MIL ranking loss ignores temporal structure and can produce degenerate predictions in which most video segments are labeled normal.
Abstract
from arXiv · showhide
Motivated by our observation that motion information is the key to good anomaly detection performance in video, we propose a temporal augmented network to learn a motion-aware feature. This feature alone can achieve competitive performance with previous state-of-the-art methods, and when combined with them, can achieve significant performance improvements. Furthermore, we incorporate temporal context into the Multiple Instance Learning (MIL) ranking model by using an attention block. The learned attention weights can help to differentiate between anomalous and normal video segments better. With the proposed motion-aware feature and the temporal MIL ranking model, we outperform previous approaches by a large margin on both anomaly detection and anomalous action recognition tasks in the UCF Crime dataset.
1 Introduction
Video anomaly detection is difficult because anomalies are rare, variable, subjective, and often recorded in low-resolution surveillance footage. The paper addresses this by learning motion-aware features and incorporating temporal context into MIL ranking for improved anomaly detection and anomalous action recognition.
- Introduction: Video anomaly detection has broad surveillance applications, but the volume of passively recorded footage makes automated monitoring necessary.Applications include detecting illegal activities, traffic accidents, and unusual events.
- Introduction: Recognizing anomalies in unconstrained videos is challenging because anomalies are rare, annotations are insufficient, events vary widely, definitions are subjective, and footage can be low resolution.The passage also notes large inter- and intra-class variations.
- Introduction: Features incorporating motion information perform better than features extracted from individual images on the UCF Crime dataset.The comparison evaluates different input features within the same MIL framework while changing only the input features.
- Introduction: The paper proposes an unsupervised temporal augmented network that learns efficient motion-aware features competitive with C3D and complementary to existing features.Combining the learned feature with other features yields significant performance improvement.
- Introduction: An attention-based temporal MIL ranking model incorporates temporal context to better differentiate anomalous from normal events.The attention weights help distinguish normal and anomalous video segments.
- Introduction: The proposed approach outperforms several state-of-the-art methods on anomaly detection and anomalous action recognition in the UCF Crime dataset.The comparison covers both tasks on the same benchmark.
2 Related Work
Prior video modeling methods capture temporal relationships through handcrafted tracking features or deep architectures such as temporal and 3D convolutions, temporal segment networks, and two-stream networks. This work instead learns a compact motion-aware feature with a temporal augmented autoencoder and adds temporal context to MIL ranking through attention.
- Video anomaly detection research models temporal relationships using tracking-based handcrafted features and deep methods including temporal convolution, 3D convolution, temporal segment networks, and two-stream networks.
- The temporal augmented network uses an autoencoder to learn a compact motion-aware feature rather than directly using optical flow.The network reconstructs a stack of 15 optical flow maps and derives a 1024×1 feature through global average pooling.
- The learned feature is generic, efficient, and easily integrated with other methods through early fusion.
- Temporal context is incorporated into classical MIL ranking models באמצעות an attention mechanism.The attention mechanism is intended to extend temporal modeling within the ranking framework.
3 Methodology
The method learns a compact motion-aware feature from optical flow with a temporal augmented autoencoder, then uses weakly supervised MIL with attention to incorporate temporal context into anomaly scoring. This addresses limited frame-level annotations and the limitations of segment-wise max ranking.
- Weakly Supervised Learning: The approach uses video-level labels because precise frame-level anomaly annotations are costly and most anomaly detection datasets provide only video-level supervision.Anomalous videos form positive bags and normal videos form negative bags, with temporal segments treated as individual instances.
- Motion-Aware Feature: The temporal augmented network is an autoencoder that takes precomputed optical flow as input, forcing it to learn complex motion patterns.The bottleneck representation is trained to reconstruct the input and serves as the motion-aware feature for anomaly detection.
- Motion-Aware Feature: For each 16-frame clip, global average pooling at the bottleneck produces a 1024×1 motion-aware feature that can be concatenated with other features.The feature extractor is obtained by forwarding clips through the trained network until the bottleneck layer.
- Temporal MIL Ranking Model: The baseline MIL ranking loss uses the highest-scoring segment in each positive and negative bag, but ignores temporal structure and can mishandle multiple or visually ambiguous segments.These limitations motivate reasoning over temporal context rather than relying on a single max operation.
- Temporal MIL Ranking Model: An attention-based temporal MIL model learns segment weights end-to-end and compares video-level anomaly scores, while sparsity constraints reflect the rarity of anomalies.The attention block contains three fully connected layers with two intermediate tanh activations and outputs a 1×m attention score for a video with m segments.
4 Experiments
Experiments on the large-scale, diverse UCF Crime benchmark show that the motion-aware feature is compact, fast, and competitive, while improving multiple anomaly-detection features and reducing false alarms. Qualitative results and class-level gains further indicate strong performance on anomalous frames and fast-motion categories.
- Dataset: The model is evaluated on UCF Crime, a large-scale real-world benchmark with balanced normal and anomalous surveillance videos spanning 13 anomaly classes.The official split includes 800 normal and 810 anomalous training videos, plus 150 normal and 140 anomalous testing videos.
- Anomaly detection: 72.1 AUC is achieved by the 1024-dimensional motion-aware feature versus 75.4 by the previous best, while running at 400+ fps versus 300+ fps.The motion-aware feature is smaller and faster than the 4096-dimensional previous-best feature.
- Anomaly detection: 3.6% improvement raises the previous best from 75.4 to 79.0 when combined with the proposed motion-aware feature.Combining the feature with the other baselines also yields performance boosts of 12.1% and 7.9%.
- Per-class analysis: Fast-motion classes benefit strongly, with Arrest improving from 46.0 to 57.4, Assault from 20.8 to 41.2, and Fighting from 32.4 to 47.1.The class-level results indicate that motion-aware features are particularly useful for rapidly moving anomalies.
- Feature combination: Motion-aware features consistently improve VGG16 from 68.7 to 74.2, Inception from 69.2 to 74.9, and I3D from 75.4 to 79.8.These gains support complementarity between the proposed feature and widely adopted visual features.
- Qualitative results: The model produces high anomaly scores for anomalous frames and consistently low, almost-zero scores for normal frames.ROC visualizations also show higher true-positive rates at low false-positive rates when the motion-aware feature is added.
5 Discussion
The discussion shows that temporal attention improves anomaly discrimination, while PWCNet provides an effective speed–performance trade-off for motion-aware features that substantially improve anomalous action recognition when combined with existing features.
- Temporal MIL attention: Adding temporal attention consistently improves AUC by 1% to 2%, supporting temporal context’s role in distinguishing anomalous from normal events.The analysis compares the attention-based formulation with the alternative temporal MIL ranking formulation.
- Motion representation ablation: FlowNet2 achieves the best AUC among evaluated flow estimators, but PWCNet performs competitively while running significantly faster.FlowNet2’s advantage is attributed to accurate, sharp flow predictions, whereas PWCNet is identified as a practical trade-off.
- Motion representation ablation: PWCNet achieves the best performance among compared motion representations, while motion vectors and video saliency perform badly.The passage attributes the weak motion-vector results to coarse resolution and discusses limitations of video saliency.
- Anomalous action recognition: 3.1% for C3D and 2.6% for TCNN are the performance improvements obtained by combining the motion-aware feature with state-of-the-art video features.The motion-aware feature alone also achieves reasonable anomalous action recognition performance under the official four-split evaluation.
6 Conclusion
The paper proposes a temporal augmented network for motion-aware video anomaly detection and an attention-based temporal MIL ranking model. Together, these components achieve new state-of-the-art results on anomaly detection and anomalous action recognition in the UCF Crime dataset, although challenges remain in some known scenarios.
- Contributions: A temporal augmented network learns a motion-aware feature that performs competitively with previous state-of-the-art methods alone and improves performance when combined with them.The feature is designed to capture motion information for video anomaly detection.
- Contributions: An attention block incorporates temporal context into the MIL ranking model, with learned attention weights better differentiating anomalous from normal video segments.The attention mechanism extends the temporal modeling used for segment ranking.
- Results: The proposed motion-aware feature and temporal MIL ranking model achieve new state-of-the-art results for anomaly detection and anomalous action recognition on the UCF Crime dataset.The reported gains apply to both the anomaly detection and anomalous action recognition tasks.
- Limitations: The model still has difficulties in some known challenging scenarios.The supplied passage begins listing such scenarios but is truncated before completing the examples.