Source-linked AI summary
Adaptive Multi-Granularity Temporal Modeling for Weakly Supervised Video Anomaly Detection
Changyi Li, Yu Xiao
TL;DR
Weakly supervised VAD must operate with coarse video-level labels despite variable anomaly durations and temporal dynamics that fixed MIL assumptions handle poorly. The paper addresses this with adaptive temporal modeling across snippet, event, and video levels, and reports consistent SOTA improvements on two benchmarks. Its results support replacing rigid temporal heuristics with adaptive representations and similarity-based score fusion for this setting.
Problem
Existing weakly supervised VAD methods rely on fixed temporal assumptions and aggregation heuristics, while snippet-level predictions inferred from video labels can be unreliable for variable real-world anomalies.
Method
The framework combines a Temporal Refinement Module, an Event Segmentation Module, and similarity-based fusion to model temporal structure and aggregate predictions adaptively.
Results
The framework consistently outperforms state-of-the-art methods on the UCF-Crime and XD-Violence benchmarks.
Takeaways & Limitations
Adaptive modeling across snippet, event, and video levels provides the paper’s supported approach to handling variable anomaly frequency, duration, and temporal dynamics.
Abstract
from arXiv · showhide
As the scale of video surveillance data outpaces manual annotation capacities, weakly supervised video anomaly detection (WSVAD) has emerged as a critical research frontier. Most existing approaches formulate WSVAD within a Multiple Instance Learning (MIL) framework that relies on rigid, hand-crafted temporal priors to supervise anomaly scoring. However, such formulations exhibit limited adaptability to the wide variation in anomaly durations and temporal dynamics observed in real-world videos, often leading to unstable or unreliable snippet-level predictions. To address this limitation, we propose an adaptive temporal modeling framework for WSVAD that explicitly accounts for variations in video dynamics across multiple temporal granularities. First, we introduce a Temporal Refinement Module (TRM) that leverages dynamic positional encoding and a learnable class token to model long-range temporal dependencies while distilling a stable global video-level representation. Second, to capture anomalous events with varying frequency and duration, we develop an adaptive Event Segmentation Module (ESM) that identifies event boundaries through temporal discontinuity analysis and aggregates snippet features into discriminative event-level representations. Finally, for snippet-level and event-level predictions, we propose an adaptive similarity-based fusion strategy that dynamically integrates anomaly scores into video-level predictions, replacing fixed top-k aggregation heuristics with global semantic relevance. Extensive experiments on two benchmarks demonstrate that the proposed framework consistently outperforms state-of-the-art methods.
1 Introduction
Weakly supervised VAD reduces annotation demands but inherits fixed temporal assumptions and unreliable snippet predictions. The proposed framework adapts modeling and score fusion across snippet, event, and video levels, and outperforms SOTA methods on two benchmarks.
- Motivation: Weakly supervised VAD uses video-level labels with MIL aggregation over temporal snippets, avoiding fine-grained temporal annotation.This makes the paradigm practical for large-scale surveillance data.
- Limitations: Fixed temporal assumptions and top-k heuristics poorly match variable anomaly dynamics, allowing noisy snippets to dominate video-level optimization.Snippet reliability is especially limited before discriminative representations emerge.
- Proposed framework: The Temporal Refinement Module uses a class token and Dynamic Position Encoding to model long-range dependencies and stabilize global video-level supervision.The class token distills a robust global representation from the snippet sequence.
- Proposed framework: The Event Segmentation Module detects temporal discontinuities, aggregates snippets into event representations, and accommodates short-lived and long-duration anomalies.It elevates temporal modeling from snippets to events.
- Proposed framework: Similarity-based fusion dynamically weights snippet- and event-level scores using their relevance to the global video representation instead of fixed top-k aggregation.This provides an alternative to heuristic video-level prediction.
- Evaluation: The framework consistently outperforms state-of-the-art methods on UCF-Crime and XD-Violence, with ablations validating the complementary roles of its components.The experiments evaluate the complete framework and individual modules.
2 Proposed Method
The framework models video anomalies across snippet, event, and video granularities. TRM provides context-aware representations, ESM captures variable-length events, and adaptive fusion combines multi-level predictions for video-level supervision.
- The framework extracts snippet features, refines them with TRM, segments events with ESM, and jointly optimizes snippet-, event-, and video-level outputs.The architecture produces global video, refined snippet, and event representations before multi-scale prediction.
- 2.1 Temporal Refinement Module (TRM): TRM prepends a learnable class token and applies self-attention with dynamic positional encoding to capture long-range context while preserving local snippet cues.The class token yields a global video representation, while residual refinement produces context-aware snippet features.
- 2.2 Event Segmentation Module (ESM): ESM detects semantic transitions through smoothed feature divergence, local background normalization, and MAD-based adaptive thresholding to identify event boundaries.Detected event spans are aggregated into event-level representations using uniform or attention-derived weights.
- 2.2 Event Segmentation Module (ESM): Event-level representations aggregate temporally coherent snippets within detected start and end indices, producing features that capture dominant dynamics of each event.The aggregation can use uniform averaging or attention scores from TRM.
- 2.3 Adaptive Score Fuse: The model independently classifies global, refined snippet, and event representations, then applies video-level supervision to predictions from all temporal granularities.The classifiers produce video-, snippet-, and event-level anomaly scores, while the loss is applied to their resulting video-level predictions.
- 2.3 Adaptive Score Fuse: Adaptive fusion replaces fixed top-k pooling by weighting snippet and event scores according to semantic similarity with the global video representation.This design addresses top-k sensitivity when anomaly frequency and duration vary across videos and scenes.
3 Experiment
Experiments on UCF-Crime and XD-Violence compare the proposed framework with SOTA methods and evaluate its components through ablations and qualitative visualizations. The method achieves strong benchmark performance, with adaptive modules contributing measurable gains.
- Datasets: UCF-Crime contains 1,900 long, untrimmed videos spanning 13 anomalous-event types, with video-level training labels and frame-level test annotations.
- Datasets: XD-Violence contains 4,754 videos across six violent-event categories and presents broader scenarios, heterogeneous recording devices, and higher intra-class variability than UCF-Crime.
- Evaluation Metrics: AUC is used for UCF-Crime, while XD-Violence uses AP; FAR is additionally reported to assess robustness on normal test videos.
- Comparison with SOTA Methods: 87.24% AUC on UCF-Crime surpasses the previous SOTA by 0.27%, while FAR is 0.54% versus 1.05% for UR-DMU.
- Comparison with SOTA Methods: 83.89% AP and 0.46% FAR on XD-Violence represent the proposed method’s best reported results on that benchmark.
- Ablation Study: Ablations on UCF-Crime show a 2.28% improvement from TRM, a 0.74% AUC gain from the class token, and 87.24% performance after adding ESM.
- Qualitative Analysis: Visualizations across both benchmarks show anomaly scores aligned with normal and anomalous temporal intervals for videos containing single or multiple anomalies.
4 conclusion
The paper concludes that its adaptive WSVAD framework addresses limitations of snippet-level analysis and fixed aggregation through complementary temporal modules and adaptive score fusion. Experiments support the model and its components, while future work considers multimodal signals for complex environments.
- The framework combines TRM and ESM to address conventional snippet-level analysis and fixed aggregation heuristics in WSVAD.TRM models long-range dependencies and distills a holistic video representation; ESM identifies event boundaries and captures event-level features.
- TRM uses a learnable class token and dynamic position encoding to model long-range temporal dependencies and produce holistic video-level representations.
- ESM adaptively identifies event boundaries to represent anomalies with variable frequency and duration.
- Adaptive similarity-based fusion replaces traditional top-k pooling for more robust and generalized video-level predictions.
- Future work will integrate multimodal signals, including audio and optical flow, to enhance robustness in complex, cluttered environments.