Source-linked AI summary
CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detection
Hyekang Kevin Joo, Khoa Vo, Kashu Yamazaki, Ngan Le
TL;DR
Weakly supervised VAD must localize anomalous snippets in untrimmed videos while relying on limited video-level supervision. CLIP-TSA combines CLIP visual features with Temporal Self-Attention and Difference Maximization training, and experiments evaluate it across three VAD datasets. The paper reports empirical validation against state-of-the-art methods and through ablation studies, while its training objective assumes a bound on the number of abnormal snippets.
Problem
Weakly supervised VAD must localize anomalies when normal frames dominate anomalous videos and anomalies may not stand out against normality.
Method
CLIP-TSA combines CLIP vision-language feature encoding, Temporal Self-Attention for temporal snippet modeling, and Difference Maximization Trainer for weak training.
Results
Experiments and ablation studies empirically validate CLIP-TSA against state-of-the-art methods on UCF-Crime, ShanghaiTech, and XD-Violence.
Takeaways & Limitations
The paper demonstrates the applicability of Temporal Self-Attention to CLIP-extracted features within an end-to-end weakly supervised VAD framework.
Takeaways & Limitations
The expected separability guarantee assumes the selected top-α instances satisfy α ≤ ϵ, where ϵ is the number of abnormal samples in a positive bag.
Abstract
from arXiv · showhide
Video anomaly detection (VAD) -- commonly formulated as a multiple-instance learning problem in a weakly-supervised manner due to its labor-intensive nature -- is a challenging problem in video surveillance where the frames of anomaly need to be localized in an untrimmed video. In this paper, we first propose to utilize the ViT-encoded visual features from CLIP, in contrast with the conventional C3D or I3D features in the domain, to efficiently extract discriminative representations in the novel technique. We then model temporal dependencies and nominate the snippets of interest by leveraging our proposed Temporal Self-Attention (TSA). The ablation study confirms the effectiveness of TSA and ViT feature. The extensive experiments show that our proposed CLIP-TSA outperforms the existing state-of-the-art (SOTA) methods by a large margin on three commonly-used benchmark datasets in the VAD problem (UCF-Crime, ShanghaiTech Campus, and XD-Violence). Our source code is available at https://github.com/joos2010kj/CLIP-TSA.
1. INTRODUCTION
Weakly supervised VAD reduces annotation costs but must localize anomalies amid dominant normal content and subtle deviations. CLIP-TSA addresses these challenges by combining CLIP-based visual features, Temporal Self-Attention, and weak training with Difference Maximization.
- Motivation: Weakly supervised VAD uses video-level labels to reduce the costly frame-level annotations required by fully supervised VAD.Unsupervised VAD can perform poorly because it lacks abnormality knowledge and cannot capture all normality variations.
- Motivation: Untrimmed anomalous videos contain many normal frames, and anomalies may not stand out against normality, making frame localization difficult.The problem is commonly formulated as multiple-instance learning over video snippets.
- Method: Existing MIL-based methods struggle with an arbitrary number of abnormal snippets, motivating a differentiable top-κ function for localizing snippets of interest.The top-κ function is inspired by differentiable top-K selection and uses differentiable hard attention.
- Method: Temporal Self-Attention generates reweighted attention features by measuring each snippet’s degree of abnormality.The resulting attention features are passed to a Difference Maximization Trainer for weak anomaly-classifier training.
- Method: CLIP-TSA replaces conventional C3D or I3D representations with CLIP-based vision-language features to address their domain gap with VAD.The framework consists of CLIP feature encoding, temporal snippet-coherency modeling with TSA, and weak training with Difference Maximization Trainer.
2. PROPOSED METHOD
The method encodes video snippets with CLIP-based visual features, applies Temporal Self-Attention to identify and reweight relevant snippets, then models temporal relationships and trains separation between abnormal and normal instances.
- Feature Encoding: Each snippet is represented by a CLIP visual feature extracted from its middle frame and projected into an image embedding.The resulting video feature set is temporally normalized for batch training.
- Temporal Self-Attention: TSA converts snippet features into scores, selects top-κ snippets, and produces soft one-hot attention vectors emphasizing high-scoring locations.The score vector is generated by a shallow three-layer MLP; noisy score clones support the soft selection process.
- Temporal Self-Attention: The selected attention weights are fused with replicated video features to form reweighted snippet features that constitute the anomaly attention representation.Element-wise multiplication combines the soft selection weights with the feature representations before summation across selected snippets.
- Temporal Modeling: A convolutional module with dilated convolutions and a non-local block models long- and short-term relationships before an MLP classifier predicts snippet-level anomaly scores.The classifier outputs a score for each feature snippet, and these scores are retained for the loss computation.
- Difference Maximization Trainer: Difference Maximization Trainer separates top-α instances from contrasting normal and abnormal bags, using representative features and score-based binary cross-entropy in the loss.The expected-separability formulation assumes the positive bag contains ε abnormal samples and requires α ≤ ε to maximize top-α separability.
3. EXPERIMENTAL RESULTS
Experiments evaluate CLIP-TSA on three VAD datasets and analyze its implementation and components. The reported ablation indicates that both TSA and CLIP features contribute to performance.
- Experiments use UCF-Crime, ShanghaiTech Campus, and XD-Violence to evaluate CLIP-TSA.
- Each video is divided into 32 snippets, with snippet length δ = 16, and CLIP features have dimension d = 512.Training uses an MLP scorer with layers (512, 256, 1), Adam optimization, and batch size 16.
- Performance comparisons examine CLIP-TSA against existing state-of-the-art methods across the three benchmark datasets.The paper states that the comparisons are reported in Tables 2, 3, and 4.
- The ablation study reports performance improvement across all scenarios when TSA is enabled.It compares CLIP, C3D, and I3D features with TSA switched on or off.
- The best ablation performance is obtained by CLIP-TSA, attributed to the combination of CLIP features and the TSA module.C3D-TSA and I3D-TSA also outperform other state-of-the-art models using the corresponding features on UCF-Crime and ShanghaiTech.