VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection

Peng Wu; Xuerong Zhou; Guansong Pang; Lingru Zhou; Qingsen Yan; Peng Wang; Yanning Zhang

Source-linked AI summary

VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection

Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, Yanning Zhang

arXiv:2308.11681v3Aug 22, 2023cs.CVcs.MM

TL;DR

VadCLIP addresses the challenge of adapting image-domain CLIP knowledge and vision-language associations to weakly supervised video anomaly detection. It uses frozen CLIP with temporal adaptation, prompt mechanisms, and dual coarse- and fine-grained branches, achieving state-of-the-art results across two WSVAD benchmarks.

Problem
Adapting image-focused CLIP to video anomaly detection requires modeling temporal dependencies, exploiting vision-language connections, and preserving performance under weak supervision.
Source: Introduction · pp. 1–2
Method
VadCLIP uses frozen CLIP with an LGT-Adapter, prompt mechanisms, and dual branches combining visual classification with language-visual alignment.
Source: Introduction · p. 2, Local Module · p. 3, Dual Branch · p. 4
Results
VadCLIP achieves state-of-the-art performance across coarse- and fine-grained WSVAD on two benchmarks, including 84.51% AP on XD-Violence.
Source: Coarse-grained WSVAD Results · p. 6, Fine-grained WSVAD Results · p. 6, Conclusion · p. 7
Takeaways & Limitations
The dual-branch design transfers pretrained vision-language knowledge to WSVAD while supporting both coarse-grained and fine-grained anomaly detection.
Source: Introduction · p. 2, Dual Branch · p. 4, Conclusion · p. 7
The CLIP image and text encoder weights remain fixed during training, so only the devised adapter and prompt parameters are optimized.
Source: Introduction · p. 2

Abstract

from arXiv · show

The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features are released at https://github.com/nwpu-zxr/VadCLIP.

Introduction

WSVAD must infer frame-level anomaly confidences from video-level labels, but classification-based pipelines underuse vision-language associations. VadCLIP addresses this with frozen CLIP, temporal adaptation, prompts, and dual coarse- and fine-grained branches.

WSVAD requires frame-level anomaly confidences when only video-level annotations are available.Source: Introduction · p. 1
Classification-based pipelines extract frame features and feed them to MIL binary classifiers, but fail to fully exploit cross-modal relationships.Source: Introduction · p. 1
Adapting image-focused CLIP to weakly supervised video anomaly detection remains an open challenge.Source: Introduction · pp. 1–2
VadCLIP uses LGT-Adapter, prompt mechanisms, and MIL-Align to address temporal modeling, knowledge adaptation, and weakly supervised alignment.Source: Introduction · p. 2
VadCLIP combines visual binary classification with vision-language alignment, enabling coarse- and fine-grained WSVAD.Source: Introduction · p. 2
84.51% AP on XD-Violence and 88.02% AUC on UCF-Crime are reported as state-of-the-art results.Source: Introduction · p. 2

Related Work

Prior WSVAD research primarily uses weakly supervised classification and temporal modeling, while CLIP-based work has largely relied on visual features without fully using textual semantics.

Sultani et al. formulate video anomaly detection as multiple instance learning, treating videos as bags and segments as instances.Source: Weakly Supervised Video Anomaly… · p. 2
Subsequent methods model temporal relations with self-attention, transformers, and graph convolutional networks.Source: Weakly Supervised Video Anomaly… · p. 2
CLIP-based methods extract discriminative visual features and model long- and short-range temporal dependencies.Source: Weakly Supervised Video Anomaly… · p. 3
Classification paradigms predict anomalous-frame probabilities but do not fully utilize textual-label semantics.Source: Weakly Supervised Video Anomaly… · p. 3
Vision-language pre-training learns semantic correspondence between visual content and language from large-scale data.Source: Vision-Language Pre-training · p. 3

Problem Definition

WSVAD trains with video-level labels while targeting frame-level anomaly confidences. Existing pipelines commonly use pretrained 3D features and MIL classifiers, whereas VadCLIP additionally uses CLIP’s text encoder.

A video is labeled normal if no frame is anomalous and abnormal if at least one frame contains an abnormal event.Source: Problem Definition · p. 3
The task trains a detector to predict frame-level anomaly confidences using only video-level annotations.Source: Problem Definition · p. 3
Previous methods typically extract video features with pretrained C3D or I3D models and feed them into MIL-based binary classifiers.Source: Problem Definition · p. 3
VadCLIP uses CLIP’s image encoder for video features and its text encoder to exploit visual-textual associations.Source: Problem Definition · p. 3

Local and Global Temporal Adapter

VadCLIP models video dynamics with a local transformer and a lightweight global GCN over frozen CLIP frame features. The global module combines feature similarity and positional distance while residual connections limit over-smoothing.

Local Module.

The adapter begins with frozen CLIP frame features Xclip ∈ R^n×d, where n is video length and d is set to 512.Source: Local Module · p. 3, Local and Global Temporal Adapter · p. 3
The local transformer restricts self-attention to overlapping temporal windows, providing a local receptive field with lower computation complexity.Source: Local Module · p. 3

Global Module.

A lightweight GCN follows the local module to capture global temporal dependencies.Source: Global Module · p. 3
The global module combines adjacency matrices based on feature similarity and relative temporal distance, with row-wise Softmax normalization and one learnable transform weight.Source: Global Module · pp. 3–4
Distance-based proximity depends on relative frame positions, while residual connections in both modules prevent feature over-smoothing.Source: Global Module · p. 4

Dual Branch and Prompt

VadCLIP combines visual classification with language-visual alignment, then enriches CLIP text representations using learnable textual and anomaly-focused visual prompts.

Dual Branch.

The C-Branch produces frame-level anomaly confidence from temporally modeled video features, while the A-Branch compares those features with CLIP class embeddings.The two branches support binary classification and video-text alignment, respectively.Source: Dual Branch · p. 4
In the A-Branch, anomaly labels become CLIP text embeddings rather than one-hot vectors, preserving language knowledge for anomaly detection.Source: Dual Branch · p. 4

Learnable Prompt.

Learnable prompts prepend context tokens to tokenized anomaly labels, forming complete text inputs for the CLIP text encoder.The label is tokenized as tinit, concatenated with learnable context tokens, and encoded as a class embedding.Source: Learnable Prompt · p. 4
The class token is placed in the middle of the sequence before positional information is added and the text encoder generates the class embedding.Source: Learnable Prompt · p. 4

Anomaly-Focus Visual Prompt.

The anomaly-focus visual prompt aggregates video embeddings from abnormal segments to make succinct text labels more accurate.Source: Anomaly-Focus Visual Prompt · p. 4
C-Branch anomaly confidence serves as attention, whose dot product with video features produces the normalized visual prompt V.Source: Anomaly-Focus Visual Prompt · p. 4
VadCLIP adds V to the class embedding and uses an FFN with a skip connection to obtain an instance-specific class embedding.The resulting class embeddings can extract related visual context from videos.Source: Anomaly-Focus Visual Prompt · p. 5
The model computes similarities between class embeddings and frame-level visual features to form the alignment map M.Source: Anomaly-Focus Visual Prompt · p. 5

Objective Function

VadCLIP trains its dual branches with classification, alignment, and contrastive objectives that jointly supervise anomaly prediction and class separation.

The C-Branch selects the top K anomaly confidences from abnormal and normal videos and applies binary cross entropy to video-level predictions.Source: Objective Function · p. 5
The A-Branch uses MIL-Align to average the top K similarities in each class row of the alignment map and measure video-class alignment.Source: Objective Function · p. 5
The alignment loss Lnce applies cross entropy to class predictions scaled by the temperature hyper-parameter τ.Source: Objective Function · p. 5
A contrastive loss pushes the normal class embedding away from abnormal class embeddings using their cosine similarity.The normal embedding is tn and abnormal embeddings are ta.Source: Objective Function · p. 5
VadCLIP’s total objective combines binary classification, alignment, and weighted contrastive losses: L = Lbce + Lnce + λLcts.Source: Objective Function · p. 5

Inference

VadCLIP uses its alignment map for fine-grained event detection and combines alternative branch outputs to estimate coarse-grained frame-level anomaly degree.

VadCLIP’s dual branch supports both fine-grained and coarse-grained weakly supervised video anomaly detection.Source: Inference · p. 5
For fine-grained detection, thresholding the alignment map M predicts anomalous events.Source: Inference · p. 5
For coarse-grained detection, anomaly degree comes either from C-Branch confidence or from one minus similarity to the normal class in A-Branch.Source: Inference · p. 5
VadCLIP selects the better of the two coarse-grained anomaly-degree calculations for frame-level prediction.Source: Inference · p. 5

Experiments

VadCLIP is evaluated on coarse- and fine-grained WSVAD across XD-Violence and UCF-Crime, using video-level training labels and CLIP-based visual features. Results, ablations, and qualitative visualizations support the dual-branch design, temporal adapter, and prompt mechanisms.

Evaluation setup

VadCLIP is compared on coarse- and fine-grained WSVAD across XD-Violence and UCF-Crime using the same CLIP visual features for all methods.Training videos on both datasets have only video-level labels.Source: Comparison with State-of-the-Art Methods · p. 5, Datasets · p. 5

Coarse-grained WSVAD

84.51% AP on XD-Violence and 82.08% AUC on UCF-Crime establish new state-of-the-art coarse-grained results.VadCLIP outperforms semi-supervised and classification-based weakly supervised methods across the reported metrics.Source: Coarse-grained WSVAD Results · p. 6

Fine-grained WSVAD

13.1% and 4.5% AVG improvements over Sultani et al. and AVVD on XD-Violence demonstrate VadCLIP’s advantage on fine-grained WSVAD.The fine-grained task requires both multicategory classification accuracy and detection-segment continuity.Source: Fine-grained WSVAD Results · p. 6

Ablation studies

12.3% AP and 9.1% AVG drops without LGT-Adapter show the importance of temporal modeling.The ablation compares the full design with a baseline lacking the temporal adapter.Source: Effectiveness of LGT-Adapter · p. 6
Local transformer encoder plus GCN is identified as the optimum temporal combination for capturing different temporal dependency ranges.The local-transformer-plus-GCN combination improves AVG by 3.9% over the global-transformer-plus-GCN combination, while the latter has a 0.4% AP advantage.Source: Effectiveness of LGT-Adapter · p. 6
The dual branch combines coarse-grained classification with fine-grained alignment, while learnable and visual prompts further improve performance.Adding coarse-grained classification yields about 7% AP improvement for A-Branch; hand-crafted and averaged visual prompts reduce AP and AVG.Source: Effectiveness of Dual Branch · p. 7, Effectiveness of Prompt · p. 7

Qualitative Analyses

VadCLIP produces more discriminative feature representations and accurately localizes abnormal temporal regions across benchmarks.

VadCLIP creates more distinguishable feature boundaries that surround corresponding textual class features after specialized optimization.t-SNE visualizations show that raw CLIP features do not effectively separate WSVAD categories.Source: Feature Discrimination Visualization · p. 7
VadCLIP precisely detects abnormal regions across categories while producing low anomaly predictions for normal videos.The qualitative visualizations cover coarse-grained WSVAD on two benchmarks.Source: Coarse-grained Qualitative Visualization · p. 7

Conclusion

VadCLIP adapts frozen CLIP knowledge and vision-language associations to weakly supervised video anomaly detection through temporal modeling, prompt mechanisms, and alignment optimization. Experiments on two benchmarks show state-of-the-art performance, while future work targets open-set VAD.

VadCLIP adapts frozen CLIP knowledge and vision-language associations to WSVAD using temporal modeling, prompt mechanisms, and MIL-Align.LGT-Adapter enhances temporal modeling, prompts adapt general knowledge, and MIL-Align supports vision-language alignment under weak supervision.Source: Conclusion · p. 7
VadCLIP achieves state-of-the-art performance on two WSVAD benchmarks with sufficient ablation studies.Source: Conclusion · p. 7
Future work will explore vision-language pre-trained knowledge and open-set video anomaly detection.Source: Conclusion · p. 7