PaperlayerView exact revision on arXiv

Source-linked AI summary

VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection

Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, Yanning Zhang

arXiv:2308.11681v3cs.CVcs.MM

TL;DR

VadCLIP addresses the challenge of adapting image-domain CLIP knowledge and vision-language associations to weakly supervised video anomaly detection. It uses frozen CLIP with temporal adaptation, prompt mechanisms, and dual coarse- and fine-grained branches, achieving state-of-the-art results across two WSVAD benchmarks.

  • Problem

    Adapting image-focused CLIP to video anomaly detection requires modeling temporal dependencies, exploiting vision-language connections, and preserving performance under weak supervision.

  • Method

    VadCLIP uses frozen CLIP with an LGT-Adapter, prompt mechanisms, and dual branches combining visual classification with language-visual alignment.

  • Results

    VadCLIP achieves state-of-the-art performance across coarse- and fine-grained WSVAD on two benchmarks, including 84.51% AP on XD-Violence.

  • Takeaways & Limitations

    The dual-branch design transfers pretrained vision-language knowledge to WSVAD while supporting both coarse-grained and fine-grained anomaly detection.

  • Takeaways & Limitations

    The CLIP image and text encoder weights remain fixed during training, so only the devised adapter and prompt parameters are optimized.

Abstract

from arXiv · show

The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features are released at https://github.com/nwpu-zxr/VadCLIP.

Introduction

WSVAD must infer frame-level anomaly confidences from video-level labels, but classification-based pipelines underuse vision-language associations. VadCLIP addresses this with frozen CLIP, temporal adaptation, prompts, and dual coarse- and fine-grained branches.

  • WSVAD requires frame-level anomaly confidences when only video-level annotations are available.
  • Classification-based pipelines extract frame features and feed them to MIL binary classifiers, but fail to fully exploit cross-modal relationships.
  • Adapting image-focused CLIP to weakly supervised video anomaly detection remains an open challenge.
  • VadCLIP uses LGT-Adapter, prompt mechanisms, and MIL-Align to address temporal modeling, knowledge adaptation, and weakly supervised alignment.
  • VadCLIP combines visual binary classification with vision-language alignment, enabling coarse- and fine-grained WSVAD.
  • 84.51% AP on XD-Violence and 88.02% AUC on UCF-Crime are reported as state-of-the-art results.

Related Work

Prior WSVAD research primarily uses weakly supervised classification and temporal modeling, while CLIP-based work has largely relied on visual features without fully using textual semantics.

  • Sultani et al. formulate video anomaly detection as multiple instance learning, treating videos as bags and segments as instances.
  • Subsequent methods model temporal relations with self-attention, transformers, and graph convolutional networks.
  • CLIP-based methods extract discriminative visual features and model long- and short-range temporal dependencies.
  • Classification paradigms predict anomalous-frame probabilities but do not fully utilize textual-label semantics.
  • Vision-language pre-training learns semantic correspondence between visual content and language from large-scale data.

Problem Definition

WSVAD trains with video-level labels while targeting frame-level anomaly confidences. Existing pipelines commonly use pretrained 3D features and MIL classifiers, whereas VadCLIP additionally uses CLIP’s text encoder.

  • A video is labeled normal if no frame is anomalous and abnormal if at least one frame contains an abnormal event.
  • The task trains a detector to predict frame-level anomaly confidences using only video-level annotations.
  • Previous methods typically extract video features with pretrained C3D or I3D models and feed them into MIL-based binary classifiers.
  • VadCLIP uses CLIP’s image encoder for video features and its text encoder to exploit visual-textual associations.

Local and Global Temporal Adapter

VadCLIP models video dynamics with a local transformer and a lightweight global GCN over frozen CLIP frame features. The global module combines feature similarity and positional distance while residual connections limit over-smoothing.

  • Local Module.: The adapter begins with frozen CLIP frame features Xclip ∈ R^n×d, where n is video length and d is set to 512.
  • Local Module.: The local transformer restricts self-attention to overlapping temporal windows, providing a local receptive field with lower computation complexity.
  • Global Module.: A lightweight GCN follows the local module to capture global temporal dependencies.
  • Global Module.: The global module combines adjacency matrices based on feature similarity and relative temporal distance, with row-wise Softmax normalization and one learnable transform weight.
  • Global Module.: Distance-based proximity depends on relative frame positions, while residual connections in both modules prevent feature over-smoothing.

Dual Branch and Prompt

VadCLIP combines visual classification with language-visual alignment, then enriches CLIP text representations using learnable textual and anomaly-focused visual prompts.

  • Dual Branch.: The C-Branch produces frame-level anomaly confidence from temporally modeled video features, while the A-Branch compares those features with CLIP class embeddings.The two branches support binary classification and video-text alignment, respectively.
  • Dual Branch.: In the A-Branch, anomaly labels become CLIP text embeddings rather than one-hot vectors, preserving language knowledge for anomaly detection.
  • Learnable Prompt.: Learnable prompts prepend context tokens to tokenized anomaly labels, forming complete text inputs for the CLIP text encoder.The label is tokenized as tinit, concatenated with learnable context tokens, and encoded as a class embedding.
  • Learnable Prompt.: The class token is placed in the middle of the sequence before positional information is added and the text encoder generates the class embedding.
  • Anomaly-Focus Visual Prompt.: The anomaly-focus visual prompt aggregates video embeddings from abnormal segments to make succinct text labels more accurate.
  • Anomaly-Focus Visual Prompt.: C-Branch anomaly confidence serves as attention, whose dot product with video features produces the normalized visual prompt V.
  • Anomaly-Focus Visual Prompt.: VadCLIP adds V to the class embedding and uses an FFN with a skip connection to obtain an instance-specific class embedding.The resulting class embeddings can extract related visual context from videos.
  • Anomaly-Focus Visual Prompt.: The model computes similarities between class embeddings and frame-level visual features to form the alignment map M.

Objective Function

VadCLIP trains its dual branches with classification, alignment, and contrastive objectives that jointly supervise anomaly prediction and class separation.

  • The C-Branch selects the top K anomaly confidences from abnormal and normal videos and applies binary cross entropy to video-level predictions.
  • The A-Branch uses MIL-Align to average the top K similarities in each class row of the alignment map and measure video-class alignment.
  • The alignment loss Lnce applies cross entropy to class predictions scaled by the temperature hyper-parameter τ.
  • A contrastive loss pushes the normal class embedding away from abnormal class embeddings using their cosine similarity.The normal embedding is tn and abnormal embeddings are ta.
  • VadCLIP’s total objective combines binary classification, alignment, and weighted contrastive losses: L = Lbce + Lnce + λLcts.

Inference

VadCLIP uses its alignment map for fine-grained event detection and combines alternative branch outputs to estimate coarse-grained frame-level anomaly degree.

  • VadCLIP’s dual branch supports both fine-grained and coarse-grained weakly supervised video anomaly detection.
  • For fine-grained detection, thresholding the alignment map M predicts anomalous events.
  • For coarse-grained detection, anomaly degree comes either from C-Branch confidence or from one minus similarity to the normal class in A-Branch.
  • VadCLIP selects the better of the two coarse-grained anomaly-degree calculations for frame-level prediction.

Experiments

VadCLIP is evaluated on coarse- and fine-grained WSVAD across XD-Violence and UCF-Crime, using video-level training labels and CLIP-based visual features. Results, ablations, and qualitative visualizations support the dual-branch design, temporal adapter, and prompt mechanisms.

  • Evaluation setup: VadCLIP is compared on coarse- and fine-grained WSVAD across XD-Violence and UCF-Crime using the same CLIP visual features for all methods.Training videos on both datasets have only video-level labels.
  • Coarse-grained WSVAD: 84.51% AP on XD-Violence and 82.08% AUC on UCF-Crime establish new state-of-the-art coarse-grained results.VadCLIP outperforms semi-supervised and classification-based weakly supervised methods across the reported metrics.
  • Fine-grained WSVAD: 13.1% and 4.5% AVG improvements over Sultani et al. and AVVD on XD-Violence demonstrate VadCLIP’s advantage on fine-grained WSVAD.The fine-grained task requires both multicategory classification accuracy and detection-segment continuity.
  • Ablation studies: 12.3% AP and 9.1% AVG drops without LGT-Adapter show the importance of temporal modeling.The ablation compares the full design with a baseline lacking the temporal adapter.
  • Ablation studies: Local transformer encoder plus GCN is identified as the optimum temporal combination for capturing different temporal dependency ranges.The local-transformer-plus-GCN combination improves AVG by 3.9% over the global-transformer-plus-GCN combination, while the latter has a 0.4% AP advantage.
  • Ablation studies: The dual branch combines coarse-grained classification with fine-grained alignment, while learnable and visual prompts further improve performance.Adding coarse-grained classification yields about 7% AP improvement for A-Branch; hand-crafted and averaged visual prompts reduce AP and AVG.

Qualitative Analyses

VadCLIP produces more discriminative feature representations and accurately localizes abnormal temporal regions across benchmarks.

  • VadCLIP creates more distinguishable feature boundaries that surround corresponding textual class features after specialized optimization.t-SNE visualizations show that raw CLIP features do not effectively separate WSVAD categories.
  • VadCLIP precisely detects abnormal regions across categories while producing low anomaly predictions for normal videos.The qualitative visualizations cover coarse-grained WSVAD on two benchmarks.

Conclusion

VadCLIP adapts frozen CLIP knowledge and vision-language associations to weakly supervised video anomaly detection through temporal modeling, prompt mechanisms, and alignment optimization. Experiments on two benchmarks show state-of-the-art performance, while future work targets open-set VAD.

  • VadCLIP adapts frozen CLIP knowledge and vision-language associations to WSVAD using temporal modeling, prompt mechanisms, and MIL-Align.LGT-Adapter enhances temporal modeling, prompts adapt general knowledge, and MIL-Align supports vision-language alignment under weak supervision.
  • VadCLIP achieves state-of-the-art performance on two WSVAD benchmarks with sufficient ablation studies.
  • Future work will explore vision-language pre-trained knowledge and open-set video anomaly detection.
Loading 2308.11681v3…