PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Large Language Models for Training-free Video Anomaly Detection

Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, Elisa Ricci

arXiv:2404.01014v1cs.CV

TL;DR

Video anomaly detection typically depends on training data, creating domain-generalization and data-collection challenges. LAVAD replaces that requirement with a training-free pipeline that combines VLM captioning, LLM temporal scoring, and cross-modal refinement. On UCF-Crime and XD-Violence, it achieves competitive or superior performance against unsupervised and one-class methods without training or additional data collection.

  • Problem

    Existing VAD methods require training data, which limits generalization across settings and complicates data collection in privacy-sensitive domains.

  • Method

    LAVAD uses pre-trained VLMs to caption video frames, an LLM to summarize temporal context and estimate anomaly scores, and cross-modal similarity to clean captions and refine scores.

  • Results

    LAVAD achieves competitive or superior performance against unsupervised and one-class VAD methods on UCF-Crime and XD-Violence without training or additional data collection.

  • Takeaways & Limitations

    Training-free VAD can be implemented by combining existing foundation models for language-based temporal scoring and visual-language refinement.

  • Takeaways & Limitations

    LAVAD is limited by noisy frame captions and the lack of global scene context and dynamics in individual captions.

Abstract

from arXiv · show

Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Existing works mostly rely on training deep models to learn the distribution of normality with either video-level supervision, one-class supervision, or in an unsupervised setting. Training-based methods are prone to be domain-specific, thus being costly for practical deployment as any domain change will involve data collection and model training. In this paper, we radically depart from previous efforts and propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm, exploiting the capabilities of pre-trained large language models (LLMs) and existing vision-language models (VLMs). We leverage VLM-based captioning models to generate textual descriptions for each frame of any test video. With the textual scene description, we then devise a prompting mechanism to unlock the capability of LLMs in terms of temporal aggregation and anomaly score estimation, turning LLMs into an effective video anomaly detector. We further leverage modality-aligned VLMs and propose effective techniques based on cross-modal similarity for cleaning noisy captions and refining the LLM-based anomaly scores. We evaluate LAVAD on two large datasets featuring real-world surveillance scenarios (UCF-Crime and XD-Violence), showing that it outperforms both unsupervised and one-class methods without requiring any training or data collection.

1. Introduction

Video anomaly detection must localize rare, context-dependent deviations, but existing approaches require training data and therefore face generalization and data-collection constraints. LAVAD addresses this gap with a training-free pipeline combining VLM-generated captions, LLM temporal reasoning, and cross-modal refinement.

  • Motivation: VAD is difficult because anomalies are undefined, context-dependent, and rare in real-world videos.Existing methods therefore learn normality with fully supervised, weakly supervised, one-class, or unsupervised training regimes.
  • Motivation: Training-based VAD methods can underperform across recording conditions, while surveillance data collection is constrained by privacy concerns.These limitations motivate the question of whether VAD can be performed without training.
  • LAVAD: LAVAD is introduced as the first training-free, language-based VAD method using pre-trained VLMs and LLMs.It targets deployment settings where task-specific supervision and training data are unavailable.
  • LAVAD: LAVAD generates frame-level textual descriptions, cleans noisy captions through cross-modal frame-caption similarity, and summarizes temporal windows with an LLM.The temporal summaries prompt frame-level anomaly scores, which are further refined by aggregating semantically similar frames.
  • Results: Experiments show competitive results against unsupervised and one-class VAD methods without task-specific supervision or training.The evaluation uses real-world surveillance benchmarks, supporting the proposed training-free setting.

2. Related Work

Prior VAD methods are organized by supervision level and generally require training, while related LLM-based approaches still involve human refinement or adaptation. LAVAD instead combines VLMs and LLMs for temporal anomaly detection without training or data collection.

  • Video Anomaly Detection: Existing VAD methods span supervised, weakly supervised, one-class, and unsupervised settings, all relying on collected data and model training.These categories differ in whether they use frame labels, video labels, only normal videos, or unlabeled videos.
  • Video Anomaly Detection: LAVAD sidesteps data collection and training by exploiting existing large-scale foundation models in a training-free VAD pipeline.This directly distinguishes it from the training-based literature reviewed in the section.
  • LLMs for VAD: Earlier LLM-related anomaly methods use LLMs mainly for textual descriptions or require human-in-the-loop refinement and further VLM training.These approaches address different settings or retain adaptation requirements.
  • LLMs for VAD: LAVAD combines LLMs and VLMs for temporal anomaly detection in videos and requires neither training nor data collection.The method is presented as the first training-free approach for this VAD setting.

3. Training-Free VAD

The paper introduces training-free VAD, which uses only pre-trained models at inference time to estimate frame-level anomaly scores. LAVAD combines caption cleaning, LLM-based temporal scoring, and video-text score refinement to address noisy captions and missing scene dynamics.

  • 3.1. Problem formulation: Training-free VAD estimates anomaly scores using only pre-trained models at inference time, without training or fine-tuning on a dataset.
  • 3.2. Are LLMs good for VAD?: The preliminary method captions each frame with BLIP-2 and prompts an LLM to select one of 11 uniformly sampled anomaly scores from 0 to 1.The prompt distinguishes normal scenes with score 0 from anomalous scenes with score 1.
  • 3.2. Are LLMs good for VAD?: On UCF-Crime, the preliminary LLM-based approach is evaluated with multiple BLIP-2 variants and LLMs against an unsupervised method and random scoring, but remains much lower than trained state-of-the-art methods.
  • 3.2. Are LLMs good for VAD?: Frame-level captions can be noisy and omit global scene context and dynamics, limiting the reliability of LLM-based anomaly scores.Figure 3 illustrates a semantically incorrect caption inside a ground-truth anomaly that produces a low anomaly score and a wrong classification.
  • 3.3. LAVAD: LAnguage-based VAD: LAVAD cleans captions by replacing each frame caption with a semantically aligned caption from the video, using image-text representations to correct noisy descriptions.This can propagate captions that better match visual content regardless of their temporal positions.
  • 3.3. LAVAD: LAnguage-based VAD: LAVAD aggregates cleaned captions within temporal windows, prompts an LLM for temporal summaries and initial scores, then refines scores using video-text similarity across semantically similar frames.The final refined scores are thresholded to identify anomalous temporal windows, and the architecture uses frozen off-the-shelf models.

4. Experiments

LAVAD is evaluated on two real-world surveillance datasets against supervised, unsupervised, one-class, weakly-supervised, and training-free baselines, with component and prompt ablations. It achieves strong benchmark performance without training, while temporal summarization and score refinement materially improve detection.

  • Experimental setup: Experiments use UCF-Crime and XD-Violence surveillance datasets, frame-level AUC ROC, and for XD-Violence also average precision, with BLIP-2, Llama-2-13b-chat, and ImageBind modules.Videos are sampled every 16 frames, and ImageBind supplies multimodal encoders.
  • Comparison with state of the art: LAVAD outperforms one-class and unsupervised methods on UCF-Crime, improving AUC ROC by +6.08% over GCL and +0.52% over DyAnNet.The comparison includes state-of-the-art and training-free baselines.
  • Comparison with state of the art: On XD-Violence, LAVAD achieves superior performance compared with all one-class and unsupervised methods.The supplied passage identifies RareAnom as the best-scoring unsupervised comparator and describes LAVAD’s improvement as substantial.
  • Comparison with state of the art: Naive zero-shot VLM baselines perform poorly, while LLaVA improves over them but remains below LAVAD’s temporal-description approach.The authors attribute the gap partly to VLMs emphasizing foreground objects rather than actions or contextual background information.
  • Ablation study: Removing caption cleaning lowers AUC ROC by −3.8%, while removing temporal summarization or score refinement lowers it by −7.58% and −7.49%, respectively.These ablations evaluate the three proposed components on UCF-Crime.
  • Ablation study: AUC ROC consistently increases as K grows and saturates near 9 semantically similar frames, supporting score refinement through related temporal summaries.The experiment uses UCF-Crime and evaluates K in the Video-Text Score Refinement component.

5. Conclusions

The paper introduces LAVAD as a training-free VAD method that combines VLMs and LLMs across caption cleaning, temporal anomaly estimation, and score refinement. Across UCF-Crime and XD-Violence, it outperforms training-based unsupervised and one-class methods without training or additional data collection.

  • Conclusion: LAVAD cleans captions with image-text similarity, estimates temporal anomaly scores with an LLM, and refines them using video-text similarity across semantically close frames.These are the method’s three main components.
  • Conclusion: On UCF-Crime and XD-Violence, LAVAD outperforms training-based unsupervised and one-class methods without training or additional data collection.The conclusion frames this as evidence for the training-free VAD setting.

Supplementary Material

The supplementary material provides prompt specifications and additional analyses of task priors, captioning-model variants, and hyperparameters.

  • Supplementary analyses: The supplement analyzes task-related prompting priors on XD-Violence, BLIP-2 captioning variants on both datasets, and additional hyperparameter settings.It first gives the exact prompts used by LAVAD, then reports these analyses.

A. Prompts

LAVAD uses separate prompts for contextual priors, output formatting, and temporal summarization. These prompts respectively guide anomaly interpretation, constrain score output, and produce concise temporal summaries for the LLM.

  • Contextual prompt: The contextual prompt PC supplies dataset-specific VAD priors, including law-enforcement framing and suspicious-activity interpretation for UCF-Crime.The prompt differs between UCF-Crime and XD-Violence.
  • Output-format prompt: The output prompt PF requests a Python list containing exactly one anomaly score from 0 to 1 in fixed increments.It is kept consistent across both datasets to facilitate automated parsing.
  • Temporal-summary prompt: The summarization prompt PS asks the LLM to condense each frame’s temporal scene description into a few sentences without unnecessary details.The resulting summary S_i is used in the anomaly-scoring pipeline.

B. Additional analyses

Additional analyses show that LAVAD’s performance depends on prompt priors, caption selection, and temporal sampling choices across UCF-Crime and XD-Violence.

  • Additional analyses: Table 5 reports LAVAD results on XD-Violence under different context-prompt priors, while Tables 6–8 cover captioner and temporal-sampling analyses.These tables define the evaluated ablation scopes rather than supplying additional comparisons beyond the accompanying analyses.
  • Task priors in the context prompt: Anomaly priors improve XD-Violence AP by +1.67%, whereas impersonation decreases AP by −1.51%; combining both improves performance mainly through the anomaly prior.The impersonation prior may be less effective beyond CCTV footage because XD-Violence includes movies, sports, games, and other sources.
  • Impact of different BLIP-2 models: The BLIP-2 ensemble is most effective on UCF-Crime, where semantically selecting captions from multiple models helps address low-resolution CCTV footage.For XD-Violence, flan-t5-xxl captions achieve the best AP, while ensemble selection can overlook anomaly-relevant background elements.
  • Temporal window’s duration and number of sampled frames: A 10-second temporal window produces the highest UCF-Crime AUC when sampling 10 frames, and 10 frames is optimal among the tested counts.The sampling trade-off balances caption coverage against excessive, non-diverse content that can overwhelm the temporal summary.

C. Qualitative results

Qualitative examples show LAVAD often associates high anomaly scores with abnormal content, while also exposing errors caused by misleading temporal descriptions and captioning interpretations.

  • C. Qualitative results: LAVAD detects several abnormal segments qualitatively, but it also produces false positives when summaries describe future content or misinterpret suspicious behavior.In Shooting047, early and final segments receive high scores for different reasons; in Burglary079, a caption incorrectly suggests car theft.

D. Limitations

LAVAD is constrained by the reliability and cross-domain alignment of its pretrained VLM and LLM components, while LLM prompting remains insufficiently understood for VAD.

  • D. Limitations: LAVAD’s performance depends on caption quality, LLM anomaly-score reliability, and multimodal encoder alignment across video domains.The authors also identify a lack of systematic understanding of LLM prompting strategies for VAD.

E. Broader Societal Impacts

The paper identifies unresolved ethical issues for deploying LLM-based VAD in safety-related private and public contexts.

  • E. Broader Societal Impacts: Before deployment, the authors call for investigation of LLM-based VAD behavior, mitigation of potential bias, and improved explainability.The concern applies to safety-related uses serving private or public interests.
Loading 2404.01014v1…