PaperlayerView exact revision on arXiv

Source-linked AI summary

Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization

Mulugeta Weldezgina Asres, Lei Jiao, Christian Walter Omlin

arXiv:2610.32073v1cs.CVcs.AI

TL;DR

Privacy-preserving VAD remains underexplored in explainability, despite the need for transparent anomaly decisions in surveillance. XAVAD combines zero-shot anonymization with category, heatmap, and context explanations. The authors report substantial gains over benchmark approaches and discuss trade-offs between anonymization and XAI.

  • Problem

    Unified zero-shot VAD explainability combining anomaly categorization, spatial localization, and context, while also protecting privacy, remains underexplored.

    Source: §I · p. 2

  • Method

    XAVAD integrates full-body anonymization with zero-shot anomaly categorization, heatmap localization, and VLM-generated context descriptions.

    Source: §I · p. 2

  • Results

    The authors report substantial accuracy and computational gains over benchmark approaches and identify trade-offs between anonymization and XAI.

    Source: §I · p. 2

  • Takeaways & Limitations

    The study concludes that zero-shot VAD explainability depends on masking and multimodal models as well as supporting pre- and post-processing techniques.

    Source: §V · p. 11

Abstract

from arXiv · show

Despite the promising development of deep learning for video anomaly detection (VAD), explainability (XAI) of these models remains a challenge. Explainable and privacy-protecting VAD builds transparency and trust that facilitates adoption of computer vision systems, including safety and security monitoring. However, anomaly explainability in privacy-preserving VAD systems is underexplored. In this study, we present the XAVAD approach, a unified descriptive anomaly XAI for an anonymized VAD, that integrates flexible zero-shot anonymization, anomaly type categorization, visual localization, and scene context description. We provide empirical and qualitative assessments of multifaceted explanations integrated with full-body visual anonymization for privacy protection. We adopt a multimodal CLIP embedding for videos, using improved LLM-assisted prompting and pool-scoring mechanisms for anomaly-type identification. We enhance Grad-ECLIP for spatial anomaly heatmap localization, leveraging computation and accuracy through our novel prompt-guided patching autofocus mechanism. We employ VLMs to generate context and reasoning descriptions of detected anomalies through curated prompt engineering. We have evaluated the accuracy and computational cost of XAVAD on several publicly available CCTV and video datasets. Our experiments demonstrate substantial gains of our XAVAD over state-of-the-art benchmark approaches and provide insights into privacy protection trade-offs in XAI for VAD. We further present ablation studies and assessments across diverse configurations and modeling alternatives to demonstrate the efficacy of our proposed methods. Code: https://github.com/muleina/XAVAD}{https://github.com/muleina/XAVAD .

I. INTRODUCTION

The paper addresses gaps in unified zero-shot explainability that combines anomaly categories, spatial localization, and context while also protecting privacy. XAVAD brings these functions together for anonymized VAD, and the authors report substantial accuracy and computational gains over benchmarks.

I. INTRODUCTION

  • Prior work has limited unified zero-shot XAI combining semantic categorization, spatial heatmaps, and context in model-agnostic VAD, and has not adequately integrated privacy protection with XAI and detection.The paper also notes that multimodal models can propagate biases and incur memory and latency costs in real-time deployments.Source: §I · p. 2, A. Explainable Video Anomaly Detection · p. 3
  • The authors report substantial accuracy and computational gains over benchmark approaches and describe trade-offs between anonymization and XAI.They evaluate XAVAD on publicly available CCTV and video datasets and include comparisons and ablations across configurations and modeling alternatives.Source: §I · p. 2
  • XAVAD combines anonymization, VAD, and textual and visual XAI in one zero-shot pipeline, with anomaly categorization, localization, and context description.Its video categorization uses CLIP with LLM-assisted prompts and pooled scores; localization adapts Grad-ECLIP, and VLMs generate context descriptions.Source: §I · p. 2
  • The proposed prompt-guided patched autofocus improves heatmap accuracy and reduces computation when handling sources with different video resolutions.The paper identifies image patching as a way to address resolution variation that otherwise carries computational overhead.Source: §I · p. 2
  • Zero-shot open-vocabulary negative segmentation (GDSN) improves full-body anonymization masks by reducing concealment of overlapping objects such as weapons.The approach targets a limitation of closed segmentation methods when objects overlap.Source: §I · p. 2

A. Explainable Video Anomaly Detection

  • Earlier explainable VAD approaches use multiple VLM queries for temporal scoring, box localization, and captioning, while pretrained VLM explanations often lack spatial heatmaps.The cited framework’s multiple-query overhead is identified as a possible challenge for real-time deployment.Source: A. Explainable Video Anomaly Detection · p. 3

B. Multimodel Embedding for Zero-Shot Modeling

  • CLIP-style multimodal embeddings support zero-shot classification by matching video or image embeddings with class-specific text descriptions, without task-specific training.The paper focuses on CLIP ViT variants for performance over ResNet variants.Source: B. Multimodel Embedding for Zero-Shot… · p. 3

C. Zero-Shot Spatial Semantic Localization

The methodology describes XAVAD’s spatial localization and context-generation approaches, then formulates privacy-preserving VAD through anonymization and explanations for active alerts. Its framework uses full-body masking and combines anomaly categories, context descriptions, and heatmaps.

C. Zero-Shot Spatial Semantic Localization

  • Grad-ECLIP improves CLIP image-text localization over classification-oriented explanations; this study adapts it for high-resolution video VAD XAI, while heatmaps offer pixel-level scoring and greater precision than bounding boxes.Bounding boxes may cover non-target areas, whereas segmentation maps improve boundary localization.Source: C. Zero-Shot Spatial Semantic… · p. 3

D. Zero-Shot Context and Reason Generation

  • VLMs generate anomaly context and reasoning only after lightweight VAD detects an anomaly, reducing the cost of running expensive VLM inference on every video stream.The text notes that VLM use for both detection and explanation can propagate model bias into VAD decisions.Source: D. Zero-Shot Context and Reason… · pp. 3–4

A. Problem Formulation

  • The privacy-preserving system Γ aims to retain VAD efficacy while reducing privacy leakage, with an anonymization system removing sensitive information before the VAD produces an alert score.The system diagram identifies Γ as the anonymized VAD and Ω as the multifaceted anomaly explainer.Source: A. Problem Formulation · p. 4, Figure 2 · p. 4
  • Full-body masking applies anonymization to human subjects using a binary segmentation mask, leaving unmasked frame content unchanged.The mask is generated by a segmentation model and applied frame by frame.Source: A. Problem Formulation · p. 4
  • For alerts above the VAD threshold, Ω provides an anomaly category, context description, and heatmap.The framework is presented as a unified, full-body privacy-protecting VAD system.Source: A. Problem Formulation · p. 4, Figure 1 · p. 4

B. Lightweight Full-Body Anonymization

XAVAD applies full-body anonymization and combines zero-shot anomaly categorization, spatial localization, and context description. Its methods include both closed-vocabulary segmentation and an open-vocabulary approach designed to avoid masking relevant overlapping objects.

B. Lightweight Full-Body Anonymization

1) Close-Vocabulary Segmentation Models:

  • Closed-vocabulary YOLO and RF-DETR segmentation can miss relevant overlapping objects; SAHI processes image patches to improve detection of small or distant objects without retraining.Source: 1) Close-Vocabulary Segmentation Models · p. 4

2) Open-Vocabulary Segmentation Models:

C. Zero-Shot Anomaly Explanation System

D. Zero-Shot Anomaly Categorization Approach

1) Anomaly Category CLIP Embedding:

  • LLMs generate crowd-monitoring descriptions for anomaly classes; averaging multiple CLIP text embeddings addresses single-description noise, with five descriptions and ten image-quality templates per anomaly type.Source: 1) Anomaly Category CLIP Embedding · p. 5

2) Video CLIP Embedding:

The categorization method embeds video frames with CLIP and scores them against anomaly classes. It uses the strongest frame-level match to identify the anomaly class and locate a relevant frame.

2) Video CLIP Embedding:

  • The method embeds consecutive video frames with CLIP and selects the maximum-similarity frame, since averaging can suppress brief alerts and miscategorize them.Source: 2) Video CLIP Embedding · p. 5

3) Zero-Shot Categorization Scoring:

E. Zero-Shot Anomaly Spatial Localization

The spatial explanation method adapts Grad-ECLIP for anomaly heatmaps and introduces prompt-guided patched autofocus to focus computation on a candidate region. It also adjusts heatmap scaling to address weak or outlying scores.

E. Zero-Shot Anomaly Spatial Localization

  • Grad-ECLIP generates a saliency heatmap for the category predicted by zero-shot categorization, and the method modifies it for computational efficiency and localization precision.Source: E. Zero-Shot Anomaly Spatial… · p. 7
  • Small relevant objects can be hard to distinguish after frame resizing, while patch-based alternatives may require training or incur costs as patch counts grow.Source: E. Zero-Shot Anomaly Spatial… · p. 7
  • PPA first uses lightweight CLIP prompt matching to select a relevant patch, then applies Grad-ECLIP to that patch for the anomaly heatmap.Source: E. Zero-Shot Anomaly Spatial… · p. 7
  • PPA slides a window over the frame with the strongest anomaly-category match and selects a candidate patch from the resulting slices.Source: E. Zero-Shot Anomaly Spatial… · p. 7
  • The scaling adjustment addresses weak peak scores that can be artificially amplified and outlying scores that can shrink other relevant heatmap regions.Source: E. Zero-Shot Anomaly Spatial… · p. 7

F. Zero-Shot Anomaly Context-Description Generation

For anomaly context descriptions, XAVAD uses structured prompts with vision-language models. The prompts constrain outputs, while model settings and sampled frames guide generation.

F. Zero-Shot Anomaly Context-Description Generation

  • The authors primarily use InternVL for speed, accuracy, and structured-output compliance; they report hallucinations or output-format difficulties with some alternatives.Source: F. Zero-Shot Anomaly… · p. 7
  • Structured prompts specify task rules and output formatting, and the ZSAD function returns anomaly status, category, and context explanation.Source: F. Zero-Shot Anomaly… · p. 7
  • Low-temperature settings, constrained sampling parameters, and uniform frame sampling are used to encourage grounded context generation.Source: F. Zero-Shot Anomaly… · pp. 7–8

IV. RESULTS AND DISCUSSION

The evaluation assesses anomaly categorization, localization, context description, and computational cost, with results showing that LLM prompting generally improves zero-shot anomaly categorization while anonymization reduces scores. Results are evaluated across several public datasets using modality-specific metrics.

A. Evaluation Approach and Performance Metrics

  • The evaluation uses standard test splits for XD Violence, UCF-Crime, and HIVAU-70k, but the full Simuletic dataset because it is small.It assesses categorization with mF1 and confusion matrices, localization with AP, AR, and IoU, context with BLEU, ROUGE, and BERTScore, and computational cost with peak GPU memory and processing time.Source: §IV · p. 8, A. Evaluation Approach and Performance… · p. 8
  • Higher scores indicate better retention of VAD-XAI utility after anonymization, and ∆(%) measures relative score change between MASKED-AN and No-AN.Tables and figures mark better-score direction with arrows and the best score in bold.Source: A. Evaluation Approach and Performance… · p. 8

B. Evaluation on Anomaly Type Categorization

  • LLM-Prompt mF1@Top1 reaches 0.569–0.689 on XD Violence, outperforming Keyword-Prompt at 0.469–0.516 and ConceptNet at 0.542–0.656 across CLIP models without anonymization.For MASKED-AN, the reported ∆mF1@Top1 drops are 7.28%–12.74%, 3.45%–5.69%, and 5.29%–10.19%, respectively; ViT-L/14 has the smallest drop, 3.45%–7.28%.Source: B. Evaluation on Anomaly Type… · p. 8
  • On UCF-Crime, ViT-L/14 models outperform ViT-B models, while LLM prompting is generally better and more stable across No-AN and MASKED-AN configurations.After anonymization, ∆mF1@Top3 ranges are 3.51%–9.17% for LLM, 3%–20.84% for Keyword, and 6.17%–15.90% for ConceptNet prompts.Source: B. Evaluation on Anomaly Type… · p. 8

1) Impact of Temporal Score Pooling:

The results examine temporal score pooling and spatial anomaly localization. Prompt-guided patching autofocus improves localization performance, while anonymization and overlap-related segmentation challenges constrain utility.

1) Impact of Temporal Score Pooling:

  • LLM-assisted prompting increases the benefit of max-pooling over average-pooling: on XD, mF1 gains are 33% and 28% before AN at Top1 and Top2, and 28% and 22% after AN.On UCF-Crime, the corresponding gains are −4% and 1% before AN, and 13% and 18% after AN at Top1 and Top3; max-pooling alone yields gains of at most 3%.Source: 1) Impact of Temporal Score Pooling · p. 8, Table III · p. 9

C. Evaluation on Anomaly Visual Localization

  • On the Simuletic CCTV Weapon dataset, PPA raises AP from 0.307 to 0.569 and AR from 0.473 to 0.715, with AP increasing from 0.18 to above 0.90 as IoU falls from 50% to 10%.For small objects, Grad-ECLIP has APsmall = 0.0, while PPA reaches APsmall = 0.710 and maintains higher accuracy for medium and large objects.Source: C. Evaluation on Anomaly Visual… · pp. 9–10, Table IV · p. 10
  • On the Simuletic CCTV Knife dataset, PPA raises AP from 0.098 to 0.589, and AP rises from 0.088 to 0.952 as IoU falls from 50% to 10%.The reported IoU@50% performance is associated with the dataset’s predominantly medium- and large-sized objects.Source: Table V · p. 10, C. Evaluation on Anomaly Visual… · p. 10
  • MASKED-AN degrades AP by 43% on weapon data and 36% on knife data, with larger drops at IoU@50% than at IoU@10%.The reported drops at IoU@50% are 51% and 75%, while those at IoU@10% are around 27% and 23%, respectively.Source: C. Evaluation on Anomaly Visual… · p. 10
  • Pre-trained YOLO segments unknown objects poorly when they overlap a person’s body, and small far-view figures produce imprecise mask boundaries around handheld objects.The black anonymization mask can also affect saliency for predominantly dark weapons.Source: C. Evaluation on Anomaly Visual… · p. 10

1) Robustness of Masking Models on Overlapped Objects:

XAVAD’s masking robustness depends on preventing anonymization from concealing overlapping threat objects, while its context descriptions and computational profile reveal additional accuracy and efficiency trade-offs. Across the conclusion, the authors report substantial benchmark gains and emphasize that zero-shot VAD explainability depends on masking, multimodal models, and supporting processing techniques.

1) Robustness of Masking Models on Overlapped Objects:

D. Evaluation on Anomaly Context Generation

  • InternVL3.5 reaches BERTScore F1 0.422 on HIVAU-70k, with MASKED-AN drops of 5.70%–5.97% on XD and 2.41%–6.71% on UCF-Crime.The authors note that anonymization removes personal and visual attributes present in the ground-truth captions, lowering the score.Source: D. Evaluation on Anomaly Context… · p. 11

E. Computational Cost Analysis

  • PPA uses 4× less memory and processes 30% faster while improving ZSAM AP accuracy by 4–6× over Grad-ECLIP; simple patching with Grad-ECLIP costs 1.8× more memory and 6.8× more time.GDSN has relatively higher memory cost because it uses large SAM models, though its speed difference from YOLO-M and YOLO-X is marginal.Source: E. Computational Cost Analysis · p. 11

V. CONCLUSION

  • XAVAD reports substantial accuracy and computational gains over benchmark approaches, with results showing that masking, multimodal models, and pre- and post-processing all matter for zero-shot VAD explainability.The study also discusses trade-offs between anonymization and explainability.Source: §V · p. 11

APPENDIX

The appendix situates XAVAD among prior explainable VAD work, describes CLIP model alternatives, and details prompting choices for categorization and context description. Its VLM instructions emphasize literal, visually grounded analysis and structured outputs.

A. Overview of Related Studies

  • Prior explainable VAD methods address textual or spatial grounding, but anomaly categorization and localization evaluations mostly remain qualitative; XAVAD evaluates all explanation modalities qualitatively and quantitatively while integrating anonymization.The approaches differ in evaluation rigor, training requirements, and privacy protection through visual anonymization.Source: A. Overview of Related Studies · p. 14

B. Highlights of CLIP Models

C. LLM-Assisted CLIP Prompting for ZSAC

  • Small LLMs generate CCTV-specific, visually observable, action-oriented CLIP prompts for anomaly categories, while avoiding legal interpretations.The prompts focus on visible actions, body motion, and object interactions.Source: C. LLM-Assisted CLIP Prompting for ZSAC · p. 14

D. VLM Prompting and Settings for ZSAD

  • ZSAD prompts require VLMs to use visible evidence, avoid inferring intent or identity, and return a structured anomaly status, category, and reason.The instructions include an UNKNOWN outcome when visual data is inadequate and use low temperature to limit hallucination.Source: D. VLM Prompting and Settings for ZSAD · pp. 14–15

E. Further Results for ZSAC

The ZSAC results show strong weighted categorization scores, with top-k performance improving as more candidate categories are considered. The authors also identify category confusion and video quality as sources of difficulty.

E. Further Results for ZSAC

  • XD Violence errors mainly confuse Abuse with Fighting or Shooting, and Top2/Top3 results improve for both No-AN and MASKED-AN.The authors relate this to category similarity and videos containing multiple anomalous scenes.Source: E. Further Results for ZSAC · p. 15
  • On UCF-Crime, darker scenes contribute to Burglary confusions, while low-quality video challenges Arson and Shooting even at Top5.The cited limitations include low resolution, poor lighting, and grayscale video.Source: E. Further Results for ZSAC · p. 15
  • F1@Top1 : 0.894, F1@Top2 : 0.961, and F1@Top3 : 0.980 on XD Violence before AN; these scores drop by 7.16%, 3.65%, and 1.64%, respectively, after AN.The scores use the ViT-L/14 CLIP model with LLM-Prompt.Source: E. Further Results for ZSAC · p. 15
  • On the UCF-Crime dataset, the reported F1@Top5 is 0.947 before AN; after applying AN, it decreases by less than 1.65%.Source: E. Further Results for ZSAC · p. 15

F. Further Results for ZSAM

Further results show that PPA and GDSN improve weapon localization, including after anonymization, while annotation mismatch and occlusion complicate evaluation. The VLM context results also vary with hyperparameters and input length.

F. Further Results for ZSAM

G. Further Results for ZSAD

  • With temperature set to 0, results are consistent across top-k and top-p settings; increasing Nvlm can improve accuracy at lower values but deteriorate at higher values, possibly from context-window saturation.The authors suggest presampling at lower FPS to cover a longer timespan and avoid filling the context window with similar frames.Source: G. Further Results for ZSAD · p. 15, Figure 12 · p. 21

Related papers

Loading 2610.32073v1…