PaperlayerView exact revision on arXiv

Source-linked AI summary

Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding

Mohd Ubaid Wani, Sara Atito, Josef Kittler, Muhammad Awais

arXiv:2610.01754v1cs.CVcs.AI

TL;DR

VAD needs to localize abnormal events, but existing approaches often depend on dataset-specific training, while zero-shot LVLM methods can lack temporal continuity and structured reasoning. Cog-VADU uses sequential cognitive reasoning with structured rationale propagation and cross-modal re-ranking. It reports competitive zero-shot performance across benchmarks, including 90.15% AUC on XD-Violence versus LAVAD’s 85.36%.

  • Problem

    Existing VAD often relies on dataset-specific training, and zero-shot LVLM methods can lack temporal continuity and structured reasoning.

    Source: §1 · p. 3

  • Method

    Cog-VADU uses a frozen pretrained LVLM for sequential reasoning across video segments, with CoADTP rationale propagation and cross-modal re-ranking.

    Source: §1 · p. 3

  • Results

    Cog-VADU reports competitive zero-shot performance across four datasets, including 90.15% AUC on XD-Violence versus LAVAD’s 85.36%.

    Source: Quantitative Results · p. 10, §5 · p. 15

  • Takeaways & Limitations

    The paper concludes that structured cognitive reasoning supports competitive zero-shot anomaly detection and interpretable understanding across the evaluated datasets.

    Source: §5 · p. 15

    Incomplete ground truth that labels only initiating causes can penalize Cog-VADU’s high scores on downstream consequences as false positives.

    Source: Quantitative Results · p. 11

Abstract

from arXiv · show

Video Anomaly Detection (VAD) aims to temporally localize abnormal events in videos. Most existing approaches rely on dataset-specific training and curated annotations, limiting generalization in open-set scenarios. Recent zero-shot methods based on Large Vision- Language Models (LVLMs) alleviate this dependency but often lack temporal continuity and structured reasoning. We propose Cog-VADU, a fully training-free framework that reformulates VAD as a sequential cognitive reasoning task. Cog-VADU introduces Chain-of- Anomaly Detection Thought Prompting (CoADTP), which unrolls an LVLM into a recurrent reasoning chain across video segments. By propagating structured rationales over time, the model maintains implicit temporal memory, enabling robust discrimination between com- plex anomalies and high-motion normal activities. To improve reliability, we further design a cross-modal re-ranking stage that aligns textual rationales with visual embeddings, enforcing semantic consistency and temporal coherence for refined and stable predictions. Extensive experiments on multiple public VAD benchmarks demonstrate that Cog-VADU achieves competitive zero-shot performance. Moreover, cross-model evaluations show that CoADTP consistently enhances reasoning-based anomaly detection in a model-agnostic manner, pro- viding interpretable and generalizable anomaly understanding for real-world applications.

1 Introduction

Cog-VADU addresses the need for contextual, temporally aware anomaly reasoning with a training-free framework that propagates structured rationales across video segments and refines predictions through cross-modal alignment. The introduction also reports competitive zero-shot performance, interpretable outputs, and unified re-annotations of two datasets.

1 Introduction

  • The framework targets contextual distinctions that low-level appearance and motion cues alone cannot resolve, such as running crowds that may signal panic or celebration.The introduction frames VAD as requiring scene context, situational interpretation, temporal evolution, and reasoning about expected behavior.Source: §1 · pp. 1, 3
  • Vision-only methods lack semantic reasoning, while multimodal VAD can offer post-hoc explanations but often depends on task-specific training or curated data.Training-free captioning and similarity methods can also lack multi-step reasoning and struggle with temporal dependencies and evolving scene dynamics.Source: Figure 1 · p. 2, §1 · p. 2
  • The paper asks whether pretrained vision-language models can support training-free, zero-shot VAD through reasoning and understanding.The stated gap is that existing VAD methods often lack structured reasoning and temporal awareness.Source: §1 · p. 3
  • Cog-VADU reformulates anomaly detection as sequential cognitive reasoning with a pretrained LVLM, without task-specific training.CoADTP propagates step-wise rationales over time, while cross-modal re-ranking aligns textual reasoning with visual representations to stabilize predictions.Source: §1 · p. 3
  • Competitive zero-shot performance across benchmarks is reported alongside interpretable reasoning outputs.The introduction also describes unified re-annotations of UCF-Crime and XD-Violence that account for causal and temporal relationships.Source: §1 · p. 3

2 Related Work

Related work spans visual anomaly scoring, supervised and weakly supervised learning, and multimodal methods for explainable VAD. The section identifies a gap in training-free approaches that combine structured reasoning with temporal feedback.

Multimodal and Explainable VAD

  • Traditional visual-cue methods are categorized as non-explainable, while multimodal and explainable VAD incorporates vision-language models to provide semantic descriptions.Related work also describes temporal sampling combined with VLM-based reasoning for interpretable anomaly descriptions.Source: §2 · pp. 3–4

Video Anomaly Detection

  • Unsupervised methods learn normal patterns but can struggle with subtle anomalies and unseen environments; fully supervised methods enable precise localization but require extensive labeling.Weakly supervised VAD commonly uses Multiple Instance Learning, treating a video as a bag of segments and assuming a positive bag contains at least one anomalous segment.Source: §2 · pp. 3–4

Zero-shot and Multimodal Methods

  • Pretrained-VLM approaches include caption-based scoring, instruction-tuned detection, frozen vision-language alignment, and coarse-to-fine reasoning, with some relying on training or showing limited generalization.Other methods fuse visual and textual representations or refine semantic alignment, but may still depend on training or dataset-specific design choices.Source: §2 · p. 4

Chain-of-Thought and Reasoning

  • Chain-of-thought prompting elicits intermediate reasoning, and multimodal work extends it to visual understanding, but its use in VAD remains largely unexplored.The section identifies the gap as integrating structured reasoning with temporal feedback for surveillance analysis.Source: §2 · p. 4

3 Our Methodology: Cog-VADU

Cog-VADU is a two-stage, training-free framework that uses a frozen LVLM for zero-shot anomaly localization, explanation, and structured reasoning. CoADTP carries reasoning across video segments, and cross-modal refinement recalibrates scores for semantic consistency and temporal coherence.

3.1 Problem Formulation

  • Cog-VADU predicts a frame-level anomaly score sequence in [0, 1], where higher values indicate greater abnormality, and also provides semantic descriptions and structured rationales.The framework maintains temporal coherence across frames while producing its zero-shot predictions.Source: §3.1 · pp. 4–5

3.2 How Effective Is Reasoning-Based Prompting for VAD?

  • Naive captioning gives weak anomaly discrimination, while format-structured prompts improve task awareness but remain unstable and lack step-wise reasoning, continuous severity modeling, and temporal consistency.CoADTP addresses these gaps with structured reasoning, severity-aware scoring, and temporal feedback.Source: §3.2 · p. 5

3.3 CoADTP: Chain of Anomaly-Detection Thought Prompt

  • CoADTP combines step-wise reasoning, a verbal anomaly database, and feedback from previous clips to generate consistent anomaly scores and interpretable predictions without training.The LVLM identifies scene elements, incorporates priors and prior-clip context, then produces observations, a description, an assessment, a rationale, and a calibrated score.Source: §3.3 · p. 6

3.4 Raw Stage

  • The raw stage samples video segments at fixed-stride centers, analyzes uniformly sampled frames with CoADTP, and outputs anomaly scores, categories, descriptions, and rationales for refinement.Each prompt includes the previous segment's summary and score; raw outputs are passed to the refinement stage.Source: §3.4 · p. 7
  • CoADTP outperforms Prompt 1 by +7.77% AUC and +16.23% AP on UCF-Crime, while format-only structure yields just +2.92% AUC and −0.31% AP.Performance scales with reasoning-level structure rather than surface wording; cross-backbone results also show improvements without prompt re-tuning.Source: §3.4 · p. 7, Table 1 · p. 8

3.5 Refinement Stage: Cross-Modal Re-Ranking

  • Cross-modal refinement retrieves semantically related descriptions, adds temporally weighted visual similarity, and aggregates re-ranked neighbors to produce smoother, context-aware scores without trainable parameters.The raw scores may lack global semantic awareness and long-range temporal consistency; visual similarity emphasizes segments that are also temporally close.Source: §3.5 · pp. 8–9

4 Experimental Setup

Cog-VADU is evaluated as a training-free approach to detection, hierarchical anomaly reasoning, and cross-benchmark generalization. Results report competitive zero-shot detection and reasoning, while ablations examine refinement strategies and parameters.

4 Experimental Setup

  • The evaluation spans detection and reasoning benchmarks, using frame-level AUC and XD-Violence AP, VANE-Bench accuracy, and HIVAU-70K BLEU, METEOR, and ROUGE-L.HIVAU-70K assesses clip-, event-, and video-level reasoning.Source: §4 · pp. 9–10

4.1 Benchmarks and Unified Re-annotation

  • 39.1% more anomalous frames in UCF-Crime and 22.5% more in XD-Violence follow unified re-annotation of complete cause-and-effect event spans.The protocol addresses omitted effects and inconsistent labels for repeated anomalies.Source: §4.1 · p. 10, Figure 5 · p. 10

Quantitative Results

  • Under original labels, AP is lower than for supervised models because trigger-only annotations omit downstream effects; under unified labels, the precision gap largely disappears.Cog-VADU scores both causes and consequences, which incomplete ground truth can count as false positives.Source: Quantitative Results · pp. 10–11
  • 90.15% AUC on XD-Violence surpasses training-free LAVAD (85.36%) and outperforms several training-based explainable approaches.The comparison uses Cog-VADU's fully training-free setting.Source: Quantitative Results · p. 10, Table 2 · p. 11

4.2 Anomaly Understanding & Reasoning (Q2): Model-Agnostic Evaluation

  • On HIVAU-70K, CoADTP improves reasoning across clip, event, and video levels; Cog-VADU scores BERTScore F1 0.316 versus HolmesVAD 0.245 and HAWK 0.077.The reported BERTScore evaluation uses the 350 IDs shared across methods.Source: §4.2 · pp. 11, 13, Table 5 · p. 12, Table 4 · p. 12

4.3 Performance Comparison on VANE-Bench & Generalization (Q3)

  • On VANE-Bench, zero-shot Cog-VADU is competitive against strong baselines; examples show grounded explanations linking scene understanding, event localization, and causal reasoning.The qualitative comparison contrasts Cog-VADU's structured reasoning with weaker-grounded prior explanations.Source: Table 6 · p. 13, §4.3 · pp. 13–14, Figure 6 · p. 14

4.4 Ablation Studies

  • Cross-modal re-ranking achieves the highest AUC and AP among tested refinement strategies by combining global semantic alignment with local temporal consistency.Visual-visual matching emphasizes local consistency, while visual-textual matching emphasizes global semantics.Source: Table 7 · p. 14, §4.4 · p. 14
  • Ablations favor M = 30, temporal K ∈ [8, 10], and balanced αvis = αtext = 0.5; excessive M may add noise and excessive smoothing may blur short anomalies.Balanced fusion integrates local visual coherence and global textual context.Source: Table 8 · p. 15, §4.4 · p. 15

5 Concluding Remarks

Cog-VADU combines training-free sequential reasoning and cross-modal re-ranking to produce interpretable anomaly scores. The paper presents competitive zero-shot results while emphasizing responsible deployment, human oversight, and awareness of documented failure modes.

5 Concluding Remarks

  • Cog-VADU uses model-agnostic CoADTP and cross-modal re-ranking to produce temporally coherent, interpretable anomaly scores without fine-tuning.The conclusion reports competitive zero-shot performance on UCF-Crime, XD-Violence, VANE, and HIVAU-70K.Source: §5 · p. 15

Broader Impact Statement

  • The authors frame Cog-VADU as decision support, not autonomous decision-making, and condition deployment benefits on human oversight and responsible practices.They call for attention to privacy, fairness, and documented failure modes.Source: Broader Impact Statement · pp. 15–16

A Appendix

  • The appendix collects re-annotation details, experimental information, extended results, visualizations, and limitations and future directions.Source: A Appendix · p. 22

B.1 Motivation and Benchmark Limitations

The paper identifies trigger-centric and incomplete annotations in UCF-Crime and XD-Violence, then re-annotates them to cover causal, temporal, and contextual anomaly states. It also describes the protocol, structural changes, and safeguards used to produce and assess the labels.

B.1 Motivation and Benchmark Limitations

  • Original labels often omit effects and context, creating a mismatch: reasoning models can detect these states but are penalized by incomplete ground truth.The paper says this discrepancy particularly affects AP.Source: §B.1 · p. 22

B.2 Systematic Annotation Errors

  • The identified errors include unlabeled downstream effects, repeated anomalies marked only once, and inconsistent labeling of cause-effect chains.Examples include injured victims or evacuation scenes left unlabeled after an initiating event.Source: §B.2 · p. 22

B.3 Unified Annotation Principle

  • The protocol labels causes, effects, connected escalation, and aftermath, with consistent coverage of repeated events and subtle contextual anomalies.It applies to both UCF-Crime and XD-Violence.Source: §B.3 · p. 23
  • The unified protocol adds 121,639 anomalous frames to XD-Violence (+22.5%) and 34,494 to UCF-Crime (+39.1%).The datasets are reported under both original and unified annotations for comparison.Source: §B.3 · p. 23

B.4 Structural Analysis of Re-Annotations

  • Word clouds show under-represented semantic states and causal, escalation, and aftermath relations, supporting the claim that original labels favor triggers.Figure 10 contrasts anomaly categories with temporal relations.Source: §B.4 · p. 23, Figure 10 · p. 25

B.5 Annotation Procedure and Quality Control

  • Re-annotations were manual and model-independent; the process used repeated inspection and reported results under both original and unified labels.The authors state that the safeguards were intended to distinguish improved anomaly coverage from relabeling artifacts.Source: §B.5 · pp. 25–26

C Chain-of-Anomaly Detection Thought Prompt (CoADTP)

Cog-VADU combines step-wise reasoning, anomaly knowledge, and temporal feedback, with cross-modal re-ranking refining predictions. Its ablation results, model comparisons, sampling study, and open-web cases examine component contributions and behavior across settings.

C Chain-of-Anomaly Detection Thought Prompt (CoADTP)

C.1 Choice of Large Vision Language Model

  • VideoLLaMA3 models show more stable anomaly scores and stronger temporal reasoning than InternVL2-8B; VideoLLaMA3-7B is reported as the more promising variant.The authors attribute effectiveness more to cross-modal alignment and temporal comprehension than parameter count alone.Source: §C.1 · p. 27

C.2 Effect of Subsampling and Temporal Window

  • A 10-second window sampled at 1 FPS gives the best reported performance among the tested temporal-window and sampling settings.The authors note that this aligns with VideoLLaMA3’s training FPS.Source: §C.2 · p. 27, Table 11 · p. 28

D Faithfulness and Failure Analysis

  • The qualitative study uses three open-web videos outside the evaluated baselines’ training distributions to examine rationale faithfulness, semantic discrimination, and a Cog-VADU failure mode.The cases are framed as complementary purposes for the study.Source: D Faithfulness and Failure Analysis · p. 28

D.1 Symmetric Discriminative Cases

  • The symmetric cases pair visually anomalous but semantically normal wrestling with a visually subtle, semantically anomalous missile interception.Together, the cases are presented as evidence of context-driven rather than default-pattern reasoning.Source: §D.1 · p. 28

D.2 Faithfulness of Cog-VADU Rationales

In two open-web examples, Cog-VADU connects visual observations to context-sensitive decisions: it treats professional wrestling as normal and a missile interception as anomalous. The reported scores align with those conclusions.

D.2 Faithfulness of Cog-VADU Rationales

  • Cog-VADU assigns the wrestling video a calibrated score of 0.15 (Normal) and the Tel Aviv interception a score of 0.95 (Explosion).The authors describe both rationales as factually consistent with visual evidence and aligned with the final scores.Source: §D.2 · p. 28
  • For the wrestling case, Cog-VADU recognizes professional-wrestling entertainment context despite physical exertion and apparent injury.Its rationale treats those cues as consistent with the sport’s nature.Source: §D.2 · pp. 28–29
  • For the interception, Cog-VADU interprets rapid light trajectories as possible projectiles or explosions rather than ordinary lights.The resulting decision is Explosion, with a score of 0.95.Source: §D.2 · pp. 28, 30

D.3 Failure Modes of Competing Methods

The two cases expose three different baseline failure patterns: motion-driven false alarms and misses, fabricated rationales, and decisions that contradict or override contextual reasoning.

D.3 Failure Modes of Competing Methods

  • URDMU scores the wrestling segment at ≈0.999 and misses the Tel Aviv event with a peak ≈0.054, illustrating pattern-matching failure.The paper attributes these outputs to reliance on motion cues.Source: §D.3 · p. 31
  • HolmesVAD and HAWK fabricate scenes absent from the videos, illustrating confident but ungrounded explanations.Examples include a man walking down a street and a vehicle rear-ending a police car.Source: §D.3 · p. 31
  • HolmesVAU recognizes wrestling context yet labels it anomalous, then misclassifies the missile interception as fireworks.The authors characterize these outputs as reasoning-decision misalignment and contextual evidence overridden by training-induced priors.Source: §D.3 · p. 31

D.4 A Documented Failure of Cog-VADU

Cog-VADU can preserve correct event reasoning and scoring while mislabeling an anomaly when its predefined category vocabulary is incomplete. The summary also characterizes its failures as diagnosable and frames error propagation as a concern to test.

D.4 A Documented Failure of Cog-VADU

  • A flooding video receives a high anomaly score of 0.92 and correct safety reasoning, but its type is labeled Suspicious rather than Flood or Accident because the database lacks Flood.The framework falls back to the generic Suspicious class; extending the anomaly database can correct this without retraining.Source: §D.4 · p. 31

D.5 Summary

  • Across three case studies, Cog-VADU’s rationales align with predictions, it distinguishes visually similar events, and its structural failures are addressable without retraining.The summary contrasts this diagnosable failure mode with competing supervised methods.Source: §D.5 · p. 32

E Error Propagation Analysis

  • The error-propagation analysis tests whether corrupted rationales accumulate across segments and examines recovery dynamics and heavy-tailed cases.It is motivated by the possibility that early reasoning errors might compound over time.Source: E Error Propagation Analysis · p. 32

E.1 Experimental Setup

The section probes rationale-error recovery, its computational trade-offs, and additional benchmark results. Recovery was universal in the controlled pilot, while runtime depends on whether the method uses sparse video-level sampling or segment-level localization.

E.1 Experimental Setup

  • The controlled test compares baseline runs with runs whose segment-1 temporal context is replaced by one of three incorrect rationales, and defines recovery using a two-segment threshold.Twenty UCF-Crime test videos were sampled; the tested corruptions are wrong category, inverted normality, and hallucinated entity.Source: §E.1 · pp. 32–33

E.2 Recovery Statistics and Propagation Curves / E.3 Key Findings

  • Across 60 video–corruption pairs, all runs recover, with median recovery taking one segment for each corruption type; score deviations remain bounded over 40 segments.The mean deviations do not show upward drift and are comparable to scoring jitter between identical runs.Source: Table 12 · p. 33, Figure 16 · p. 33, §E.3 · p. 34

E.3 Key Findings / E.5 Why Error Accumulation Is Bounded

  • Fresh visual evidence at each segment and cross-modal re-ranking re-anchor predictions, allowing later observations to override a contaminated rationale.The prior rationale biases the reasoning but does not dictate it; re-ranking uses visual embeddings as an additional correction signal.Source: §E.3 · p. 34, §E.5 · p. 34

E.4 Heavy-Tailed Cases / E.6 Limitations

  • In the 20-video UCF-Crime pilot, hallucinated-entity corruption has a right-skewed recovery distribution, with a mean of 5.4 segments and standard deviation of 10.6.Slower cases involve ambiguous visual evidence compatible with the injected rationale; long-form footage over 10 minutes was not stress-tested.Source: §E.4 · p. 34, §E.6 · p. 34

F.1 Video-Level Reasoning Mode / F.2 Frame-Level Localisation Mode / F.3 Discussion

  • Video-level reasoning samples 16 frames and has nearly constant inference time, whereas frame-level localization analyzes segments every 16th frame and scales linearly with video length.The two modes trade temporal precision for efficiency; stride is tunable, and systematic stride–accuracy and backbone-size studies are left for future work.Source: §F.1 · p. 35, §F.2 · p. 35, §F.3 · p. 35

G.1 Quantitative Results: Average Precision

  • Cog-VADU achieves the highest AP among training-free approaches and surpasses several supervised baselines on UCF-Crime.The comparison reports Average Precision (AP) and includes original and re-annotated annotation results.Source: §G.1 · p. 35, Table 13 · p. 36

G.2 Qualitative results: UCF and XD Violence

  • On UCF-Crime and XD-Violence examples, abnormal regions receive higher scores than normal regions, and temporal summaries align with relevant frames.The qualitative results describe score increases when abnormal events occur and low scores for normal regions or videos.Source: §G.2 · pp. 35–36

G.3 Qualitative Results on the VANE Benchmark

On the VANE SORA subset, Cog-VADU produces structured explanations connecting visual observations, anomaly reasoning, and decisions. The qualitative examples contrast its reasoning with inconsistencies or hallucinations in competing methods.

G.3 Qualitative Results on the VANE Benchmark

  • On SORA’s realistic synthetic videos with subtle anomalies, Cog-VADU produces structured, grounded explanations linking visual observations to anomaly reasoning and final decisions.Examples include unusual creature appearances and unrealistic environmental events.Source: §G.3 · p. 36
  • For a puppy video with an unnatural nose and tongue change, Cog-VADU identifies the anomaly, while HAWK, HolmesVAU, and VAD-R1 show unrelated hallucinations or fail to localize it.VAD-R1’s reasoning includes minor hallucinations despite similar visual evidence.Source: §G.3 · p. 36
  • The qualitative results characterize Cog-VADU’s predictions as accurate and its reasoning as concise and reliable.Source: §G.3 · p. 36

H Limitations and Future Directions

The paper introduces Video Anomaly Reasoning for zero-shot semantic analysis of anomalous video events. It identifies additional inference time as a limitation and points to efficiency improvements and causal-temporal evaluation as future work.

H Limitations and Future Directions

Loading 2610.01754v1…