PaperlayerView exact revision on arXiv

Source-linked AI summary

VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models

Ying Cheng, Yu-Ho Lin, Min-Hung Chen, Fu-En Yang, Shang-Hong Lai

arXiv:2511.07299v2cs.CV

TL;DR

Video anomaly understanding seeks detailed interpretation of events, but existing approaches may overlook deeper causal relationships and object interactions. VADER combines context-aware keyframe sampling and dynamic relational modeling with visual cues in an LLM. Across challenging benchmarks, it reports state-of-the-art or highly competitive performance on anomaly description, causal explanation, and question answering.

  • Problem

    Video anomaly understanding requires more than detection and localization, yet existing approaches often neglect deeper causal relationships and object interactions relevant to anomalous behavior.

    Source: §1 · p. 1

  • Method

    VADER uses anomaly scoring and context-aware keyframe sampling, encodes dynamic object relations, and integrates relational and visual cues into an LLM.

    Source: §1 · pp. 1–2, §3.1.2 · p. 3, §3.2 · p. 4

  • Results

    VADER achieves state-of-the-art or highly competitive performance across anomaly description, causal explanation, and question answering on multiple challenging benchmarks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results support VADER as a framework for detailed, causally grounded interpretations of anomalous events across description, explanation, and reasoning tasks.

    Source: §6 · p. 8

    Reliance on relational volatility biases VADER toward high-motion events and makes it less sensitive to subtle or low-motion anomalies.

    Source: §5 · p. 8

Abstract

from arXiv · show

Video anomaly understanding (VAU) aims to provide detailed interpretation and semantic comprehension of anomalous events within videos, addressing limitations of traditional methods that focus solely on detecting and localizing anomalies. However, existing approaches often neglect the deeper causal relationships and interactions between objects, which are critical for understanding anomalous behaviors. In this paper, we propose VADER, an LLM-driven framework for Video Anomaly unDErstanding, which integrates keyframe object Relation features with visual cues to enhance anomaly comprehension from video. Specifically, VADER first applies an Anomaly Scorer to assign per-frame anomaly scores, followed by a Context-AwarE Sampling (CAES) strategy to capture the causal context of each anomalous event. A Relation Feature Extractor and a COntrastive Relation Encoder (CORE) jointly model dynamic object interactions, producing compact relational representations for downstream reasoning. These visual and relational cues are integrated with LLMs to generate detailed, causally grounded descriptions and support robust anomaly-related question answering. Experiments on multiple real-world VAU benchmarks demonstrate that VADER achieves strong results across anomaly description, explanation, and causal reasoning tasks, advancing the frontier of explainable video anomaly analysis.

1. Introduction

VADER addresses the limited semantic interpretability of conventional anomaly detection and the neglect of deeper causal relationships in existing LLM-based video anomaly understanding. It combines context-aware keyframe sampling and object-relation features with visual cues for causally grounded anomaly descriptions and question answering, and reports strong results across three VAU benchmarks.

Motivation

  • Existing LLM-based VAU approaches often neglect deeper causal relationships and dynamic object interactions that matter for understanding unusual behaviors.Earlier conventional methods focus primarily on temporal localization and classification, often lacking semantic interpretability.Source: §1 · p. 1

VADER framework

  • VADER combines context-aware keyframe sampling, dynamic object-relation modeling, and visual cues to support causally grounded video anomaly understanding with LLMs.CAES selects keyframes; CORE turns interaction dynamics into compact relational tokens for the LLM.Source: §1 · pp. 1–2
  • VADER integrates visual and relational cues into an LLM to generate detailed, causally consistent narratives and answer anomaly-related questions.Its pipeline assigns frame-level anomaly scores, samples keyframes to capture event context, and encodes relational features.Source: §1 · pp. 1–2

Evaluation

  • Across three VAU benchmarks, VADER reports outperforming strong baselines and recent state-of-the-art systems on multiple metrics and tasks.The evaluations cover anomaly description, question answering, and causal reasoning; the paper also reports state-of-the-art or highly competitive performance across description, explanation, and reasoning tasks.Source: §1 · p. 2

2. Related Work

Video anomaly understanding extends detection toward describing anomalous events and their context. Recent multimodal approaches advance video reasoning, while scene graphs offer structured object-relation representations; the paper identifies deeper causal relationships and object interactions as an unresolved focus.

2.1. Video Anomaly Detection and Understanding

  • Video anomaly understanding goes beyond detecting deviations to describing anomaly content and investigating event contexts.Prior work includes benchmarks for VAU, while traditional VAD methods often use handcrafted features, deep spatiotemporal models, or reconstruction and weak supervision.Source: §2.1 · p. 2

2.2. Video Anomaly Analysis with MLLMs

  • Despite advances in multimodal video analysis and VAU systems, existing approaches still overlook deeper causal relationships and object interactions.Related systems include causation-focused reasoning, interaction-aware modeling, and anomaly-focused sampling.Source: §2.2 · p. 2

2.3. Modeling Relationships with Scene Graphs

  • Scene graphs represent objects as nodes and relationships as edges, supporting relational video analysis beyond appearance features alone.Prior work applies scene graphs and knowledge graphs to anomaly detection; VADER builds on relational features extracted from DETR-based scene graph generation.Source: §2.3 · p. 2

3. Method

VADER selects informative video keyframes and models evolving object interactions before integrating visual and relational cues into an LLM. Its pipeline combines anomaly scoring, context-aware sampling, relation extraction and encoding, and multimodal fine-tuning.

3. Method

  • VADER has two main components: narrative-driven keyframe detection and selection, followed by fine-grained relation extraction and integration.The framework overview shows an Anomaly Scorer and CAES selecting keyframes, relational and visual feature encoding, and LLM fusion.Source: §3 · p. 2, Figure 2 · p. 3

3.1. Keyframe Detection and Selection

  • An Anomaly Scorer assigns each frame a score, and keyframe selection filters candidate frames to focus computation on informative content.The scorer is inspired by a CLIP-based framework and pretrained once on an aggregated dataset covering diverse video scenarios.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.1. Anomaly Scoring

  • The anomaly scorer recenters visual and text features around a normality prototype, then combines frame anomaly probability with conditional anomaly-class probabilities.Predicted anomaly classes also provide semantic cues for downstream LLM reasoning.Source: §3.1.1 · p. 3

3.1.2. Context-AwarE Sampling (CAES)

  • CAES detects anomalous intervals, expands them into pre-event and post-event context, and samples frames across all three event phases.It uses adaptive thresholds and rise and calm thresholds on score slopes; when anomalies exceed the frame budget, highest-scoring frames are prioritized and remaining slots can come from background segments.Source: §3.1.2 · pp. 3–4, Figure 3 · p. 4

3.2.1. Relational Feature Extraction

  • For each CAES keyframe, a pretrained DETR-based scene-graph model extracts object appearance embeddings and intermediate pairwise relational features.These relational features provide a snapshot of scene interactions for subsequent temporal analysis.Source: §3.2.1 · p. 4
  • Object association links identities across frames, while relational volatility measures the largest change in relational features among co-tracked object pairs.The resulting volatility scores and before/after relational vectors are cached for downstream processing.Source: §3.2.1 · p. 4

3.2.2. COntrative Relation Encoder (CORE)

  • CORE treats top smoothed volatility peaks in abnormal videos as positive relation-change pairs and valleys or normal-video pairs as negatives, then maps transitions to compact relation tokens.It is trained with a triplet margin loss and semi-hard negative mining.Source: §3.2.2 · p. 4

3.3. LLM Integration and Fine-tuning

  • The multimodal LLM jointly receives keyframe visual tokens, CORE relation tokens, and task text tokens to reason over visual content and evolving object relationships.Training fine-tunes multimodal projectors and LoRA adapters while keeping the backbone frozen.Source: §3.3 · p. 4

4. Experiments

Across three VAU benchmarks, VADER is evaluated on anomaly description, question answering, and causal reasoning. Results, ablations, and qualitative comparisons report strong performance and causally grounded outputs.

4.1. Benchmark Datasets and Evaluation Metrics

  • The benchmarks cover HIVAU-70k’s hierarchical perception and reasoning annotations, HAWK description and question answering, and CUVA what/why/how causation; evaluation uses lexical, semantic, and judge-based metrics.The three benchmark metric groups include text-overlap measures, pretrained-model similarity measures, and LLM/VLM judgments.Source: §4.1 · p. 5

4.2. Experimental Results

  • On CUVA, VADER’s MMEval score for Causes is highest, improving by 7.38 points, while its other task scores are comparable.The prose reports slightly lower BLEURT scores than some baselines but stronger human-aligned UniEval and MMEval performance.Source: Table 2 · p. 5, §4.2 · p. 5
  • On HIVAU-70k, VADER achieves state-of-the-art results across metrics and clip, event, and video evaluation levels, with particularly significant gains at event and video levels.The benchmark table describes hierarchical evaluation across those three levels.Source: §4.2 · pp. 5–6, Table 3 · p. 6
  • For HAWK question answering, VADER exceeds prior methods by 0.164 BLEU-1 and 0.077 BLEU-4; description gains are 0.045 BLEU-1 and 0.026 BLEU-4.The reported leading GPT-guided scores are Detail for description and Consistency for question answering.Source: Table 4 · p. 6, §4.2 · p. 6

4.3. Ablation Study

  • Removing CORE, CAES, or fine-tuning lowers text-level and GPT-guided metrics, although the untuned model’s GPT detail score is anomalously high with lengthy, less focused responses.The paper attributes the unusually high detail score likely to lack of domain adaptation.Source: §4.3 · p. 6, Table 5 · p. 7
  • CAES outperforms uniform, Top-K, and ATS by balancing anomaly saliency with context diversity; its Dynamic Window sets context boundaries using anomaly-score gradients.Uniform sampling may miss key anomaly frames, while Top-K may overfocus on anomalous segments and neglect context.Source: §4.3 · pp. 6–7, Table 6 · p. 7, Table 7 · p. 7
  • CORE relation tokens outperform relational visual cues and scene-graph text across text-level and GPT-guided metrics, with continuous fine-grained temporal cues for evolving interactions.Static visual features and generic relationship labels do not capture the same evolving, context-specific interactions.Source: Table 8 · p. 7, §4.3 · p. 7

4.4. Qualitative Results

  • On break-ins and other anomalous videos, VADER captures object interactions and causal dynamics, while Otter and Video-ChatGPT descriptions contain hallucinations or incorrect analysis.The break-in example includes visual details and intruders’ actions; the comparisons span physical altercations, road accidents, and crowd behaviors.Source: §4.4 · pp. 7–8, Figure 5 · p. 8

5. Limitation

VADER has three stated limitations: dependence on upstream modules, a bias toward high-motion events, and an object-centric view that misses some scene- or group-level anomalies.

5. Limitation

  • Errors in the Anomaly Scorer or Relation Feature Extractor can propagate and affect VADER’s final reasoning.The limitation concerns upstream detection and object association.Source: §5 · p. 8
  • Relational volatility biases VADER toward strong-motion anomalies, reducing sensitivity to subtle or low-motion events.Source: §5 · p. 8
  • VADER’s object-centric design limits its capture of scene- or group-level anomalies, including environmental changes and collective crowd behaviors.Source: §5 · p. 8

6. Conclusion

VADER combines context-aware sampling and dynamic relational modeling for causally grounded video-anomaly interpretations. The authors report competitive or state-of-the-art performance across several VAU tasks and identify key component roles through ablations.

6. Conclusion

  • VADER combines Context-AwarE Sampling (CAES) with dynamic relational modeling to interpret anomalous events causally.Source: §6 · p. 8
  • Across anomaly description, causal explanation, and question answering, VADER achieves state-of-the-art or highly competitive performance on multiple challenging benchmarks.Source: §6 · p. 8
  • Ablation studies highlight the roles of adaptive context sampling and relational dynamics encoding.Source: §6 · p. 8

A. Illustrative Example of Motivation

A roadside dog attack illustrates why anomaly descriptions require both causal relationships and object interactions. In this example, VADER’s description is more coherent and human-aligned than the cited alternatives, while lexical overlap alone does not capture factual correctness.

A. Illustrative Example of Motivation

  • The dog attack example requires understanding both what happens and how it happens through causal relationships and dynamic object interactions.Source: A. Illustrative Example of Motivation · p. 12
  • CUVA [11] misdescribes the event despite relatively high ROUGE, while Holmes-VAU [67] leaves the cause-effect chain and temporal progression ambiguous.Source: A. Illustrative Example of Motivation · p. 12
  • VADER’s description clearly identifies the participants and how the dog attack unfolded, aligning with human judgments on semantic-level and human-evaluation metrics.The passage names BLEURT, UniEval, and mmEval as examples of those metrics.Source: A. Illustrative Example of Motivation · p. 12
  • Lexical overlap metrics may not capture factual correctness, whereas human-aligned evaluations reflect description clarity and interpretability.VADER receives the highest ratings by providing a precise, logically structured explanation.Source: A. Illustrative Example of Motivation · p. 12

B. Ablation Study on Sample Mining Hyperparameters

The sample-mining ablation finds the best reported AUC with σ = 2.0 and top-k = 5%, and the authors use these settings in subsequent experiments.

B. Ablation Study on Sample Mining Hyperparameters

C. Failure Cases

VADER’s failure cases expose motion bias and limits to pairwise object reasoning. The proposed mitigations target upstream reliability, subtle anomalies, and broader scene context.

D. Potential Mitigations for Limitations

D.1. Mitigating Dependency on Upstream Modules

  • To reduce upstream error propagation, the authors propose joint LLM-to-module training and strengthening modules with diverse data and advanced tracking.Source: §D.1 · p. 13

D.2. Reducing Bias Toward High-Motion Events

  • Because relational volatility favors strong motion, proposed complements include object-state tracking for static anomalies and scene-flow modeling for loitering or inactivity.Source: §D.2 · p. 13

D.3. Moving Beyond Object-Centric Reasoning

  • Pairwise interactions leave group- and scene-level anomalies out of scope; suggested extensions add whole-frame features and group reasoning for smoke or crowd panic.Source: §D.3 · pp. 13–14

C. Failure Cases

E. Computational Efficiency

F. Intermediate Visualization of Module Contributions

G. Implementation Details

Implementation details specify how VADER samples temporal context, forms relational training signals, and fine-tunes its language model backbone.

  • CAES detects intervals using a per-video 97th-percentile anomaly threshold, sets context with 95th- and 85th-percentile score slopes, samples 4 context and 8 event frames, then fills 64.Source: G. Implementation Details · p. 15
  • Object association combines appearance cosine similarity weighted 0.8 with IoU and Hungarian matching; relational volatility uses adjacent-frame relation-pair L2 distance, smoothed before selecting top 5% peaks.Source: G. Implementation Details · p. 15
  • NVILA fine-tuning updates only the projector and LoRA adapters, freezes other parameters, and uses a 2 × 10^-5 learning rate with cosine scheduling and 0.03 warm-up.Source: G. Implementation Details · p. 15

Related papers

Loading 2511.07299v2…