PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Zero-Shot Video Anomaly Detection via Signal-Aware Adaptive Fusion (SAAF)

Khaled Waleed Dawoud, Muhammad Haris Khan

arXiv:2610.32486v1cs.CVcs.AIcs.LG

TL;DR

VLM-based video anomaly detection faces a computational burden at real-world video scale, while anomalies are sparse and can be abrupt or gradual. SAAF combines semantic dual-signal sampling with gated temporal reasoning and signal-aware fusion; it reports leading training-free zero-shot results on three benchmarks while reducing computation.

  • Problem

    VLM reasoning is computationally expensive at video scale, and fixed-interval sampling does not adapt to the sparse, varied temporal signatures of anomalies.

    Source: §1 · pp. 1–2

  • Method

    SAAF uses semantic delta and drift signals to select frames, then gates video-level temporal analysis on frame-level confidence and fuses scores according to the triggering signal.

    Source: §1 · p. 2, §3.3.1 · p. 5, §4.1 · p. 6

  • Results

    SAAF sets training-free zero-shot state-of-the-art results on XD (92.07% AUC-ROC), MSAD (86.70% AUC-ROC), and ShanghaiTech (80.54% AUC-ROC).

    Source: §4.2 · pp. 6–7, §7 · p. 8

  • Takeaways & Limitations

    SAAF’s results suggest that content-aware allocation guided by lightweight semantic signals is a promising direction for deploying VLMs at scale.

    Source: §7 · p. 8

    If anomalies begin immediately or dominate early frames, warmup statistics can treat them as normal and suppress later signal activations.

    Source: D. Limitations · p. 19

Abstract

from arXiv · show

Deploying Vision-Language Models for video anomaly detection confronts a fundamental constraint. Their semantic reasoning demands computational budgets incompatible with the scale of real-world video streams. We present Signal-Aware Adaptive Fusion (SAAF), a framework that achieves efficient zero-shot anomaly detection through three key innovations. First, we introduce a dual-signal adaptive sampler operating in semantic embedding space: a delta signal captures sudden events via frame-to-frame cosine distance, while a drift signal detects static and gradual anomalies through deviation from rolling temporal context. Their OR-gated combination with adaptive EMA-based thresholding provides comprehensive anomaly coverage without training data. Second, we propose a threshold-gated cascaded architecture where lightweight frame-level scoring precedes expensive video-level temporal analysis, invoking it only when confidence warrants additional context. Third, we design a signal-aware weighted fusion mechanism that dynamically adapts score combination based on which signal triggered frame selection. Across four benchmarks, SAAF delivers competitive accuracy at substantially lower cost. Among training-free zero-shot methods, SAAF sets a new state of the art on XD-Violence (92.07%), MSAD (86.70%), and ShanghaiTech (80.54%) AUC-ROC, with XD-Violence and MSAD also surpassing supervised baselines. SAAF sustains 44.95, 48.61, and 41.06 FPS on XD-Violence, UCF-Crime, and MSAD, a 2.43$\times$, 2.63$\times$, and 2.22$\times$ throughput improvement over the prior best zero-shot method. Code: https://github.com/kdawoud91/SAAF

1. Introduction

SAAF targets the mismatch between costly VLM reasoning and sparse, temporally diverse anomalies by steering frame selection and temporal analysis with semantic signals. It combines dual-signal sampling, confidence-gated temporal reasoning, and signal-aware fusion, and reports stronger zero-shot benchmark performance at lower cost.

1. Introduction

  • VLM reasoning is computationally costly at video scale, while anomalies are rare and vary in how much each frame contributes to a decision.Existing pipelines invoke selection, scoring, and temporal reasoning at fixed intervals rather than redirecting effort based on video content.Source: §1 · pp. 1–2
  • The dual-signal sampler uses frame-to-frame change to capture sudden events and rolling-context deviation to detect static or gradual anomalies.Both signals operate on semantic embeddings; their OR-gated combination with adaptive EMA thresholding supports coverage of distinct anomaly types without training data.Source: §1 · p. 2
  • Frame-level scoring gates expensive video-level analysis, which is invoked only for retained anchor frames whose scores exceed a confidence threshold.This focuses temporal reasoning where it is most likely to add discriminative value.Source: §1 · p. 2
  • Signal-aware fusion weights frame-level scores higher for delta-triggered sudden events and video-level scores higher for drift-triggered static or gradual anomalies.When both signals fire, the method applies equal weighting.Source: §1 · p. 2
  • SAAF sets a new state of the art among training-free zero-shot methods on XD, MSAD, and ShanghaiTech, while improving throughput over the prior best zero-shot method.The reported throughput improvements are 2.43× on XD, 2.63× on UCF, 2.22× on MSAD, and 1.81× on ShanghaiTech.Source: §1 · p. 2

2. Literature Review

Prior VAD pipelines include training-dependent black-box detectors and zero-shot VLM systems, but existing sampling and reasoning strategies often allocate computation without using video content. SAAF is distinguished by deriving selection signals intrinsically from semantic embeddings and addressing different anomaly signatures with complementary signals.

2. Literature Review

  • Traditional reconstruction-, prediction-, and distance-based VAD methods typically require training data and can be sensitive to scene and anomaly-category distribution shifts.They produce anomaly scores without explaining what is anomalous or why.Source: §2 · p. 2
  • Zero-shot VLM systems offer semantic interpretability, but LAVAD uses content-blind uniform sampling and unconditional frame- and temporal-level analysis, while Unified retains uniform initial sampling and type-agnostic refinement.The cited comparison describes score-based gates in Unified for triggering additional reasoning on ambiguous predictions.Source: §2 · p. 2
  • Recent zero-shot and weakly supervised pipelines predominantly use fixed-stride or other content-independent sampling strategies.The cited review states that no method in the table lets video content drive first-stage selection, where each eliminated frame removes a downstream VLM call.Source: §2 · pp. 2–3, Table 1 · p. 3
  • Unlike learned closed-set selectors and single-mode saliency methods, SAAF derives delta and drift signals from statistical deviation in frozen semantic embeddings without training data.The two signals measure frame-to-frame change and deviation from rolling temporal context, supporting use across traffic, industrial, and crowd scenarios without per-domain adaptation.Source: §2 · p. 3

3. Methodology

SAAF allocates VLM analysis through complementary semantic-change signals, then uses confidence-gated temporal reasoning and signal-aware fusion to produce dense anomaly scores. Its stages target where analysis is needed, when temporal context is warranted, and how frame- and video-level scores are combined.

3. Methodology

  • SAAF produces dense frame-level anomaly scores through adaptive frame selection, frame scoring, conditional video-level fusion, and score propagation.Source: §3 · pp. 3–4

3.1. Dual-Signal Adaptive Frame Selection

  • The delta signal detects abrupt changes between consecutive semantic embeddings, while drift measures deviation from a rolling temporal context to capture persistent or gradual anomalies.Source: §3.1.2 · p. 4, §3.1.3 · p. 4, §3.1 · p. 4

3.1.4. Adaptive Thresholding

  • EMA-based adaptive thresholds select frames when either signal crosses its threshold, and freeze updates on selected frames to protect baseline statistics from anomalous observations.Source: §3.1.4 · p. 4

3.1.5. OR-Gated Selection and Signal Attribution

  • The OR-gated selector retains frames triggered by delta, drift, or both, enabling complementary coverage and recording which signal selected each frame.Source: §3.1.5 · p. 5

3.2. Frame-Level Anomaly Scoring

3.3.1. Threshold-Gated Temporal Reasoning

  • Video-level temporal reasoning runs only when a frame-level score exceeds a confidence threshold, reducing computational cost by focusing context analysis on high-confidence candidates.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.2. Signal-Aware Weighted Fusion

  • For frames receiving both scores, fusion weights depend on the triggering signal: delta favors frame-level scores, drift favors video-level scores, and joint activation uses equal weighting.Source: §3.3.2 · p. 5

3.4. Score Propagation

  • Anchor scores are propagated forward up to the next anchor or maximum stride, and frames outside propagation windows receive a default fill value.Source: §3.4 · p. 5

4. Experiments

SAAF combines adaptive frame selection with threshold-gated video analysis and is evaluated across four benchmarks. It improves on prior zero-shot results on XD-Violence, MSAD, and ShanghaiTech while retaining competitive results on UCF-Crime; qualitative examples show event-focused selection and low scores on a dynamic normal video.

4.1. Implementation Details

  • Implementation uses stride-16 candidate sampling, CLIP ViT-B/32 embeddings, adaptive EMA thresholds, and a 10-frame temporal clip when frame-level confidence exceeds τf = 0.4.Source: §4.1 · p. 6

4.2. Results & Discussion

  • SAAF reaches 92.07% AUC-ROC on XD, 86.70% on MSAD, and 80.54% on Shang; it leads zero-shot methods on all three and surpasses supervised methods on XD and MSAD.Source: §4.2 · pp. 6–7
  • SAAF reaches 82.59% AUC-ROC on UCF, exceeding LAVAD, AnyAnomaly, and Eventvad while remaining below Unified's 84.28%.It processes 23.48% fewer frames than the comparison setup described in the results.Source: §4.2 · p. 6
  • Abrupt categories score well at frame level, while gradual categories gain from video-level fusion; smoothing Stage 2 alone remains below the full pipeline across benchmarks.Source: §4.2 · p. 7

4.3. Qualitative results

  • Qualitative examples show selection around anomalous events and skipped gaps, while a normal video with continuous motion selects 43% of frames and remains low-scoring.Source: §4.3 · p. 7

5. Computation Reduction Analysis

SAAF reduces VLM calls through adaptive selection and reports higher throughput than the prior best zero-shot method. Savings vary by video, with the largest reductions associated with predominantly static content.

5. Computation Reduction Analysis

  • The per-component timing analysis reports costs across benchmarks; the pipeline selectively sends only anchors passing the frame-level threshold to video-level fusion.Source: Table 3 · p. 7, §5 · p. 7
  • SAAF reaches 44.95 FPS on XD, 48.61 FPS on UCF, and 41.06 FPS on MSAD, delivering 2.43×, 2.63×, and 2.22× speedups over Unified's 18.5 FPS.Source: §5 · p. 7, Table 4 · p. 8
  • Most videos show 20–60% fewer VLM calls than uniform stride-16 sampling, while a long tail of predominantly static videos reaches 80–90% reduction without sacrificing detection capability.Source: Figure 5 · p. 7, §5 · p. 7

6. Ablation Study

The ablations examine signal attribution, encoder and drift-window choices, stage-wise gains, and the refinement stage. The reported results support complementary signal selection and added gains from later pipeline stages.

Signal Attribution Distribution

  • Across four benchmarks, δ-only accounts for 11–16% of selected frames and ρ-only for 14–24%, with both signals firing on 59–75%.The two single-signal categories together account for 25–40% of selections.Source: Figure 6 · p. 8, §6 · p. 8

Backbone Selection for Signal Extraction

  • Encoder AUC-ROC spans a narrow 0.46% range; CLIP marginally leads, while Dinov2 ViT-B/14 trails by 0.29%.CLIP ViT-B/32 is adopted as the default.Source: §6 · p. 8

Rolling Window Size for Drift Signal

  • Larger drift windows improve performance slightly; W = 20 provides more context to distinguish genuine drift from short-term fluctuations.The study adopts W = 20 as the default.Source: Table 6 · p. 8, §6 · p. 8

Stage-Wise Breakdown Analysis

  • Stage-wise gating adds +10.23% on XD and +11.57% on UCF, while refinement adds +6.32% on UCF and +5.04% on Shang.The reported gains are cumulative AUC-ROC changes across pipeline stages.Source: Table 7 · p. 8, §6 · p. 8

Ablation of the Refinement Stage

  • Full Stage 3 exceeds smoothing-only AUC-ROC by 2.72, 9.29, 4.92, and 3.35 points on XD, UCF, MSAD, and Shang, respectively.Smoothing alone improves AUC-ROC on every benchmark, but the full pipeline retains these margins.Source: Table 8 · p. 8, §6 · p. 8

7. Conclusion

SAAF addresses the tension between VLM semantic capability and computational feasibility in zero-shot video anomaly detection. Its benchmark results suggest content-aware allocation guided by lightweight semantic signals as a direction for deploying VLMs at scale.

Conclusion

  • Among training-free zero-shot methods, SAAF achieves state-of-the-art results on XD, MSAD, and Shang while reducing computation.The conclusion presents content-aware allocation guided by lightweight semantic signals as a promising direction for deploying VLMs at scale.Source: §7 · p. 8

Supplementary Material for

The supplied passage gives the paper title.

Supplementary Material for

Supplementary Roadmap

The supplementary material outlines additional ablations and evaluates event-level selection, signal combinations, and parameter choices. Across these analyses, it reports trade-offs in event coverage, detection performance, and computational cost.

Supplementary Roadmap

  • The supplementary document covers additional ablations, qualitative results, prompt templates, limitations, and broader impact.Source: Supplementary Roadmap · p. 12

A. Additional ablation study

A.1. Event-Level Recall of Stage 1 Selection

  • Stage 1 recovers 98.3% of MSAD, 92.3% of UCF, and 88.9% of XD events, while ShanghaiTech reaches 77.8%; 22 of its 43 missed events are unhittable on the stride-16 grid.An event is a contiguous run of anomalous frames, counted as hit when at least one selected frame falls within it; unhittable events have no stride-16 candidate frame.Source: §A.1 · p. 12, Table 9 · p. 12

A.2. Signal Component Ablation

  • OR-gated selection performs best across datasets, while AND-gating performs worst; drift-only outperforms delta-only on XD and MSAD.The study attributes the OR-gated advantage to complementary anomaly patterns captured by the two signals.Source: Table 10 · p. 12, §A.2 · p. 12

A.3. Selection Threshold (N)

  • At N = 0.5, AUC-ROC is highest but 76–84% of candidate frames are selected; at N = 2, selection falls to 43–59%, with XD dropping 9.70 points from the default.The adopted N = 1 setting selects 65–77% of frames as a balance between coverage and efficiency.Source: §A.3 · pp. 12–13, Table 11 · p. 13

A.4. Default Fill Value (sd)

  • Setting sd = 0.2 is optimal on XD-Violence, UCF-Crime, and ShanghaiTech, while MSAD peaks at sd = 0.1; the default sd = 0.3 is within 0.7% of optimal.The ablation reports degraded performance at sd = 0 and improvement with sd ≥ 0.1.Source: Table 12 · p. 13, §A.4 · p. 13

A.5. Fusion Threshold Sensitivity (τf).

  • On XD-Violence, raising τf reduces invocation from 10.9% to 4.5% with an AUC-ROC decrease of only 0.86%; UCF-Crime and MSAD stay within 0.7%, and ShanghaiTech within 0.3%, across the sweep.At τf ≥ 0.8, ShanghaiTech invocation falls from 19.5% to 0.1%; the default τf = 0.4 maximizes recall.Source: Table 13 · p. 13, §A.5 · p. 13

A.6. Signal-Aware Fusion Weights (λδ, λρ)

  • Across XD-Violence, UCF-Crime, and MSAD, fusion configurations vary by less than 0.4 points; on ShanghaiTech, (0.2, 0.8) gains 0.93 points.The default (λδ, λρ) = (0.6, 0.4) is retained to avoid dataset-specific tuning.Source: §A.6 · pp. 13–14, Table 14 · p. 14

A.7. Additional Per-Category Analysis

SAAF performs especially well on high-motion anomalies, while gradual or subtle categories remain harder and benefit from later temporal stages. Qualitative cases show that it concentrates VLM analysis on event-relevant frames while skipping much normal content, including dynamic scenes.

A.7. Additional Per-Category Analysis

  • Fighting (99.8%), Explosion (98.5%), Robbery (98.0%), Shooting (97.8%), Assault (96.6%), and Object falling (95.7%) exceed 95% AUC-ROC after refinement.These high-motion events have visual signatures that the delta signal and frame-level VLM scoring capture reliably.Source: §A.7 · p. 14
  • Vandalism (85.7%) and Fire (89.2%) remain more challenging because subtle actions may lack distinctive signals and slow fire progression requires drift to accumulate.For vandalism, distinctive δ or ρ signals may appear only after the act is underway.Source: §A.7 · p. 14
  • Object falling improves by 21.5% and People falling by 15.1% with Stage 2 gating; Stage 3 refinement adds 2–10% across categories.The reported pattern associates video-level reasoning with categories needing temporal context, while sudden events benefit from frame-level detection.Source: §A.7 · p. 14

B.1. Qualitative Success Cases

  • The road-accident example selects 15 of 63 candidate frames, yielding 76.2% computational savings while concentrating activations and high scores around the event.The event occurs at frames 650–750, where the VLM score reaches 0.65.Source: §B.1 · pp. 14–15, Figure 8 · p. 15
  • The ShanghaiTech cycling anomaly produces scores of 0.8–0.85 in the event window, with 42.1% computational savings from skipping normal pedestrian scenes.Both signals activate in the anomalous region; later activity receives lower scores of 0.3–0.55.Source: §B.1 · pp. 14–15
  • On normal static content, only 12 of 113 stride-16 candidates are selected, yielding 89.4% computational savings while scores remain low.The example shows brief activations during initial dynamics and minor late scene changes, while most frames are skipped.Source: §B.1 · pp. 14–15
  • In a busy metro scene, SAAF skips ongoing pedestrian movement between frames 200–1,200 and gives selected frames low scores of 0.02–0.08.Only 16 of 103 stride-16 candidates require analysis, yielding 84.5% computational savings.Source: §B.1 · p. 16

B.2. Failure Case Analysis

The failure cases expose limits in baseline initialization, temporal sampling, and VLM recognition. The appendices also specify the prompts and illustrate how frame scoring gates temporal analysis and signal-dependent fusion.

B.2. Failure Case Analysis

  • When a robbery begins at frame zero, anomalous warmup content contaminates the baseline, causing sparse signal activations despite high scores on selected frames.The failure is in frame selection; the adaptive thresholds assume representative normal content during warmup.Source: §B.2 · pp. 16–17
  • A brief 50-frame collision amid continuous traffic and pedestrian motion produces dense activations and poor separation from normal content.The collision cues resemble ordinary abrupt braking or turning, creating ambiguity for the VLM.Source: §B.2 · p. 17
  • In low-light arson footage, signals select the event region but faint smoke and small flames against Christmas lights keep anomaly scores low.Accurate frame selection cannot compensate when the visual signature challenges current VLM recognition.Source: §B.2 · p. 17
  • For industrial MSAD fire, frame selection concentrates on the event, yet scores remain low and fail to distinguish it from normal frames.The example identifies downstream scoring, rather than selection, as the difficulty.Source: §B.2 · p. 17

C. Prompts and Templates

  • The frame-captioning prompt requests people, actions, objects, setting, and safety cues; the scoring prompt uses anomaly and normal indicators to produce a 0–1 score.The indicators cover benchmark-aligned anomaly categories and routine activities.Source: §C.1 · p. 17, §C.2 · pp. 17–18

C.3. Video-Level Temporal Captioning Prompt

  • The temporal prompt is shorter than frame captioning because multiple frames provide context for reasoning about actions and dynamics.It asks the video model to describe events and identify suspicious or anomalous behavior.Source: §C.3 · p. 18

C.4. Example Pipeline Outputs

Example 3: Normal Scene (Unselected Frame)

  • An unselected normal frame receives no captioning or scoring, illustrating that the pipeline skips VLM analysis when neither signal crosses its threshold.The examples describe this as part of the full inference chain.Source: Example 3: Normal Scene (Unselected… · p. 19

D. Limitations

SAAF’s limitations concern warmup assumptions, coarse temporal sampling, and dependence on VLM recognition. The paper also reports that sampling reduces computational cost and may lower energy use at scale.

D. Limitations

  • Warmup contamination can suppress signal activations when anomalies begin immediately or dominate early frames, because thresholds treat abnormal patterns as normal.The method assumes representative normal content during warmup to establish baseline statistics.Source: D. Limitations · p. 19
  • Stride-16 sampling yields only 2–4 signal observations for ShanghaiTech anomalies spanning 40–60 frames, reducing reliability for brief events.The limitation is temporal resolution relative to longer-duration anomalies.Source: D. Limitations · p. 19
  • Subtle signatures, including low-light gradual fire and fine-grained actions, can exceed VLM reasoning capability even when frame selection is correct.The paper states that improved VLM capabilities would benefit SAAF without architectural changes.Source: D. Limitations · p. 19

E. Broader Impact

  • SAAF’s sampling reduces computational cost compared with uniform processing and may lower the energy footprint of large-scale video analysis, though underlying VLM biases may still influence scoring.The paper notes that its zero-shot design avoids encoding dataset-specific biases into learned parameters.Source: E. Broader Impact · p. 19

Related papers

Loading 2610.32486v1…