PaperlayerView exact revision on arXiv

Source-linked AI summary

MoG-VLM: Efficient Video Anomaly Detection through Motion-Guided Vision-Language Models

Khaled Waleed Dawoud, Muhammad Zaigham Zaheer, Khan Muhammad, Muhammad Haris Khan

arXiv:2610.31478v1cs.CVcs.AIcs.LG

TL;DR

VLM-based anomaly detection offers semantic reasoning, but costly frame-level inference and content-blind uniform sampling limit its real-time practicality. MoG-VLM uses motion-guided sampling and anchor-triggered refinement to allocate VLM computation selectively. It establishes zero-shot, training-free state of the art on three benchmarks, with a reported XD-Violence gain and speedup.

  • Problem

    VLM-based anomaly detection faces costly dense inference, while common uniform sampling ignores content dynamics and the field lacks systematic analysis of adaptive resource allocation and its cost-performance trade-off.

    Source: §1 · pp. 2–3

  • Method

    MoG-VLM uses Temporal Gradient Magnitude as a lightweight motion control signal for adaptive sampling, then selectively expands context around high-confidence anchors and fuses frame-level and temporal cues.

    Source: §1 · p. 3

  • Results

    MoG-VLM establishes zero-shot, training-free state of the art on three benchmarks; on XD-Violence it reaches 91.91% AUC-ROC, a +0.57% gain over Unified-reasoning, with a 1.31× end-to-end inference speedup.

    Source: §4.3.1 · p. 10, §5 · p. 14

  • Takeaways & Limitations

    MoG-VLM’s results support content-aware allocation of VLM capacity as a more productive direction than scaling uniform inference across frames.

    Source: §5 · p. 14

    Abrupt illumination changes and camera jitter can trigger spurious TGM activations and erode efficiency gains; multiscale processing only partially mitigates this.

    Source: D Limitations · p. 26

Abstract

from arXiv · show

Vision-Language Models (VLMs) have shown strong potential for video anomaly detection, but dense frame-level inference makes them impractical for real-time deployment. Existing acceleration commonly relies on uniform sampling, which ignores content dynamics and can either waste computation on static segments or miss localized anomalies. We propose MoG-VLM a motion-adaptive anomaly detection framework that leverages motion purely as a lightweight control signal to decide when VLM inference is necessary. To this end, we introduce Temporal Gradient Magnitude (TGM) which performs content-aware sampling to concentrate computation on informative regions while avoiding redundant processing. We further introduce a hierarchical dual-stream design in which dense multi-frame VLM reasoning is activated only around frames with elevated anomaly likelihood, via anchor-triggered context-window expansion with early termination. A fusion module then reconciles these instantaneous semantic cues with temporal action dynamics. Across four benchmarks, our method achieves 91.91%, 84.30%, 87.88%, and 78.96% AUC-ROC on XD-Violence, UCF-Crime, MSAD, and ShanghaiTech, respectively, establishing new state of the art on three benchmarks under the zero-shot, training-free setting. On XD-Violence, we improve over the previous best result by +0.57% while delivering a 1.31x inference speedup. These results demonstrate that content-aware allocation of VLM capacity, processing fewer frames more intelligently rather than all frames uniformly, is a more effective axis of progress than uniformly densifying VLM inference across frames. Code is available on GitHub: https://github.com/kdawoud91/MoG-VLM.

1 INTRODUCTION

VLM-based anomaly detection offers semantic understanding but faces high inference costs, while uniform sampling ignores the sparsity of anomalous events. MoG-VLM uses motion as a lightweight control signal to adapt sampling and selectively refine high-confidence detections.

1 INTRODUCTION

  • Prior VLM-based anomaly detection uses fixed-stride or exhaustive processing, overlooking static intervals and the spatial and temporal sparsity of surveillance footage.Expensive VLM inference makes this uniform allocation difficult to sustain for real-time deployment.Source: §1 · p. 2
  • The paper identifies a gap in systematic study of content-aware computation allocation and the trade-off between computational cost and detection performance.It frames the question around selectively adding computation to a coarse uniform baseline when video dynamics warrant it.Source: §1 · p. 3
  • MoG-VLM combines multiscale Temporal Gradient Magnitude sampling, anchor-triggered hierarchical refinement, and tier-aware fusion of frame-level and temporal-level streams.Motion controls when VLM inference is invoked rather than serving as a direct classification feature; high-confidence sampled frames trigger progressive multi-window analysis.Source: §1 · p. 3
  • 84.30%, 91.91%, and 87.88% AUC-ROC on UCF-Crime, XD-Violence, and MSAD establish zero-shot, training-free state of the art on three benchmarks.The contribution summary also reports 78.96% on ShanghaiTech, but its benchmark name and state-of-the-art status are truncated in the passage.Source: §1 · p. 3

2 Related Work

Prior VLM-based anomaly detection work emphasizes reasoning quality and detection accuracy, with limited attention to computational efficiency. MoG-VLM adapts a two-stage selective-computation approach to untrimmed anomaly footage using training-free motion gating.

2 Related Work

  • VLM-based VAD methods include zero-shot and training-free approaches, supervised prompt optimization, instruction-tuned models, and context-aware or graph-based methods.The passage notes that some approaches require annotated training data or remain tied to seen anomaly types.Source: §2 · p. 4
  • Most prior VLM-based VAD work prioritizes detection accuracy over efficiency, and only Unified reports timing among the methods discussed.The passage says deployment feasibility depends on throughput as well as AUC-ROC.Source: §2 · p. 4
  • MoG-VLM applies a two-stage selective-computation pattern with training-free multiscale TGM gating to untrimmed footage, using motion only to control VLM invocation.The passage contrasts this with learned sampling policies for trimmed clips and says motion does not enter the anomaly score.Source: §2 · p. 4

3 Methodology

MoG-VLM uses motion as a lightweight control signal to allocate VLM inference selectively, combining frame-level analysis, anchor-triggered refinement, and temporal reasoning. Tier-aware fusion reconciles instantaneous frame semantics with action dynamics.

3 Methodology

  • MoG-VLM pairs motion-adaptive hierarchical VLM processing with a dual-stream design and tier-aware fusion.Motion guides where inference is allocated; it is not used as a direct classification feature.Source: §3 · p. 4, Figure 2 · p. 5, §3.1 · p. 5

3.1.1 Temporal Gradient Magnitude Analysis

  • Multiscale TGM combines velocity and acceleration signals, weighting stronger scale responses more heavily to quantify temporal motion.The per-scale signal uses a 90th-percentile reduction, and β = 0.35 controls acceleration’s contribution.Source: §3.1.1 · p. 5

3.1.2 Adaptive Z-Score Anomaly Detection

  • An adaptive z-score compares TGM with scene-specific EMA statistics, which freeze when z(t) exceeds θanom to avoid anomalous behavior contaminating the reference.This is intended to preserve sensitivity to sustained anomalous activity.Source: §3.1.2 · p. 6

3.1.3 Motion-Guided Adaptive Sampling

  • When z(t) exceeds θanom, sampling increases to rhigh within [t − ∆t, t + ∆t] and remains at rbase elsewhere.This concentrates processing on dynamic regions while avoiding redundant inference on static or slowly changing segments.Source: §3.1.3 · p. 6

3.2 Vision Language Frame-Level Captioning

  • The frame stream captions adaptively sampled images with a prompt emphasizing anomaly-relevant visual attributes, then uses language-model reasoning to assign frame-level anomaly scores.The scoring prompt targets unusual actions, dangerous situations, suspicious behavior, violence, accidents, or other deviations from typical footage.Source: §3.2 · p. 6, §3.3 · pp. 6–7

3.4 Anchor-Triggered Multi-Frame Analysis

  • Sampled frames whose anomaly scores exceed θtrigger become anchors for progressively expanding full-resolution lookback windows.The three non-overlapping windows grow in duration; progression beyond the first two stages requires the window’s mean score to exceed θtrigger.Source: §3.4 · p. 7, §3.4.1 · p. 7, §3.4.2 · p. 7

3.5 Temporal-Level Anomaly Detection via Multi-Frame Context

  • A complementary temporal stream samples fixed-size frame sets across regular-duration windows for Video-VLM reasoning about motion patterns, action progressions, and inter-frame dependencies.Window-level anomaly scores are propagated forward until the next window.Source: §3.5 · p. 8, §3.5.1 · p. 8, §3.5.2 · p. 8

3.6 Tier-Aware Adaptive Fusion

  • Tier-aware fusion combines instantaneous frame scores with multi-frame temporal scores, increasing trust in frame-level evidence from baseline to motion to anchor tiers.The fusion weights are smoothed before application to reduce tier-boundary discontinuities.Source: §3.6 · pp. 8–9

4 Experiments

MoG-VLM is evaluated on four video-anomaly benchmarks using frame-level AUC-ROC, with comparisons, component analyses, qualitative examples, efficiency measurements, and sensitivity studies. It reaches state-of-the-art results on three benchmarks in the zero-shot, training-free setting while improving accuracy and inference speed on XD-Violence.

4.1 Datasets & Evaluation Metrics

  • The evaluation uses frame-level AUC-ROC on XD-Violence, UCF-Crime, MSAD, and ShanghaiTech, with ablation and efficiency analyses anchored on XD-Violence.The implementation samples at a 1 FPS baseline, raises density to 3 FPS around detected motion anomalies, and triggers hierarchical lookback when frame anomaly scores exceed 0.5.Source: §4.1 · p. 9, §4.2 · pp. 9–10

4.3.1 Comparison to State-of-the-Art Methods and Discussion

  • 91.91% AUC-ROC on XD-Violence improves on Unified-reasoning by +0.57% with a 1.31× speedup, and MoG-VLM is state of the art on three of four benchmarks.It also reaches 87.88% on MSAD, 1.98% above Unified-reasoning, and 84.30% on UCF-Crime, slightly above Unified-reasoning’s 84.28%.Source: §4.3.1 · pp. 10–11, Table 1 · p. 10
  • Category AUC-ROC exceeds 90% for most MSAD and XD categories, while UCF ranges from 94.8% for Assault to 59.8% for Shoplifting.ShanghaiTech is omitted from the category breakdown because its abnormal events are not annotated by category.Source: §4.3.1 · p. 11, Figure 3 · p. 13

4.3.2 Progressive Pipeline Component Analysis

  • TGM sampling improves AUC-ROC by +0.69% to +3.29% across datasets, while anchor refinement adds +1.76% on UCF, +2.6% on XD, and +0.80% on MSAD.Tier-aware fusion adds +7.92% on UCF and +2.77% on ShanghaiTech; the temporal stream alone scores 61.64% on ShanghaiTech versus the 71.54% uniform baseline.Source: §4.3.2 · pp. 11–12, Table 2 · p. 11

4.4 Qualitative Results

  • Qualitative examples show TGM misses ShanghaiTech skating that resembles normal walking, leaving AUC-ROC at the 1 FPS baseline of 72.9%.In the UCF road-accident example, the brief hit is missed, but the score remains elevated during the prolonged post-impact interval.Source: §4.4 · pp. 12–13, Figure 4 · p. 13

4.5 Computational Efficiency Analysis

  • MoG-VLM reaches 43.84, 36.25, 39.01, and 40.46 FPS across the four benchmarks, giving 4.30–4.82× speedups over LAVAD and 1.17–1.31× over Unified-reasoning.On XD-Violence it saves 4.60 hours of GPU inference time while improving AUC-ROC by +0.57%; the final dual configuration takes 14.80, 7.80, 1.62, and 0.28 hours, respectively.Source: Table 5 · p. 12, Table 4 · p. 12, §4.5 · pp. 13–14

4.6 Ablation

  • TGM performance is stable across tested window configurations, while raising the anchor threshold from 0.4 to 0.8 drops AUC-ROC by 2.76% on XD and 2.60% on UCF.The widest TGM windows perform best on UCF and XD, whereas the default {1,2,4} windows favor MSAD and ShanghaiTech.Source: §4.6 · p. 14, Table 6 · p. 14, Table 7 · p. 14

5 Conclusion

MoG-VLM is a zero-shot, training-free framework that uses motion to govern when VLM inference runs; it achieves state-of-the-art results on three benchmarks and remains competitive on ShanghaiTech. Its fusion-weight analysis finds that sensitivity differs across weights and datasets.

5 Conclusion

  • MoG-VLM sets state of the art on UCF-Crime, XD-Violence, and MSAD, remains competitive on ShanghaiTech, and speeds up inference over the strongest zero-shot baseline.The conclusion characterizes content-aware allocation as more productive than scaling uniform VLM inference.Source: §5 · p. 14

A.1 Tier-Aware Fusion Weight Sensitivity

  • Raising λanchor from 0.2 to 0.7 lifts mean AUC by 0.63%, while raising λbase from 0.3 to 0.6 raises mean AUC to 86.31% (+0.46%).Both changes benefit ShanghaiTech while reducing UCF-Crime AUC; the authors retain λbase = 0.3 because higher values inflate the mean at UCF-Crime’s expense.Source: §A.1 · p. 19
  • The λmotion sweep spans only 0.19% in mean AUC, with per-dataset variation under 1.2%.The authors keep λmotion = 0.4 as the inflection point where ShanghaiTech and UCF-Crime trends cross.Source: §A.1 · p. 19

A.2 Ablation: EMA Freezing during Anomaly Windows

Freezing the EMA statistics during elevated motion prevents sustained anomalous motion from being absorbed into the normal baseline. The mechanism improves frame-level AUC-ROC, with larger gains on datasets where anomalies unfold over multiple seconds.

A.2 Ablation: EMA Freezing during Anomaly Windows

  • Freezing µt and σt when z(t) > θanom prevents self-suppression and improves frame-level AUC-ROC, with gains of +2.01 pp on MSAD and +1.95 pp on UCF-Crime.The gain is +0.44 pp on XD-Violence, where punctate events may finish before the EMA adapts; freezing adds no inference cost or hyperparameters.Source: §A.2 · pp. 19–20

A.3 Ablation: Anomaly Threshold θanom Sensitivity

The motion gate uses θanom = 2.3 across all datasets without per-dataset tuning. Frame-level AUC-ROC is relatively stable across the tested threshold range.

A.3 Ablation: Anomaly Threshold θanom Sensitivity

  • Frame-level AUC-ROC varies by less than 1% across θanom values from 1.5 to 2.3 on every benchmark.All four datasets favor lower thresholds that admit more candidates; θanom = 2.3 was fixed across datasets and selected before sensitivity analysis.Source: §A.3 · p. 20

A.4 Ablation: Gaussian Smoothing Strength

The Gaussian filter smooths the fused score to suppress high-frequency noise from caption variability and tier transitions. The optimal width depends on anomaly density, and the authors use σ = 1.0 s across datasets to preserve temporal precision for sparse anomalies.

A.4 Ablation: Gaussian Smoothing Strength

  • At σ = 3.0, ShanghaiTech falls 2.32% to 76.24%, while dense-anomaly benchmarks are robust and mildly benefit from wider kernels.XD-Violence plateaus near 92.0%, MSAD peaks at 88.17%, and UCF-Crime at 84.73%; the authors fix σ = 1.0 s across datasets.Source: §A.4 · p. 20

A.5 Ablation: Prompt Sensitivity

The frame-level pipeline was tested with four prompt variants on XD, and AUC-ROC varied by only 0.71% across them.

A.5 Ablation: Prompt Sensitivity

  • 0.71% is the full AUC-ROC span across four frame-level prompt variants on XD, including the default prompt.The variants remove indicator lists, remove lists and rules and structure, or reword the default content.Source: §A.5 · pp. 20–21, Table 13 · p. 21

A.6 Ablation: Isolating the Sampling Contribution

With the pipeline fixed, motion-adaptive sampling is faster than uniform sampling at 3 FPS while maintaining or improving frame-level AUC-ROC on three benchmarks.

A.6 Ablation: Isolating the Sampling Contribution

  • 2.55–2.86× faster than uniform sampling at 3 FPS, motion-adaptive sampling improves frame-level AUC-ROC on XD-Violence, UCF-Crime, and MSAD.ShanghaiTech has a small drop, which the full pipeline recovers.Source: §A.6 · p. 21, Table 14 · p. 22
  • The schedule ablation holds the pipeline fixed and replaces motion-adaptive sampling with uniform sampling at 3 FPS, MoG-VLM’s densest rate.This isolates the sampling schedule as the varied factor.Source: §A.6 · p. 21, Table 14 · p. 22
  • The reported gains are attributed to where computation is spent rather than to uniformly processing more frames.The comparison uses the same pipeline under the two sampling schedules.Source: §A.6 · p. 21

B Additional Qualitative Results

Qualitative examples include detections of both motion-driven and semantically anomalous events, as well as failures involving subtle behavior and missing scene context. The accompanying prompts target security-relevant frame descriptions, anomaly classification, and temporal action dynamics.

B Additional Qualitative Results

  • Robbery_016 scores 0.96 as sustained motion tracks the ground-truth anomaly boundary, while a vehicle inside a retail interior scores 0.92 through contextual incongruity.The latter event has comparatively modest motion, showing a semantically anomalous case alongside a kinematically driven one.Source: B Additional Qualitative Results · p. 22, Figure 6 · p. 23
  • Two failures involve a circular-walking irregularity scored 0.1 and a motorcycle in a pedestrian plaza scored 0.2.The first reflects difficulty distinguishing trajectory-level semantics from appearance; the second reflects missing explicit pedestrian-only context in zero-shot semantic grounding.Source: B Additional Qualitative Results · p. 22, Figure 6 · p. 23

C.1 Frame-Level Captioning Prompt

  • The prompt design seeks zero-shot generalization across surveillance scenarios without dataset-specific terminology.It requests entity counts, positions, activities, interactions, and open-ended safety concerns in frame descriptions.Source: §C.1 · p. 24

C.2 Frame-Level Anomaly Scoring Prompt

  • The frame-level classifier uses explicit normal and anomaly indicators to reduce ambiguity across scenarios and operates on frames selected by TGM-based motion detection.The indicator lists include security events such as violence and accidents, alongside routine activities and appropriate interactions.Source: §C.2 · pp. 24–25

C.3 Temporal-Level Multi-Frame Captioning Prompt

  • The temporal prompt directs Video-LLaMA to describe actions, movements, and events, with suspicious behaviors explicitly listed to focus on security-relevant patterns.The design rationale emphasizes behavioral evolution across frames rather than static scene elements.Source: §C.3 · pp. 25–26

D Limitations

Raw temporal gradients can generate spurious activations from abrupt illumination changes and camera jitter, reducing efficiency gains.

D Limitations

  • Abrupt illumination changes and camera jitter can create spurious TGM activations that erode efficiency gains.The adaptive EMA baseline (α = 0.95) and z-score normalization absorb gradual drift but cannot distinguish a true motion burst from a global photometric spike within a single window.Source: D Limitations · p. 26
  • The multiscale design W = {1,2,4} partially mitigates brief photometric spikes by diluting their response at w = 2 and w = 4.The aggregated signal is consequently less likely to exceed the z-score threshold than a sustained physical motion event would.Source: D Limitations · p. 26
  • Within a single window, the method cannot distinguish a true motion burst from a global photometric spike.This remains a limitation despite the EMA baseline, z-score normalization, and multiscale design.Source: D Limitations · p. 26

Related papers

Loading 2610.31478v1…