PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Score Matching Paradigm for Video Anomaly Detection and Anticipation

Congqi Cao, Zhenhe Liang, Hanwen Zhang, Yifan Zhao, Qinyi Lv, Lingtong Min, Yanning Zhang

arXiv:2610.11149v1cs.CVeess.IV

TL;DR

Video anomaly detection approaches based on likelihoods can miss anomalies in local modes, and video anomaly anticipation cannot rely on future ground-truth frames. Uni-DSM applies a shared score-matching formulation with task-specific mechanisms to both tasks; experiments report state-of-the-art performance across multiple benchmarks.

  • Problem

    Likelihood-based anomaly scoring can miss anomalies in local modes, while video anomaly anticipation lacks future ground-truth frames for conventional contrastive or prediction-based paradigms.

    Source: §I · p. 1, B. Video Anomaly Anticipation · p. 4

  • Method

    Uni-DSM uses denoising score matching with a shared noise-conditioned score transformer and scene- and motion-aware scoring, plus autoregressive detection and self-distilled anticipation mechanisms.

    Source: §I · p. 2

  • Results

    Experiments report state-of-the-art performance in both detection and anticipation across multiple benchmarks, including Avenue, ShanghaiTech, and NWPU Campus.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper presents one denoising score-matching formulation for video anomaly detection and anticipation, with task-specific inference and supervision mechanisms.

    Source: §VI · p. 12

Abstract

from arXiv · show

Video anomaly detection (VAD) is a fundamental and safety-critical task in computer vision. Recent generative approaches detect anomalies from a distributional perspective, but remain limited by local anomaly modes. Meanwhile, video anomaly anticipation (VAA), as a proactive extension beyond post-hoc detection, introduces additional challenges. In particular, the contrastive inference paradigm in VAD, which relies on ground-truth frames, is not applicable to VAA, hindering its development. To address these challenges, we propose a unified score-driven framework, termed Uni-DSM, based on denoising score matching (DSM), which models anomaly patterns through likelihood estimation and score functions over the learned data distribution. Within this unified framework, we adopt a shared noise-conditioned score transformer backbone with scene-dependent embeddings and motion-aware weighting for distribution-level modeling. Instead of introducing separate architectures, Uni-DSM unifies VAD and VAA through different inference and supervision paradigms built upon the same score-based formulation. For VAD, we instantiate an autoregressive denoising score matching (ADSM) mechanism, which progressively accumulates anomalous evidence via autoregressive denoising, enabling enhanced perception of local modes beyond visual cues. For VAA, we extend the same architecture by incorporating a lightweight auxiliary decoder and a novel self-distilled denoising score matching (SDSM) mechanism. By constructing supervision from output discrepancies instead of relying on unavailable future ground truth, our method achieves efficient training suitable or early anomaly anticipation. Extensive experiments on multiple benchmark datasets demonstrate state-of-the-art performance in both VAD and VAA while maintaining high efficiency, establishing a unified and scalable pipeline from anomaly detection to anticipation.

I. INTRODUCTION

Video anomaly detection methods based on visual contrastive errors and likelihoods face limitations with appearance variation and anomalies near local modes, while anticipation lacks future ground truth. Uni-DSM uses one score-driven formulation for both tasks, with task-specific inference and supervision mechanisms.

I. INTRODUCTION

  • Visual contrastive errors can confuse normal samples with significant appearance variation, likelihood-based scores can miss anomalies in local modes, and anticipation lacks future ground truth.The latter makes conventional contrastive or prediction-based VAD paradigms unsuitable for VAA.Source: §I · pp. 1–2
  • Uni-DSM models anomalies through denoising score matching and a shared noise-conditioned score transformer with scene-dependent embeddings and motion-aware weighting.Its score-driven formulation supports both detection and anticipation.Source: §I · p. 2
  • For detection, ADSM progressively injects intensifying noise and aggregates denoising discrepancies to accumulate anomalous context and improve discrimination of local modes.The mechanism combines denoised-versus-original discrepancies with autoregressive inference.Source: §I · p. 2
  • For anticipation, SDSM uses output discrepancies for supervision instead of future ground-truth frames, within a lightweight main-auxiliary architecture with unidirectional inference.This self-distillation mechanism maintains architectural consistency with ADSM while reducing computational overhead.Source: §I · pp. 2–3
  • Experiments report state-of-the-art performance in detection and anticipation across Avenue, ShanghaiTech, and NWPU Campus.The paper describes its benchmark results as demonstrating effectiveness and generalization of the unified framework.Source: §I · p. 2

II. RELATED WORK

Semi-supervised VAD methods based on reconstruction, prediction, and likelihood estimation remain challenged by anomalies in local modes. VAA must predict future risk without future ground truth, while existing approaches trade generalization or computational efficiency against predictive modeling.

A. Video Anomaly Detection Methods

  • Likelihood-based VAD methods still struggle to distinguish anomalies residing in local modes.Prior work also lacks explicit modeling of scene context and motion patterns, potentially misaligning distribution-level estimation with visual perception.Source: A. Video Anomaly Detection Methods · p. 3
  • Uni-DSM combines autoregressive denoising with accumulated abnormal context and scene-dependent, motion-aware score constraints for VAD.The approach jointly uses visual appearance and likelihood to improve anomaly discrimination.Source: A. Video Anomaly Detection Methods · p. 3

B. Video Anomaly Anticipation Methods

  • Semantic memory banks avoid pixel-level anticipation but may generalize poorly, while bidirectional prediction adds computational overhead and may be unstable for long horizons.Memory banks can be weakly correlated with evolving anomalies; bidirectional methods use forward prediction and backward reconstruction.Source: B. Video Anomaly Anticipation Methods · p. 4
  • Uni-DSM extends its architecture to VAA with a lightweight main-auxiliary decoder and self-distilled supervision that does not require future ground truth.The design is presented as maintaining consistency with ADSM while supporting efficient predictive anticipation.Source: B. Video Anomaly Anticipation Methods · p. 4

III. TASK DEFINITION

  • VAD scores whether an anomaly is occurring now, whereas VAA predicts whether one will occur within a future time window.VAA supports proactive warning without requiring the exact anomaly onset to be predicted.Source: §III · p. 4

A. Video Anomaly Detection

The section distinguishes reactive detection from future-window anticipation and introduces Uni-DSM as a shared score-driven framework for both. Its denoising score matching objective learns score functions from noise-corrupted data, including across noise levels.

A. Video Anomaly Detection

  • VAD estimates the current frame’s anomaly score from historical frames; normal frames receive low scores and anomalous frames high scores.Because it is reactive, VAD can raise alarms only after anomalies occur.Source: A. Video Anomaly Detection · p. 4

B. Video Anomaly Anticipation

  • VAA predicts whether an anomaly will occur in the future window [t+1, t+α], unlike VAD’s current-state judgment.The absence of future ground-truth frames makes conventional contrastive or prediction-based paradigms unsuitable for this task.Source: B. Video Anomaly Anticipation · p. 4

A. Framework Overview

  • Uni-DSM uses a shared noise-conditioned score transformer with scene-dependent and motion-aware scoring, alongside ADSM for detection and SDSM for anticipation.ADSM addresses local-mode anomalies through autoregressive noise injection; SDSM provides ground-truth-free supervision through a lightweight auxiliary branch.Source: A. Framework Overview · p. 4

B. Denoising Score Matching

  • Denoising score matching perturbs clean data with Gaussian noise and trains a score network to approximate the corrupted data distribution’s score.The objective is described as equivalent to matching the score of a non-parametric Parzen density estimator constructed from the data.Source: B. Denoising Score Matching · pp. 4–5
  • A noise-conditioned score network estimates scores across noise levels, with the score interpreted as a vector field pointing in the denoising direction.The network jointly estimates score functions for the specified noise levels.Source: B. Denoising Score Matching · p. 5

C. Noise-Conditional Score Transformer

The noise-conditional score transformer (NCST) adapts denoising score matching to video through block-wise training. It conditions a DiT-based attention model on noise level and processes noisy video as patch tokens.

C. Noise-Conditional Score Transformer

  • NCST adapts the denoising score matching objective to a block-wise formulation, using attention to model video dynamics while conditioning on noise level.Its design is motivated by the need to capture complex spatiotemporal dynamics and auxiliary variables such as noise.Source: C. Noise-Conditional Score Transformer · p. 5
  • NCST adds Gaussian noise to historical video frames at scheduled levels, then partitions the perturbed frames into non-overlapping image patches.For frame resolution H × W and patch size d × d × c, the patch count is N = n·(H/d)·(W/d).Source: C. Noise-Conditional Score Transformer · p. 5
  • The score transformer estimates a score for each noisy patch using a block-wise objective and receives the diffusion step through an MLP noise embedding.Continuous multi-scale block-wise denoising supervision is described as learning a stable vector field that enhances anomaly discriminability.Source: C. Noise-Conditional Score Transformer · p. 5

D. Scene-Conditioned Embedding

Uni-DSM conditions its score transformer on scene identity and diffusion time, while motion-aware patch weights emphasize dynamic foreground and reduce static-background interference.

D. Scene-Conditioned Embedding

  • Scene-aware context matters because event normality or abnormality depends on the specific environment.The cited work reports that incorporating scene-aware context improves detection performance,,.Source: D. Scene-Conditioned Embedding · pp. 5–6
  • Scene identity and diffusion time jointly condition the transformer through block-level adaptive modulation.The scene label is based on camera ID, and S-AdaLN transmits the combined condition to each transformer block.Source: D. Scene-Conditioned Embedding · p. 6

E. Key-Frame Motion Weighting

  • Motion-aware patch weighting emphasizes dynamic foreground and reduces interference from static background areas.The method injects motion information into DSM to address score bias toward background statistics.Source: E. Key-Frame Motion Weighting · p. 6

F. Autoregressive Denoising Mechanism for Anomaly Detection

For detection, Uni-DSM progressively denoises inputs to accumulate anomalous context and combines score and appearance cues. For anticipation, self-distillation uses main–auxiliary output discrepancies without future ground-truth supervision.

F. Autoregressive Denoising Mechanism for Anomaly Detection

  • ADSM applies intensifying noise to denoised data across steps, accumulating abnormal context to improve anomaly scoring.The method also integrates discrepancies between denoised outputs and original inputs to enhance appearance perception and suppress local modes.Source: F. Autoregressive Denoising Mechanism… · p. 6
  • The VAD score combines score-based evidence with PSNR-based visual fidelity to bridge the appearance gap.The visual consistency term is described as helping address cases where likelihood and appearance cues disagree.Source: F. Autoregressive Denoising Mechanism… · pp. 6–7

G. Self-Distilled Denoising Mechanism for Anomaly Anticipation

  • SDSM uses main–auxiliary output discrepancies as an anomaly-probability proxy, avoiding dependence on future ground truth.Both branches are trained exclusively on normal data; their denoised outputs are consistent for normal samples and diverge for anomalous inputs. The auxiliary branch adds one transformer layer and diverges after the main branch’s first transformer block.Source: G. Self-Distilled Denoising Mechanism… · p. 7

H. Score-Based Anomaly Inference

The inference procedure uses score magnitude, normalized by visual fidelity, to estimate anomaly evidence over noise levels. VAD scores observable frames, while VAA uses main–auxiliary discrepancies to score a future window without future-frame comparison.

VAD inference

  • For VAD, the method evaluates each video segment across noise levels, takes the maximum per level, then normalizes and averages scores.The section describes frame-level score evaluation because the entire video sequence is observable.Source: H. Score-Based Anomaly Inference · p. 7

VAA inference

  • The score norm normalized by visual fidelity is the primary anomaly metric for VAA, where future frames are unavailable at inference.The score function remains defined over the learned distribution despite this partial observability.Source: H. Score-Based Anomaly Inference · pp. 7–8
  • For VAA, main–auxiliary prediction discrepancies provide a surrogate for anomaly probability, and the window score takes the maximum across its future frames.Scores are normalized over the anticipation window and combined across noise levels by weighted averaging.Source: H. Score-Based Anomaly Inference · p. 8

A. Experimental Setup

The evaluation uses Avenue, ShanghaiTech, and NWPU Campus, with AUC metrics for detection and anticipation. NWPU is described as supporting scene-dependent anomaly evaluation and VAA.

Datasets

  • The study evaluates VAD on Avenue, ShanghaiTech, and NWPU Campus, and identifies NWPU as supporting VAA.NWPU contains 305 training and 242 test videos across 43 scenes and is described as the first dataset explicitly considering scene-dependent anomalies and supporting VAA.Source: A. Experimental Setup · pp. 8–9

Evaluation Metric

  • AUC is used for both tasks; VAD reports macro- and micro-average AUC, while VAA uses dataset-level frame AUC.Macro-AUC averages per-video AUC values; micro-AUC computes one AUC after concatenating test frames.Source: A. Experimental Setup · p. 9

Implementation details

  • Inputs are object-centered 160 × 160 crops, with 8-frame sequences for VAD and 5-frame sequences for VAA.The crops are centered on ByteTrack-detected objects; the model uses 100 patches per frame with patch size 16, and training runs for 100 VAD epochs and 50 VAA epochs.Source: A. Experimental Setup · p. 9

B. VAD Performance Benchmarking

Uni-DSM achieves the strongest macro-AUC across Avenue, ShanghaiTech, and NWPU, with a reported improvement of up to 2.4% on ShanghaiTech. On NWPU, it also leads on both micro- and macro-AUC, using only raw video clips.

B. VAD Performance Benchmarking

  • Uni-DSM outperforms all competitors on macro-AUC across Avenue, ShanghaiTech, and NWPU, with an improvement of up to 2.4% on ShanghaiTech.On NWPU, it achieves the best performance on both micro- and macro-AUC.Source: B. VAD Performance Benchmarking · p. 9, Table I · p. 9
  • Using only raw video clips, Uni-DSM is comparable to or often better than multimodal methods and leads same-modality methods across both micro- and macro-AUC.Reported performance fluctuations across multiple runs are under 1%.Source: B. VAD Performance Benchmarking · p. 9

C. VAA Performance Benchmarking

For VAA, SDSM outperforms the reported baselines across anticipation horizons while retaining stable performance as the horizon increases. The accompanying studies report lower performance for latent-space modeling and gains from progressively adding Uni-DSM components.

C. VAA Performance Benchmarking

  • SDSM outperforms learning-based approaches across 0.5s–3s horizons, improving by around 5% over FBAE and averaging 2.1% gains over SSAE.Performance declines as the anticipation horizon extends, while remaining stable relative to the baselines.Source: C. VAA Performance Benchmarking · p. 10

D. Study on Latent Space

  • On ShanghaiTech and NWPU, latent-space ADSM performs worse than modeling original data directly, and Uni-DSM outperforms two likelihood-based baselines in original data space.The authors attribute the latent-space result to distribution inconsistency between pretrained features and the VAD dataset.Source: D. Study on Latent Space · p. 10, Table III · p. 10, Table IV · p. 10

E. Ablation Studies

  • On ShanghaiTech VAD, denoised-to-original discrepancy aggregation improves micro- and macro-scores by 5.7% and 5.8%, respectively.The ablation reports a 4.1% average gain from the autoregressive mechanism and a further 2.9% improvement from motion consistency; all components together add 2.4% micro and 3.3% macro.Source: E. Ablation Studies · pp. 10–11, Table V · p. 11
  • On NWPU VAA, the full model reaches 70.5% accuracy, versus 67.5% for the DSM baseline, as SDSM, scene embedding, and motion weighting are added.The reported intermediate accuracies are 68.1% with SDSM, 68.6% with scene embedding, and 69.3% with motion weighting.Source: E. Ablation Studies · p. 11, Table VI · p. 11

F. Computational Resource Analysis

Uni-DSM reports real-time inference speeds for both detection and anticipation, alongside resource comparisons. Qualitative analyses describe detection score dynamics and anticipation curves, and the conclusion presents one shared score-driven framework for both tasks.

F. Computational Resource Analysis

G. Qualitative Analysis

  • On NWPU, ADSM scores rise when climber and dog anomalies occur and flatten when they disappear; the cyclist anomaly is detected without false alarms.The visualization covers scene-dependent, motion, and appearance anomalies.Source: G. Qualitative Analysis · p. 12, Figure 8 · p. 12
  • NWPU SDSM curves rise before anomaly onset, remain elevated during the predicted interval, and decline as scenes return to normal.The curves are smoother than VAD curves because anticipation operates over a continuous temporal window rather than discrete frames.Source: G. Qualitative Analysis · p. 12, Figure 9 · p. 12

VI. CONCLUSION

  • Uni-DSM combines a shared noise-conditioned score transformer with ADSM for detection and SDSM for anticipation, reporting state-of-the-art performance across both tasks.The conclusion also reports improved inference speed and reduced computational overhead.Source: §VI · p. 12
Loading 2610.11149v1…