Source-linked AI summary
Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core
Yogesh Kumar
TL;DR
Prior streaming video anomaly detectors use internal clip or window buffering and lack theory connecting temporal memory to detection latency. This paper introduces a strictly causal state-space detector with on-device evaluation, finding fast measured response despite loose decay-based bounds, mixed accuracy, and dataset-dependent gating effects.
Problem
Existing SSM-based video anomaly detectors buffer clips or windows, lack a theory of detection latency, and are commonly evaluated through GPU throughput rather than target edge hardware.
Method
The paper builds a strictly causal streaming detector using a diagonal state-space recurrence with an event-boundary decay gate and identical per-frame updates during training and deployment.
Results
The detector achieves 67.9% and 70.2% frame-AUC on Ped2 and Avenue, while gate ablations reverse across datasets and real edge latency reaches 0.74–0.77 ms per frame.
Takeaways & Limitations
The results indicate that responsiveness is governed by the event-boundary gate rather than base decay, while gating benefits depend on training-data scale.
Takeaways & Limitations
Evaluation remains limited by a single Apple M3 Pro target, an untuned accuracy configuration, and the absence of a third larger benchmark such as ShanghaiTech.
Abstract
from arXiv · showhide
Recent work has applied Mamba style state space models (SSMs) to video anomaly detection, yet existing approaches still rely on buffering clips or windows internally, lack a theoretical account of how temporal memory relates to detection latency, and benchmark efficiency only through GPU throughput rather than the edge hardware these methods are intended to target. We introduce a strictly causal streaming anomaly detector whose fixed size state is updated in O(1) time and memory per incoming frame, with no lookahead and no clip buffering. Its temporal core is a diagonal linear state space recurrence with an input and state dependent decay gate, trained self supervised through causal next embedding prediction on a frozen visual backbone. We derive a closed form relationship between the recurrence decay spectrum and both detection delay and the shortest anomaly it can reliably capture, then validate empirically on UCSD Ped2 and CUHK Avenue. The settling delay bound predicted from the learned base decay (57 to 59 frames) sits far above the measured detection delay (1.6 and 18.4 frames), showing that the event boundary gate, not the base decay, governs responsiveness. We further report end to end latency and throughput measured directly on Apple M3 Pro hardware, 0.74 ms and 0.77 ms per frame (over 1300 FPS), rather than simulated GPU numbers. With an untuned initial configuration the method reaches 67.9 percent and 70.2 percent frame level AUC on Ped2 and Avenue, trailing prior non causal SSM baselines in accuracy. Ablations over decay rate, state size, and gating reveal that the gate contribution is dataset size dependent, hurting accuracy on the smaller Ped2 training set but helping on the larger Avenue one. Closing this accuracy gap and extending evaluation to a third, larger benchmark are immediate next steps.
1. Introduction
The paper targets strict causal, per-frame video anomaly detection for constrained edge devices, addressing gaps in prior SSM-based methods’ online operation, latency theory, and hardware evaluation.
- Edge VAD requires online predictions under constrained compute, memory, and power, motivating strict causality at frame time t.
- Prior Mamba-based VAD methods improve efficiency but still process clips or windows and lack a theoretical account of temporal memory and detection latency.
- The paper introduces a strictly causal SSM detector with per-frame O(1) updates, decay-spectrum analysis, and real consumer-edge latency, memory, and throughput evaluation.
2. Related Work
Prior video-anomaly SSMs and broader efficient video methods reduce computational cost but generally retain clip-level processing, leaving strict frame-by-frame bounded-state streaming as the paper’s target.
- Video anomaly detection: VAD methods commonly learn normal motion through reconstruction or prediction errors, while weakly supervised methods use video-level labels over untrimmed surveillance footage.
- State-space models for anomaly detection: S4, S5, and Mamba establish efficient state-space sequence modeling, while VADMamba, VADMamba++, and Wave-MambaAD apply related SSM designs to video anomaly detection.
- State-space models for anomaly detection: Existing SSM-based VAD methods process clips or windows and do not provide strict online O(1)-per-frame updates, theoretical latency analysis, or edge-hardware measurements.
- Efficient and streaming video understanding more broadly: AdaFrame and SCSampler select frames or clips before heavier processing, whereas this method bounds memory and updates per frame; the mechanisms could be combined.
3. Method
The method maps each frame to a frozen-backbone embedding, updates a fixed-size gated diagonal SSM state causally, and scores anomalies through one-step embedding prediction error.
- Frozen backbone: Each incoming frame is mapped to an embedding by a frozen visual backbone, with the reported configuration using a frozen ResNet-18.
- Causal streaming core: The temporal core maintains a fixed-size state and predicts the next embedding using only current and previous information.
- Training objective and anomaly scoring: At inference, the anomaly score is the prediction error from the prediction made one step earlier, using only preceding embeddings.
- Causal diagonal state-space core: The recurrence uses a learned base decay multiplied by an input- and state-dependent event-boundary gate, enabling fast forgetting when inputs mismatch the current state.
- Strict causality, by construction: The diagonal recurrence updates state in O(1) sequence-length compute per frame and avoids lookahead, clip buffering, and reprocessing past frames.
- Training objective and anomaly scoring: Training is self-supervised on normal-only clips by minimizing causal next-embedding prediction error.
4. Decay–Delay Analysis
The analysis derives a closed-form link between recurrence decay and anomaly reaction time or minimum detectable duration, then explains how the input- and state-dependent gate changes this fixed-decay picture. Empirical ablations indicate that gating benefits depend on training-data scale, while broader validation remains incomplete.
- 4.1. Settling-time bound: The analysis models one recurrence channel with fixed effective decay and derives closed-form bounds for settling delay and minimum reliably detectable anomaly duration.The fixed-decay assumption is later relaxed conceptually through the learned gate.
- 4.1. Settling-time bound: For a step anomaly, the state converges toward s∗ = u/(1 − a), with settling delay determined by the smallest integer satisfying the ε-error condition.The proof converts a^δ ≤ ε into δ ≥ ln ε / ln a.
- 4.1. Settling-time bound: Larger decay a means longer memory but strictly slower reaction, so the bound can be compared directly with measured score-threshold crossing delays.The paper computes this quantity from each trained layer’s mean learned base decay.
- 4.2. Minimum detectable duration: An anomaly shorter than the settling delay may fail to generate the calibrated full-magnitude prediction error, increasing its risk of missed detection.The analysis predicts that reducing the base decay should reduce delay and improve recall for short anomalies.
- 4.3. Role of the event-boundary gate: The gate is intended to reduce effective decay at genuine event boundaries while preserving a larger stable base decay elsewhere.This can shorten reaction time without applying a globally smaller decay that could increase false positives.
- 4.3. Role of the event-boundary gate: Gate effects vary by dataset: disabling it improves Ped2 frame-AUC from 67.9% to 76.6% but reduces Avenue frame-AUC from 70.2% to 60.0%.The authors interpret this as consistent with a capacity–data-size interaction, but only two datasets were tested.
5. Experiments
Experiments evaluate accuracy, theory-based responsiveness, ablations, and on-device efficiency on UCSD Ped2 and CUHK Avenue. Results show a large gap between predicted settling delay and measured detection delay, dataset-dependent gating effects, and real-time Apple M3 Pro performance, while also exposing evaluation limitations.
- Datasets and protocol: The evaluation uses UCSD Ped2 and CUHK Avenue with frame-level ROC-AUC and EER under standard per-clip min-max normalization.Ped2 has 16 training and 12 test clips; Avenue has 16 training and 21 test clips.
- Main results: 67.9% frame-AUC on Ped2 and 70.2% on Avenue were achieved with an untuned first-pass configuration, below the 90s-percent range typically reported for tuned methods.The configuration used a frozen backbone, no motion signal, and no hyperparameter search.
- Theory validation: 57 to 59 frames of predicted settling delay contrasted with measured detection delays of 1.6 frames on Ped2 and 18.4 on Avenue.The comparison uses the trained mean base decay and the interval from labeled anomaly onset to the first score-threshold crossing.
- Ablations: On Ped2, disabling the gate reached 76.6% versus 67.9% gated frame-AUC, while on Avenue it fell to 60.0% versus 70.2%.The differing outcomes are consistent with a capacity and data-size interaction, but only two datasets were evaluated.
- Ablations: Combining gate removal with the smaller N=32 state did not improve Ped2, reaching 67.8% versus the 67.9% baseline, while Avenue’s combined N=256 configuration reached 71.1%.The Ped2 combined result was below either individual change, whereas the Avenue result matched the state-size-alone result.
- Edge deployment: On-device M3 Pro measurements reported 0.7 to 0.8 ms per frame, supporting real-time operation, but no CoreML or second non-Apple edge-device benchmark was included.The measurements used the deployment-equivalent per-frame update routine with explicit device synchronization.
6. Conclusion
The paper concludes that its strictly causal detector combines a causal state-space core, learned event-boundary gating, theoretical analysis, and real edge-device evaluation, but accuracy and evaluation scope remain limited.
- 67.9% and 70.2% frame-level AUC on Ped2 and Avenue trails prior non-causal SSM baselines.
- The event-boundary gate ablation reverses sign across datasets, suggesting a capacity and data-size interaction rather than a settled gate property.
- The development evaluation uses a simplified frame-overlap approximation instead of the official region- and track-based criteria.Reported localization numbers should therefore be checked with the official toolkit.
- Edge-latency measurements cover only an Apple M3 Pro with PyTorch’s MPS backend, limiting how broadly the real-time claim generalizes.CoreML/Neural-Engine export and a second non-Apple edge device are identified as needed extensions.
- The settling-delay theorem assumes fixed decay, whereas the deployed gate makes decay time-varying, so the bound is loose when the gate is active.The paper also identifies ShanghaiTech evaluation and closing the accuracy gap as remaining priorities.