PaperlayerView exact revision on arXiv

Source-linked AI summary

MS2CAST: A Video Anomaly Detection Framework via Multi-Scale Spatial Cross-Attention and Efficient Spatiotemporal Networks

Ali Safaei, Mohammad Kazemi

arXiv:2610.29649v1cs.CV

TL;DR

Predictive VAD often relies on computationally intensive optical-flow networks, while combining CNN and transformer spatial features remains underexplored. MS2CAST fuses those features and uses latent spatiotemporal modules; across three datasets, it reports competitive frame-level AUC with peak GPU memory under 1 GB.

  • Problem

    Predictive VAD often relies on computationally intensive optical-flow networks, and approaches combining CNN and transformer spatial features remain underexplored.

    Source: §1 · p. 2

  • Method

    MS2CAST predicts future frames by fusing hierarchical ViT and CNN features with multi-scale cross-attention, using interchangeable latent-space SwinLSTM or VMRNN modules and an L1, GDL, and MS-SSIM objective.

    Source: §1 · pp. 3–4, §3.5 · p. 18

  • Results

    Across three benchmarks, MS2CAST achieves frame-level ROC AUC scores on par with top-tier techniques while maintaining peak GPU memory under 1 GB.

    Source: §1 · p. 4, §5 · p. 35

  • Takeaways & Limitations

    The reported ablations support the effectiveness of multi-scale spatial feature fusion paired with an efficient spatiotemporal module in this predictive VAD framework.

    Source: §5 · p. 35

Abstract

from arXiv · show

Video anomaly detection (VAD) is a challenging task for surveillance systems due to the scarcity and diversity of anomalous events. While semi-supervised VAD approaches have advanced the field, many rely on computationally intensive optical flow networks to capture motion dynamics. Convolutional neural networks (CNNs) are also widely used for spatial encoding, but their effective receptive field is highly local. To address these limitations, we introduce MS2CAST, an end-to-end semi-supervised predictive VAD framework. For spatial encoding, we employ a multi-scale cross-attention encoder that fuses the hierarchical feature representations from a vision transformer (MViTv2-S) and a CNN (ConvNeXt-T) to exploit their distinct receptive fields. To model spatiotemporal dynamics without optical flow estimation, we integrate two interchangeable spatiotemporal modules in the latent space. One module (SwinLSTM) incorporates a Swin Transformer, while the other (VMRNN) integrates Vision Mamba blocks. Our objective function combines L1 loss, Gradient Difference Loss (GDL), and Multi-Scale Structural Similarity Index Measure (MS-SSIM) to ensure pixel-wise accuracy, sharpness, and structural integrity. Evaluations across the UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets demonstrate frame-level Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) scores that are competitive with top-tier techniques while maintaining a peak GPU memory usage of under 1 GB.

1. Introduction

MS2CAST addresses predictive VAD’s motion-modeling and spatial-encoding challenges with a semi-supervised architecture that fuses ViT and CNN features and uses efficient latent spatiotemporal modules. It reports competitive frame-level AUC across three benchmarks with peak GPU memory below 1 GB.

1. Introduction

  • Predictive VAD can avoid reconstruction-based overgeneralization, but GAN training may be unstable, optical flow is computationally intensive, and pixel-space L2 can produce blurry predictions.Pre-trained optical-flow networks may also limit dataset adaptability.Source: §1 · pp. 2–3
  • MS2CAST fuses hierarchical ViT and CNN feature representations through multi-scale spatial cross-attention, leveraging their distinct receptive fields for spatial encoding.The paper notes that CNN receptive fields are local at early layers, whereas ViTs have a more global receptive field.Source: §1 · pp. 3–4
  • SwinLSTM and VMRNN are interchangeable latent-space modules for capturing spatiotemporal dynamics without computationally intensive optical-flow networks.SwinLSTM uses Swin Transformer blocks, while VMRNN integrates Vision Mamba blocks.Source: §1 · pp. 3–4
  • The prediction objective combines L1 loss, MS-SSIM, and GDL to address pixel-wise fidelity, perceptual integrity, and sharp edges.The paper motivates the combined criteria with prior findings on sharper predictions and improved results from combining these losses.Source: §1 · p. 4
  • Across UCSD Ped2, CUHK Avenue, and ShanghaiTech, MS2CAST achieves frame-level ROC AUC scores on par with top-tier techniques while peak GPU memory remains under 1 GB.The reported evaluation covers three public datasets.Source: §1 · p. 4

2. Related works

The related-work review contrasts multimodal optical-flow systems with single-modal RGB methods, then positions MS2CAST as a single-modal predictive framework combining ViT/CNN feature fusion, efficient spatiotemporal modeling, and a streamlined loss.

2. Related works

  • Related VAD work spans multimodal and single-modal inputs, with input choice affecting detection accuracy and computational demands.The section reviews both settings to contextualize MS2CAST.Source: §2 · p. 4

2.1. Multi-modal approaches

  • Multimodal methods pair RGB with traditional or pre-trained optical flow; deep flow can add computation and may degrade across domains, while memory modules can limit cross-scene generalization.Examples include TV-L1-based methods,, and deep-flow approaches such as AMMC-Net, Ning et al., and Ma et al..Source: §2.1 · p. 5, §2.0 · p. 6

2.2. Single-modal approaches

  • Single-modal RGB methods include reconstruction systems,,, predictive approaches using adversarial training or optical-flow constraints,,, and transformer-based frameworks,.Some approaches also use object detectors, which can struggle with small or occluded objects and trade accuracy against speed.Source: §2.2 · pp. 6–7
  • MS2CAST instead combines ViT and CNN feature maps through spatial cross-attention, pairs this encoding with an efficient spatiotemporal module, and uses MS-SSIM, GDL, and L1 loss.The stated design avoids reliance on adversarial training, object detectors, or optical-flow loss.Source: §2.2 · p. 7

3.1. Overall architecture

MS2CAST predicts the next frame from a four-frame RGB clip using parallel ViT and CNN spatial pathways. Cross-attention fuses their features before temporal modeling and decoding.

3.1. Overall architecture

  • The model takes four RGB frames as input and predicts the next frame within a U-Net-based architecture.The input clip is I_(t−3:t), and the target is the predicted I_(t+1).Source: §3.1 · p. 8
  • Independent ViT and CNN pathways extract multi-level spatial features, which are fused at each level using CNN features to guide ViT feature maps.The encoder uses skip connections to pass fused features from the last input frame to the decoder.Source: §3.1 · p. 8
  • The lowest-resolution fused feature map passes through the spatiotemporal module and decoder, while the other fused maps provide spatial information through skip connections.Source: §3.1 · p. 8

3.2. Multi-scale cross-attention encoder

The encoder pairs multi-scale features from MViTv2-S and ConvNeXt-T, then fuses them with cross-attention while preserving spatial topology. Its fused representation feeds the spatiotemporal module, with earlier-stage features passed to the decoder through skip connections.

3.2. Multi-scale cross-attention encoder

  • The encoder uses parallel pathways to extract multi-scale spatial maps from RGB frames, fusing the maps with spatial cross-attention.The resulting architecture is part of a pipeline that takes four consecutive frames to predict the future frame.Source: §3.2 · p. 8, Figure 1 · p. 9

3.2.1. Multi-scale spatial feature extractors

  • MViTv2-S and ConvNeXt-T provide aligned multi-scale features at four stages, enabling cross-attention fusion of paired maps with distinct receptive fields.The backbones are initialized with ImageNet-1k pre-trained weights and kept frozen during training.Source: §3.2 · p. 8, §3.2.1 · pp. 9–10

3.2.2. 2D sinusoidal spatial positional encoding

  • A non-learnable 2D sinusoidal positional encoding preserves spatial topology in the flattened feature sequences used for cross-attention.The encoding is dynamically scaled to each stage’s resolution and broadcast across batch and temporal dimensions.Source: §3.2.2 · pp. 10–11, §3.2.3 · p. 11

3.2.3. Multi-head cross-attention mechanism

  • Cross-attention fuses flattened feature sequences, and the study tests both Pre-LN and Post-LN configurations.The fused sequence goes to the spatiotemporal module, while first-three-stage features from the final input frame reach the decoder through skip connections.Source: §3.2.3 · pp. 11–12

3.3. Spatiotemporal module

The latent spatiotemporal module models motion dynamics from bottleneck features using either SwinLSTM or VMRNN. Both alternatives carry temporal memory, but use different spatial processing mechanisms.

3.3. Spatiotemporal module

  • The module receives fused bottleneck features to model motion dynamics; SwinLSTM and VMRNN are interchangeable alternatives to ConvLSTM.The paper motivates these alternatives by ConvLSTM’s CNN-based limitation in capturing long-range spatial dependencies.Source: §3.3 · p. 13

3.3.1. SwinLSTM

  • SwinLSTM combines current spatial tokens with the previous hidden state, then alternates window-based and shifted-window self-attention in its transformer blocks.The described local attention window is 7 × 7.Source: §3.3.1 · pp. 13–14

3.3.2. VMRNN

  • VMRNN’s Visual State Space blocks process spatial tokens through four directional scans and aggregate the restored outputs to capture spatial context.The four trajectories are raster, reversed raster, column-major, and reversed column-major scans.Source: §3.3.2 · p. 16

3.4. Decoder

The decoder maps the spatiotemporal module’s final hidden state to a 224 × 224 predicted frame using five hierarchical upsampling stages. It uses skip connections in the first three stages and omits them in the final two.

3.4. Decoder

  • The decoder maps the final hidden state to a predicted frame through five upsampling stages, producing 224 × 224 output.A 1 × 1 convolution reduces the output to three RGB channels, followed by sigmoid activation.Source: §3.4 · pp. 16–17
  • Stages 1–3 upsample by bilinear interpolation, reduce channels with convolution, and concatenate corresponding fused intermediate features through skip connections.The concatenated representations pass through Double Convolution blocks.Source: §3.4 · p. 17
  • Stages 4–5 continue the upsampling process without skip connections, relying on the preceding stage operations apart from concatenation.The decoder’s Double Convolution blocks use group normalization, which is independent of batch size.Source: §3.4 · p. 17

3.5. Training loss

The training objective combines pixel-level accuracy, edge sharpness, and structural integrity for predicted frames. It weights L1 and GDL together against the MS-SSIM loss component.

3.5. Training loss

  • The objective combines L1 loss, GDL, and MS-SSIM loss to target pixel-wise accuracy, sharp edges, and perceptual integrity.The paper describes the total objective as a weighted combination of the three components.Source: §3.5 · pp. 18, 19, 21
  • GDL penalizes mean absolute differences between prediction and target spatial gradients along horizontal and vertical axes.The paper uses GDL to generate sharp edges in predicted frames.Source: §3.5 · p. 19
  • MS-SSIM assesses a generated frame’s holistic structural integrity using luminance, contrast, and structure across progressively downscaled scales; its score ranges from 0 to 1, with higher scores indicating better frame quality.Source: §3.5 · pp. 19–20
  • The objective gives L1 and GDL the same weight, balancing their combined pixel-level errors against structural integrity measured by MS-SSIM.The coefficient α is optimized as discussed in Section 4.3.Source: §3.5 · p. 21

3.6. Anomaly score

The anomaly score combines pixel-level prediction error with structural deviation to identify anomalous frames. Normalized PSNR- and MS-SSIM-based scores are combined using a convex weight and smoothed over time.

3.6. Anomaly score

  • PSNR captures pixel-level prediction errors, while MS-SSIM measures structural deviations between predicted and actual future frames.The two metrics are used together to score anomalies from prediction errors.Source: §3.6 · pp. 21–22
  • The model is trained exclusively on normal samples, with the expectation that anomalous samples produce high prediction errors at inference.Source: §3.6 · p. 21
  • PSNR is used for anomaly scoring because squared L2 error magnifies large pixel-wise deviations quadratically.The training objective instead uses L1 loss to preserve pixel-wise accuracy.Source: §3.6 · p. 21
  • The final anomaly score is a convex combination, with coefficient w in [0, 1], of normalized PSNR- and MS-SSIM-based scores, then smoothed using a 1D Gaussian filter.Source: §3.6 · p. 22

4. Experiments

Experiments evaluate MS2CAST on three standard VAD benchmarks using frame-level ROC AUC, then examine loss weighting, encoder and temporal-module choices, noise resilience, and efficiency. The selected Pre-LN cross-attention–SwinLSTM configuration uses 806 MB peak GPU memory while maintaining strong anomaly-detection capability.

4.1. Datasets

  • The experiments use UCSD Ped2, CUHK Avenue, and ShanghaiTech, with ShanghaiTech spanning 13 scenes and 330 training and 107 testing videos.Performance is evaluated using frame-level ROC AUC.Source: §4.1 · p. 23, §4.2 · p. 24

4.3. Loss function optimization

  • Across UCSD Ped2 and CUHK Avenue, peak performance occurred at α=0.2, highlighting the importance of the MS-SSIM perceptual-loss component over L1 and GDL.The authors selected α=0.2 for subsequent experiments.Source: §4.3 · p. 24, Table 2 · p. 25

4.4.1. Spatial encoder and spatiotemporal module effect evaluation

  • Cross-attention improved AUC on UCSD Ped2, while on CUHK Avenue the Post-LN cross-attention–SwinLSTM combination ranked highest among ten configurations.Fine-tuning the ImageNet-1k-pretrained MViTv2-S degraded AUC in all cases.Source: §4.4.1 · pp. 25–27

4.4.2. Robustness to noise

  • AUC decreased monotonically as AWGN increased across configurations, but degradation severity varied by configuration.The Pre-LN cross-attention–SwinLSTM pairing generally showed high noise resilience on both datasets, despite the Post-LN pairing’s clean-condition advantage.Source: §4.4.2 · pp. 28–29

4.4.3. Computational efficiency analysis

  • Single-encoder configurations had lower latency and higher throughput than multi-scale cross-attention, while SwinLSTM was faster than VMRNN but used more GPU memory.The authors describe a trade-off between anomaly-detection capability and computational efficiency.Source: §4.4.3 · pp. 29–30, Table 7 · p. 30
  • 806 MB peak GPU memory was measured for the selected Pre-LN cross-attention–SwinLSTM architecture, which the authors report maintains strong anomaly-detection capability under 1 GB.Source: §4.4.3 · p. 30

4.5. Comparison with existing methods

MS2CAST performs comparably to top-tier semi-supervised methods on UCSD Ped2 and CUHK Avenue and outperforms them on ShanghaiTech. The selected configuration’s anomaly-score weighting also has dataset-dependent optima, with higher PSNR weighting favored on the single-scene datasets and higher MS-SSIM weighting on ShanghaiTech.

4.5. Comparison with existing methods

  • MS2CAST performs comparably to top-tier semi-supervised methods on UCSD Ped2 and CUHK Avenue and outperforms all of them on ShanghaiTech.The selected Pre-LN cross-attention–SwinLSTM configuration achieved 75.3% AUC on ShanghaiTech at w=0.2, and used under 1 GB peak GPU memory.Source: §4.5 · pp. 30–31, Table 8 · p. 31
  • Figures 4 and 5 visualize predicted frames and pixel-wise error maps, alongside anomaly scores over time with ground-truth anomalous frames marked in red.Source: §4.5 · pp. 31–32, Figure 4 · p. 32, Figure 5 · p. 33

4.6. Hyperparameter sensitivity analysis

  • 96.1% AUC was the UCSD Ped2 maximum at w=0.6, while CUHK Avenue peaked at 85.8% AUC at w=0.9.On ShanghaiTech, AUC peaked at w=0.2 and remained at its maximum through w=0.4.Source: §4.6 · p. 34, Figure 6 · p. 34
  • Higher PSNR weighting yielded maximum AUC on UCSD Ped2 and CUHK Avenue, whereas higher MS-SSIM weighting generally yielded higher AUC on ShanghaiTech.The authors conclude that the dual-metric scoring mechanism offers flexibility across surveillance environments.Source: §4.6 · pp. 34–35

5. Conclusion

MS2CAST combines multi-scale cross-attention between ViT and CNN features with an efficient spatiotemporal module for semi-supervised predictive VAD. The authors report competitive results on UCSD Ped2 and CUHK Avenue, better performance than top-tier methods on ShanghaiTech, and peak GPU memory below 1 GB.

5. Conclusion

  • MS2CAST with a multiscale Pre-LN cross-attention encoder and SwinLSTM performs competitively on UCSD Ped2 and CUHK Avenue and outperforms top-tier VAD methods on ShanghaiTech.The reported configuration maintains peak GPU memory usage under 1 GB.Source: §5 · p. 35
  • The framework fuses hierarchical ViT and CNN representations for spatial encoding and pairs them with an efficient spatiotemporal module to capture motion without deep optical flow networks.Its objective balances pixel-wise accuracy, structural integrity, and sharp predicted-frame edges using spatial gradients.Source: §5 · p. 35
  • Ablation studies support the effectiveness of multi-scale spatial cross-attention, while the reported configuration shows robustness across additive-noise types.Source: §5 · p. 35
  • Anomaly scoring remains dataset-sensitively dependent on hyperparameter w, and future work will explore alternative metrics and a unified parameter-free score.Source: §5 · p. 35

Related papers

Loading 2610.29649v1…