PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection

Alessandro Flaborea, Luca Collorone, Guido D'Amely, Stefano D'Arrigo, Bardh Prenkaj, Fabio Galasso

arXiv:2307.07205v3cs.CV

TL;DR

OCC methods often constrain normal motion to limited latent regions, although normal actions can also vary in many ways. MoCoDAD uses diffusion to generate multimodal future skeletal poses conditioned on past motion; it reports state-of-the-art results across four benchmarks.

  • Problem

    OCC approaches that constrain normal actions to limited latent volumes can overlook diverse-but-still-normal behaviors, despite anomalies being rare and normal-only training being common.

    Source: §1 · p. 1

  • Method

    MoCoDAD uses diffusion to generate multimodal future skeletal poses conditioned on past motion and statistically aggregates the generations to assess their pertinence to the actual future.

    Source: §1 · p. 2

  • Results

    MoCoDAD outperforms the state of the art by 5.1%, 4.4%, .5%, and .8% on UBnormal, HR-UBnormal, HR-STC, and HR-Avenue, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MoCoDAD sets a new state of the art among OCC techniques and catches up with supervised techniques without using anomaly-training labels.

    Source: §6 · p. 9

Abstract

from arXiv · show

Anomalies are rare and anomaly detection is often therefore framed as One-Class Classification (OCC), i.e. trained solely on normalcy. Leading OCC techniques constrain the latent representations of normal motions to limited volumes and detect as abnormal anything outside, which accounts satisfactorily for the openset'ness of anomalies. But normalcy shares the same openset'ness property since humans can perform the same action in several ways, which the leading techniques neglect. We propose a novel generative model for video anomaly detection (VAD), which assumes that both normality and abnormality are multimodal. We consider skeletal representations and leverage state-of-the-art diffusion probabilistic models to generate multimodal future human poses. We contribute a novel conditioning on the past motion of people and exploit the improved mode coverage capabilities of diffusion processes to generate different-but-plausible future motions. Upon the statistical aggregation of future modes, an anomaly is detected when the generated set of motions is not pertinent to the actual future. We validate our model on 4 established benchmarks: UBnormal, HR-UBnormal, HR-STC, and HR-Avenue, with extensive experiments surpassing state-of-the-art results.

1. Introduction

The paper addresses OCC’s limited treatment of diverse normal motions by modeling both normal and abnormal motion as multimodal. MoCoDAD uses diffusion to generate future poses conditioned on past skeletal motion and reports state-of-the-art performance on four benchmarks.

1. Introduction

  • OCC methods constrain normal actions to limited latent volumes, which can classify diverse-but-still-normal behaviors as anomalous.OCC is motivated by anomaly rarity and commonly learns from regular samples only.Source: §1 · pp. 1–2
  • MoCoDAD models normality and abnormality as multimodal and uses diffusion’s mode coverage to generate diverse future motions.Here, multimodal refers to distributions with multiple modes, and generated futures are statistically aggregated.Source: §1 · p. 2
  • The model conditions denoising of corrupted future joint coordinates on uncorrupted past poses; an autoencoder representation performed best among three tested strategies.The alternatives were direct pose concatenation and an encoding module.Source: §1 · p. 2
  • By avoiding appearance inputs, MoCoDAD offers increased privacy protection and computational efficiency through lightweight body-kinematic representations.The paper specifies that it uses no visual facial or body features.Source: §1 · p. 2
  • MoCoDAD outperforms the state of the art by 5.1%, 4.4%, .5%, and .8% on UBnormal, HR-UBnormal, HR-STC, and HR-Avenue, respectively.The reported order of gains matches the dataset order.Source: §1 · p. 2

2. Related Work

Related work spans video anomaly detection and diffusion models for motion synthesis. MoCoDAD differs by modeling multimodal skeletal futures conditioned on past motion rather than relying on fixed modes or unconditioned noise removal.

2. Related Work

  • The related-work discussion groups prior research into video anomaly detection and diffusion models for motion synthesis.These are the two perspectives used to organize the section.Source: §2 · p. 2

2.1. Video Anomaly Detection Techniques

  • Prior VAD methods use constrained latent regions, reconstruction proxies, or fixed-mode representations, while MoCoDAD addresses multimodality without choosing k in advance.The paper also situates MoCoDAD among skeleton-based approaches.Source: §2.1 · pp. 2–3

2.2. Diffusion Models

  • Existing diffusion motion-synthesis work conditions generation on text or action classes, or applies diffusion in a learned motion latent space; MoCoDAD conditions on past frames.The cited approaches include constrained and unconstrained synthesis.Source: §2.2 · p. 3
  • Unlike AnoDDPM’s unconditioned image-noise removal, MoCoDAD forecasts multimodal motions and compares them with the ground-truth future for pertinence.The paper presents MoCoDAD as the first to use diffusion’s multimodal generation and mode coverage for VAD.Source: §2.2 · p. 3

3. Methodology

MoCoDAD corrupts future skeletal poses and learns to denoise them conditioned on clean past poses. At inference, it generates multiple candidate futures and aggregates their reconstruction errors to detect anomalies.

3.1. Diffusion on Trajectories

  • For each motion sequence, MoCoDAD corrupts future joint coordinates with timestep-scaled random translations and trains a denoiser, conditioned on the clean past, to estimate the displacement noise.The denoising objective compares estimated and sampled noise; the training loss is smoothed.Source: §3.1 · p. 4
  • At inference, it generates multiple futures from random displacements, scores each against the actual future, and aggregates the scores into a sequence-level anomaly score.The score is assigned to corresponding frames, and scores are averaged for frames with multiple actors.Source: §3.1 · p. 4
  • The minimum reconstruction distance is the best average aggregation statistic; using diversity itself as the anomaly score performs nearly at random chance.Normal-conditioned futures are as diverse as abnormal-conditioned ones but more biased toward the actual motion, making close samples more likely.Source: §3.1 · pp. 4–5

3.2. Motion Conditioning for multimodal Pose Forecasting

  • Among the three past-motion conditioning choices—input concatenation, E2E-embedding, and AE-embedding—the AE-embedding performs best and exceeds state-of-the-art results.The AE-embedding adds an auxiliary reconstruction loss for the past; the encoded history and diffusion-timestep embedding are fed to the denoiser.Source: §3.2 · pp. 5–6

3.3. Architecture Description

  • The denoising network uses a U-Net-like architecture with space-time separable GCN layers, while an STS-GCN autoencoder embeds the past motion for conditioning.The U-Net contracts and expands the pose’s spatial dimension; the embedded history and timestep signal condition each layer.Source: §3.3 · p. 6

4. Experiments

MoCoDAD is evaluated on UBnormal and three human-filtered benchmarks, with training restricted to normal actions under an open-set split. It achieves leading results across the benchmarks and is competitive with supervised methods on UBnormal.

4.1. Datasets

  • UBnormal’s train, validation, and test sets contain disjoint anomaly types, and only normal actions are included in training.The evaluation also uses HR-UBnormal, HR-STC, and HR-Avenue; STC has 101 test clips with 130 anomalous events, while Avenue has 21 test videos with 47 anomalous events.Source: §4.1 · p. 6

4.2. Comparison with state-of-the-art

  • MoCoDAD records AUC scores of 77.6 on HR-STC, 89.0 on HR-Avenue, and 68.4 on HR-UBnormal, and reaches 68.3 on UBnormal, surpassing COSKAD by 5.1%.It outperforms the current best method on the human-related datasets by up to 4.4%.Source: §4.2 · p. 7
  • On UBnormal, MoCoDAD achieves an AUC of 68.3, exceeding weakly supervised method (59.3); despite lacking supervision or visual information, it is competitive with (68.5).Source: §4.2 · p. 7

5. Discussion

The discussion examines generated-motion diversity, score aggregation, conditioning choices, proxy tasks, and latent-space diffusion. It finds that normal and abnormal futures are similarly diverse, while their pertinence to the actual future differs.

5. Discussion

  • The discussion evaluates multimodal futures, aggregation of their scores, past-motion conditioning, alternative imputation tasks, and latent-space variants.It frames these analyses as tests of the model’s components and design choices.Source: §5 · p. 7

5.1. Multimodality

  • Normal- and abnormal-conditioned generations have similar variance, but the true future lies among a main mode for normal conditioning and at the edge of predictions for abnormal conditioning.The diversity ratio grows with more generated samples in both cases and remains comparable, with slightly greater growth for abnormal cases.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2. Statistical aggregations of generated motions

  • AUC improves with more generations for quantiles below 0.5, but declines for the mean and quantiles above 0.5; performance saturates beyond 50 generations.Normal-conditioned samples are centered around the true future and more likely to yield lower reconstruction errors than abnormal-conditioned samples.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3. Conditioning

  • AE-embedding gives the best conditioning results, reaching AUC scores of 68.3 on UBnormal and 68.4 on HR-UBnormal; E2E-embedding does not improve performance.The AE strategy uses an auxiliary reconstruction loss to supervise the past-motion encoder.Source: §5.3 · p. 8

5.4. Proxy Task

  • Forecasting is the best proxy task, with AUC scores of 68.3 on UBnormal and 68.4 on HR-UBnormal; random imputation is worst at 65.10 and 65.21, respectively.In-between imputation reaches 65.65 on UBnormal and 65.72 on HR-UBnormal.Source: §5.4 · p. 9

5.5. Latent space

  • Diffusion on latent space underperforms the original MoCoDAD on UBnormal and HR-UBnormal.The authors suspect latent diffusion is ineffective here because skeletons have lightweight encodings.Source: §5.5 · p. 9, Table 5 · p. 9

6. Conclusion

MoCoDAD models and exploits the diversity of normal and abnormal motions, using diffusion models for VAD without anomaly training labels.

6. Conclusion

  • MoCoDAD models multimodal future motions, sets a new state of the art among OCC techniques, and catches up with supervised methods without anomaly training labels.For normal sequences, forecast motions are diverse and pertinent to the observed sequence; for abnormal sequences, they do not show a bias toward the true future.Source: §6 · p. 9

Supplementary Materials

The supplement outlines additional analyses and implementation material, then reviews the forward corruption and reverse denoising processes used by diffusion models.

Supplementary Materials

  • The supplement adds diffusion-process ablations, conditioning analyses, framework pseudocode, implementation details, and qualitative illustrations.It includes generated distribution manifolds and examples of input poses under different corruption types.Source: Supplementary Materials · p. 13

A.1. Background on Diffusion Models

  • DDPMs gradually corrupt data toward a simple prior, then use a learned reverse process to generate samples by iterative denoising.The training objective optimizes a neural network to predict the noise added to the data.Source: §A.1 · p. 13

A.2. Diffusive steps

MoCoDAD performs best with 10 diffusion steps, balancing pose refinement for normal samples against limited refinement of abnormal ones.

A.2. Diffusive steps

  • 10 diffusion steps give optimal performance; higher or lower step counts deteriorate results.The evaluated settings were 2, 5, 10, 25, and 50 steps.Source: §A.2 · p. 13, Table 6 · p. 13
  • The pose-specific inductive bias can turn random joint positions into a pose-like structure in one step, while small T may fail to refine abnormal poses.The authors use T = 10 to balance improving normal poses against refining abnormal ones.Source: §A.2 · p. 13
  • The non-iterative T = 2 model underperforms MoCoDAD, supporting the value of multi-step denoising.The authors note that this step count supplies only clean or completely corrupted input poses.Source: §A.2 · p. 13

B. Weaker forms of conditioning

Weaker or absent conditioning and alternative corruption strategies underperform MoCoDAD; the supplementary analyses also describe its inference scoring and test whether diversity alone separates classes.

B. Weaker forms of conditioning

  • The alternative sampling strategy starts inference from a partially corrupted sample, weakening conditioning by removing the need to condition reverse diffusion on past frames.The experiments also evaluate MoCoDAD without past-motion conditioning.Source: B. Weaker forms of conditioning · p. 14
  • Partially corrupted inputs yield AUC-ROC results of 52 and 57.35, as they are more similar to the target and lower reconstruction error for both classes.The table compares noise distributions and sampling strategies; γ denotes the inference corruption step and T the complete-corruption step.Source: B. Weaker forms of conditioning · p. 14, Table 7 · p. 14
  • Simplex noise retains more input-motion information than Gaussian noise and yields performance of 52, while Gaussian displaces each joint with random intensity.The authors conclude that neither noise distribution matches MoCoDAD’s performance and that a conditioning signal is needed.Source: B. Weaker forms of conditioning · p. 14, Figure 7 · p. 15
  • Without past-motion conditioning, performance falls to 54.11, near the 50% chance level, because generated normal futures may be unrelated to the specific past.The model instead samples from its learned training normal distribution.Source: B. Weaker forms of conditioning · p. 14

C. MoCoDAD Algorithms

  • Inference draws multiple futures from random noise, compares them with the ground-truth future, and aggregates their scores into a subject-level anomaly score.A sliding-window procedure extends the single-window algorithm across the video.Source: C. MoCoDAD Algorithms · pp. 14–15
  • The frame-level anomaly score combines the mean of actors’ maximum errors with a log-scaled term measuring the error range.The logarithm prevents the range term from dominating the score.Source: C. MoCoDAD Algorithms · p. 15

D. Further notes on multimodality

  • Normal and abnormal classes have similar diversity, so diversity metrics alone cannot separate them; multimodality measures variance among motions generated from the same conditioning sequence.The supplementary analysis compares the rF and Multimodality metrics.Source: D. Further notes on multimodality · p. 15
  • Using a diversity metric as the anomaly score brings detection performance nearly to chance, and rF falls below chance for fewer than 10 generations.Figure 6 shows the performance trend under diversity-based scoring.Source: Figure 6 · p. 14, D. Further notes on multimodality · p. 15

E. Implementation details

MoCoDAD uses past motion to condition future-pose generation, with a six-frame window split between conditioning and diffusion. The implementation also specifies its inference scoring and network training configuration.

E. Implementation details

  • Inference generates multiple futures, scores each against the actual future with Lsmooth, aggregates generation scores, and imputes frame anomaly scores.Source: E. Implementation details · p. 16
  • Each six-frame motion window uses its first three frames for conditioning and the remaining three for diffusion.The same setup is used for the imputation proxy tasks.Source: E. Implementation details · pp. 15–16
  • Training uses Adam at a 1e−4 learning rate with exponential decay for 36 epochs, and a cosine variance schedule with T = 10.The diffusion endpoints are β1 = 1e−4 and βT = 2e−2; λ1 = λ2 = 1.Source: E. Implementation details · p. 16
  • The U-Net-GCN downsamples 17 joints to 10 and expands channels through (32, 32, 64, 64, 128, 64).Training took approximately 7 hours on an Nvidia Quadro P6000 GPU.Source: E. Implementation details · p. 16

F. Results on the UBnormal Validation set

On UBnormal validation, MoCoDAD outperforms the other OCC approaches and is second only to TimeSformer among the reported supervised and weakly supervised methods. The supplement also illustrates its iterative motion-generation procedure.

F. Results on the UBnormal Validation set

  • 77.6 AUC: MoCoDAD outperforms all other OCC approaches on the UBnormal validation set and is second only to TimeSformer among supervised and weakly supervised methods.The validation results are described as in line with the main-paper test-set results.Source: Table 8 · p. 16, F. Results on the UBnormal Validation… · p. 16

G. Generating motion sequences

  • Generation starts from random noise and iteratively predicts a clean sample before diffusing it back to the previous timestep, from T to 1.This illustrates the reverse procedure used to generate motion sequences.Source: G. Generating motion sequences · p. 16

H. Qualitative results

MoCoDAD’s generated futures are pertinent to normal-conditioned motion, while anomalous-conditioned futures can lie far from the generated distribution. The figures also illustrate how iterative sampling produces and refines those futures.

H. Qualitative results

  • Under normal conditioning, generated futures cluster near the true motion; under anomalous conditioning, the true future lies far from the generated distribution and predictions are poorer.Source: H. Qualitative results · pp. 16–17
  • Sampling begins from random noise and repeatedly predicts a clean pose before adding displacement to produce a less-corrupted pose for the preceding timestep.The process runs iteratively from T to 1 and refines the prediction for comparison with the actual future.Source: Figure 8 · p. 17
  • The diffusion process generates multiple plausible futures in both normal and anomalous contexts.Source: H. Qualitative results · p. 17, Figure 9 · p. 18

Related papers

Loading 2307.07205v3…