Source-linked AI summary
Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos
Romero Morais, Vuong Le, Truyen Tran, Budhaditya Saha, Moussa Mansour, Svetha Venkatesh
TL;DR
Video anomaly detection needs representations that avoid the noise and semantic entanglement of appearance features while capturing human behavioral irregularity. The paper decomposes skeleton motion into global movement and local posture and jointly models them with MPED-RNN. Experiments report competitive or equal-or-better detection performance, interpretability, and identifiable error modes, with accuracy dependent on skeleton quality and limited when abnormal and normal motions look geometrically similar.
Problem
Appearance features used for video anomaly detection are complex, entangled, noise-sensitive, and semantically distant from event meaning.
Method
The method decomposes dynamic skeleton motion into global body movement and local body posture, then jointly models both with an interacting Message-Passing Encoder-Decoder Recurrent Network.
Results
Skeleton features with MPED-RNN achieve competitive detection performance and are reported to provide highly interpretable anomaly decisions.
Takeaways & Limitations
Compact semantic skeleton features can support human-related anomaly detection while enabling open-box examination of contributing factors and failure modes.
Takeaways & Limitations
Performance depends on skeleton detection and tracking quality, and geometrically similar abnormal and normal motions can produce false negatives when appearance is excluded.
Abstract
from arXiv · showhide
Appearance features have been widely used in video anomaly detection even though they contain complex entangled factors. We propose a new method to model the normal patterns of human movements in surveillance video for anomaly detection using dynamic skeleton features. We decompose the skeletal movements into two sub-components: global body movement and local body posture. We model the dynamics and interaction of the coupled features in our novel Message-Passing Encoder-Decoder Recurrent Network. We observed that the decoupled features collaboratively interact in our spatio-temporal model to accurately identify human-related irregular events from surveillance video sequences. Compared to traditional appearance-based models, our method achieves superior outlier detection performance. Our model also offers "open-box" examination and decision explanation made possible by the semantically understandable features and a network architecture supporting interpretability.
1. Introduction
The paper addresses unsupervised video anomaly detection using compact, semantically rich skeleton features instead of noisy appearance representations. It factorizes human motion into global movement and local posture, then jointly models them with MPED-RNN for interpretable anomaly detection.
- Motivation: Video anomaly detection learns regular patterns from normal videos to identify irregular test events without human supervision.The task remains difficult because human-perceivable abnormality is ambiguous.
- Motivation: Pixel-based features are high-dimensional, unstructured, noise-sensitive signals whose redundancy burdens models and can mask scene information.Appearance features may also create a semantic gap that limits interpretability and error analysis.
- Skeleton Representation: Skeleton trajectories provide compact, structured, semantically rich descriptions of human action and movement in surveillance videos.They explicitly exploit scenes containing humans and attached objects moving against static backgrounds.
- Skeleton Representation: Human behavioral irregularity is factorized into global body movement and local body posture, with the latter expressed after global movement is removed.The decomposition captures factors such as location, velocity, direction, pose, and action.
- Proposed Method: MPED-RNN uses two interacting recurrent branches to model the global and local components and supports open-box explanations through factor contribution weights and visualizations.The model is trained end-to-end to distill compact normal patterns and detect abnormal events.
2. Related Work
Prior work progressed from one-class models and appearance-based CNN features toward structured representations and recurrent human-motion modeling. This paper extends that direction with skeleton-based, factorized, interpretable anomaly detection.
- Video Anomaly Detection: Traditional video anomaly methods model frames with one-class SVM or probabilistic PCA, but often perform suboptimally on large datasets with diverse anomaly types.
- Video Anomaly Detection: Recent methods use CNNs, autoencoders, 3D convolutional networks, and recurrent coding to model high-level frame-intensity features.Optical-flow prediction can filter some appearance noise, but optical flow is costly and remains semantically distant from events.
- Structured Representations: Structured representations use object trajectories, object attributes, and action labels to focus detection or improve understanding of abnormality scores.
- Structured Representations: The paper positions skeleton features as low-dimensional, semantic-rich inputs that support explanations in a factorized semantic space.
- Human Trajectory Modeling: Earlier recurrent human-motion approaches model whole-body xy-location sequences or joint locations, while this work addresses their separation of global movement and local posture.
- Human Trajectory Modeling: Du et al. divide skeleton joints into five parts and jointly model them with a five-branch bidirectional recurrent network.
3. Method
The method decomposes skeleton trajectories into global body movement and local posture, then models their individual dynamics and cross-component interactions with MPED-RNN. It trains on fixed-length trajectory segments using reconstruction and future prediction losses in multiple coordinate frames.
- Input representation: Skeleton trajectories are assumed to be extracted first, then represented as temporal sequences of joint locations for anomaly detection.The method operates on fixed-size trajectory segments obtained with a sliding window.
- Skeleton motion decomposition: The decomposition separates whole-body movement from posture because skeleton scale and activity context can change their relative contributions in surveillance videos.Near-field motion may be dominated by local factors, whereas far-field motion is more influenced by global movement.
- Skeleton motion decomposition: Global features describe bounding-box shape, size, and rigid movement, while local features describe internal skeleton deformation relative to the box.The local component ignores absolute position, and bounding-box width and height normalize local features.
- MPED-RNN architecture: The model captures both component-specific dynamics and their contextual relationship, since breaking normal cross-component correlation can indicate abnormality.The two processes may occur independently in generic motion but are strongly correlated in a specific context.
- MPED-RNN architecture: MPED-RNN uses separate recurrent encoder-decoder branches for global and local components that exchange cross-branch messages at each time step.Each branch contains an encoder, reconstructing decoder, and predicting decoder; projected branch outputs are combined to generate perceptual features.
- Training objective: Training combines perceptual, global, and local losses, with reconstructing and predicting decoders jointly learning a compact representation of normal trajectories.During testing, abnormal patterns receive poor prediction because they were not observed or do not follow normal dynamics.
4. Experiments
Experiments evaluate MPED-RNN on human-related surveillance anomalies across ShanghaiTech and CUHK Avenue, comparing performance, interpretability, ablations, and error modes. MPED-RNN outperforms compared methods while remaining vulnerable to skeleton-input errors and visually similar abnormal actions.
- Comparison with Appearance-based Methods: MPED-RNN achieves the highest frame-level ROC AUC on both HR-ShanghaiTech and the original ShanghaiTech dataset.The comparison trains MPED-RNN on all training videos and evaluates both the human-related subset and the full dataset.
- Comparison with Appearance-based Methods: Skeleton-focused anomaly maps avoid many irrelevant background aspects that affect whole-scene appearance-based methods.MPED-RNN assigns scores to skeletons, whereas Conv-AE and Liu et al. predict the whole scene and are more susceptible to scene noise.
- Open-box Interpretation: In normal sequences, predicted global, local, and image-space features closely follow inputs, whereas running produces a lagging predicted global bounding box.The visualization contrasts a normal walking-area sequence with an anomalous sequence containing a running person.
- Ablation Study: Ablations show that recurrent modeling, message passing between global and local processes, and dual decoders each contribute to anomaly detection.The study reports that RNNs are needed, cross-process message passing is necessary when both feature components are modeled, and dual decoders help regularization and detection.
- Error Mode Analysis: Major errors arise from inaccurate skeleton detection or tracking, while similar skeleton patterns can cause false negatives for abnormal actions.Reported cases include reflections, low resolution, lighting, shadows, occlusions, crossing people, and bicycle riding resembling walking.
- CUHK Avenue Dataset: On HR-Avenue, MPED-RNN obtains a frame-level ROC AUC of 0.863, versus 0.862 for Liu et al. and 0.848 for Conv-AE.HR-Avenue excludes segments with non-human or pose-detector-undetectable anomalies.
5. Discussion
Skeleton features remain compact while matching or exceeding state-of-the-art performance, but MPED-RNN depends on reliable skeleton detection and tracking and is limited for interaction anomalies.
- Less than 100 dimensions per frame still provide equal or better performance than current state-of-the-art anomaly-detection methods.ResNet and AlexNet fc7 features use 2048 and 4096 dimensions, respectively.
- MPED-RNN performance depends on skeleton detection and tracking quality, especially in low-quality videos.This dependence prevented evaluation on UCSD Ped1/Ped2 because the videos were too low quality for skeleton detection.
- Appearance features can complement skeleton features when skeletons are unavailable, motivating cascaded models that cover each representation’s weaknesses.The message-passing scheme can also incorporate non-skeleton feature subprocesses.
- Single-person movement and posture omit interactions among people and between humans and objects, limiting coverage of multi-entity anomalies.The framework is described as extensible through interentity messages and part-based object configurations.
6. Conclusions
Experiments indicate that skeleton motion sequences can identify human-related video anomalies, with global-local decomposition and message passing separating anomalous from normal sequences.
- Skeleton motion sequences are effective for identifying human-related video anomalous events.
- Decomposing skeleton sequences into global movement and local deformation, combined with message-passing encoder-decoder RNNs, separates anomalous from normal sequences.
- MPED-RNN is described as simple, competitive in performance, and highly interpretable.
- Future work targets inter-human interactions, combining skeleton and appearance features, and extending the component-based model to non-human objects.
A. HR-ShanghaiTech and HR-Avenue Datasets
The HR-ShanghaiTech dataset removes six test videos containing non-human anomalies, while HR-Avenue ignores segments with undetectable or non-human anomalies.
- Six of ShanghaiTech’s 107 test videos contain anomalies unrelated to humans.
- HR-ShanghaiTech was created by removing those six videos from ShanghaiTech.
- HR-Avenue ignores video segments where the pose detector could not detect anomalies or where anomalies were unrelated to humans.
- The ignored HR-Avenue segments are Video 02 frames 272–319 and 723–763, Video 03 frames 293–340, and Video 06 frames 561–624 and 814–1006.