Source-linked AI summary
Graph Embedded Pose Clustering for Anomaly Detection
Amir Markovitz, Gilad Sharir, Itamar Friedman, Lihi Zelnik-Manor, Shai Avidan
TL;DR
Video anomaly detection needs to distinguish abnormal action variations as well as actions outside a selected normal set. The paper embeds and clusters human pose graphs, models their soft-assignment distributions with a Dirichlet process mixture, and reports state-of-the-art results on ShanghaiTech while outperforming existing unsupervised methods in its coarse-grained setting.
Problem
The paper addresses unified anomaly detection for fine-grained abnormal action variations and coarse-grained settings where selected actions are normal and others are abnormal.
Method
The method embeds and clusters temporal human pose graphs, represents actions by soft assignments to action-words, and models those proportional vectors with a Dirichlet process mixture.
Results
The method achieves state-of-the-art results on ShanghaiTech and outperforms existing unsupervised methods on the proposed coarse-grained anomaly detection test.
Takeaways & Limitations
The approach supports both fine-grained and coarse-grained anomaly detection through a unified pose-graph and soft-assignment representation.
Takeaways & Limitations
A classifier baseline is supervised and uses class-action labels unavailable to the other methods, so it is not directly comparable.
Abstract
from arXiv · showhide
We propose a new method for anomaly detection of human actions. Our method works directly on human pose graphs that can be computed from an input video sequence. This makes the analysis independent of nuisance parameters such as viewpoint or illumination. We map these graphs to a latent space and cluster them. Each action is then represented by its soft-assignment to each of the clusters. This gives a kind of "bag of words" representation to the data, where every action is represented by its similarity to a group of base action-words. Then, we use a Dirichlet process based mixture, that is useful for handling proportional data such as our soft-assignment vectors, to determine if an action is normal or not. We evaluate our method on two types of data sets. The first is a fine-grained anomaly detection data set (e.g. ShanghaiTech) where we wish to detect unusual variations of some action. The second is a coarse-grained anomaly detection data set (e.g., a Kinetics-based data set) where few actions are considered normal, and every other action should be considered abnormal. Extensive experiments on the benchmarks show that our method performs considerably better than other state of the art methods.
1. Introduction
The paper seeks a unified anomaly detector for fine-grained abnormal variations and coarse-grained action distinctions, using pose graphs and distributional modeling. It evaluates this approach on ShanghaiTech and two coarse-grained action-classification datasets, reporting strong benchmark performance.
- Problem: The paper aims to distinguish normal from abnormal actions using only an unlabeled training set of normal videos, across fine- and coarse-grained settings.Fine-grained detection targets abnormal variations of an action, while coarse-grained detection treats selected actions as normal and other actions as abnormal.
- Representation: Human pose graphs reduce dependence on viewpoint, illumination, and background clutter while providing a compact representation for faster analysis.The method assumes pose graphs extracted from video rather than raw visual features.
- Method: The method jointly embeds and soft-clusters temporal pose graphs, representing each action by its similarity to learned action-words.This produces a bag-of-words-like soft-assignment vector over clusters.
- Method: A Dirichlet process mixture models the distribution of soft-assignment vectors and produces a normality score for classifying actions.The model is chosen because the soft-assignment vectors capture proportional data.
- Evaluation: The evaluation covers fine-grained ShanghaiTech detection and a proposed coarse-grained setting using NTU-RGB+D and 250 Kinetics400 actions.The coarse-grained experiments train on subsets of normal actions and test discrimination against the remaining videos.
- Results: The paper reports state-of-the-art AUC of 0.761 on ShanghaiTech and presents embedded pose graphs with a Dirichlet process mixture as key contributions.It also introduces the coarse-grained anomaly detection setting.
2. Background
The background situates the work among reconstructive, predictive, generative, pose-based, graph-learning, and deep-clustering approaches to video anomaly detection. It also identifies practical costs and methodological distinctions in prior methods.
- Video anomaly detection: Video anomaly detection methods include reconstructive, predictive, and generative deep-learning models.Reconstructive methods learn embeddings and reconstruct samples; predictive methods model current frames from prior frames; generative methods model or reconstruct data distributions.
- Prior limitations: Some prior approaches combine future-frame prediction with gradient-based and optical-flow differences, but require optical-flow computation and complete-scene generation.The cited method is described as costly and less robust to large scenery changes.
- Pose-based methods: Pose-based anomaly detection can use recurrent models with reconstruction and prediction branches to score anomalies from past- and future-pose errors.This represents an alternative to the paper’s graph-based approach.
- Graph learning: Graph representations describe inner-pose relations with weighted adjacency matrices, which may be static or learnable.Graph-convolution research extended these representations with temporal, multiple-adjacency, attention-based, and inferred-adjacency mechanisms.
- Deep clustering: Deep clustering jointly learns representations and cluster assignments, with DEC alternating target-distribution construction and assignment optimization.Later extensions address DEC’s susceptibility to degenerate solutions through regularization and post-processing.
3. Method
The method converts temporal human pose graphs into clustered latent representations, then models their soft-assignment distributions to detect abnormal actions. Its graph encoder combines multiple adjacency types with temporal processing, while clustering is jointly refined with reconstruction.
- Normality modeling: Each action becomes a soft-assignment vector over latent clusters, and a Dirichlet process mixture models these proportions to score normality.The fitted mixture contains modes corresponding to normal behaviors, and test samples are scored by log probability.
- Pose-graph representation: Human pose sequences are represented as temporal graphs whose nodes are body keypoints and whose edges encode spatial and temporal joint relations.Temporal adjacency connects joints across successive frames, enabling graph convolutions over both spatial and temporal dimensions.
- Pose-graph embedding: ST-GCAE embeds temporal pose graphs using a spatio-temporal graph autoencoder based on sequential spatial graph and temporal convolutions.The encoder preserves graph structure while compressing the input sequence into a latent vector used by the clustering branch.
- Spatial attention graph convolution: The spatial attention operator combines static, globally learned, and sample-specific inferred adjacency matrices through separate GCNs and learnable output weighting.These adjacency types encode body-part connectivity, dataset-level relations, and sample-specific relations, respectively.
- Joint embedding and clustering: Training jointly reconstructs inputs and improves clustering, with the autoencoder embedding subsequently fine-tuned during clustering optimization.The clustering branch uses an encoder, decoder, and soft clustering layer, initialized from the encoded training data.
4. Experiments
Experiments evaluate the method in fine-grained ShanghaiTech anomaly detection and a proposed coarse-grained setting using NTU-RGB+D and Kinetics-250. The method outperforms unsupervised competitors across these settings, while pose quality and abnormal training noise affect performance.
- 4.1. ShanghaiTech Campus: Fine-grained evaluation uses ShanghaiTech, where normal behavior is a single class and the goal is detecting abnormal variations.The experimental protocol trains on normal examples only and reports frame-level AUC over concatenated test-frame scores.
- 4.1. ShanghaiTech Campus: The proposed model outperforms previous state-of-the-art methods on ShanghaiTech using both pose and patch-based inputs.Table 1 reports frame-level AUC, with Pose denoting keypoint coordinates and Patch denoting patch embeddings.
- 4.2. Coarse-Grained Anomaly Detection: The coarse-grained setting trains without labels on a few normal action classes and tests whether unseen clips belong to those normal actions.Experiments use 3–5 selected classes, with random and subjectively meaningful splits, and evaluate Few vs. Many and Many vs. Few configurations.
- 4.2.4. Evaluation: The algorithm outperforms all four competing unsupervised methods across random and meaningful splits and both Few vs. Many and Many vs. Few settings.Dirichlet-based scoring performs better than max-softmax scoring for most cases and is used for the reported results.
- 4.2.1. Experimental Setting: The coarse-grained experiments use NTU-RGB+D and 250 Kinetics400 actions, with subsets defining normal training actions and the remaining videos used for testing.The datasets differ in pose quality because NTU-RGB+D uses Kinect depth sensing, whereas Kinetics-250 relies on estimated poses.
- 4.4. Ablation Study: Performance is robust to 5% abnormal training noise, losing on average less than 10%, but declines considerably with 20% noise.At 20%, abnormal samples may exceed the amount of data in an individual underlying class.
5. Conclusion
The paper presents a pose-graph anomaly detector that jointly embeds and clusters actions, then uses Dirichlet-process modeling of soft assignments for normality scoring. It handles both fine- and coarse-grained detection and reports state-of-the-art or improved results across the evaluated settings.
- The method represents human actions as temporal pose graphs, jointly embeds and clusters them, and models their soft-assignment vectors with a Dirichlet Process Mixture Model.
- The algorithm supports fine-grained detection of variations within one action and coarse-grained detection separating designated normal actions from other actions.
- The method achieves state-of-the-art results on ShanghaiTech and outperforms existing unsupervised methods on the proposed coarse-grained anomaly detection test.
A. Supplementary Material
The supplementary material extends the paper with ablations, visualizations, operator and implementation details, and complete coarse-grained experiment information.
- The supplement provides additional ablation experiments, experiment splits and results, spatial attention graph convolution details, and implementations of the method and baselines.
- Section B evaluates the model through further ablations, while Section C presents learned base action-words for both fine- and coarse-grained settings.
- Sections D–F detail the spatial attention graph convolution, the proposed method’s implementation, and baseline implementations.
- Sections G–I provide coarse-grained per-split results, class lists, and the complete Kinetics-250 class list.
B. Ablation Experiments - Cont.
The ablations examine input representations, spatial graph operators, clustering choices, initialization, and fine-tuning. They report meaningful gains from patches and spatial attention, frequent advantages for K=20 and Dirichlet scoring, and considerable improvements from fine-tuning.
- Input and Spatial Convolution: The ablations compare Pose coordinates with Patch CNN features and evaluate GCN against the proposed spatial attention graph convolution.
- Input and Spatial Convolution: Both patch inputs and the spatial attention graph convolution play a key role in the reported results.
- Clustering Components: The clustering ablations vary the number of clusters, initialization, normality score, and fine-tuning stage.
- Clustering Components: K = 20 usually achieves the best results in the coarse setting, and the Dirichlet-based normality score consistently outperforms the maximum-softmax score except for one reported case.
- Clustering Components: K-means initialization improves results compared with random initialization, while fine-tuning improves results considerably except when K = 5.
C. Visualization of Action-words
The action-word visualizations show that learned cluster neighborhoods reflect the structure of each anomaly-detection setting: viewpoint-related variations for fine-grained walking and broader action variation for coarse-grained data.
- Fine-grained: In the fine-grained setting, samples near cluster centroids capture common variations of the normal walking action, mainly differing by walking direction or viewpoint.
- Coarse-grained: In the coarse-grained setting, centroid-near samples vary considerably and depict essential manifestations of the underlying action classes.
- Fine-grained: The fine-grained action-word dictionary contains clear, unoccluded, full-body samples from normal actions.
- The graph model processes spatial pose structure independently across frames before temporal modeling.
E. Implementation Details
The implementation extracts 2D or 3D pose-based inputs, supports patch features, and uses an augmented symmetric clustering architecture with dataset-specific cluster counts.
- Pose graphs are extracted with AlphaPose and PoseFlow for ShanghaiTech, OpenPose keypoints for Kinetics-250, and Kinect annotations for NTU-RGB+D.ShanghaiTech and Kinetics-250 use 2D keypoints with confidence values, while NTU-RGB+D provides 3D annotations with 25 keypoints per person.
- Patch Inputs: Patch-based ShanghaiTech inputs crop a 16 × 16 image patch around each keypoint and embed it with a public ResNet model.Intermediate detector features can also provide patch embeddings without added computation beyond increased input dimensionality.
- Architecture: ST-GCAE uses a symmetric encoder-decoder with temporal downsampling factors of 2 and 3, random rotation and flip augmentation, and augmented evaluation averaging.The model uses K = 20 clusters for NTU-RGB+D and Kinetics-250, and K = 10 for ShanghaiTech.
- Training: Training consists of pre-training without clustering loss followed by similarly long fine-tuning with combined loss and λ = 0.5.The Adam optimizer is used for all experiments.
F. Baseline Implementation Details
Baseline evaluations use public implementations for video anomaly methods, a supervised ST-GCN classifier, and Dirichlet-based scoring of per-class assignment vectors.
- Video anomaly detection methods: Future Frame Prediction and Trajectory-Based Anomaly Detection baselines use the authors’ public implementations with default parameters adapted to the datasets.The trajectory method is evaluated only where 2D pose is available.
- Classifier softmax scores: The supervised baseline uses a classifier based on Yan et al.’s ST-GCN architecture, with a smaller six-block model for Few vs. Many experiments.The six blocks use channel widths of 64, 64, 128, 128, 256, and 256.
- Classifier softmax scores: Per-class assignment probabilities are scored by a Dirichlet-based normality model fitted on training samples considered normal.Each test sample receives a score using the fitted parameters.
G. Detailed Experiment Results
The experiments report AUC and compare the proposed model with pose-based, pixel-based, unsupervised, and supervised baselines across datasets and settings.
- The detailed results cover NTU-RGB+D and Kinetics-250, while the figures also identify the supervised classifier baseline as fundamentally label-supervised.
- Across all settings, the method is the top performer in most splits relative to unsupervised methods, often by a large margin.
H. Class Splits Table
The coarse-grained evaluation uses random and subjectively meaningful action splits for NTU-RGB+D and Kinetics-250, with AUC tables and a Kinetics-250 class-selection cutoff.
- Class Splits Table: NTU-RGB+D and Kinetics-250 split lists are provided in Tables 8 and 9, respectively.
- Class Splits Table: Random splits test performance on specific subsets of unrelated actions, whereas meaningful splits group actions by shared physical or environmental properties.Examples include musicians playing and actions likely to occur in a gym.
- Coarse-Grained Experiment Results: The NTU-RGB+D coarse-grained results table reports area under the curve and identifies the best unsupervised and overall methods.The table also includes the supervised and Future Frame Prediction baselines.
- Class Selection: Kinetics-250 classes are selected using a cutoff on sorted Kinetics-400 top-1 training accuracy from Yan et al.’s classifier.Figure 7 presents the class rankings and accuracy values used to show this cutoff.
- Coarse-Grained Experiment Results: The Kinetics-250 results table reports AUC and includes supervised, Future Frame Prediction, and Trajectory-Based Anomaly Detection baselines.