Source-linked AI summary
Part-based Graph Convolutional Network for Action Recognition
Kalpit Thakkar, P J Narayanan
TL;DR
Skeletal action recognition commonly models the whole skeleton as one graph, while actions also involve motion of distinct body parts and object interactions can remain ambiguous. This paper introduces a part-based graph convolutional network with task-relevant motion features, achieving state-of-the-art performance on NTURGB+D and HDM05.
Problem
Whole-skeleton graph models do not explicitly represent body-part structure, and skeletal data alone may not distinguish actions with similar part dynamics but different object interactions.
Method
The paper partitions the skeleton into body-part subgraphs with shared joints and uses relative coordinates and temporal displacements as features in a part-based graph convolutional network.
Results
The model achieves state-of-the-art skeletal action-recognition performance on NTURGB+D and HDM05, outperforming the prior state-of-the-art graph-based method by approximately 6% and 5% under two NTURGB+D protocols.
Takeaways & Limitations
Partitioning the skeleton graph and providing relative-coordinate and temporal-displacement cues improves recognition compared with treating the entire skeleton as one graph.
Takeaways & Limitations
Skeletal data alone may be insufficient when different actions have similar body-part dynamics but differ in their interactions with objects; RGB information may help disambiguate them.
Abstract
from arXiv · showhide
Human actions comprise of joint motion of articulated body parts or `gestures'. Human skeleton is intuitively represented as a sparse graph with joints as nodes and natural connections between them as edges. Graph convolutional networks have been used to recognize actions from skeletal videos. We introduce a part-based graph convolutional network (PB-GCN) for this task, inspired by Deformable Part-based Models (DPMs). We divide the skeleton graph into four subgraphs with joints shared across them and learn a recognition model using a part-based graph convolutional network. We show that such a model improves performance of recognition, compared to a model using entire skeleton graph. Instead of using 3D joint coordinates as node features, we show that using relative coordinates and temporal displacements boosts performance. Our model achieves state-of-the-art performance on two challenging benchmark datasets NTURGB+D and HDM05, for skeletal action recognition.
1 Introduction
The paper proposes part-based graph convolution for skeletal action recognition because whole-skeleton graphs overlook body-part structure. It also replaces raw 3D joint coordinates with relative and temporal-motion features.
- Graph convolutions model skeletal videos, but treating the whole skeleton as one graph overlooks the structure and relations of body parts.
- Figure 1 organizes the representation from geometric and kinematic features to two-, four-, and six-part skeleton decompositions.
- The proposed model represents the skeleton using geometric relative coordinates and kinematic temporal displacements instead of raw 3D joint locations.
- The paper contributes a general PB-GCN, task-relevant geometric and motion features, and state-of-the-art results on NTURGB+D and HDM05.
2 Related Work
Prior skeletal action-recognition methods include handcrafted features, LSTMs, manifold techniques, and graph convolutions. This paper combines part-based representation with graph convolution to model spatio-temporal structure while using task-relevant coordinate features.
- Handcrafted methods use joint angles, distances, and kinematic features but have limited capacity for modeling spatio-temporal action properties.
- LSTM approaches model skeletal videos as sequences, but they require either carefully trained complex architectures or simpler part-based representations.
- Manifold methods represent actions on Lie groups or Riemannian manifolds, whereas graph deep learning offers a more direct framework for human skeleton graphs.
- Earlier graph-based methods treat the human skeleton as a single graph, while this work explores partitioned skeleton graphs with part-based convolutions.
- Relative coordinates are presented as geometric features that improve action-recognition performance.
3 Background
The paper defines graph and part-based graph representations, then extends graph convolution to subgraphs whose information is aggregated across parts. This framework is used to learn rich representations from spatial and temporal skeleton neighborhoods.
- Graph Background: A graph is represented as vertices and edges, with adjacency, neighborhood, and labeling functions supporting graph convolution.
- Part-based Graph: A part-based graph divides a graph into subgraphs that may share vertices or connect through cross-part edges.
- Spatio-temporal Convolution: The model uses spatial and temporal neighborhoods to extend graph convolution over spatio-temporal skeleton graphs.
- Part-based Graph Convolutions: Part-based convolutions capture properties within each subgraph and aggregate information across subgraphs to learn relations between parts.
4 Spatio-temporal Part-based Graph Convolutions
PB-GCN represents skeletal videos as spatio-temporal graphs partitioned into body-part subgraphs, using labeled spatial and temporal neighborhoods for convolution. The architecture aggregates part-specific spatial features and then applies temporal convolution to produce video-level class scores.
- Spatio-temporal graph construction: Skeletal videos are modeled as spatio-temporal graphs by connecting corresponding joints across time within each body-part subgraph.The model uses four final parts—head, hands, torso, and legs—with some joints shared across parts.
- Neighborhoods and labeling: Each vertex uses a 1-neighborhood spatially and a τ-neighborhood temporally, with spatial and temporal labels organizing the receptive fields.Spatial neighbors are weighted equally, while temporal labels distinguish vertices across frames.
- Convolution: The convolution first aggregates each part’s spatial graph features and then applies temporal convolution across τ frames.Part-specific channel kernels process input features, while a shared temporal kernel operates on the aggregated spatial output.
- Fusion: A weighted-sum fusion combines outputs from the partition graphs.The fused representation is used within the spatio-temporal graph convolution pipeline.
- Architecture: The network represents each subgraph with a degree-normalized adjacency matrix and outputs class scores for the input video.Its architecture contains nine spatio-temporal graph convolution units with part-specific kernels, a temporal kernel, and residual connections.
5 Geometric & Kinematic Signals
The model replaces absolute joint locations with relative coordinates and temporal displacements as geometric and kinematic signals. Experiments compare these signals, their concatenation, and different skeleton partitions using PB-GCN on NTURGB+D.
- Signal design: PB-GCN combines relative joint coordinates with temporal displacements instead of using absolute 3D joint locations directly.Relative coordinates are computed with respect to shoulders and hips, and the combined representation provides translation invariance.
- Experimental comparison: Table 1 compares absolute joint locations, relative coordinates, temporal displacements, and their concatenation across PB-GCN partition schemes.The table reports Cross Subject and Cross View performance on NTURGB+D.
- Results: The geometric and kinematic signals improve performance for both one-part and four-part PB-GCN models.The study evaluates the individual signals and their combination, reporting noteworthy improvement from the geometric and kinematic signals.
6 Experimental Setup and Results
The experiments evaluate PB-GCN on NTURGB+D and HDM05, testing part granularity, graph-based comparisons, and geometric and kinematic input signals. Four-part modeling and explicit relative-coordinate and temporal-displacement cues improve recognition, while PB-GCN surpasses prior graph-based methods on both datasets.
- Datasets: PB-GCN is evaluated on NTURGB+D and HDM05, using the datasets' established action-recognition protocols.NTURGB+D contains 56,880 sequences and 60 classes, while HDM05 contains 2,337 sequences spanning 130 motion classes.
- Part-based model: Four-part partitioning outperforms one- and two-part schemes, whereas splitting upper and lower skeletons by laterality does not improve performance.The final model therefore uses four parts and remains agnostic to laterality.
- Graph-based comparisons: PB-GCN outperforms prior graph-based models on both datasets, although its comparison with Li et al. may partly reflect a deeper network.The model uses nine layers versus two in the cited Li et al. model.
- Geometric and kinematic signals: Temporal displacements provide explicit motion information, and the final configuration concatenates geometric and kinematic signals.The displacement signal is motivated as analogous to optical flow or sparse 3D scene flow.
- Geometric and kinematic signals: Relative coordinates improve both global and four-part models by providing translation-invariant geometric features.The improvement is observed for the one-part and four-part configurations in the ablation results.
- State-of-the-art comparison: PB-GCN exceeds prior state of the art by ~6% and ~5% on NTURGB+D's two protocols and by ~3% mean accuracy on HDM05.The cited comparisons are against STGCN on NTURGB+D and Deep STGC on HDM05.
7 Conclusion
The paper formalizes spatio-temporal convolutions over partitioned skeleton graphs through PB-GCN and combines this representation with explicit geometric and motion cues. It reports state-of-the-art performance on two challenging action-recognition datasets and identifies other tasks as future applications.
- Conclusion: PB-GCN partitions the skeleton graph to model relations between body parts and their importance in human actions.The approach is contrasted with treating the entire body as a single graph.
- Conclusion: Relative coordinates and temporal displacements provide explicit geometric and motion cues for skeletal action recognition.These signals are presented as beneficial task-specific inputs to the convolutional model.
- Conclusion: The model achieves state-of-the-art performance on two challenging action-recognition datasets.The conclusion summarizes the reported results across the paper's two benchmark datasets.
- Future work: The authors propose exploring part-based graph models for tasks including object detection and image similarity.These applications are identified as future work rather than evaluated contributions.
arXiv:1809.04983v1 [cs.CV] 13 Sep 2018
The supplementary material presents additional quantitative analysis of action-recognition results, including confusion matrices and observations about model behavior. It also discusses the importance of geometric and kinematic features.
- Supplementary analysis: The supplementary material analyzes confusion matrices for different action-recognition models.The analysis is used to examine useful model properties.
- Supplementary analysis: Graph-based models handle actions with more motion better than actions with little skeleton motion and object interactions.This observation comes from the supplementary quantitative analysis.
- Supplementary analysis: The supplementary material further examines the importance of geometric and kinematic features.The supplied passage introduces this analysis without reporting its detailed findings.
1 Quantitative Analysis
The experiments compare confusion patterns across one-part, four-part, and Yan’s graph models, while examining recurring ambiguities and the effects of part-based structure and input signals. Skeleton-only models remain limited when actions differ mainly through object interactions.
- Experimental setup: The confusion matrices compare the part-based model, a one-part graph model, and Yan’s graph model under the cross-subject protocol.The cross-subject protocol is treated as more challenging than cross-view evaluation.
- Commonly confused classes: Actions such as reading and writing are consistently confused because they involve little skeleton motion and differ mainly through object interactions.Skeleton-based models lack direct object information for distinguishing these classes.
- Limitations: Skeleton-only recognition is insufficient when different actions have similar body dynamics but different interactions with objects.RGB information is identified as a way to provide object cues and disambiguate such interactions.
- Model-1 vs Model-2: Model-2 improves over Model-1 on actions involving correlated movements of both hands and legs, including brushing teeth, cheer up, and making or answering a phone call.The part-based representation learns from body parts specifically rather than treating the entire skeleton as one part.
- Model-3 vs Model-2: Adding relative coordinates and temporal displacements to Yan’s graph model boosts recognition performance on NTURGB+D.The experiment indicates that these geometric and kinematic signals improve recognition for graph-based skeleton models.
2 Conclusion
The conclusion favors partitioning the skeleton into parts and using relative coordinates with temporal displacements, while recognizing that skeletal data alone cannot resolve all object-related ambiguities.
- Conclusion: Part-based modeling works better than treating the skeleton graph as an unpartitioned whole.Relative coordinates and temporal displacements can substantially improve skeletal action recognition, while RGB can help distinguish object interactions.