Source-linked AI summary
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Sijie Yan, Yuanjun Xiong, Dahua Lin
TL;DR
Skeleton-based action recognition lacks a principled way to learn spatial joint configurations and temporal dynamics without hand-crafted parts or traversal rules. ST-GCN models skeleton sequences as spatial-temporal graphs and outperforms previous skeleton-based methods on Kinetics and NTU-RGBD.
Problem
Existing skeleton models rely on hand-crafted parts or traversal rules, limiting automatic learning of spatial configurations and temporal dynamics.
Method
ST-GCN represents joints as nodes with spatial and temporal edges, then applies layered graph convolutions to integrate information across both dimensions.
Results
ST-GCN outperforms previous state-of-the-art skeleton-based methods on two challenging large-scale action-recognition datasets.
Takeaways & Limitations
ST-GCN provides a generic skeleton representation that avoids manual part assignment or traversal rules and can capture motion information complementary to RGB.
Takeaways & Limitations
On Kinetics, skeleton-based methods remain inferior to video frame-based models.
Abstract
from arXiv · showhide
Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.
1 Introduction
The paper introduces ST-GCN, a spatial-temporal graph model that represents skeleton sequences through connected joints across space and time. It removes hand-crafted structural rules by learning spatial configurations and temporal dynamics with graph convolutions, aiming for stronger expressive power and generalization.
- Motivation: The method addresses prior approaches that depend on hand-crafted body parts or traversal rules, which are difficult to generalize across applications.Earlier methods also analyzed joint coordinates as feature vectors over time, limiting their ability to exploit joint connectivity.
- Method: ST-GCN extends graph neural networks to dynamic human-skeleton graphs, a setting that prior work with fixed input graphs had not explored on large-scale datasets.The model is formulated over a sequence of skeleton graphs rather than a single fixed graph.
- Proposed representation: ST-GCN represents a skeleton sequence as a graph whose nodes are body joints, with spatial edges following natural joint connectivity and inter-frame edges linking the same joints across consecutive frames.Joint coordinates serve as the model inputs.
- Method: Its hierarchical formulation eliminates hand-crafted part assignments and traversal rules while supporting greater expressive power and easier generalization to different contexts.The paper also studies graph-convolution kernel-design strategies inspired by image models.
- Contributions: The paper claims three contributions: a generic graph-based dynamic-skeleton formulation, convolution-kernel design principles for skeleton modeling, and superior performance on two large-scale action-recognition datasets.The formulation is presented as the first graph-based neural-network approach for this task.
2 Related work
Prior work extends neural networks to graph-structured data through recurrent and convolutional architectures. Skeleton-based action recognition methods use robust, accessible skeletal information and include handcrafted-feature and deep-learning approaches.
- Neural Networks on Graphs: Graph neural-network research includes recurrent and convolutional architectures for graph-structured data.The cited convolutional approaches include Bruna et al. (2014), Henaff, Bruna, and LeCun (2015), Duvenaud et al. (2015), Li et al. (2016), and Defferrard, Bresson, and Vandergheynst (2016).
- Neural Networks on Graphs: One graph-CNN approach constructs spatial-domain filters by limiting each filter’s application to the 1-neighbor of every node.
- Skeleton Based Action Recognition: Skeleton-based action-recognition methods exploit skeleton and joint trajectories because they resist illumination and scene variation and are readily obtained from depth sensors or pose estimation.These approaches are categorized into handcrafted-feature-based and deep-learning methods.
3 Spatial Temporal Graph ConvNet
ST-GCN represents skeleton sequences as spatial-temporal graphs and applies graph convolutions to learn hierarchical spatial and temporal patterns. The graph connects anatomically related joints within frames and corresponding joints across consecutive frames, using one-hop neighborhoods for convolution.
- Network architecture: The resulting multiple spatial-temporal graph convolution layers progressively generate higher-level graph feature maps before standard Softmax classification.The architecture applies several convolutional layers to the constructed skeleton graph and then predicts the corresponding action category.
- Graph construction: The graph includes intra-frame edges between naturally connected body joints and inter-frame edges linking each joint to itself in the next frame.The inter-frame edges represent each joint’s trajectory over time.
- Graph convolution: Graph convolution samples neighboring nodes within graph distance D, and the model uses D = 1 for all cases.The one-neighbor set consists of joint nodes directly adjacent to the center node; larger D values are left for future work.
- Spatial-temporal modeling: Spatial-temporal modeling extends spatial graph convolution by including temporally connected joints in each node’s neighborhood.This extension uses the graph’s consecutive-frame connections to model spatial-temporal dynamics in skeleton sequences.
4 Experiments
Experiments evaluate ST-GCN on unconstrained Kinetics and constrained NTU-RGB+D, including component ablations and comparisons with prior methods. The results show benefits from spatial-temporal graph convolution, multi-subset partitioning, and learnable edge importance weighting, while skeleton inputs remain below frame-based models on Kinetics.
- Experimental setup: Experiments use Kinetics for unconstrained action recognition and NTU-RGB+D for constrained recognition, covering ablations, cross-method comparisons, and different skeleton acquisition settings.Kinetics uses estimated 2D joints from unconstrained videos, whereas NTU-RGB+D provides Kinect-derived 3D joints in a constrained lab environment.
- Ablation study: Spatial-temporal graph convolution outperforms a temporal-convolution-only baseline that concatenates all joint locations at each frame.The baseline is called “baseline TCN” and replaces spatial-temporal convolutions with only temporal convolution.
- Ablation study: Multiple-subset partitioning generally outperforms uni-labeling, while distance partitioning* provides an intermediate strategy between distance partitioning and uni-labeling.Uni-labeling is equivalent to averaging features before convolution, motivating the comparison with alternative partition strategies.
- Ablation study: Adding learnable edge importance weighting to vanilla ST-GCN with spatial configuration partitioning raises recognition performance by more than 1 percent.The component is reported as “ST-GCN+Imp.” and reflects differing importance among joints in different body parts.
- Comparisons: On Kinetics, ST-GCN is compared with Feature Encoding, Deep LSTM, and Temporal ConvNet using top-1 and top-5 accuracy, while NTU-RGB+D uses top-1 accuracy on X-Sub and X-View.On NTU-RGB+D, ST-GCN is trained without data augmentation and evaluated under the standard cross-subject and cross-view protocols.
- Discussion: 70.4% RGB CNN, 72.8% Flow CNN, and 72.4% ST-GCN are reported on the Kinetics comparison.The experiments note that skeleton-based methods are inferior to video frame-based models on Kinetics, despite ST-GCN working across both datasets.
5 Conclusion
The paper introduces ST-GCN, a spatial-temporal graph convolutional model for skeleton-based action recognition that outperforms prior state-of-the-art skeleton methods and captures motion complementary to RGB.
- ST-GCN constructs spatial-temporal graph convolutions on skeleton sequences for action recognition.The model is presented as a novel approach to skeleton-based action recognition.
- ST-GCN outperforms the previous state-of-the-art skeleton-based model on two challenging large-scale datasets.
- ST-GCN captures motion information in dynamic skeleton sequences that is complementary to the RGB modality.