Source-linked AI summary
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition
Zhan Chen, Sicheng Li, Bing Yang, Qinghan Li, Hong Liu
TL;DR
Local graph convolutions limit direct modeling of distant joints and long-range temporal information needed for skeleton-based action recognition. The paper introduces multi-scale spatial and temporal graph-convolution modules and combines them into MST-GCN, which achieves competitive or state-of-the-art results across three large-scale datasets and improves RGB-model performance when added to a two-stream system.
Problem
Local graph convolutions primarily capture short-range joint dependencies and short-term trajectories, leaving distant spatial relations and long-range temporal information insufficiently modeled.
Method
MS-GC and MT-GC decompose local graph convolutions into cascaded sub-convolutions with hierarchical residual-like connections, enlarging spatial and temporal receptive fields without additional parameters.
Results
MST-GCN achieves competitive or state-of-the-art results on three large-scale datasets and increases an RGB-based model by 1.7% when added, versus 0.8% for ST-GCN.
Takeaways & Limitations
Coupling multi-scale spatial and temporal modeling yields a powerful skeleton-action representation and provides useful motion information for RGB-based ensemble models.
Abstract
from arXiv · showhide
Graph convolutional networks have been widely used for skeleton-based action recognition due to their excellent modeling ability of non-Euclidean data. As the graph convolution is a local operation, it can only utilize the short-range joint dependencies and short-term trajectory but fails to directly model the distant joints relations and long-range temporal information that are vital to distinguishing various actions. To solve this problem, we present a multi-scale spatial graph convolution (MS-GC) module and a multi-scale temporal graph convolution (MT-GC) module to enrich the receptive field of the model in spatial and temporal dimensions. Concretely, the MS-GC and MT-GC modules decompose the corresponding local graph convolution into a set of sub-graph convolution, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-graph convolutions, and each node could complete multiple spatial and temporal aggregations with its neighborhoods. The final equivalent receptive field is accordingly enlarged, which is capable of capturing both short- and long-range dependencies in spatial and temporal domains. By coupling these two modules as a basic block, we further propose a multi-scale spatial temporal graph convolutional network (MST-GCN), which stacks multiple blocks to learn effective motion representations for action recognition. The proposed MST-GCN achieves remarkable performance on three challenging benchmark datasets, NTU RGB+D, NTU-120 RGB+D and Kinetics-Skeleton, for skeleton-based action recognition.
Introduction
Skeleton-based action recognition uses graph representations to model human joints, but local graph convolutions inadequately capture distant spatial relations and long-range temporal dependencies. The paper proposes MS-GC and MT-GC modules, coupled in MST-GCN, to enlarge spatial and temporal receptive fields.
- Introduction: Skeleton data represents human structure efficiently and is robust to appearance variation and environmental noise.It uses 2D or 3D coordinates of a few dozen joints rather than pixel-level RGB information.
- Introduction: Existing GCN-based action-recognition methods mainly model short-range joint connections, although different actions require coordination across different body parts.Walking involves whole-body coordination, whereas hand waving can be performed primarily with the hand.
- Introduction: MS-GC decomposes spatial graph convolution into sub-graph convolutions with residual connections, enabling multiple exchanges between neighboring nodes and distant-joint modeling.The resulting equivalent spatial receptive field is increased without changing the module's basic local-convolution formulation.
- Introduction: MT-GC extends the multi-scale design to temporal modeling, targeting long-range relationships that local temporal operations do not directly capture.The two modules provide complementary spatial and temporal multi-scale modeling abilities.
- Introduction: MST-GCN couples MS-GC and MT-GC into blocks and stacks them to learn motion representations for skeleton-based action recognition.Experiments on three large-scale datasets report better performance than ST-GCN with fewer than one third of its parameters and remarkable results at equivalent parameter counts.
Related Works
Graph-based methods model skeletons as non-Euclidean data, while prior skeleton-action approaches also include handcrafted features and sequence-based recurrent models. The paper follows spatial graph convolution to learn representations directly from graph nodes and their neighbors.
- Neural Networks with Graph.: Spatial-based GCNs generalize convolution from Euclidean to non-Euclidean space by operating directly on graph nodes and their neighbors.The paper follows the spatial-based approach rather than spectral methods that rely on eigen-decomposition.
- Skeleton-Based Human Action Recognition: Handcrafted skeleton-action methods design features from physical intuition, distances, or kinematic information but require substantial manual effort.The passage notes that manually designed features may not consider all factors related to an action.
- Skeleton-Based Human Action Recognition: Deep-learning approaches learn action patterns from skeleton data, including RNN models that represent skeletons as coordinate-vector sequences and capture consecutive-frame dynamics.These approaches complement graph-based modeling by treating the data as temporal sequences.
Methodology
MST-GCN models skeleton actions with graph convolutions that enlarge spatial and temporal receptive fields through multi-scale sub-graph processing. Its MS-GC and MT-GC modules are combined into blocks that alternately extract spatial and temporal features while retaining efficiency.
- Graph representation: The skeleton sequence is represented as a graph whose nodes are joints and whose edges encode intraskeleton and inter-frame connections.Spatial and temporal neighborhoods are defined separately, with sampling ranges controlled by D and Γ.
- Multi-scale spatial modeling: MS-GC splits channels into fragments processed by sub-spatial graph convolutions with hierarchical residual connections, enlarging the equivalent spatial receptive field.Cascaded fragments can aggregate information from progressively more distant joints before concatenation and a module-level residual connection.
- Multi-scale temporal modeling: MT-GC extends the multi-scale design temporally by cascading sub-temporal graph convolutions over fragments and concatenating their outputs.The resulting representation captures both short-range and long-range temporal relationships without the additional residual connections used in MS-GC.
- Network integration: MST-GCN combines MS-GC and MT-GC either in separate replacements or within an STR-GC module that alternately updates spatial and temporal features.The first combination preserves continuity in multi-scale feature extraction, whereas STR-GC is lighter and more convenient for joint learning.
Datasets
The evaluation uses NTU RGB+D, NTU-120 RGB+D, and Kinetics-Skeleton, with skeleton data collected or estimated under varied subjects, views, setups, and action classes.
- NTU RGB+D contains 56,880 samples across 60 action classes, performed by 40 subjects and captured from three Kinect v2 camera views.Each pose contains 25 joints, and the dataset supports cross-subject and cross-view evaluation protocols.
- NTU-120 RGB+D expands the data to 114,480 samples and 120 action classes from 106 subjects across three camera views and 32 setups.
- Kinetics-Skeleton contains roughly 300,000 Internet video clips across 400 classes, with 18-joint 2D skeletons estimated using OpenPose.The stated split includes 240,436 training samples and 19,794 testing samples.
- The model stacks 10 building blocks, doubles channels and downsamples temporal length in the fifth and eighth blocks, while comparison channels are adjusted for parameter fairness.Training uses SGD with Nesterov momentum and batch size 24, with an initial learning rate of 0.1.
Ablation Study
The ablation study evaluates MS-GC, MT-GC, and their combination while adjusting channels to keep parameter counts consistent.
- The ablation compares the proposed MS-GC module, MT-GC module, and their combination against an adaptive ST-GCN baseline under matched parameter settings.
3.2 implementation.
The implementation study examines spatial, temporal, and joint multi-scale modules through subset-size ablations, efficiency comparisons, feature visualization, and combined-model evaluation.
- MS-GC module: With four spatial subsets, MS-GC improves the adaptive ST-GCN baseline by 1.0% on NTU RGB+D cross-subject evaluation.Increasing the number of subsets generally improves performance with similar parameters, while lightweight MS-GCN also outperforms the baseline.
- MT-GC module: With four temporal subsets, MT-GC improves accuracy by 1.2%, while lightweight MT-GCN achieves a 0.7% absolute improvement over ST-GCN.The experiments report consistently better results as the number of temporal subsets increases with similar parameter counts.
- MST-GCN module: MST-GCN improves adaptive ST-GCN by 1.8% with similar parameters and by 0.9% using only one third as many parameters.The study attributes these gains to the complementary spatial and temporal multi-scale modules.
- Feature visualization.: Feature responses concentrate on action-relevant body parts, such as arms for clapping and hand waving, while walking activates coordinated whole-body representations.For wearing a shoe, the model focuses on both the left arm and left foot, and it captures useful long-range dependencies.
Comparison with State of the Arts
MST-GCN is evaluated with one-, two-, and four-stream inputs across three skeleton-action benchmarks, achieving competitive or state-of-the-art results depending on dataset and metric.
- The four-stream setting combines joint, bone, joint-motion, and bone-motion inputs, while one- and two-stream results are also reported.
- On NTU RGB+D, MST-GCN achieves competitive performance on both cross-view and cross-subject benchmarks.
- On NTU-120 RGB+D, MST-GCN achieves state-of-the-art results on both cross-subject and cross-setup benchmarks.
- On Kinetics-Skeleton, MST-GCN achieves a state-of-the-art Top-1 result and a competitive Top-5 result.
Discussion
MST-GCN provides useful motion information for RGB-based action recognition, improving the combined model more than ST-GCN on the Kinetics validation set.
- Skeleton-based models provide complementary motion information that boosts RGB-based model performance.
- Table 7 reports accuracy for combinations of the RGB model with different skeleton models on the Kinetics validation set.
- 1.7% increase from adding MST-GCN to the RGB model exceeds the 0.8% increase from adding ST-GCN.Both skeleton models use only the joint stream for fair comparison.
Conclusion
The paper combines multi-scale spatial and temporal graph convolutions to capture short- and long-range dependencies, achieving competitive or state-of-the-art results on three challenging datasets.
- MS-GC and MT-GC capture short- and long-range dependencies in spatial and temporal domains, respectively.
- The modules decompose local convolutions into sub-convolutions and combine them through residual connections to enhance multi-scale representations.
- MST-GCN couples MS-GC and MT-GC to extract multi-scale spatial and temporal features.
- The proposed network achieves competitive or state-of-the-art results on three large-scale challenging datasets.