Source-linked AI summary
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, Nanning Zheng
TL;DR
Skeleton action recognition needs models that exploit joint semantics without the high complexity of many deep feedforward approaches. SGN combines joint type and frame-index semantics with hierarchical joint- and frame-level modeling, and reports superior benchmark performance with an order of magnitude smaller model size than many competing methods.
Problem
Many skeleton-recognition approaches underexplore joint type and frame-index semantics and rely on deep networks with high model complexity.
Method
SGN combines joint dynamics with joint type and frame-index semantics in hierarchical joint-level and frame-level modules.
Results
SGN consistently achieves superior performance over many competing algorithms on three large benchmark datasets while using an order of magnitude smaller model size than many algorithms.
Takeaways & Limitations
SGN provides a lightweight strong baseline for studying efficient skeleton-based action recognition.
Abstract
from arXiv · showhide
Skeleton-based human action recognition has attracted great interest thanks to the easy accessibility of the human skeleton data. Recently, there is a trend of using very deep feedforward neural networks to model the 3D coordinates of joints without considering the computational efficiency. In this paper, we propose a simple yet effective semantics-guided neural network (SGN) for skeleton-based action recognition. We explicitly introduce the high level semantics of joints (joint type and frame index) into the network to enhance the feature representation capability. In addition, we exploit the relationship of joints hierarchically through two modules, i.e., a joint-level module for modeling the correlations of joints in the same frame and a framelevel module for modeling the dependencies of frames by taking the joints in the same frame as a whole. A strong baseline is proposed to facilitate the study of this field. With an order of magnitude smaller model size than most previous works, SGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets. The source code is available at https://github.com/microsoft/SGN.
1. Introduction
Skeleton-based action recognition offers accessible, structured, and computationally efficient data, but many feedforward approaches use deep, high-complexity networks while underusing joint semantics. SGN addresses this with explicit semantics and hierarchical joint- and frame-level modeling, achieving strong accuracy with substantially fewer parameters.
- Motivation: Skeleton data encode each joint by type, frame index, and 3D position, providing a structured representation for action recognition.Skeletons also abstract pose and motion, are robust to viewpoint and appearance variation, and have low-dimensional representations.
- Proposed approach: SGN uses joint type in GCN-based joint modeling and frame index in frame-level modeling, with dynamics represented from joint position and velocity.The framework uses three GCN layers for joint dependencies and two CNN layers for frame dependencies.
- Motivation: Deep learning methods model spatio-temporal skeleton evolution using RNNs, CNNs, and GCNs, while recent feedforward CNN approaches often incur high complexity.Many methods arrange joint coordinates into resized 2D maps and rely on large receptive fields to capture dependencies and semantics.
- Motivation: Joint type and frame index provide semantic information that distinguishes otherwise similar joint coordinates and captures differences in frame occurrence order.The paper notes that most existing approaches overlook or underexplore these semantics.
- Proposed approach: SGN explicitly exploits semantics and dynamics through hierarchical joint-level and frame-level modules for efficient skeleton-based action recognition.The joint-level module models within-frame joint correlations, while the frame-level module models dependencies across frames by treating each frame's joints as a whole.
- Results: SGN consistently outperforms many competing methods on three large benchmark datasets while using an order of magnitude smaller model size than many algorithms.The paper presents the model as a lightweight strong baseline, supported by extensive ablation studies and benchmark comparisons.
2. Related Work
Prior skeleton-action-recognition research has used recurrent models to capture temporal dynamics and graph models to represent joint relationships. These approaches include predefined or data-driven graph construction strategies alongside other neural architectures.
- Recurrent neural networks: Recurrent neural networks, including LSTM and GRU models, have commonly been used to model temporal dynamics in skeleton sequences.Earlier approaches favored recurrent architectures for short- and long-term temporal modeling.
- Graph convolutional networks: Graph convolutional approaches represent joints as graph nodes and model their relationships through edges within each frame.Some methods extend human-defined graphs with additional edges, while others learn graph connections from data.
3. Semantics-Guided Neural Networks
SGN represents skeleton joints using both dynamics and explicit semantics, then hierarchically models within-frame joint correlations and across-frame dependencies.
- Semantics and dynamics: SGN combines each joint’s 3D position and velocity with explicit joint-type and frame-index semantics.Position and velocity are embedded into a shared space and summed; joint type and frame index are separately encoded.
- Joint-level module: The joint-level module builds a content-adaptive graph to model correlations among joints in the same frame.Joint representations include dynamics and joint-type embeddings; pairwise affinities are normalized into an adjacency matrix for residual graph convolution.
- Joint-level module: Joint-type semantics help the network learn suitable connection weights between different joints within a frame.The model uses joint type and dynamics when learning graph connections among nodes.
- Frame-level module: The frame-level module embeds frame indices, aggregates joints by spatial max pooling, and applies temporal convolutions to model frame dependencies.A second convolution maps features to a higher-dimensional space, followed by temporal max pooling over frames.
4.1. Datasets
The evaluation uses NTU60, NTU120, and SYSU skeleton action-recognition datasets with subject, view, or setup-based testing protocols.
- NTU60: NTU60 contains 56,880 skeleton sequences across 60 action classes and 40 subjects, with 25 joints per skeleton.Cross Subject and Cross View settings divide subjects or camera views between training and testing.
- NTU120: NTU120 extends NTU60 to 114,480 sequences, 120 action classes, and 106 subjects.Its evaluation uses Cross Subject and Cross Setup splits.
- SYSU: SYSU is included as a 3D human-object interaction dataset in the evaluation suite.
4.2. Implementation Details
The implementation uses fixed-dimensional fully connected embeddings, Adam optimization with scheduled learning-rate decay, and sequence-level rotation augmentation.
- Network setting: Dynamic position and velocity embeddings use 64 neurons per fully connected layer, while frame-index encoding produces C3 = 256 features.Position and velocity layers do not share weights; joint-type encoding also uses two 64-neuron layers.
- Training: Training uses Adam with an initial learning rate of 0.001, decay by 10 at epochs 60, 90, and 110, and termination at epoch 120.Weight decay is 0.0001, with batch sizes 64, 64, and 16 for NTU60, NTU120, and SYSU.
- Data augmentation: Sequence-level augmentation randomly rotates 3D skeletons to improve robustness to viewpoint variation.Rotation ranges are ±17° for NTU60 Cross Subject, NTU120, and SYSU, and ±30° for NTU60 Cross View.
4.3. Ablation Study
The ablation studies show that explicit joint-type and frame-index semantics improve SGN, while hierarchical joint/frame modeling and spatial max pooling provide additional gains. The lightweight baseline also combines effective techniques with limited parameter cost, and SGN remains competitive against larger methods.
- Effectiveness of Exploiting Semantics: 1.7% and 1.3%: adding joint-type semantics to GCN message passing improves CS and CV accuracy, respectively.The authors attribute this benefit to making message passing sensitive to joint types rather than only coordinates.
- Effectiveness of Exploiting Semantics: 1.0% and 0.9%: frame-index semantics improve CS and CV accuracy when temporal convolution is disabled.With temporal convolution enabled, the gains are smaller but remain positive at 0.3% and 0.4%; the final scheme uses temporal convolution with frame-index semantics.
- Effectiveness of Hierarchical Model: 0.7% and 0.6%: SGN with spatial MaxPooling outperforms the same scheme without SMP on CS and CV, respectively.SMP aggregates joint information within each frame and reduces subsequent computation burden.
- Strong Baseline: SGN achieves slightly better performance than compared RNN-, GCN-, and CNN-based methods with much fewer parameters.The paper presents a lightweight strong baseline and reports that SGN uses only ten percent of the parameter counts of two compared methods.
4.4. Comparison with the State-of-the-arts
SGN achieves strong accuracy across NTU60, NTU120, and SYSU while using substantially fewer parameters than competing methods. Explicit semantics provide measurable gains over the baseline.
- SGN improves accuracy over its semantics-free baseline by 2.1% on CS and 1.7% on CV for NTU60.
- SGN surpasses ElAtt-GRU and Clips+CNN+MTLN by 8.3% and 9.4% accuracy, respectively, in NTU60 CS.
- Table 6 compares SYSU methods by accuracy (%), with asterisks marking models using parameters pretrained on NTU60.
- SGN exceeds mixed CNN-GCN and LSTM-GCN methods by 5.5% and 4.2% accuracy, respectively, in NTU60 CS.
- SGN delivers competitive performance with only ten percent of the parameters of [37] and [58].
- SGN achieves the best accuracy on NTU120 and SYSU, with semantics adding 1.8% and 2.3% accuracy in NTU120 C-Subject and C-Setup.
5. Conclusion
The paper presents SGN, which incorporates joint type and frame index semantics into hierarchical joint- and frame-level modeling. It reports state-of-the-art results on three benchmark datasets with an order of magnitude smaller model size than some previous works.
- SGN explicitly uses joint type and frame index as high-level semantic inputs alongside skeleton dynamics.
- Its joint-level module models correlations among joints within each frame, while its frame-level module models dependencies across frames.
- The paper develops a strong baseline that is better than most previous methods.
- SGN achieves state-of-the-art results on three benchmark datasets with an order of magnitude smaller model size than some previous works.