Source-linked AI summary
Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition
Yi-Fan Song, Zhang Zhang, Caifeng Shan, Liang Wang
TL;DR
Skeleton-based action recognition requires discriminative features without the excessive complexity and weak explainability of many SOTA models. The paper proposes an efficient GCN baseline with early-fused Multiple Input Branches, residual bottlenecks, and Part-wise Attention, achieving SOTA or competitive results on NTU RGB+D 60 and 120 with far fewer parameters.
Problem
Skeleton-based action recognition needs rich discriminative features, while many SOTA models are over-parameterized, costly to train and infer, and give limited consideration to explainability.
Method
The baseline combines early-fused Joint, Velocity, and Bone branches with residual GCN bottleneck blocks and Part-wise Attention over manually defined body parts.
Results
PA-ResGCN achieves SOTA performance on NTU RGB+D 120 and competitive performance on NTU RGB+D 60, while bottleneck ResGCN uses 0.77 million parameters, nearly 34 times fewer than DGNN.
Takeaways & Limitations
The baseline combines strong recognition performance with lower training and inference complexity and more explainable body-part representations.
Takeaways & Limitations
Reading and writing remain difficult, with accuracies below 70%, because the datasets record only two joints per hand for these similar two-handed actions.
Abstract
from arXiv · showhide
One essential problem in skeleton-based action recognition is how to extract discriminative features over all skeleton joints. However, the complexity of the State-Of-The-Art (SOTA) models of this task tends to be exceedingly sophisticated and over-parameterized, where the low efficiency in model training and inference has obstructed the development in the field, especially for large-scale action datasets. In this work, we propose an efficient but strong baseline based on Graph Convolutional Network (GCN), where three main improvements are aggregated, i.e., early fused Multiple Input Branches (MIB), Residual GCN (ResGCN) with bottleneck structure and Part-wise Attention (PartAtt) block. Firstly, an MIB is designed to enrich informative skeleton features and remain compact representations at an early fusion stage. Then, inspired by the success of the ResNet architecture in Convolutional Neural Network (CNN), a ResGCN module is introduced in GCN to alleviate computational costs and reduce learning difficulties in model training while maintain the model accuracy. Finally, a PartAtt block is proposed to discover the most essential body parts over a whole action sequence and obtain more explainable representations for different skeleton action sequences. Extensive experiments on two large-scale datasets, i.e., NTU RGB+D 60 and 120, validate that the proposed baseline slightly outperforms other SOTA models and meanwhile requires much fewer parameters during training and inference procedures, e.g., at most 34 times less than DGNN, which is one of the best SOTA methods.
1 INTRODUCTION
Skeleton-based action recognition needs rich, discriminative features, but existing high-performing models are often costly, difficult to train, and insufficiently explainable. The paper proposes an efficient GCN baseline combining early-fused inputs, residual bottlenecks, and part-wise attention.
- Motivation: Skeleton sequences compactly represent 3D joint dynamics and are robust to illumination, viewpoint, and background variations.These properties motivate skeleton-based methods for extracting informative motion features.
- Motivation: Current SOTA models are often multi-stream and over-parameterized, creating difficult training and high computational costs.2s-AGCN has about 6.94 million parameters and requires nearly 4 GPU-days on NTU RGB+D 60; DGNN exceeds 26 million parameters.
- Approach: The proposed baseline early-fuses Joint, Velocity, and Bone branches before using a main stream of ResGCN modules.This retains multiple spatio-temporal inputs while reducing the complexity associated with separate streams.
- Approach: Residual GCN bottleneck blocks reduce parameter-tuning and inference costs, while residual links ease model optimization.The architecture combines residual connections with channel-reducing bottlenecks.
- Approach: PartAtt discovers important body parts across complete action sequences, improving feature discrimination and providing visual explanations through class activation maps.It operates over manually defined body parts rather than individual joints.
- Results: PA-ResGCN achieves SOTA performance on NTU RGB+D 120 and competitive performance on NTU RGB+D 60, while bottleneck ResGCN uses 0.77 million parameters.The bottleneck model uses nearly 34 times fewer parameters than DGNN, with slightly lower accuracy than other SOTA models.
2 RELATED WORK
Related work progresses from RNN- and CNN-based skeleton models toward graph-based, efficient, attention, and part-based approaches. The paper differentiates its method by combining efficient GCN design with attention over body parts.
- Skeleton-based Action Recognition: ST-GCN introduced graph convolution for jointly modeling skeleton spatial configurations and temporal dynamics.Subsequent GCN studies address issues including occlusion and richer joint activation.
- Efficient Models: Efficiency-focused studies use lightweight CNN blocks or extensive preprocessing to reduce model complexity and speed inference.These approaches trade architectural simplicity or preprocessing complexity against recognition performance.
- Attention Models: Attention-based methods allocate importance across spatial and temporal elements to identify informative skeleton information.Prior work includes spatial-temporal attention LSTMs and attention mechanisms for skeleton recognition.
- Part-based Models: Part-based models separately process body parts, whereas this paper uses attention to discover the most informative parts.The paper distinguishes part discovery from simply extracting features from each part individually.
- Proposed ResGCN: Figure 2 depicts a ResGCN bottleneck example using module configurations, spatial and temporal blocks, and three residual-link types.The example configuration is [B1,N2,N3,N3], with Block, Module, and Dense residual links.
3 METHODS
The method combines graph convolution, early-fused multi-input skeleton representations, residual bottleneck modules, and part-wise attention to build an efficient and more explainable action-recognition baseline.
- Graph Convolutional Network: Spatial GCN aggregates joint features using distance-specific adjacency matrices, normalized neighborhoods, learnable convolution weights, and learnable edge-importance parameters.Temporal convolution separately aggregates contextual features from adjacent frames using an L × 1 window.
- Residual GCN: Bottleneck blocks reduce feature channels around the main convolution, lowering computational cost while preserving the spatial and temporal processing structure.With 256 input and output channels, reduction rate r = 4, and temporal window L = 9, the example uses 69,632 rather than 589,824 parameters.
- Residual GCN: Residual links connect features across GCN blocks or modules, while denser links may improve connectivity at the cost of compactness and memory.The architecture therefore requires selecting an appropriate residual-link type.
- Model Architecture: The complete architecture applies ResGCN modules to three input branches, concatenates them, then uses PA-ResGCN modules that pair ResGCN with PartAtt.Temporal stride 2 is used at the beginning of later modules to reduce complexity and was also found useful for avoiding over-fitting.
- Multiple Input Branches: The baseline processes joint, velocity, and bone feature sequences through early-fused multiple input branches before shared feature extraction.Joint inputs include relative and absolute positions; velocity inputs use temporal motion; bone inputs encode lengths and angles.
- Part-wise Attention: PartAtt divides features into five manually selected body parts and computes part-level attention from global temporal context before element-wise feature reweighting.The block uses temporal and part pooling, shared dimension reduction, part-specific weights, ReLU, and part-level Softmax.
4 EXPERIMENTAL RESULTS
Experiments on NTU RGB+D 60 and 120 evaluate the proposed models against state-of-the-art methods and through component ablations. The results show strong accuracy, substantially lower complexity, faster inference, benefits from all three input branches, and difficulty with subtle two-hand actions.
- Experimental Setup: The evaluation covers PA-ResGCN and ResGCN on the NTU RGB+D 60 and 120 datasets, including ablation studies of individual components.ResGCN-N51 is used as the base model for the ablation studies.
- NTU RGB+D 60 Results: 90.9% accuracy on X-sub and 96.0% on X-view are achieved by PA-ResGCN-B19 on NTU RGB+D 60.ResGCN-N51 reaches 89.1% and 93.5% on the same benchmarks with one quarter the parameters of PA-ResGCN-B19.
- Accuracy and Efficiency: 0.77 million parameters give ResGCN-N51 slightly lower accuracy than DGNN while requiring about 34 times fewer parameters.Compared with ST-GCN, ResGCN-N51 improves accuracy by 7.6% on X-sub and 5.2% on X-view.
- Accuracy and Efficiency: ResGCN-N51 greatly improves inference speed over ST-GCN while outperforming lightweight SGN, which reaches 86.6% on X-sub.Inference speed is measured as sequences evaluated per second using one GPU.
- Comparisons with Attention Models: PA-ResGCN outperforms STA-LSTM by over 10% on both benchmarks and exceeds AGC-LSTM by 1.7% on X-sub and 1.0% on X-view.Its PartAtt block operates over whole sequences and body parts, unlike attention applied per frame and joint in the compared models.
- NTU RGB+D 120 Results: On NTU RGB+D 120, PA-ResGCN-B19 outperforms 2s-AGCN by 4.8% on one benchmark and 4.1% on the other.The authors associate this gap with PartAtt discovering features from informative body parts.
- Ablation Studies: A proper bottleneck reduction rate, especially r = 4, reduces model complexity while maintaining accuracy.Except at r = 8, ResGCN-N51 retains competitive accuracies with half or one quarter of the parameters of ResGCN-B19.
- Ablation Studies: All three MIB input branches are necessary: models using only one branch perform significantly worse than the model using all input data.The branches represent joint, velocity, and bone information.
5 CONCLUSION
The paper proposes an efficient GCN baseline combining MIB, residual bottleneck blocks, and PartAtt blocks. On NTU RGB+D 60 and 120, PA-ResGCN achieves SOTA performance with fewer parameters and higher inference speed.
- The baseline combines early-fused MIB, residual bottleneck blocks, and PartAtt blocks for skeleton-based action recognition.These components target feature enrichment, computational efficiency, and body-part-focused attention.
- The PartAtt block focuses on essential body parts rather than individual joints, helping avoid superfluous or interfering features.
- The bottleneck technique reduces learnable parameters by at most 34 times compared with other models, saving training and inference time.
- PA-ResGCN achieves SOTA performance on the NTU RGB+D 60 and 120 datasets, with inference speed obviously higher than other models.