Source-linked AI summary
PYSKL: Towards Good Practices for Skeleton Action Recognition
Haodong Duan, Jiaqi Wang, Kai Chen, Dahua Lin
TL;DR
Skeleton action recognition lacks consistently aligned comparisons across algorithms and training practices. PYSKL provides a unified PyTorch toolbox with six approaches, standardized good practices, and broad benchmarks, achieving strong results across the evaluated tasks.
Problem
Existing skeleton action recognition implementations use inconsistent architectures, streams, preprocessing, and training practices, limiting unified comparison.
Method
PYSKL unifies six GCN- and CNN-based approaches, standardizes training practices, supports nine benchmarks, and introduces the simple ST-GCN++ model.
Results
PYSKL achieves state-of-the-art recognition performance on eight of nine benchmarks, while ST-GCN++ is competitive with state-of-the-art GCN approaches.
Takeaways & Limitations
PYSKL provides shared practices, trained models, and detailed benchmarks, while ST-GCN++ offers a simple strong baseline for future research.
Takeaways & Limitations
CTR-GCN performance drops slightly in the benchmark because its original spatial augmentations are omitted.
Abstract
from arXiv · showhide
We present PYSKL: an open-source toolbox for skeleton-based action recognition based on PyTorch. The toolbox supports a wide variety of skeleton action recognition algorithms, including approaches based on GCN and CNN. In contrast to existing open-source skeleton action recognition projects that include only one or two algorithms, PYSKL implements six different algorithms under a unified framework with both the latest and original good practices to ease the comparison of efficacy and efficiency. We also provide an original GCN-based skeleton action recognition model named ST-GCN++, which achieves competitive recognition performance without any complicated attention schemes, serving as a strong baseline. Meanwhile, PYSKL supports the training and testing of nine skeleton-based action recognition benchmarks and achieves state-of-the-art recognition performance on eight of them. To facilitate future research on skeleton action recognition, we also provide a large number of trained models and detailed benchmark results to give some insights. PYSKL is released at https://github.com/kennymckormick/pyskl and is actively maintained. We will update this report when we add new features or benchmarks. The current version corresponds to PYSKL v0.2.
1. Introduction
PYSKL addresses inconsistent comparisons in skeleton action recognition by unifying representative GCN and CNN approaches with shared practices and benchmarks. It also introduces ST-GCN++ and provides resources for reproducible research.
- Skeleton action recognition uses compact, informative, and illumination-robust 2D or 3D joint coordinates, with methods mainly based on GCNs or CNNs.
- Existing GCN studies differ in streams, preprocessing, and training practices, while prior repositories generally implement only one algorithm without unified comparisons.
- PYSKL reports that good practices contribute more to strong recognition performance than complicated architectural design.
- ST-GCN++ uses simple modifications to ST-GCN, achieves competitive performance without complicated attention, and serves as a strong baseline.
- PYSKL implements six representative approaches and supports nine skeleton-based action recognition benchmarks, including extensive GCN evaluations.
- The toolbox includes source code, a model zoo, training and testing instructions, visualization tools, and custom-dataset support.
2.1. Good Practices for GCN-based approaches
PYSKL standardizes preprocessing, temporal augmentation, and optimization settings for GCN-based skeleton action recognition. These practices are intended to make model comparisons more consistent and can improve recognition performance.
- PYSKL follows CTR-GCN preprocessing for 3D skeletons and normalizes 2D skeletons into a fixed range, with pose-based tracking for NTURGB+D.
- Temporal preprocessing differs across prior methods, while PYSKL uses uniform sampling to generate sequences of length M=100.
- PYSKL trains all GCN models with shared settings: learning rate 0.1, batch size 128, 80 epochs, cosine annealing, momentum 0.9, and weight decay 5×10^-4.
- For most GCN networks, the shared hyper-parameter configuration performs better than earlier settings using a MultiStep learning-rate scheduler.
2.2. The Design of ST-GCN++
ST-GCN++ simplifies ST-GCN’s spatial and temporal modules while retaining strong recognition performance. Its multi-branch temporal design improves modeling capability while reducing computational cost and parameters.
- The Design of ST-GCN++: ST-GCN++ modifies ST-GCN’s interleaved spatial graph-convolution and temporal 1D-convolution modules.
- Spatial Module Design: ST-GCN++ initializes coefficient matrices from a predefined joint topology and updates them iteratively by gradient descent during training.
- Temporal Module Design: A six-branch temporal ConvNet replaces ST-GCN’s single kernel-size-9 temporal convolution.
- Spatial Augmentations: Random rotation works for 3D skeletons, whereas random scaling works for both 2D and 3D skeletons.
- Benchmark Results: ST-GCN++ with good practices and spatial augmentations surpasses CTR-GCN on 3 of 4 NTURGB+D benchmarks.
- Temporal Module Design: The redesigned temporal module improves temporal modeling capability while saving computational cost and parameters through narrower branch widths.
2.3. Benchmarking GCN Algorithms
PYSKL benchmarks representative GCN algorithms under common settings across NTURGB+D benchmarks and skeleton modalities. Under this unified evaluation, accuracy differences are smaller than originally reported, while ST-GCN++ remains competitive with simpler computation.
- PYSKL benchmarks ST-GCN, AAGCN, MS-G3D, CTR-GCN, and ST-GCN++ on four NTURGB+D benchmarks using 3D and 2D skeleton annotations.
- The evaluation reports Top-1 Accuracy for joint, bone, two-stream, and four-stream configurations.
- On all NTURGB+D benchmarks, the extreme deviation in Top-1 Accuracy across GCN approaches is less than 2%.
- ST-GCN++ achieves competitive recognition performance with fewer parameters and fewer FLOPs than more complicated state-of-the-art GCN approaches.
2.4. Spatial Augmentations
PYSKL evaluates three spatial augmentations for skeleton action recognition and finds that their effectiveness depends on skeleton dimensionality. Random rotating and scaling help in specified settings, while Gaussian noise does not.
- PYSKL implements random rotating, random scaling, and random Gaussian noise as spatial augmentations.
- Random rotating works for 3D skeletons, whereas random scaling works for both 2D and 3D skeletons.
- Random Gaussian noise does not work for either 2D or 3D skeletons.
- ST-GCN++ is trained with 3D skeletons for 120 epochs using random rotating and random scaling.
- On 3 of 4 NTURGB+D benchmarks, ST-GCN++ surpasses the current state-of-the-art CTR-GCN.
3. CNN-based approaches
PoseC3D is a 3D-CNN approach that converts 2D skeleton coordinates into Gaussian-map heatmap volumes for convolutional processing. PYSKL supports multiple 3D-CNN backbones and releases models across seven datasets.
- PoseC3D takes 2D human skeletons, generates Gaussian maps from joint coordinates, and organizes them into a 3D heatmap volume.
- PoseC3D can process the heatmap volume with an arbitrary 3D-CNN backbone.
- PYSKL supports C3D, SlowOnly, and X3D as PoseC3D backbones.
- PYSKL releases PoseC3D models trained on seven different datasets.
4. Conclusion
PYSKL provides a unified, publicly released framework for benchmarking skeleton-based action recognition. It combines six algorithms, nine benchmarks, extensive training resources, and the ST-GCN++ baseline.
- PYSKL implements six representative algorithms under a unified framework and trains them on nine skeleton-based action recognition benchmarks.
- PYSKL achieves state-of-the-art recognition performance on eight benchmarks.
- The toolbox offers training good practices, extensive benchmarks, released configurations, and model weights.
- ST-GCN++ is introduced as a simple, strong baseline that surpasses previous state-of-the-art results on NTURGB+D benchmarks.