Source-linked AI summary
A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
Bin Ren, Mengyuan Liu, Runwei Ding, Hong Liu
TL;DR
Prior reviews largely emphasized RGB or RGB-D action recognition and did not comprehensively examine deep learning for 3D skeleton sequences. This survey organizes skeleton-based action recognition around four neural architectures, reviews methods and datasets, and concludes that NTU-RGB+D 120 remains challenging while GCN methods lead among compared architectures. It identifies long-term recognition, representation, real-time operation, labeling, and limited exploitation of spatial-temporal correlations as ongoing challenges.
Problem
Prior surveys often lacked detailed examination of skeleton data and neural-network perspectives, despite extensive deep-learning work in action recognition.
Method
The survey provides a data-driven synthesis of RNN-, CNN-, GCN-, and Transformer-based 3D skeleton action-recognition methods, architectures, datasets, and future directions.
Results
GCN-based methods achieved leading results among compared architectures, while NTU-RGB+D 120 remains a significant challenge requiring further advancement.
Takeaways & Limitations
The topology graph is the most intuitive representation of human skeleton joints, although effectively extracting pertinent information remains a challenge across learning structures.
Takeaways & Limitations
Long-term recognition, optimized sequence representations, real-time operation, and labor-intensive labeling remain open challenges for 3D skeleton action recognition.
Abstract
from arXiv · showhide
3D skeleton-based action recognition (3D SAR) has gained significant attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or RGB data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3D skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on four fundamental deep architectures, i.e., Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Graph Convolutional Network (GCN), and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.
1. Introduction
3D skeleton data offers a compact, structurally informative modality for action recognition, while deep learning has enabled architectures tailored to its spatial-temporal information. This survey organizes the field around RNNs, CNNs, GCNs, and Transformers, presenting methods, datasets, and future directions.
- Motivation: Action recognition supports applications including intelligent surveillance, human-computer interaction, virtual reality, and robotics.
- Motivation: 3D skeleton data represents human bodies through joints and bones, providing spatial structure and temporal correlations while reducing sensitivity to complex backgrounds and variable conditions.The passage contrasts skeleton data with computationally intensive RGB, depth, and video modalities affected by body scale, viewpoint, and motion-speed variation.
- Survey scope: The survey addresses a review gap because prior work largely emphasized traditional methods or RGB/RGB-D data rather than skeleton sequences and deep architectures.
- Survey scope: The survey reviews 3D skeleton action recognition through four deep architectures: RNNs, CNNs, GCNs, and Transformers.It also discusses motivations for model selection and potential future research directions.
- Pipeline: The general pipeline obtains skeletons from depth sensors or pose estimation, processes them with deep networks, and outputs an action category.
- Survey scope: Its data-driven discussion covers spatial-temporal modeling, skeleton representations, joint and bone correlations, and co-occurrence feature learning across contemporary methods.
2. 3D SAR With Deep Learning
The survey organizes 3D skeleton-based action recognition around RNN, CNN, GCN, and Transformer architectures, emphasizing spatial-temporal modeling and data-driven feature extraction. It discusses architecture-specific methods for representing skeleton sequences, learning joint relationships, and addressing limitations in spatial or temporal modeling.
- Overview: Existing surveys provide limited neural-network-centered coverage of 3D skeleton-based action recognition, motivating a dedicated comparison of deep architectures.The survey examines RNN-, CNN-, GCN-, and Transformer-based methods for 3D skeleton data.
- RNN-Based Methods: RNNs model sequential joint movements but can struggle with long-term dependencies, sequential computation, and spatial modeling.LSTM and GRU variants address some gradient and dependency problems, while two-stream and hybrid designs strengthen spatial modeling.
- RNN-Based Methods: Data-driven RNN methods selectively emphasize informative joints and transform skeleton coordinates before extracting salient motion features.Global context-aware attention focuses on important joints, while coordinate transformations improve robustness to scale, rotation, and translation.
- CNN-Based Methods: CNN-based methods transform vectorized skeleton sequences into pseudo-images so convolution can learn spatial-temporal features.Approaches encode joints into multiple 2D pseudo-images or texture images, but representation design must preserve both spatial and temporal information.
- GCN- and Transformer-Based Methods: GCN-based action recognition exploits skeleton topology, while Transformer methods use multi-head self-attention to model spatial-temporal relations.The survey notes that effectively extracting latent information and exploiting spatial-temporal correlation remain open challenges; Transformer spatial modeling is also constrained by high-dimensional skeleton semantics.
3. Latest Datasets And Performance
The survey focuses on NTU-RGB+D and NTU-RGB+D 120, reporting recent methods and evaluation protocols for both datasets. Existing algorithms perform impressively on NTU-RGB+D, while NTU-RGB+D 120 remains more challenging, with GCN-based and hybrid Transformer methods showing strong results.
- Datasets: The survey primarily analyzes NTU-RGB+D and its expanded edition, NTU-RGB+D 120.These datasets are presented as the focus of the survey's dataset and performance discussion.
- Datasets: NTU-RGB+D contains 56,880 video samples and 25-joint 3D coordinates, with Cross-Subject and Cross-View evaluation protocols.NTU-RGB+D 120 expands the setting to 120 action classes, 114,480 skeleton sequences, and 155 viewpoints.
- Performance: The reported performance tables cover recent skeleton-based methods on NTU-RGB+D and NTU-RGB+D 120.For NTU-RGB+D 120, the listed protocols are Cross-Subject and Cross-Setup.
- Performance: Existing algorithms achieve impressive performance on NTU-RGB+D, whereas NTU-RGB+D 120 presents a significant challenge requiring further advances.The newer dataset is described as more complex than the original benchmark.
- Performance: GCN-based methods achieve leading results, while Transformer-based methods perform strongly and hybrid Transformer architectures further boost overall 3D-SAR performance.The comparison spans the architectures discussed for both datasets.
4. Discussion
The discussion compares deep architectures by their modeling strengths and computational characteristics, while identifying unresolved challenges in 3D skeleton action recognition. It also considers combining architectures and learning with fewer manual labels.
- Architecture Selection: GCNs can capture spatial and temporal relationships among joints, RNNs capture temporal dynamics, and CNNs identify spatial features.The discussion frames these as architecture-selection considerations for accuracy and robustness.
- Architecture Selection: CNNs process faster through parallelism, whereas RNNs are slower because they use sequential processing.The comparison concerns computational efficiency rather than recognition accuracy.
- Hybrid Models: Combining architectures may allow a model to absorb the advantages of each fundamental architecture.The paper presents integration as a possible practical solution.
- Open Challenges: Long-term recognition, improved 3D-skeleton sequence representations, real-time operation, and less labor-intensive annotation remain open challenges.The discussion points to unsupervised, weakly supervised, and zero-shot learning as possible directions.
5. Conclusion
The survey reviews 3D skeleton-based action recognition through four deep architectures and organizes methods in a data-driven manner. It highlights information extraction as a central challenge, topology graphs as an intuitive representation, and hybrid or multitask strategies as promising directions.
- Survey Scope: The survey comprehensively examines 3D skeleton action recognition using RNN-, CNN-, GCN-, and Transformer-based architectures.It presents recent algorithms and discusses their corresponding architectures in a data-driven manner.
- Architecture Analysis: RNN and CNN methods address spatial-temporal information through skeleton representations and network design, while GCN methods exploit joint and bone correlations.Transformer methods are often combined with other architectures for action recognition.
- Conclusion: Effectively extracting pertinent information from 3D skeleton data is a fundamental challenge across diverse learning structures.The conclusion identifies information extraction as common to the reviewed architectures.
- Conclusion: Topology graphs provide an intuitive representation of human skeleton joints, although CNN- and RNN-based methods remain suitable for the task.The conclusion relates this assessment to performance on datasets such as NTU-RGB+D.
- Conclusion: Multi-task learning shows promise for improving cross-view or cross-subject evaluation, while further accuracy gains on NTU-RGB+D are increasingly difficult.The conclusion attributes this difficulty to already high performance levels on the dataset.