Source-linked AI summary
NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang
TL;DR
Existing 3D activity-recognition benchmarks lack large, varied training data and diverse action classes and camera views. This paper introduces a large-scale RGB+D dataset and a part-aware LSTM, with experiments showing data-driven methods outperform hand-crafted features and P-LSTM improves over traditional recurrent models.
Problem
Existing 3D activity-recognition benchmarks lack large datasets, subject variation, action classes, and camera-view diversity needed for challenging evaluation.
Method
The paper introduces a 56,880-sample RGB+D dataset from 40 subjects and proposes a part-aware LSTM that models long-term patterns for different body parts.
Results
Data-driven learning frameworks outperform hand-crafted features, while the proposed P-LSTM outperforms traditional recurrent models.
Takeaways & Limitations
The dataset’s scale and variety support data-driven learning for RGB+D action recognition, and its part-aware model further improves the recurrent learning framework.
Abstract
from arXiv · showhide
Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art hand-crafted features on the suggested cross-subject and cross-view evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.
1. Introduction
Existing 3D action benchmarks are limited by sample size, subject and action diversity, and camera coverage, constraining evaluation and data-hungry learning. NTU RGB+D addresses these gaps with a large, varied RGB+D benchmark and a part-aware recurrent model.
- Motivation: 3D action benchmarks lack large-scale publicly shared video data, limiting comparison of methods and evaluation of data-hungry learning approaches.The paper identifies overfitting and reduced learning-parameter sizes in prior data-driven attempts as consequences of insufficient samples.
- Motivation: Limited subject numbers and narrow performer-age ranges reduce intra-class variation, although age, gender, culture, and physical condition shape human activities.The paper therefore treats subject variation as crucial for benchmark design.
- Motivation: More action classes make classification harder because motion patterns and interacting objects are shared across classes.
- Motivation: Existing datasets commonly use restricted front or side camera views, limiting viewpoint diversity.
- Dataset and method: 56,880 RGB+D samples from 40 subjects span 80 camera viewpoints and include RGB, depth, skeleton, and infrared data.The skeleton data contain 3D locations of 25 major body joints; subjects range from 10 to 35 years old, and varied backgrounds add environmental variation despite indoor-only capture.
- Dataset and method: The proposed part-aware LSTM splits memory into body-part sub-cells to learn long-term patterns for each part and combine their outputs.Experiments report advantages of data-driven learning over state-of-the-art hand-crafted features on the proposed dataset.
2. Related work
Prior 3D action benchmarks and recognition methods offer varied modalities, views, and representations but remain limited in scale or coverage. NTU RGB+D broadens dataset diversity, while Part-aware LSTM introduces body-part memory sub-cells distinct from earlier recurrent approaches.
- 3D activity analysis datasets: NTU RGB+D provides more action classes, samples per class, intra-class variation, camera views, camera-to-subject distances, and improved Kinect v2 measurements.
- 3D action recognition methods: Prior methods extract depth normals, oriented components, range-sample features, or supernormal vectors to address representation and viewpoint variation.
- 3D action recognition methods: Skeleton-based methods encode body-part configurations or skeletal curves, while multimodal methods combine skeleton and depth features for classification.
- 3D action recognition methods: Earlier recurrent approaches include derivative-gated LSTM and hierarchical subnetworks that process body parts and combine their hidden representations.
- 3D action recognition methods: Part-aware LSTM differs by using internal part-based memory sub-cells with a novel gating mechanism.
3. The Dataset
NTU RGB+D combines four Kinect v2 modalities with 60 action classes, 40 subjects, and multiple camera views. It defines cross-subject and cross-view evaluation protocols for standardized classification accuracy reporting.
- Data modalities: The dataset provides depth maps, 3D joint information, RGB frames, and infrared sequences collected with Microsoft Kinect v2 sensors.Each sample includes multimodal observations from the same sensor platform.
- Action classes: 60 action classes comprise 40 daily, 9 health-related, and 11 mutual actions.The classes include examples such as drinking, sneezing, falling down, punching, and hugging.
- Subjects: 40 subjects aged 10–35 provide variation in age, gender, and height.Each subject retains a consistent identity number throughout the dataset.
- Views: Three simultaneous cameras capture horizontal views at −45°, 0°, and +45° during each action performance.The protocol produces front, side, and 45-degree views, with camera and setup numbers recorded for every sample.
- Collection setup: Table 2 records the height and distance of the three cameras for each collection setup in meters.The table specifies camera geometry for the acquisition configurations.
- Evaluation: The benchmark reports classification accuracy under cross-subject and cross-view evaluation criteria.Cross-subject uses 20 subjects for training and 20 for testing; cross-view trains on cameras 2 and 3 and tests on camera 1.
4. Part-Aware LSTM Network
The paper models 3D joint sequences with recurrent networks and introduces P-LSTM, which maintains separate long-term memories for body parts. Its design splits body dynamics into part-based cells and combines their learned representations for action recognition.
- Sequence modeling: Human actions are represented as sequences of 3D body-joint configurations and modeled with recurrent neural networks.RNNs update an internal state from the current input and previous state, while stacked layers form deep recurrent networks.
- Traditional RNN and LSTM: LSTM extends traditional RNNs with a memory cell that learns when to remember or forget long-term sequence information.Its gates include input, forget, output, and input modulation gates.
- Training: The recurrent output is fed to a softmax layer, with each input sample’s training label fixed across time.This trains the recurrent network for action recognition.
- Proposed Part-Aware LSTM: P-LSTM splits the LSTM memory into part-based cells because body joints move together in groups.The model groups joints into torso, two hands, and two legs, using separate input, forget, and modulation gates with a shared output gate.
- Proposed Part-Aware LSTM: At each frame, P-LSTM concatenates the 3D coordinates of joints within each body part as that part’s input representation.The resulting part-specific inputs drive the proposed recurrent unit.
- Model motivation: The baseline LSTM fully connects memory cells and input features, creating many parameters that are prone to overfitting.P-LSTM regularizes this structure by dropping unnecessary links, learning part-specific temporal patterns before combining them globally for classification.
5. Experiments
Experiments compare depth-based, skeleton-based, and recurrent methods under cross-subject and cross-view evaluation. P-LSTM achieves the strongest reported performance by modeling long-term context separately for body parts.
- The evaluation includes six published depth-based methods alongside RNN, LSTM, and P-LSTM under the benchmark’s two settings.The baselines comprise three depth-map methods and three skeleton-based methods.
- Depth-map features perform better cross-subject than cross-view because view-dependent appearances and motion patterns hinder cross-view generalization.
- Skeleton-based features outperform depth-map features in both evaluation settings, with a notable gap.Their 3D skeletal representation generalizes more strongly across views but remains prone to body-tracker errors.
- LSTM improves significantly over RNN by utilizing long-term context, while stacking another LSTM layer provides only a slight additional improvement.
- 62.93% cross-subject and 70.27% cross-view accuracy are achieved by P-LSTM, outperforming all other evaluated methods.P-LSTM isolates context memory for each body part and trains the classifier on their combination.
6. Conclusion
The paper introduces a large and varied RGB+D action-recognition dataset and a Part-aware LSTM model. Experiments show that large-scale data supports data-driven methods over hand-crafted features, while P-LSTM further improves recurrent modeling.
- 56,880 video samples from 60 action classes are collected in highly varied camera settings.The dataset is presented as larger and more varied than current datasets for this task.
- The dataset’s scale enables data-driven methods such as Long Short-Term Memory networks to achieve better performance than hand-crafted features.
- The proposed Part-aware LSTM uses the physical structure of the human body to improve the LSTM learning framework.
- Experimental results show data-driven learning frameworks outperform hand-crafted features, and P-LSTM outperforms traditional recurrent models.