NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Motivation

• 3D action benchmarks lack large-scale publicly shared video data, limiting comparison of methods and evaluation of data-hungry learning approaches.The paper identifies overfitting and reduced learning-parameter sizes in prior data-driven attempts as consequences of insufficient samples.Source: §1 · p. 1
• Limited subject numbers and narrow performer-age ranges reduce intra-class variation, although age, gender, culture, and physical condition shape human activities.The paper therefore treats subject variation as crucial for benchmark design.Source: §1 · p. 1
• More action classes make classification harder because motion patterns and interacting objects are shared across classes.Source: §1 · p. 1
• Existing datasets commonly use restricted front or side camera views, limiting viewpoint diversity.Source: §1 · p. 1

Dataset and method

• 56,880 RGB+D samples from 40 subjects span 80 camera viewpoints and include RGB, depth, skeleton, and infrared data.The skeleton data contain 3D locations of 25 major body joints; subjects range from 10 to 35 years old, and varied backgrounds add environmental variation despite indoor-only capture.Source: §1 · pp. 1–2
• The proposed part-aware LSTM splits memory into body-part sub-cells to learn long-term patterns for each part and combine their outputs.Experiments report advantages of data-driven learning over state-of-the-art hand-crafted features on the proposed dataset.Source: §1 · p. 2

3D activity analysis datasets

• NTU RGB+D provides more action classes, samples per class, intra-class variation, camera views, camera-to-subject distances, and improved Kinect v2 measurements.Source: §2.1 · p. 3

3D action recognition methods

• Prior methods extract depth normals, oriented components, range-sample features, or supernormal vectors to address representation and viewpoint variation.Source: §2.2 · p. 3
• Skeleton-based methods encode body-part configurations or skeletal curves, while multimodal methods combine skeleton and depth features for classification.Source: §2.2 · p. 3
• Earlier recurrent approaches include derivative-gated LSTM and hierarchical subnetworks that process body parts and combine their hidden representations.Source: §2.2 · p. 4
• Part-aware LSTM differs by using internal part-based memory sub-cells with a novel gating mechanism.Source: §2.2 · p. 4

Data modalities

• The dataset provides depth maps, 3D joint information, RGB frames, and infrared sequences collected with Microsoft Kinect v2 sensors.Each sample includes multimodal observations from the same sensor platform.Source: §3.1 · p. 4

Action classes

• 60 action classes comprise 40 daily, 9 health-related, and 11 mutual actions.The classes include examples such as drinking, sneezing, falling down, punching, and hugging.Source: §3.1 · p. 4

Subjects

• 40 subjects aged 10–35 provide variation in age, gender, and height.Each subject retains a consistent identity number throughout the dataset.Source: §3.1 · p. 4

Views

• Three simultaneous cameras capture horizontal views at −45°, 0°, and +45° during each action performance.The protocol produces front, side, and 45-degree views, with camera and setup numbers recorded for every sample.Source: §3.1 · p. 4

Collection setup

• Table 2 records the height and distance of the three cameras for each collection setup in meters.The table specifies camera geometry for the acquisition configurations.Source: Table 2 · p. 5

Evaluation

• The benchmark reports classification accuracy under cross-subject and cross-view evaluation criteria.Cross-subject uses 20 subjects for training and 20 for testing; cross-view trains on cameras 2 and 3 and tests on camera 1.Source: §3.2 · p. 5, §3.2.1 · p. 5, §3.2.2 · p. 5

Sequence modeling

• Human actions are represented as sequences of 3D body-joint configurations and modeled with recurrent neural networks.RNNs update an internal state from the current input and previous state, while stacked layers form deep recurrent networks.Source: §4 · p. 5, §4.1 · p. 5

Traditional RNN and LSTM

• LSTM extends traditional RNNs with a memory cell that learns when to remember or forget long-term sequence information.Its gates include input, forget, output, and input modulation gates.Source: §4.1 · p. 5, Figure 2 · p. 6

Training

• The recurrent output is fed to a softmax layer, with each input sample’s training label fixed across time.This trains the recurrent network for action recognition.Source: §4.1 · p. 5

Proposed Part-Aware LSTM

• P-LSTM splits the LSTM memory into part-based cells because body joints move together in groups.The model groups joints into torso, two hands, and two legs, using separate input, forget, and modulation gates with a shared output gate.Source: §4.2 · p. 6
• At each frame, P-LSTM concatenates the 3D coordinates of joints within each body part as that part’s input representation.The resulting part-specific inputs drive the proposed recurrent unit.Source: §4.2 · p. 6

Model motivation

• The baseline LSTM fully connects memory cells and input features, creating many parameters that are prone to overfitting.P-LSTM regularizes this structure by dropping unnecessary links, learning part-specific temporal patterns before combining them globally for classification.Source: §4.2 · p. 6

• The evaluation includes six published depth-based methods alongside RNN, LSTM, and P-LSTM under the benchmark’s two settings.The baselines comprise three depth-map methods and three skeleton-based methods.Source: §5 · p. 6, §5.1 · p. 6, Table 3 · p. 7
• Depth-map features perform better cross-subject than cross-view because view-dependent appearances and motion patterns hinder cross-view generalization.Source: §5.2 · p. 7
• Skeleton-based features outperform depth-map features in both evaluation settings, with a notable gap.Their 3D skeletal representation generalizes more strongly across views but remains prone to body-tracker errors.Source: §5.2 · p. 7
• LSTM improves significantly over RNN by utilizing long-term context, while stacking another LSTM layer provides only a slight additional improvement.Source: §5.2 · p. 7
• 62.93% cross-subject and 70.27% cross-view accuracy are achieved by P-LSTM, outperforming all other evaluated methods.P-LSTM isolates context memory for each body part and trains the classifier on their combination.Source: §5.2 · p. 7

• 56,880 video samples from 60 action classes are collected in highly varied camera settings.The dataset is presented as larger and more varied than current datasets for this task.Source: §6 · pp. 7–8
• The dataset’s scale enables data-driven methods such as Long Short-Term Memory networks to achieve better performance than hand-crafted features.Source: §6 · p. 8
• The proposed Part-aware LSTM uses the physical structure of the human body to improve the LSTM learning framework.Source: §6 · p. 8
• Experimental results show data-driven learning frameworks outperform hand-crafted features, and P-LSTM outperforms traditional recurrent models.Source: §6 · p. 8