PaperlayerView exact revision on arXiv

Source-linked AI summary

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, Alex C. Kot

arXiv:1905.04757v2cs.CV

TL;DR

Existing 3D action-recognition benchmarks lack scale, class diversity, varied camera views, and subject variation. The paper introduces a large RGB+D dataset, evaluates existing methods, and proposes APSR for one-shot recognition, with promising results enabled by large-scale data.

  • Problem

    Existing 3D action-recognition benchmarks lack large-scale samples, many realistic action classes, varied camera views, and diverse human subjects.

  • Method

    The paper introduces a large-scale RGB+D action-recognition dataset, evaluates existing 3D methods and modality fusion, and proposes APSR for one-shot recognition.

  • Results

    114,480 video samples across 120 action classes support promising data-driven performance, while APSR achieves promising results for one-shot 3D action recognition.

  • Takeaways & Limitations

    The dataset supports applying data-driven learning frameworks to depth-based and RGB+D-based human activity understanding.

  • Takeaways & Limitations

    Skeleton-only recognition can misclassify fine-grained hand and finger motions because Kinect provides few hand joints and noisy tracking.

Abstract

from arXiv · show

Research on depth-based human activity analysis achieved outstanding performance and demonstrated the effectiveness of 3D representation for action recognition. The existing depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of large-scale training samples, realistic number of distinct class categories, diversity in camera views, varied environmental conditions, and variety of human subjects. In this work, we introduce a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames. This dataset contains 120 different action classes including daily, mutual, and health-related activities. We evaluate the performance of a series of existing 3D activity analysis methods on this dataset, and show the advantage of applying deep learning methods for 3D-based human action recognition. Furthermore, we investigate a novel one-shot 3D activity recognition problem on our dataset, and a simple yet effective Action-Part Semantic Relevance-aware (APSR) framework is proposed for this task, which yields promising results for recognition of the novel action classes. We believe the introduction of this large-scale dataset will enable the community to apply, adapt, and develop various data-hungry learning techniques for depth-based and RGB+D-based human activity understanding. [The dataset is available at: http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp]

1 INTRODUCTION

NTU RGB+D 120 addresses major limitations in 3D action-recognition benchmarks by providing large scale, class diversity, varied subjects, camera viewpoints, and environmental conditions. It also supports deep-learning evaluation and introduces APSR for one-shot 3D action recognition.

  • Benchmark limitations: Existing 3D action benchmarks lack large-scale samples, realistic class diversity, varied camera views, environmental variation, and performer diversity.Limited samples also hinder data-driven methods through overfitting and reduced model scale.
  • Dataset contribution: 114,480 RGB+D video samples from 106 subjects provide RGB, depth, skeleton, and infrared data captured from 155 camera viewpoints.Skeleton sequences contain 3D locations for 25 major body joints.
  • Dataset contribution: Ninety-six backgrounds with illumination variation add environmental diversity beyond datasets collected under fixed or limited conditions.The authors contrast this variation with other datasets using single or few backgrounds and fixed illumination.
  • Evaluation and impact: The benchmark enables more sensible cross-subject and cross-setup evaluations and supports data-hungry methods such as deep learning.The paper evaluates state-of-the-art approaches and fusion across RGB, depth, and skeleton modalities.
  • One-shot recognition: The APSR framework addresses one-shot 3D action recognition by using semantic relevance between body parts and action classes to emphasize relevant motions.The framework uses distributed word embeddings and action descriptions to guide body-part relevance for novel classes.

2 RELATED WORK

Existing 3D activity benchmarks vary in scale, action coverage, viewpoints, environments, and modalities. NTU RGB+D 120 extends prior work with broader dataset variation and introduces APSR for one-shot recognition.

  • 3D Activity Analysis Datasets: NTU RGB+D 120 adds more action classes, samples, intra-class variation, environments, camera views, and camera-to-subject distances than earlier datasets.The dataset also uses Kinect v2 for more accurate depth maps and 3D joints, especially in multi-camera setups.
  • 3D Activity Analysis Datasets: The paper extends a preliminary 60-class conference dataset into a larger benchmark with greater variation and additional one-shot recognition challenges.The extended dataset is designed to provide more challenging 3D human activity analysis.
  • 3D Action Recognition Methods: Earlier methods use hand-crafted depth, skeleton, or multimodal features, while recent work applies deep networks to 3D human activity analysis.Multimodal methods combine complementary appearance and geometrical information from different data sources.
  • 3D Action Recognition Methods: APSR emphasizes body-part features according to semantic relevance between each action class and body part for one-shot 3D action recognition.The framework uses action and body-part descriptions to guide recognition of novel action categories.

3 THE NTU RGB+D 120 DATASET

This section introduces the NTU RGB+D 120 action-recognition dataset and its evaluation criteria. It frames the dataset as a benchmark for 3D human activity analysis.

  • The section presents the NTU RGB+D 120 action-recognition dataset.
  • The dataset’s structure is introduced alongside defined evaluation criteria for tested methods.

3.1 Dataset Structure

NTU RGB+D 120 combines four Kinect modalities with 120 action categories, diverse subjects, viewpoints, environments, and action types. Its collection design increases variation across samples.

  • Dataset Modalities: The dataset contains depth maps, 3D joint information, RGB frames, and infrared sequences collected using Microsoft Kinect sensors.Depth frames and infrared sequences have 512 × 424 resolution, while RGB videos use 1920 × 1080 resolution.
  • Dataset Modalities: Each tracked human body is represented by 3D locations of 25 major body joints, with corresponding RGB and depth pixels provided.
  • Action Classes: 120 action categories comprise 82 daily, 12 health-related, and 26 mutual actions.
  • Action Classes: The extended dataset adds fine-grained hand and finger motions, object-related individual and mutual actions, and classes sharing similar postures or objects.Examples include making an OK sign, counting money, wielding a knife toward another person, and taking something from a bag.
  • Subjects and Collection: The collection includes 106 subjects from 15 countries, ages 10–57, and heights from 1.3m to 1.9m.
  • Subjects and Collection: Samples are captured across 32 setups using three horizontal camera views, varied camera heights and distances, and repeated performances toward left and right cameras.The horizontal camera angles are −45°, 0°, and +45°.

3.2 Benchmark Evaluations

The benchmark defines standardized classification evaluations using accuracy percentages. It separates subjects for cross-subject testing and collection setups for cross-setup testing while documenting camera geometry.

  • Evaluation Criteria: The benchmark reports classification accuracy in percentage for two defined action-classification evaluation criteria.
  • Cross-Setup Evaluation: Cross-setup evaluation trains on samples from 16 even-numbered setups and tests on samples from 16 odd-numbered setups.
  • Cross-Setup Evaluation: Table 3 reports each collection setup’s camera height and distance to the subjects.

4 APSR FRAMEWORK FOR ONE-SHOT 3D ACTION RECOGNITION

The APSR framework addresses one-shot 3D action recognition by using action–body-part semantic relevance to emphasize informative body-part features. It generates spatio-temporal body-part representations, weights them using text-derived relevance scores, and applies the resulting representations to novel-class recognition.

  • One-shot scenario: One-shot recognition uses an auxiliary set of known classes and one exemplar per novel evaluation class, with remaining novel samples reserved for testing.The auxiliary and evaluation sets have no overlapping classes.
  • Recognition: During one-shot recognition, relevance scores guide weighted pooling for both exemplars and test samples, and cosine distance compares their aggregated representations.The test sample is pooled using the relevance score set associated with the exemplar’s class.
  • Feature generation: The feature generation network produces body-part features at each frame from skeletal joint coordinates while incorporating spatial and temporal context.It extends 2D ST-LSTM with bidirectional context passing to obtain more discriminative part features.
  • Semantic relevance: APSR estimates action–body-part relevance from the semantic similarity of their text descriptions using pre-trained Word2Vec embeddings.Multi-word descriptions are represented by averaging word embeddings, and negative relevance scores are reset to zero.
  • Semantic relevance: Figure 4 indicates that semantically relevant body parts can be reliably identified for novel actions.Examples associate feet with hopping and hands with wielding a knife toward another person.
  • Training: Semantic relevance scores weight classification losses so the network learns more discriminative features for body parts that matter more to each action.Each body-part feature is classified through a softmax classifier during end-to-end training.

5 EXPERIMENTS

The experiments evaluate state-of-the-art 3D action-recognition methods under cross-subject and cross-setup criteria, compare RGB, depth, and skeleton modalities, and assess APSR for one-shot recognition.

  • Benchmark evaluation: Experiments evaluate state-of-the-art 3D action-recognition methods using cross-subject and cross-setup criteria.The evaluation is conducted on the proposed dataset.
  • Data modalities: The study compares RGB, depth, and skeleton modalities and evaluates their fusion for activity analysis.The modalities provide complementary appearance and geometrical information.
  • One-shot recognition: The experiments also evaluate the proposed APSR framework for one-shot 3D action recognition.

5.1 Experimental Evaluations of 3D Action Recognition

Evaluations across modalities, training-set sizes, action types, and methods show that recognition benefits from complementary information and larger training sets, while view changes and fine-grained or similar actions remain challenging.

  • Evaluation across modalities: Cross-setup performance is weaker than cross-subject performance for RGB and depth inputs, whereas 3D skeleton input generalizes better across views but remains prone to body-tracker errors.RGB and depth methods are affected by changing camera viewpoints and backgrounds; skeleton representations are more view-invariant.
  • Training-data size: More training samples improve action-recognition accuracy across evaluated methods and modalities; FSNet rises from 40.6% with 20% of training samples to 62.4% with the full set.The increase is observed consistently across the six example methods and different data modalities.
  • Action-level analysis: Actions with significant motions and discriminative posture patterns are more accurately recognized, including “walk apart from each other” and “walk towards each other.”Both actions appear among the top 10 accurate classes across the three modalities and all evaluated methods.
  • Action-level analysis: Object-related actions can be confused when modalities omit object appearance, while depth distinguishes similar object shapes and skeleton data struggles with fine-grained hand and finger motions.Skeleton-only methods confuse object interactions and fine-grained gestures; RGB can confuse “open a box” with “fold paper,” whereas depth better represents 3D object shape.
  • Action-level analysis: Actions with highly similar motions and appearances, such as “take off a shoe” and “put on a shoe,” remain difficult to distinguish across modalities and methods.The same pair is reported as a difficult confusion case in modality- and method-level analyses.
  • Method-level analysis: Skeleton-based methods frequently confuse object-related and fine-grained hand or finger actions, while FSNet also struggles with mutual actions when separately extracted person features are averaged.The reported FSNet limitation concerns insufficient representation of interaction patterns between two people.

5.2 Experimental Evaluations of One-Shot Recognition

The paper evaluates one-shot 3D action recognition by learning representations from auxiliary classes and comparing pooling, fully connected, attention, and APSR approaches. APSR performs best on novel classes, while reducing auxiliary training data weakens generalization.

  • Experimental setup: One-shot recognition uses 100 auxiliary action classes for feature-generation training and the remaining 20 classes for evaluation.The evaluation targets novel action categories absent from the auxiliary training set.
  • Compared methods: Four approaches are compared: Average Pooling, Fully Connected, Attention Network, and the proposed APSR framework.The methods differ in how body-part features are aggregated or weighted for comparing test videos with exemplars.
  • Results: APSR achieves the best results among the compared one-shot methods, indicating generalization capability on novel action categories.The attention network performs worse than average pooling, possibly because its attention is trained without the novel actions.
  • Training-set analysis: Using fewer auxiliary classes and samples lowers one-shot recognition accuracy and weakens generalization to novel classes.The feature-generation network is otherwise trained on an auxiliary set containing about 100 thousand videos.

5.3 Discussions

The dataset supports research across modalities, fusion, transfer, robustness, and interaction settings because it combines varied subjects, environments, views, and action types. It also enables specialized studies such as mutual, early, and one-shot recognition.

  • Research scope: The dataset’s high variability facilitates applying, adapting, developing, and evaluating learning-based techniques for future human-activity analysis.The stated variability includes subjects, environments, camera views, and action categories.
  • Data modalities: The dataset provides depth, 3D skeleton, RGB, and infrared modalities for investigating modality-specific action-recognition algorithms.The modalities have different data structures and application advantages.
  • Data modalities: Complementary appearance and 3D geometrical information supports research on heterogeneous feature fusion across modalities.The dataset can be used to identify modality strengths and investigate fusion techniques.
  • Learning and transfer: Its large and diverse sample collection can support deep-network pre-training to address over-fitting on smaller RGB+D datasets.The paper expects initialization on this dataset to improve generalization to other relatively small datasets.
  • Robustness evaluation: Variation across 106 subjects, 96 backgrounds, and 155 camera views enables cross-subject, cross-environment, and cross-view evaluation.These settings support analysis of robustness to subject, background, illumination, and viewpoint variation.
  • Specialized tasks: The dataset contains 25 thousand two-person mutual-action videos spanning 26 interaction classes, supporting human-human interaction analysis.It also supports research on real-time skeleton-based early recognition and one-shot recognition of novel classes.

6 CONCLUSION

The paper introduces a large-scale RGB+D action-recognition dataset with 114,480 samples, 120 action classes, and highly varied camera settings, and proposes APSR for one-shot 3D action recognition. The dataset’s scale supports data-driven learning, while experiments report promising APSR results on novel classes.

  • Dataset contribution: 114,480 video samples from 120 action classes are collected in highly variant camera settings for large-scale RGB+D action recognition.The dataset is substantially larger and more varied than existing datasets according to the paper.
  • One-shot recognition: The proposed APSR framework addresses one-shot 3D action recognition, with experiments showing promising results on novel action classes.The conclusion presents APSR as a second contribution alongside the dataset.
  • Implications: The dataset’s scale facilitates applying data-driven learning methods to RGB+D action recognition.The conclusion links the large collection to the use of data-driven methods.
Loading 1905.04757v2…