Source-linked AI summary
Exploiting deep residual networks for human action recognition from skeletal data
Huy-Hieu Pham, Louahdi Khoudour, Alain Crouzil, Pablo Zegers, Sergio A. Velastin
TL;DR
The paper addresses accurate human action recognition from skeletal data amid challenges in real videos and limitations of existing representations and architectures. It converts 3D joint coordinates into RGB images and learns them with a novel deep ResNet architecture. The method achieves state-of-the-art performance on three benchmark datasets while requiring less computation.
Problem
Human action recognition remains challenging in realistic videos, while 2D data lacks 3D structure and existing approaches can be computationally complex.
Method
The method encodes 3D skeletal-joint coordinates as RGB images and applies a novel deep ResNet architecture for end-to-end action recognition.
Results
State-of-the-art recognition performance was achieved on MSR Action 3D, KARD, and NTU-RGB+D while requiring less computation.
Takeaways & Limitations
The framework supports effective skeletal action recognition across three established datasets with reduced computation requirements.
Takeaways & Limitations
The paper notes that 2D data lacks 3D structure from the scene, limiting approaches based on 2D inputs.
Abstract
from arXiv · showhide
The computer vision community is currently focusing on solving action recognition problems in real videos, which contain thousands of samples with many challenges. In this process, Deep Convolutional Neural Networks (D-CNNs) have played a significant role in advancing the state-of-the-art in various vision-based action recognition systems. Recently, the introduction of residual connections in conjunction with a more traditional CNN model in a single architecture called Residual Network (ResNet) has shown impressive performance and great potential for image recognition tasks. In this paper, we investigate and apply deep ResNets for human action recognition using skeletal data provided by depth sensors. Firstly, the 3D coordinates of the human body joints carried in skeleton sequences are transformed into image-based representations and stored as RGB images. These color images are able to capture the spatial-temporal evolutions of 3D motions from skeleton sequences and can be efficiently learned by D-CNNs. We then propose a novel deep learning architecture based on ResNets to learn features from obtained color-based representations and classify them into action classes. The proposed method is evaluated on three challenging benchmark datasets including MSR Action 3D, KARD, and NTU-RGB+D datasets. Experimental results demonstrate that our method achieves state-of-the-art performance for all these benchmarks whilst requiring less computation resource. In particular, the proposed method surpasses previous approaches by a significant margin of 3.4% on MSR Action 3D dataset, 0.67% on KARD dataset, and 2.5% on NTU-RGB+D dataset.
1. Introduction
The paper targets accurate, practical human action recognition from skeletal data by combining 3D skeleton representations with deep residual CNNs. It addresses limitations of 2D structure, computational cost, and underuse of very deep architectures, reporting state-of-the-art results on three benchmarks.
- Motivation: Skeletal data preserves 3D human-motion structure while offering lower-dimensional representations than RGB or RGB-D inputs.This can make recognition systems simpler and faster.
- Motivation: 2D and single-modality RGB approaches lack 3D scene structure and are insufficient for challenging realistic videos.The paper identifies viewpoint, occlusion, and lighting conditions as challenges in human action recognition.
- Motivation: Existing CNN-based action-recognition methods had not fully exploited very deep architectures and often used computationally expensive RGB, depth, or RGB-D sequences.Large input dimensions can make models more complex, slower, and less practical for large-scale or real-time applications.
- Method: The framework converts 3D skeletal-joint coordinates into RGB images that encode spatial-temporal motion structure for end-to-end D-CNN learning.The resulting color images are designed to be compatible with deep learning networks.
- Method: The proposed method uses a novel deep ResNet architecture and building unit to learn image features and recognize skeletal actions.The architecture is designed to be easier to optimize and better prevent overfitting than the original ResNet model.
- Results: State-of-the-art performance was achieved on MSR Action 3D, KARD, and NTU-RGB+D while requiring less computation.The paper also reports effectiveness in preventing overfitting and reducing degradation during training of very deep networks.
2. Related Work
Prior skeleton-based action-recognition methods use engineered features with graphical models, shape analysis, RNN-LSTMs, or CNN–RNN combinations. The paper motivates deep ResNets as an alternative that learns from simple skeleton representations while addressing deep-network training challenges.
- Skeleton-based action recognition: Skeleton-based methods mainly combine hand-crafted features with probabilistic graphical models, including HMMs, LDA, CRFs, or Fourier Temporal Pyramids.These approaches model spatio-temporal representations or temporal dynamics before predicting action labels.
- Skeleton-based action recognition: Shape-analysis approaches represent actions as sequences of skeletal shapes and classify them using methods such as KNN or SVM.Some methods analyze skeletal sequences on statistical shape manifolds, including Kendall’s shape manifold.
- Limitations of prior methods: Existing approaches have reported good performance but may require substantial feature engineering, sequence segmentation and alignment, or increased computational time as input features grow.FTP-based approaches also cannot globally capture action temporal sequences, while RNNs may overfit when input features are insufficient.
- Skeleton-based action recognition: RNN-LSTM methods learn motion features from skeletal time series and can store and access long-range temporal context.Some approaches combine CNN visual feature extraction with RNN-LSTM motion modeling in a unified framework.
- D-CNNs for visual recognition: Deep CNNs improve image-recognition performance but are difficult to train because of vanishing gradients and degradation in deeper networks.Residual Networks use shortcut connections to improve information flow across layers and reduce degradation.
- D-CNNs for visual recognition: Unlike prior CNN-based studies that emphasize skeletal representations with simple CNNs, this paper exploits deep D-CNNs and designs a ResNet framework for simple skeleton-based representations.The proposed approach transforms skeleton sequences into RGB images and reports state-of-the-art performance on MSR Action 3D, KARD, and NTU-RGB+D.
3. Method
The method converts skeleton sequences into RGB image representations that preserve spatial-temporal motion information, then applies deep residual networks for action recognition. It also introduces an improved residual building block intended to support direct forward and backward information propagation.
- 3.1. Encoding skeletal data into RGB images: The representation preserves static postures and temporal dynamics by encoding skeleton-sequence information into 2D image structures.The resulting data are converted into 3D tensors and supplied as model inputs.
- 3.1. Encoding skeletal data into RGB images: Skeleton sequences are transformed into RGB images so CNNs can learn spatial-temporal features for action classification.The 3D joint coordinates are normalized into [0, 255] and treated as RGB components.
- 3.1. Encoding skeletal data into RGB images: Joints are rearranged according to human-body structure, dividing each frame into two arms, a trunk, and two legs.Within each part, joints are concatenated according to physical connections before the parts are arranged sequentially.
- 3.2. Deep residual network: ResNet units combine convolutional transformations with shortcut connections that connect input features directly to outputs.The residual formulation uses nonlinear transformations alongside identity mappings.
- 3.3. An improved residual network for skeleton-based action recognition: The proposed improved residual block replaces ReLU layers after element-wise additions with identity mappings to enable direct forward and backward propagation.The authors present this modification as addressing signal-propagation limitations in the original design.
4. Experiments
The study evaluates the proposed deep learning framework on three 3D skeleton benchmarks using their established evaluation criteria. It describes dataset characteristics, augmentation, and end-to-end ResNet implementation details.
- Experimental setup: The framework is evaluated on MSR Action 3D, KARD, and NTU-RGB+D using evaluation criteria from the original dataset papers.The experiments first present datasets and criteria, then describe augmentation and implementation.
- Datasets: KARD contains 540 skeleton sequences from 10 subjects performing 18 actions, with three proposed train-test experiments per subset.The experiments use one-third, two-thirds, or one-half of each subset for training, with the remainder used for testing.
5. Experimental results
The proposed ResNet models outperform prior approaches across MSR Action 3D, KARD, and both NTU-RGB+D evaluation protocols under matched evaluation criteria. The strongest reported gains are 3.4% on MSR Action 3D, +2.0% on Cross-Subject NTU-RGB+D, and +2.5% on Cross-View.
- MSR Action 3D: 3.4% higher accuracy than the best previous published result is achieved on MSR Action 3D.The proposed 44-layer ResNet reaches 99.9% on AS1, 99.8% on AS2, 100% on AS3, and 99.9% average accuracy.
- KARD: The proposed 44-layer ResNet achieves the best reported results on KARD and outperforms prior state-of-the-art methods on the whole dataset.The passage reports the model selection and comparative outcome but does not provide the numerical KARD margin here.
- NTU-RGB+D: +2.0% over the best published result is reported for the Cross-Subject evaluation on NTU-RGB+D.The comparison is made against the result reported by Li et al. (2017).
- NTU-RGB+D: +2.5% over the state-of-the-art accuracy reported by Kim and Reiter (2017) is reported for Cross-View NTU-RGB+D.The proposed method outperforms various state-of-the-art approaches in both Cross-Subject and Cross-View evaluations.
6. Discussion
The discussion examines accuracy, overfitting, degradation, representation choices, and computational efficiency. The proposed architecture reduces overfitting and degradation relative to the original ResNet while supporting fast prediction.
- Discussion scope: Accuracy, overfitting prevention, computational efficiency, and degradation are the principal discussion criteria for the proposed method.These aspects are evaluated experimentally across the benchmark datasets.
- Overfitting and degradation: Degradation effects are reduced in both training and test phases, and proposed 56-layer networks outperform 20- and 32-layer networks in the reported comparison.The discussion contrasts this behavior with the original ResNet architecture.
- Overfitting and degradation: Degradation depends considerably on dataset size, and the 110-layer network consequently produces higher errors than several other networks.The difference between training and test error is used to assess overfitting prevention.
- Overfitting and degradation: The proposed ResNet architectures reduce overfitting compared with the original architecture across three action benchmarks.The authors associate this result with batch normalization before convolutional layers and dropout in each ResNet unit.
- Pipeline: The pipeline has three stages: encoding skeletons into RGB images, supervised training, and prediction.The stages are illustrated for the NTU-RGB+D Cross-View evaluation with Proposed-ResNet-56.
- Computational efficiency: The method takes 0.128s per skeleton sequence for encoding and classification during prediction, supporting the authors’ real-time application claim.Training takes 7.83 × 10^-3s per skeleton sequence in the stated single-GPU setup.
7. Conclusion and future work
The paper concludes that encoding skeleton sequences as RGB images and learning them with a novel ResNet architecture yields strong human action recognition performance with reduced computational resource requirements. Future work targets richer joint relationships and newer ResNet-based convolutional architectures.
- Conclusion: State-of-the-art performance is achieved on three established datasets while requiring less computation resource.The conclusion presents this as the principal empirical outcome of the framework.
- Conclusion: The proposed framework combines a spatio-temporal 3D-motion representation with a deep learning model for skeletal action recognition.Skeleton sequences are encoded into RGB images before learning with the proposed ResNet architecture.
- Future work: Future encoding work will exploit Euclidean distance and orientation relationships between joints.These relationships are proposed as extensions of the skeleton encoding method.
- Future work: Future architecture work will explore ResNet-based D-CNNs including Inception-ResNet-v2 and DenseNet.The authors state that preliminary results are encouraging.
- Conclusion: The study aims to support further research on very deep networks for human action recognition.The authors describe the work as opening a new research direction for exploiting deep networks in this setting.
A. List of action classes from the NTU-RGB+D dataset
The NTU-RGB+D dataset contains 60 actions captured by Kinect v2 sensors, spanning individual activities, object interactions, and person-to-person interactions.
- The NTU-RGB+D dataset contains 60 different actions captured by Kinect v2 sensors.
- Individual activities include drinking, eating, brushing, dropping, picking up, throwing, sitting, standing, clapping, reading, and writing.
- The dataset also includes phone use, typing, pointing, taking selfies, checking time, bowing, saluting, and other upper-body gestures.
- It covers bodily reactions and movements such as sneezing, coughing, staggering, falling, touching body regions, vomiting, and fanning oneself.
- Interpersonal actions include punching, kicking, pushing, patting, pointing, hugging, giving objects, touching pockets, handshaking, and walking together or apart.
B. Network structures
The paper describes stacked residual-network architectures for action classification, varying depth from 20 to 110 layers and using residual units with increasing filter widths.
- The proposed network family stacks residual building units to form architectures with 20, 32, 44, 56, and 110 layers.
- Residual units use the sequence BN-ReLU-Conv.-BN-ReLU-Dropout-Conv. and are configured with 16, 32, or 64 filters.
- Each network ends with a fully connected layer whose n units equal the number of action classes.
- The baseline architectures begin with a 3x3 convolution using 16 filters, followed by batch normalization and ReLU activation.
- The architectures increase channel capacity across stages, using residual units with 32 and 64 filters in deeper sections.