Source-linked AI summary
Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn
Bo Li, Mingyi He, Xuelian Cheng, Yucheng Chen, Yuchao Dai
TL;DR
The paper addresses skeleton-based action recognition, including the limited availability of effective methods for 2D skeleton video data. It maps skeleton videos to translation- and scale-invariant color images, applies a multi-scale deep CNN with transferable pretrained models, and reports state-of-the-art results on several benchmarks while performing well on 2D data.
Problem
Skeleton-based action recognition needs effective handling of skeleton video data, while effective methods for 2D skeleton video recognition remain limited.
Method
The method maps 3D skeleton videos to dataset-independent, translation-scale invariant color images and classifies them with a multi-scale deep CNN built from fine-tuned pretrained models.
Results
The method achieves state-of-the-art results on NTU RGB-D, UTD-MHAD, MSRC-12, and G3D, and performs well on 2D skeleton-based video recognition.
Takeaways & Limitations
The proposed image mapping and CNN framework is applicable across multiple skeleton-action benchmarks and extends to 2D skeleton video recognition.
Takeaways & Limitations
The conventional coordinate-to-image normalization is dataset dependent and translation-scale variant because it uses training-set coordinate statistics.
Abstract
from arXiv · showhide
This paper presents an image classification based approach for skeleton-based video action recognition problem. Firstly, A dataset independent translation-scale invariant image mapping method is proposed, which transformes the skeleton videos to colour images, named skeleton-images. Secondly, A multi-scale deep convolutional neural network (CNN) architecture is proposed which could be built and fine-tuned on the powerful pre-trained CNNs, e.g., AlexNet, VGGNet, ResNet etal.. Even though the skeleton-images are very different from natural images, the fine-tune strategy still works well. At last, we prove that our method could also work well on 2D skeleton video data. We achieve the state-of-the-art results on the popular benchmard datasets e.g. NTU RGB+D, UTD-MHAD, MSRC-12, and G3D. Especially on the largest and challenge NTU RGB+D, UTD-MHAD, and MSRC-12 dataset, our method outperforms other methods by a large margion, which proves the efficacy of the proposed method.
1 Northwestern Polytechnical University, China
The paper addresses limitations in skeleton-based action recognition by mapping skeleton videos to images and classifying them with multi-scale deep CNNs. The framework is evaluated on 3D and 2D skeleton data, with reported state-of-the-art benchmark results.
- Motivation: 3D skeletons provide compact, robust representations for action analysis, but effective CNN-compatible representations remain an open problem.Skeletons are described as robust to illumination, clustered backgrounds, and camera motion; prior CNN encodings have representation limitations.
- Motivation: Hand-crafted skeleton features are shallow and dataset-dependent, limiting their performance.
- Proposed framework: The proposed framework combines translation-scale invariant image mapping with a multi-scale deep CNN classifier.The mapping converts 3D skeleton videos into color images, while the classifier is designed for frequency adjustment.
- Proposed framework: Pre-trained AlexNet, VGGNet, and ResNet models can be fine-tuned for skeleton images despite their difference from natural images.The fine-tune strategy is especially valuable when annotated skeleton videos are insufficient and avoids retraining millions of parameters from scratch.
- Experiments: The method achieves state-of-the-art results on NTU RGB-D, UTD-MHAD, MSRC-12, and G3D, and also works well on 2D skeleton data.The paper reports extensive component analysis alongside the benchmark and 2D-skeleton experiments.
II. RELATED WORK
Prior work uses hand-crafted features, recurrent networks, CNN encodings, and transfer learning for skeleton recognition. The paper positions its approach against limitations in feature depth, representation design, and dataset dependence.
- Hand-crafted features: Hand-crafted descriptors capture spatial-temporal information but are generally shallow and dataset-dependent.Examples include spatial descriptors, geometric descriptors, and key-pose representations.
- Recurrent methods: RNN-based methods extract temporal and contextual information, but may overemphasize temporal information and overfit when training data is insufficient.
- CNN-based methods: CNN approaches represent skeleton sequences as texture or image-like inputs, but effective 3D skeleton representation for deep CNNs remains an open problem.JTM uses three color-encoded texture images, while another method encodes temporal dynamics in columns and spatial structure by frame.
- 2D skeleton recognition: Recent work estimates relatively accurate 2D skeleton joints from RGB videos, while effective recognition methods for 2D skeleton sequences remain limited.
- Transfer learning: Transfer learning shows that pre-trained deep CNN features can be transferred to new classification or recognition problems and boost performance.The paper cites ImageNet-trained CNNs and related transfer-learning work as background for its fine-tuning strategy.
- Paper extension: This journal version extends an earlier conference paper and adds performance improvements, experiments, and analysis.The earlier work placed third in the Large Scale 3D Human Activity Analysis Challenge in Depth Videos.
III. METHOD
The framework consists of translation-scale invariant image mapping, a multi-scale deep CNN classifier, and data augmentation for 3D skeleton data.
- Framework components: The proposed framework has three parts: translation-scale invariant image mapping, multi-scale deep CNN classification, and data augmentation.A conceptual illustration of the framework is provided in Fig. 1.
A. Translation-scale invariant image mapping
The method maps skeleton videos into color action images using sequence-level translation-scale invariant normalization. This produces dataset-independent representations that preserve joint structure while reducing translation and scale effects.
- Skeleton organization: Each frame is organized into five body parts, whose physically connected joints are concatenated before forming the action-image representation.The five parts are two arms, two legs, and a trunk.
- Coordinate-to-image mapping: The mapping assigns the x, y, and z coordinates of each joint to the R, G, and B components of image pixels across frames.Concatenating all joints yields the action image for the original skeleton video.
- Limitations of prior normalization: Dataset-wide normalization is dataset dependent and cannot guarantee translation or scale invariance across sequences.It uses training-set extrema, so translated actions and actions performed by differently scaled subjects can produce variant images.
- Proposed normalization: The proposed mapping normalizes coordinates using each sequence’s minimum and maximum values, making the representation dataset independent.Sequence-specific extrema avoid dependence on statistics from an entire training dataset.
- Invariance properties: Sequence-relative normalization eliminates translation effects and scale changes while preserving relative scale across the x, y, and z axes.The method is described as translation invariant and scale invariant.
B. Action recognition through multi-scale CNN
The paper uses a multi-scale CNN to classify action images, building on fully convolutional pretrained networks. Multiple input resolutions provide frequency-varied activity cues while shared weights limit parameter growth.
- Architecture: The multi-scale CNN can be built on fully CNN-based pretrained models such as AlexNet, VGGNet, and ResNet.The architecture is designed for classification of the mapped skeleton images.
- Multi-scale motivation: Different input sizes generate different frequency variations in skeleton images when the convolution-kernel size is fixed.The multi-scale input is intended to include richer activity-recognition cues.
- Parameter sharing: Weights in the fully convolutional components are shared across resolutions, reducing model parameters.Global pooling converts corresponding feature maps into same-sized feature vectors.
- Training objective: Softmax loss is applied to each resolution-specific output and to their average to further regularize training.The network is trained with multinomial logistic loss.
- Classification output: The softmax output assigns class probabilities from the input of the softmax layer.The loss formulation uses N training samples and m classes, with k denoting each sample’s corresponding label.
C. Data augmentation
The paper augments encoded skeleton-image data with geometric, noise-based, and temporal cropping transformations to improve CNN classification.
- C. Data augmentation: The augmentation strategies include random 3D coordinate rotation, Gaussian noise, and video cropping.These transformations are applied after encoding 3D skeleton data as RGB images.
- C. Data augmentation: 3D coordinates are randomly rotated within [−30o, 30o] along the x, y, and z axes.Figure 5 illustrates mapping results for x-axis rotations of −30o, −15o, 0o, 15o, and 30o.
- C. Data augmentation: Gaussian noise is randomly added to 3D coordinates with θ = 0 and σ = 0.01.
- C. Data augmentation: Videos are randomly cropped over a range of [0.7, 1] at random locations.
IV. IMPLEMENTATION DETAILS
The network is trained with momentum-based stochastic gradient descent, pretrained CNN initialization, weight decay, and a scheduled learning rate.
- IV. IMPLEMENTATION DETAILS: The network uses stochastic gradient descent with momentum 0.9 and weight decay 0.0004.
- IV. IMPLEMENTATION DETAILS: Weights are initialized from pretrained AlexNet, VGGNet, and ResNet models.
- IV. IMPLEMENTATION DETAILS: The learning rate is fixed at 0.001 for the first 8 epochs and then divided by 10 every 5 epochs.
- IV. IMPLEMENTATION DETAILS: The implementation uses the Caffe CNN toolbox with an NVIDIA Tesla Titian X GPU.
V. EXPERIMENTS
Experiments evaluate the method on benchmark skeleton-action datasets using established protocols, including multi-person handling and NTU RGB-D comparisons against state-of-the-art methods.
- V. EXPERIMENTS: The evaluation covers NTU RGB+D, UTD-MHAD, MSRC-12 Kinect Gesture, and G3D benchmark datasets.Final recognition results are compared with reported state-of-the-art results on the same datasets.
- V. EXPERIMENTS: NTU RGB-D is described as the largest action recognition dataset, and the experiments adopt the train-test protocol from.
- V. EXPERIMENTS: NTU RGB-D contains more than 56 thousands sequences, 4 million frames, and 60 actions performed by 40 subjects.Its viewpoint and intra-class variations make the dataset challenging.
- V. EXPERIMENTS: The NTU evaluation uses cross-subject and cross-view protocols, with camera-based training and testing splits specified for cross-view evaluation.The training dataset is further augmented by 2 times.
- V. EXPERIMENTS: For NTU samples containing more than one person, the two persons’ coordinates are concatenated and represented in one image.Figure 6 presents an example of this representation.
- V. EXPERIMENTS: 12% in cross-subject evaluation and 11% in cross-view evaluation are the reported margins over current state-of-the-art methods.The paper states that its method achieves the best performance in both evaluations.
B. UTD-MHAD
UTD-MHAD is a multimodal Kinect-and-inertial-sensor dataset, while MSRC-12 provides 3D skeleton gesture data; the paper presents comparison and confusion-matrix evaluations for these datasets.
- B. UTD-MHAD: UTD-MHAD combines one Microsoft Kinect camera with one wearable inertial sensor.
- B. UTD-MHAD: UTD-MHAD contains 27 actions performed by 8 subjects, with each subject performing each action 4 times.After three corrupted sequences were removed, the dataset contained 861 sequences.
- B. UTD-MHAD: The UTD-MHAD actions span sports, hand gestures, daily activities, and training exercises.
- B. UTD-MHAD: Table II is identified as a performance comparison on the UTD-MHAD dataset.
- B. UTD-MHAD: The paper identifies confusion-matrix figures for UTD-MHAD and MSRC-12 and a performance-comparison table for MSRC-12.
- B. UTD-MHAD: MSRC-12 contains 594 sequences, 12 gestures, 30 subjects, and 6244 gesture instances from 3D skeleton data captured by a Kinect sensor.A cross-subject protocol uses odd subjects for training and even subjects for testing.
D. G3D Dataset
The G3D evaluation compares the proposed mapping with other skeleton-image mappings under a common AlexNet fine-tuning setup. The method outperforms across datasets but outperforms on most datasets except G3D.
- Dataset: G3D contains 10 subjects performing 20 gaming actions in a real-time gaming scenario.The evaluation uses subject-based training, validation, and testing partitions, as described in the dataset passage.
- Comparison: The G3D results are presented as a performance comparison against competing methods.The cited table is identified as a performance comparison on the G3D dataset.
- Experimental setup: All encoded action images were fine-tuned on AlexNet for the comparison, with Wang et al.'s result quoted from prior work.This setup was used to reduce the influence of hyper-parameter settings on the comparison.
- Results: The proposed mapping outperforms across all evaluated datasets by a clear margin.The authors attribute this result to the importance of translation-scale invariance.
- Results: The proposed method outperforms on most datasets but not on G3D.The paper states that its advantage is obvious overall despite this exception.
B. Effect of our multi-scale architecture
The multi-scale network improves performance, while the method also transfers to 2D skeletons with only a small degradation relative to 3D data. Performance decreases sharply for 1D skeleton data.
- Multi-scale architecture: The multi-scale network structure significantly improves performance across comparisons of pre-trained CNNs.The comparison includes different pre-trained CNN networks and reports an improvement from the multi-scale structure.
- Adaptation to 2D skeleton: The method performs well on 2D skeleton data, which is only slightly worse than 3D skeleton data.Missing coordinates in the 2D skeletons were set to 0.
- Adaptation to lower-dimensional skeletons: 1D skeleton data causes a sharp decrease in performance.The 1D results are also reported in Table VII.
- Overall evaluation: Experiments on NTU RGB-D, UTD-MHAD, and MSRC-12 report large-margin gains over state-of-the-art methods, while the framework also performs well for 2D recognition.The paper also reports experiments on G3D and extensive analyses of the method's properties.