Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn

I. INTRODUCTION

• 3D skeletons provide a compact, high-level representation of human dynamics and are robust to illumination, cluttered backgrounds, and camera motion.The introduction presents these properties as motivation for skeleton-based action analysis.Source: §I · p. 1
• Prior approaches are limited by shallow, dataset-dependent hand-crafted features,, –, RNN overfitting risk with insufficient data, and CNN encodings that are complicated or dataset-dependent and translation-scale variant,.The introduction describes JTM as potentially losing information when projecting 3D skeletons into 2D images, while Du et al.’s encoding may be affected by translation and action scale.Source: §I · p. 1
• The proposed mapping converts 3D skeleton videos into color images while resisting translation and scale changes and remaining dataset-independent.The paper says this is intended to avoid translation- and scale-related influence on the mapping.Source: §I · pp. 1–2
• A multi-scale deep CNN is proposed to enhance the method’s frequency adjustment ability, and pre-trained AlexNet, VGGNet, and ResNet can be fine-tuned on skeleton images.The paper presents fine-tuning as useful when annotated skeleton videos are insufficient and as a way to avoid training millions of parameters from scratch.Source: §I · pp. 1–2
• The framework is extended to 2D skeleton video recognition, which the introduction identifies as lacking effective recognition methods despite increasingly accurate 2D pose estimates –.The paper reports that its method also works well on 2D skeleton data.Source: §I · pp. 1–2
• The method achieves state-of-the-art results on NTU RGB-D, UTD-MHAD, MSRC-12, and G3D, with additional component analyses reported.The introduction also claims the approach works well on 2D skeleton data.Source: §I · p. 2

• Hand-crafted spatial, geometric, and key-pose features for 3D skeleton recognition,, – are shallow and dataset-dependent, limiting performance.The authors refer readers to for a broader summary of hand-crafted features.Source: §II · p. 2
• Recurrent approaches include hierarchical skeleton-part modeling, salient-motion differential recurrence, co-occurrence-regularized deep LSTMs, and tree traversal with reliability gating for noise and occlusion.Source: §II · p. 2
• CNN methods encode skeleton sequences as color-coded Joint Trajectory Maps or images of temporal and spatial structure, while effective 3D-to-CNN representation remains open.JTM combines spatial configuration and joint-trajectory dynamics in texture images; Du et al. encode temporal dynamics in column changes and each frame’s spatial structure as a column.Source: §II · p. 2
• Deep CNN methods have also estimated reasonably accurate 2D skeleton joints from RGB videos –.Source: §II · p. 2
• Transfer-learning work finds that pretrained CNN features can boost new recognition tasks,, alongside influential ImageNet architectures: AlexNet, VGGNet, and residual networks.The passage notes AlexNet’s ImageNet classification leap, VGGNet’s 2014 challenge results, and residual structure enabling hundreds-layer CNNs to be trainable.Source: §II · p. 2
• This journal version extends the authors’ ICMEW conference paper, which placed third in the “Large Scale 3D Human Activity Analysis Challenge in Depth Videos,” with improved performance and further experiments and analysis.Source: §II · p. 2

III. METHOD

• The framework combines translation-scale invariant image mapping, multi-scale CNN classification, and data augmentation for 3D skeletons.Fig. 1 depicts mapping 3D skeleton videos to action images before classification with multi-scale CNNs.Source: §III · p. 2, Figure 1 · p. 3

A. Translation-scale invariant image mapping

• Skeleton joints are grouped into two arms, two legs, and a trunk, concatenated by physical connections, then mapped to action-image RGB channels as x, y, and z coordinates.Each image row represents one joint across the sequence’s frames; concatenating the joints yields the action image.Source: A. Translation-scale invariant image… · p. 2
• Du et al. normalize using coordinate extrema from the training dataset, making the encoding dataset-dependent, translation-variant, and unable to guarantee scale invariance.The same action can map differently when performed at another position or by subjects with different scales.Source: A. Translation-scale invariant image… · pp. 2–3
• The proposed mapping addresses these encoding problems by normalizing each skeleton sequence using its own coordinate extrema.The method is presented in Equation 2.Source: A. Translation-scale invariant image… · p. 3
• Sequence-relative normalization makes the mapping translation-invariant because it uses each sequence’s minimum coordinates rather than dataset-wide minima.The paper illustrates this property in Fig. 9.Source: A. Translation-scale invariant image… · p. 3
• Normalizing each sequence by its coordinate variation removes scale changes while preserving relative scale across x, y, and z; sequence-specific extrema also make the mapping dataset-independent.The normalization is isometric across the three coordinate axes.Source: A. Translation-scale invariant image… · p. 3

B. Action recognition through multi-scale CNN

• The multi-scale CNN can be built on fully convolutional pretrained AlexNet, VGGNet, or ResNet models, as shown in Fig. 4.Source: B. Action recognition through… · p. 3, Figure 4 · p. 4
• Different input resolutions produce different frequency variations in skeleton images, and multi-frequency inputs can provide rich activity-recognition cues.Source: B. Action recognition through… · p. 4
• To reduce parameters, the model shares fully convolutional weights across resolutions, then globally pools their feature maps to produce equal-sized feature vectors.Source: B. Action recognition through… · p. 4
• For regularization, softmax loss is applied to each resolution's output and their average; training uses multinomial logistic loss.Source: B. Action recognition through… · p. 4

C. Data augmentation

• The augmentation strategies for encoded 3D skeleton images are random coordinate rotation, Gaussian noise, and video cropping.The paper introduces them as ways to augment the dataset and improve classification performance.Source: C. Data augmentation · p. 4
• 3D coordinates are randomly rotated from −30o to 30o along the x, y, and z axes; Fig. 5 illustrates x-axis settings.The illustrated settings are −30o, −15o, 0o, 15o, and 30o.Source: C. Data augmentation · p. 4, Figure 5 · p. 4
• Gaussian noise uses θ = 0 and σ = 0.01, while video crops randomly retain a range of [0.7, 1] at random video locations.Source: C. Data augmentation · p. 4

IV. IMPLEMENTATION DETAILS

• Training uses momentum 0.9, weight decay 0.0004, and AlexNet, VGGNet, or ResNet initialization, with learning rate 0.001 for eight epochs then divided by 10 every five.The implementation specifies stochastic gradient descent with momentum; the learning rate is fixed for the first eight epochs.Source: §IV · p. 4
• The implementation uses Caffe on an NVIDIA Tesla Titian X GPU.Source: §IV · p. 4

• The evaluation covers NTU RGB+D, UTD-MHAD, MSRC-12 Kinect Gesture, and G3D, comparing final recognition results with reported state-of-the-art results on the same datasets.Source: §V · p. 4

A. NTU RGB+D Dataset

• NTU RGB-D contains more than 56,000 sequences, 4 million frames, 60 actions, and 40 subjects, with multiple camera views and substantial intra-class and viewpoint variation.Subjects range from 10 to 35 years old; views include a front view, two side views, and left and right 45-degree views.Source: A. NTU RGB+D Dataset · p. 4
• Evaluation uses cross-subject and cross-view protocols, and the training dataset is augmented twofold.Cross-view training uses cameras 2 and 3, with camera 1 reserved for testing; cross-subject training uses the specified subject split, with remaining subjects reserved for testing.Source: A. NTU RGB+D Dataset · p. 4
• For samples containing two people, their skeleton coordinates are concatenated and represented in one image.The paper illustrates one-person and two-person skeleton images in Fig. 6.Source: A. NTU RGB+D Dataset · p. 4, Figure 6 · p. 5
• The method outperforms current state-of-the-art methods by 12% in cross-subject evaluation and 11% in cross-view evaluation.The paper reports the best performance in both evaluations.Source: Table I · p. 5, A. NTU RGB+D Dataset · p. 5

B. UTD-MHAD

• UTD-MHAD combines data captured by a Microsoft Kinect camera and a wearable inertial sensor.Source: B. UTD-MHAD · p. 5
• After three corrupted sequences were removed, UTD-MHAD contained 861 sequences of 27 actions performed by 8 subjects, each repeating every action 4 times.Source: B. UTD-MHAD · p. 5
• UTD-MHAD covers sport actions, hand gestures, daily activities, and training exercises.Source: B. UTD-MHAD · p. 5
• UTD-MHAD uses a cross-subject protocol, training on subjects 1, 3, 5, and 7 and testing on 2, 4, 6, and 8, as in.Source: B. UTD-MHAD · p. 5

C. MSRC-12 Kinect Gesture Dataset

• MSRC-12 is a Kinect-captured 3D skeleton gesture/action dataset with 594 sequences, 12 gestures, 30 subjects, and 6,244 gesture instances.Source: C. MSRC-12 Kinect Gesture Dataset · p. 5
• MSRC-12 uses odd-numbered subjects for training and even-numbered subjects for testing.Source: C. MSRC-12 Kinect Gesture Dataset · p. 5

D. G3D Dataset

• G3D targets real-time action recognition in gaming scenarios and contains 10 subjects performing 20 gaming actions.Source: D. G3D Dataset · p. 6
• The G3D split assigns the first four subjects to training, the fifth to validation, and the remaining five to testing, following.Source: D. G3D Dataset · p. 6
• A performance comparison on G3D is presented in the supplied table.Source: Table IV · p. 6

A. Effect of our mapping method

• For a fair comparison, all encoded action images were fine-tuned on AlexNet, and Wang et al.'s result was quoted to avoid hyperparameter-setting effects.Source: A. Effect of our mapping method · p. 6
• Wang et al.'s results used data augmentation, and the paper also reports results for its method with data augmentation.Source: A. Effect of our mapping method · p. 6
• The mapping method outperforms on all datasets, which the paper attributes to its translation-scale property.The passage says the method's margin is clear but provides no numerical results.Source: A. Effect of our mapping method · p. 6
• The method outperforms on most datasets, except G3D, and the paper describes its advantage as obvious.Source: A. Effect of our mapping method · p. 6

B. Effect of our multi-scale architecture

• The multi-scale network structure significantly improves performance across the compared pre-trained CNNs.The comparison is reported in Table VI.Source: B. Effect of our multi-scale… · p. 6

C. Adapted to 2D skeleton

• With missing coordinates set to 0, 2D skeleton results are only a little worse than 3D skeleton results.The authors describe this as the first such experiment to their knowledge and report that the method works well on 2D skeleton data; results are in Table VII.Source: C. Adapted to 2D skeleton · p. 6
• Performance with 1D skeleton data decreases sharply.The 1D results are reported in Table VII.Source: C. Adapted to 2D skeleton · p. 6

VII. CONCLUSION AND FUTURE WORK

• The method outperforms state-of-the-art methods by a large margin on NTU RGB-D, UTD-MHAD, and MSRC-12.Source: §VII · p. 6
• The proposed approach combines translation-scale invariant image mapping with multi-scale deep CNNs for skeleton-based action recognition.Source: §VII · p. 6
• The authors extend their method to 2D skeleton-based video recognition and report that it performs well.Source: §VII · p. 6