Source-linked AI summary
Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks
Pichao Wang, Wanqing Li, Chuankun Li, Yonghong Hou
TL;DR
The paper tackles how to represent skeleton sequences for ConvNets, proposing color-encoded Joint Trajectory Maps from three orthogonal projections. The method fine-tunes existing ConvNets, fuses complementary JTM scores, and achieves state-of-the-art results on four benchmark datasets.
Problem
It remains unclear how to effectively represent skeleton sequences for deep neural networks, because hand-crafted features are shallow and dataset-dependent while RNNs may overemphasize temporal information.
Method
The method encodes joint-trajectory spatial configuration and dynamics into three color-based Joint Trajectory Maps, uses rotated skeletons for view variation, and classifies them with fine-tuned ConvNets and score fusion.
Results
The method achieved state-of-the-art recognition results on the NTU RGB+D, MSRC-12, G3D, and UTD-MHAD benchmark datasets.
Takeaways & Limitations
Three complementary orthogonal JTMs provide an effective image-based representation that allows existing image-trained ConvNets to classify skeleton sequences without retraining the networks afresh.
Abstract
from arXiv · showhide
Convolutional Neural Networks (ConvNets) have recently shown promising performance in many computer vision tasks, especially image-based recognition. How to effectively apply ConvNets to sequence-based data is still an open problem. This paper proposes an effective yet simple method to represent spatio-temporal information carried in $3D$ skeleton sequences into three $2D$ images by encoding the joint trajectories and their dynamics into color distribution in the images, referred to as Joint Trajectory Maps (JTM), and adopts ConvNets to learn the discriminative features for human action recognition. Such an image-based representation enables us to fine-tune existing ConvNets models for the classification of skeleton sequences without training the networks afresh. The three JTMs are generated in three orthogonal planes and provide complimentary information to each other. The final recognition is further improved through multiply score fusion of the three JTMs. The proposed method was evaluated on four public benchmark datasets, the large NTU RGB+D Dataset, MSRC-12 Kinect Gesture Dataset (MSRC-12), G3D Dataset and UTD Multimodal Human Action Dataset (UTD-MHAD) and achieved the state-of-the-art results.
I. INTRODUCTION
The paper addresses the unresolved problem of representing skeleton sequences for deep recognition by encoding spatio-temporal information into Joint Trajectory Maps and processing them with ConvNets.
- Hand-crafted skeleton features are shallow and dataset-dependent, while RNNs can overemphasize temporal information and overfit when training data is limited.
- Joint Trajectory Maps encode joint-trajectory spatial configuration and dynamics into three texture images through color encoding for ConvNet-based action recognition.The representation supports fine-tuning existing ImageNet-trained ConvNets without retraining the whole networks from scratch.
- The three complementary JTMs are processed by ConvNets, and late multiply score fusion improves final recognition accuracy.
- Skeleton rotation mimics multiple views and augments training data to address ConvNets’ lack of rotation invariance for cross-view recognition.
- The method achieved state-of-the-art recognition results on NTU RGB+D, MSRC-12, G3D, and UTD-MHAD benchmark datasets.
A. Skeleton-Based 3D Action Representation
Prior skeleton-based representations model joints, joint groups, or joint dynamics, while the proposed framework combines view rotation, three-plane JTMs, ConvNets, and score fusion.
- Skeleton-Based 3D Action Representation: Skeleton-based 3D action representations generally describe individual joints, groups of joints, or joint dynamics.
- Skeleton-Based 3D Action Representation: Joint representations capture spatial correlations using spatial, geometric, or key-pose descriptors, whereas joint-group methods seek discriminative subsets of joints.
- Skeleton-Based 3D Action Representation: Joint-dynamics methods model motion through projected trajectories, body-part descriptors, or temporal models of joint locations and derivatives.
- Skeleton-Based 3D Action Representation: The proposed framework rotates skeletons, constructs three orthogonal JTMs, trains three ConvNets, and fuses their scores for classification.
- Skeleton-Based 3D Action Representation: Skeleton rotation mimics multiple views and enlarges training data to support cross-view recognition and mitigate ConvNets’ limited view invariance.
B. Construction of JTMs
JTMs preserve action structure by encoding joint trajectories and their temporal motion into color across three orthogonal projections, while distinguishing body-part coordination.
- Construction of JTMs: An effective JTM should preserve joint or body-part spatial identity within the image.
- Construction of JTMs: A JTM should encode trajectory direction and speed to represent temporal evolution of joint motion.
- Construction of JTMs: JTMs should capture motion differences among body parts to reflect synchronization during an action.
- Construction of JTMs: The JTM accumulates a frame-wise encoding function, with hue, saturation, and brightness proposed to represent spatial-temporal motion patterns.
- Construction of JTMs: Skeleton trajectories are projected onto three orthogonal Cartesian planes, preserving spatial information while initially losing motion direction.
1) Joint Trajectory Maps:
Joint Trajectory Maps represent 3D skeleton motion as three 2D projections, encoding trajectories and temporal motion direction with color.
- Joint Trajectory Maps: Each frame records the 3D coordinates of all joints, and the skeleton trajectory aggregates these joint trajectories across the action.Joint trajectories are formed from successive frame-to-frame joint positions.
- Joint Trajectory Maps: The skeleton trajectory is projected onto three orthogonal Cartesian planes to form three Joint Trajectory Maps.The projections preserve spatial information while initially losing motion direction.
- Encoding Joint Motion Direction: Hue encodes temporal position along each joint trajectory, so color variations represent the direction of motion.The paper uses a jet colormap ranging from blue to red through cyan, yellow, and orange.
3) Encoding Body Parts:
The method uses distinct colormaps for body parts and modulates saturation and brightness by joint speed to encode coordination and motion magnitude as texture.
- Encoding Body Parts: Different colormaps distinguish coordinated motion among body parts, with the body divided into left, right, and middle groups.The left and right groups include the corresponding limbs; the middle group includes the head, neck, torso, and hip center.
- Encoding Motion Magnitude: Motion magnitude is encoded through saturation and brightness because larger joint speeds are expected to provide discriminative information.High-speed joints receive higher saturation and brightness, creating texture for ConvNets to learn.
- Encoding Motion Magnitude: Saturation encoding dilutes slow trajectories and saturates fast trajectories in the illustrated circle-drawing action.The example identifies knees and ankles as slow-motion trajectories and part of the circle as fast motion.
- Encoding Motion Magnitude: Brightness modulation makes motion variation more textured, and combining saturation and brightness enriches the final JTM texture.The example shows texture becoming apparent after brightness modulation and richer in the final encoding.
C. ConvNets Training
The three JTMs are processed by individually fine-tuned AlexNet ConvNets, using an ImageNet-pretrained image-classification setup rather than training the networks entirely afresh.
- ConvNets Training: Three ConvNets are fine-tuned individually after constructing JTMs on the three orthogonal image planes, with each network using AlexNet.The training uses mini-batch stochastic gradient descent with momentum and weight decay; images are resized to 256 × 256.
- ConvNets Training: The learning rate is 10−2 for training from scratch and 10−3 for fine-tuning ImageNet-pretrained models, followed by scheduled decreases.Training lasts 100 cycles, with learning-rate reductions every 30 cycles.
D. Multiply Score Fusion
At test time, the three JTM-specific ConvNet outputs are combined by element-wise multiplication, and the maximum resulting score determines the recognized action.
- Multiply Score Fusion: Each testing skeleton generates three JTMs that are fed into separate ConvNets before multiply score fusion combines their outputs.The score vectors are multiplied element-wise, and the index of the maximum score gives the class label.
IV. EXPERIMENTS
Experiments evaluated the encoding schemes, rotation, and fine-tuning choices across benchmark skeleton-action datasets. The results show that complementary JTMs, rotation, and ImageNet fine-tuning improve recognition.
- Experimental setup: Experiments evaluated encoding schemes, rotation, fine-tuning, and score fusion on four public benchmark datasets.The supplied experiment description names NTU RGB+D, MSRC-12, G3D, and UTD-MHAD.
- Different Encoding Schemes: The encoding methods progressively add spatio-temporal information, while the three complementary JTMs improve recognition through fusion.This evaluation was conducted on the G3D dataset using the schemes listed in Table I.
- Rotation: Almost 3.5 percentage points: rotation improved cross-view recognition on the NTU RGB+D and G3D datasets.Rotation also mimics multiple views and enlarges the training data, supporting use on small datasets.
- Fine-tuning vs. Training from Scratch: 5 percentage points on NTU RGB+D: fine-tuning improved recognition over training from scratch, while nearly doubling accuracy on the small G3D dataset.These comparisons used NTU RGB+D and G3D without rotation.
4) Comparison of Three Score Fusion Methods:
The paper compares multiply, average, and max score fusion, and reports consistent advantages for multiply fusion across four datasets. It also reports strong NTU RGB+D results and identifies a projection-related confusion on MSRC-12.
- Comparison of Three Score Fusion Methods: Multiply score fusion consistently outperformed average and max score fusion across the four evaluated datasets.The authors interpret this as evidence that the three JTMs provide complementary information and are likely statistically independent.
- NTU RGB+D Dataset: The NTU RGB+D evaluation used cross-subject and cross-view protocols under the same protocol as prior work.The dataset includes substantial intra-class and viewpoint variation.
- NTU RGB+D Dataset: The proposed method achieved the best results in both cross-subject and cross-view evaluation on NTU RGB+D.The comparison included hand-crafted features and deep learning methods.
- MSRC-12 Kinect Gesture Dataset: On MSRC-12, most actions were distinguished well, but “goggles” and “had enough” remained difficult to separate.The authors attribute their similar JTM appearance probably to the 3D-to-2D projection.
D. G3D Dataset
On G3D, the proposed method recognizes most actions well and outperforms LRBM, while its view-selection experiments find natural camera coordinates sufficient. The UTD-MHAD analysis also exposes a limitation: temporal normalization can make jogging and walking look similar.
- Dataset and recognition: G3D contains 20 gaming actions performed by 10 subjects, with the first four subjects used for training and the fifth for validation.
- Dataset and recognition: The proposed method recognizes most G3D actions well and outperforms LRBM.LRBM confuses several action pairs that the proposed method distinguishes more effectively, likely because JTMs preserve spatial information.
- Dataset and recognition: The UTD-MHAD dataset contains 861 sequences from 27 actions performed by eight subjects after removing three corrupted sequences.
- Dataset and recognition: On UTD-MHAD, the proposed method cannot distinguish some actions well, including jogging and walking.The paper attributes this difficulty to temporal normalization producing similar JTMs for the two actions.
- View selection and fusion: Recognition accuracy varies only slightly across tested orthogonal views, with natural camera coordinates producing the best result.The experiments rotated joint points by fixed 22.5° steps across polar and azimuthal angles from −45° to 45°.
- View selection and fusion: Fusing additional rotated views produced nearly identical accuracies, supporting the sufficiency of the three natural-coordinate orthogonal views.
V. CONCLUSION
The paper applies ConvNets to skeleton sequences by projecting joint trajectories into three orthogonal JTMs that encode spatio-temporal information as texture patterns. These complementary images enable fine-tuning existing image-trained ConvNets without retraining them from scratch.
- V. CONCLUSION: Three orthogonal Joint Trajectory Maps encode skeleton joint trajectories into texture patterns for ConvNet-based action recognition.