Source-linked AI summary
A New Representation of Skeleton Sequences for 3D Action Recognition
Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, Farid Boussaid
TL;DR
The paper tackles the difficulty of learning long-term temporal dependencies and spatial structure from 3D skeleton trajectories. It converts each sequence into three cylindrical-coordinate clips, extracts CNN features, and jointly processes them with MTLN; experiments report state-of-the-art performance across three datasets.
Problem
Existing LSTM methods have difficulty memorizing entire long skeleton sequences and constructing deep networks for high-level feature extraction.
Method
The method transforms each sequence into three cylindrical-coordinate clips, uses CNNs to extract temporal features, and jointly processes four spatial-relationship feature vectors with MTLN.
Results
The method achieves state-of-the-art performance on three skeleton datasets, including NTU RGB+D.
Takeaways & Limitations
Representing each sequence as clips lets deep CNNs learn long-term temporal information while MTLN incorporates relationships among different spatial configurations.
Takeaways & Limitations
The approach relies on CNN features transferred from ImageNet because skeleton datasets are too small or noisy to suitably train a deep network.
Abstract
from arXiv · showhide
This paper presents a new method for 3D action recognition with skeleton sequences (i.e., 3D trajectories of human skeleton joints). The proposed method first transforms each skeleton sequence into three clips each consisting of several frames for spatial temporal feature learning using deep neural networks. Each clip is generated from one channel of the cylindrical coordinates of the skeleton sequence. Each frame of the generated clips represents the temporal information of the entire skeleton sequence, and incorporates one particular spatial relationship between the joints. The entire clips include multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We propose to use deep convolutional neural networks to learn long-term temporal information of the skeleton sequence from the frames of the generated clips, and then use a Multi-Task Learning Network (MTLN) to jointly process all frames of the generated clips in parallel to incorporate spatial structural information for action recognition. Experimental results clearly show the effectiveness of the proposed new representation and feature learning method for 3D action recognition.
1. Introduction
The paper addresses the difficulty of learning long-term temporal dependencies and spatial skeleton structure from 3D joint trajectories. It represents each sequence as three cylindrical-coordinate clips and uses CNNs with MTLN to learn temporal and spatial information.
- 3D skeleton sequences provide joint trajectories that are robust to illumination changes and invariant to camera views.
- LSTM-based methods can explore spatial and temporal structure, but struggle with long sequences and extracting high-level features through deep recurrent networks.
- Each generated frame contains temporal information from the entire sequence, allowing deep CNNs to learn long-term temporal structure from a short clip.
- The method transforms each skeleton sequence into three clips corresponding to cylindrical-coordinate channels, with four frames encoding different spatial relationships.
- The MTLN jointly processes four feature vectors and exploits their intrinsic relationships, outperforming feature concatenation or pooling.
- The proposed method achieves state-of-the-art performance on three skeleton datasets, including NTU RGB+D.
2. Related Works
Prior skeleton-based action-recognition work uses handcrafted trajectory and relative-position features or deep recurrent networks. These methods model temporal patterns, body-part structure, or spatial-temporal information through techniques including FTP, PCA, and LSTM variants.
- Hand-crafted Features: Handcrafted approaches model skeleton sequences with covariance matrices of joint trajectories across hierarchical temporal levels.
- Hand-crafted Features: Other handcrafted methods use pairwise relative joint positions with Fourier Temporal Pyramid or PCA to characterize temporal and posture-related features.
- Deep Learning Methods: Deep learning approaches divide joints into body-part sets or feed them into deep LSTMs to learn co-occurrence and long-term context representations.
- Deep Learning Methods: Spatial-temporal LSTM methods learn spatial and temporal information jointly, with Trust Gate mechanisms proposed to remove noisy joints.
3. Proposed Method
The proposed pipeline converts arbitrary-length skeleton sequences into three fixed-frame clips, extracts CNN representations with temporal pooling, and jointly processes the resulting spatial-temporal features using MTLN.
- Overview: A skeleton sequence of any length is transformed into three clips of gray images, enabling deep CNN-based spatial-temporal feature learning.
- Clip Generation: Four reference joints generate relative-position arrays, which are converted from Cartesian to cylindrical coordinates and rendered as three four-frame clips.
- Clip Generation: Each generated frame encodes the temporal dynamics of the full sequence and one particular spatial relationship between skeleton joints.
- CNN Feature Learning: Pre-trained VGG19 extracts conv5_1 feature maps of size 14×14×512 from each generated frame.
- Temporal Pooling: Temporal mean pooling over feature-map rows aggregates sequence dynamics, and concatenation produces a 7168D feature vector per cylindrical-coordinate channel.
- Multi-Task Learning Network: Three same-time-step CNN features are concatenated into four feature vectors, which MTLN jointly processes to exploit relationships among spatial configurations.
4. Experiments and Analysis
Experiments on three skeleton-action datasets compare CNN-based clip processing and MTLN against prior methods and baselines. Results show gains across protocols, robustness to noisy joints, and sensitivity to representation and design choices.
- Datasets: The method was evaluated on NTU RGB+D, SBU Kinect Interaction, and CMU skeleton-action datasets.
- Results: On NTU RGB+D, accuracy reached 79.57% cross-subject and 84.83% cross-view, improving over prior results in both protocols.The cross-subject gain over ST-LSTM + Trust Gate was 10.37%, while cross-view accuracy improved from 77.7% to 84.83%.
- Ablation analysis: Using entire clips with CNN and MTLN improved NTU protocol performance by about 4% and 5% over frame-based processing.MTLN also outperformed feature concatenation and pooling by exploiting relationships among multiple frames.
- Results: On SBU, individual-frame CNN features achieved 90.88%, while MTLN over entire clips increased accuracy to 93.57%.Concatenation and pooling improved performance by about 2% before MTLN produced the highest reported accuracy.
- Robustness: The learned clip features remained effective without preprocessing noisy joints, improving accuracy from 88.6% to 93.3%.The authors attribute this robustness to convolution and pooling in the deep network.
- Results: On CMU, the proposed method achieved about 93.22% on the subset and 88.3% on the entire dataset, surpassing previous methods.The subset result was about 5% above the previous method, and the entire-dataset result improved from 81.04% to 88.3%.
- Design analysis: On CMU, one color clip reached 84.67%, six additional reference joints reached 86.01%, and Cartesian coordinates reached 86.21%, each below the proposed configuration.The paper uses three gray clips, four reference joints, and cylindrical coordinates; added joints were reported to introduce noise.
- Feature analysis: Intermediate conv5_1 features performed better than earlier conv4_1 features at 84.59% and later fc6 features at 83.52% on CMU.The authors associate the difference with earlier features being insufficiently deep and later features being more task-specific.
5. Conclusion
The paper transforms skeleton sequences into three video clips, extracts compact CNN features, and jointly processes them with an MTLN for action recognition. Experiments on three datasets show the effectiveness of this representation and feature-learning method.
- The method transforms each skeleton sequence into three video clips for robust feature learning and action recognition.
- A pre-trained CNN and temporal pooling extract compact frame representations, whose same-timestep features from the three clips are concatenated.
- An MTLN jointly learns feature vectors from all timesteps in parallel, using their intrinsic relationships for action recognition.
- Experiments on the NTU RGB+D, SBU Kinect Interaction, and CMU datasets show the effectiveness of the proposed representation and feature-learning method.