Source-linked AI summary
Skeleton-based Action Recognition with Convolutional Neural Networks
Chao Li, Qiaoyong Zhong, Di Xie, Shiliang Pu
TL;DR
Skeleton action recognition has been dominated by recurrent models, motivating a CNN framework for classification and temporal detection. The method combines coordinate and motion inputs with learned joint transformation and window proposals, achieving strong benchmark results.
Problem
Skeleton action recognition is predominantly addressed with recurrent neural networks, motivating a CNN-based framework for both classification and detection.
Method
The framework feeds raw skeleton coordinates and motion into CNNs, learns joint rearrangement with a skeleton transformer, and uses a window proposal network for temporal detection.
Results
The method achieves 89.3% accuracy on NTU RGB+D and reports a 58% absolute mAP improvement over the baseline for temporal action detection.
Takeaways & Limitations
Treating skeleton sequences as images is reported as a viable solution for both action classification and temporal action detection.
Takeaways & Limitations
The action detection approach operates in batch-processing mode, while real-time applications require online detection.
Abstract
from arXiv · showhide
Current state-of-the-art approaches to skeleton-based action recognition are mostly based on recurrent neural networks (RNN). In this paper, we propose a novel convolutional neural networks (CNN) based framework for both action classification and detection. Raw skeleton coordinates as well as skeleton motion are fed directly into CNN for label prediction. A novel skeleton transformer module is designed to rearrange and select important skeleton joints automatically. With a simple 7-layer network, we obtain 89.3% accuracy on validation set of the NTU RGB+D dataset. For action detection in untrimmed videos, we develop a window proposal network to extract temporal segment proposals, which are further classified within the same network. On the recent PKU-MMD dataset, we achieve 93.7% mAP, surpassing the baseline by a large margin.
1. INTRODUCTION
The paper replaces predominantly LSTM-based skeleton action recognition with a CNN framework for classification and temporal detection. It reports strong classification and detection improvements over baselines.
- Skeleton action-recognition methods are predominantly based on RNNs, especially LSTMs, because skeleton sequences are time series.
- The proposed CNN representation is reported to outperform a strong LSTM baseline for skeleton action recognition.
- 58% absolute mAP improvement over the baseline is reported for temporal action detection using the CNN-based detection framework.
2. RELATED WORKS
Prior work largely uses LSTMs for skeleton sequences, while CNNs have also been explored for skeleton representation and image detection. This paper extends Faster R-CNN-style detection to skeleton-based temporal action detection.
- LSTM-based methods model skeleton temporal patterns, with prior improvements addressing joint co-occurrence, attention, and viewpoint variation.
- CNN-based skeleton representation had already achieved state-of-the-art performance before this paper.
- Faster R-CNN uses a region proposal network followed by ROI-pooled proposal classification with shared features.
- This work is presented as the first adaptation of Faster R-CNN to skeleton-based temporal action detection.
3. METHOD
The method treats skeleton sequences as image-like inputs to a compact CNN, using raw coordinates, motion, learned joint transformation, and temporal proposal processing. It also supports variable-length videos and multiple people.
- Action classification: Raw joint coordinates and inter-frame skeleton motion serve as two input streams for CNN-based action classification.A skeleton sequence is represented as a T × N × 3 array and treated as a T × N-sized 3-channel image.
- Action classification: Videos with varying frame lengths are resized to a fixed length before CNN processing.
- Skeleton transformer: The skeleton transformer learns a new ordering and interpolated set of joints to select more informative body-joint representations.It applies a learned linear transformation before the convolution layers and is trained end to end.
- Multi-person maxout: Multiple people are handled with shared network layers and element-wise maxout feature merging, avoiding zero padding and model-size growth for additional people.
- Network architecture: A 7-layer network with 3 convolutional and 4 fully connected layers contains 1.3 million parameters and can be trained from scratch without pre-training.
- Action detection: For temporal detection, a window proposal network replaces the region proposal network, using one-dimensional anchors and temporal window regression.Proposal features are pooled from shared feature maps and classified by an R-CNN subnetwork.
4. EXPERIMENTS
Experiments evaluate classification on NTU RGB+D and detection on PKU-MMD. The CNN framework improves classification over LSTM-based methods and substantially raises detection mAP over the baseline.
- Datasets: The experiments use NTU RGB+D for action classification and PKU-MMD for action detection.NTU RGB+D contains 56880 trimmed clips across 60 categories; PKU-MMD contains 1076 untrimmed videos, 21545 action instances, and 51 categories.
- Ablation study: Skeleton motion improves accuracy by 1.6 points cross-subject and 3.3 points cross-view, while the skeleton transformer adds 1.8 points cross-subject.The transformer’s cross-view improvement is marginal.
- Ablation study: 83.2% and 89.3% accuracy are achieved when skeleton motion and the skeleton transformer are combined across the two partitioning schemes.These results are reported on the NTU RGB+D validation set.
- Classification comparison: The method significantly outperforms recent state-of-the-art approaches in both cross-subject and cross-view settings, improving 10 points over STA-LSTM cross-subject.It also surpasses the CNN-based method of Ke et al.
- Action detection: At IoU threshold 0.5, detection improves absolute mAP by 58% cross-subject and 40% cross-view over JCRRNN.The evaluation uses the authors’ own implementation of mean average precision because official evaluation code was unavailable.
5. CONCLUSION
The paper presents a CNN framework for skeleton-based action classification and detection by treating skeleton sequences as images. It reports new state-of-the-art performance on two large-scale datasets, while noting that its detection approach is batch-based rather than online.
- Conclusion: The framework treats skeleton sequences as images for both action classification and detection.The paper reports this as its central CNN-based approach.
- Conclusion: The method achieves new state-of-the-art performance on two recent large-scale skeleton datasets.The conclusion summarizes results across both classification and detection tasks.
- Conclusion: The action detection approach operates in batch-processing mode, while real-time applications require online detection.Online detection is left for future exploration.