Source-linked AI summary
Skeleton-based Action Recognition Using LSTM and CNN
Chuankun Li, Pichao Wang, Shuang Wang, Yonghong Hou, Wanqing Li
TL;DR
CNN encodings of skeleton sequences can lose temporal information, while RNN-based approaches may not effectively exploit multiple feature types. The paper combines spatial and temporal skeleton features in LSTM and CNN channels with late score fusion, achieving state-of-the-art NTU RGB+D results and 87.40% accuracy in the challenge.
Problem
CNN-based conversion of 3D skeleton sequences into images can lose temporal information, while combining multiple feature types in RNNs has not consistently improved recognition.
Method
The method processes spatial-domain features with LSTMs and temporal-domain texture maps with CNNs, then performs late fusion of their class scores.
Results
87.40% accuracy ranked the proposed method first in the Large Scale 3D Human Activity Analysis Challenge in Depth Videos.
Takeaways & Limitations
CNN–LSTM score fusion exploits complementary spatial and temporal modeling and performs better than LSTM–LSTM fusion for the same feature.
Abstract
from arXiv · showhide
Recent methods based on 3D skeleton data have achieved outstanding performance due to its conciseness, robustness, and view-independent representation. With the development of deep learning, Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM)-based learning methods have achieved promising performance for action recognition. However, for CNN-based methods, it is inevitable to loss temporal information when a sequence is encoded into images. In order to capture as much spatial-temporal information as possible, LSTM and CNN are adopted to conduct effective recognition with later score fusion. In addition, experimental results show that the score fusion between CNN and LSTM performs better than that between LSTM and LSTM for the same feature. Our method achieved state-of-the-art results on NTU RGB+D datasets for 3D human action analysis. The proposed method achieved 87.40% in terms of accuracy and ranked $1^{st}$ place in Large Scale 3D Human Activity Analysis Challenge in Depth Videos.
1. INTRODUCTION
Skeleton data offers concise, robust, view-independent action representations, but CNN image encodings can lose temporal information. The paper combines LSTM and CNN channels with late score fusion to address complementary spatial-temporal modeling needs.
- 3D skeleton features are concise, robust, and view-independent, motivating their use for human action recognition.
- CNN-based image representations inevitably lose temporal information when 3D skeleton sequences are converted into 2D images.Trajectory overlap can make actions such as “knock” and “catch” difficult to distinguish.
- RNN-based methods model temporal dependencies, but combining multiple feature types has not consistently improved performance over single-feature inputs.The paper attributes this to a likely weakness in distinguishing useful information from multiple features.
- The proposed method fuses class scores across LSTM channels and between CNN and LSTM channels, exploiting their complementarity.The authors report that CNN–LSTM fusion performs better than LSTM–LSTM fusion for the same feature.
- The method achieved state-of-the-art results on the NTU RGB+D Dataset.
2. PROPOSED METHOD
The method extracts spatial and temporal features from skeleton sequences, processes them with separate LSTM and CNN networks, and fuses their output scores. It uses selected skeletal relations, multi-view trajectory maps, and multiply-score fusion to combine complementary evidence.
- 2.1. Feature Extraction From Skeleton Sequence: The framework feeds three spatial-domain features to LSTM networks and seven temporal-domain texture maps to CNN networks.It trains ten networks in total and applies late score fusion to their outputs.
- 2.1. Feature Extraction From Skeleton Sequence: Relative position, joint distances, and joint-to-line distances provide the selected spatial-domain skeletal features.Joint distances provide spatial information, while relative position captures orientation information lost by distance features.
- 2.1. Feature Extraction From Skeleton Sequence: 12-joint skeletons are used to reduce noise, while selected joint-line relations reduce the computational cost of the much larger complete feature set.With 12 joints, all possible triples would generate 660 lines for one subject.
- 2.1. Feature Extraction From Skeleton Sequence: Three line-selection constraints generate six, three, or ten lines depending on the joints’ positions in the kinetic chain.
- 2.1. Feature Extraction From Skeleton Sequence: Multi-view joint trajectory maps are created by rotating skeleton data around the Y axis in fixed 45-degree steps.This addresses the view dependence of the trajectory-map method used for CNN inputs.
- 2.2. Neural Networks: LSTM models temporal contextual dependencies, whereas CNN processes image-like features with stronger spatial information.Their complementary behavior is used to improve later score fusion.
- 2.2. Neural Networks: A three-layer stacked LSTM passes lower-layer outputs upward and converts the highest-layer output into class probabilities through softmax.
- 2.3.1. Score Fusion: Multiply-score fusion combines ten score vectors element-wise and selects the index with the maximum resulting score.The paper reports better performance for multiply-score fusion than for max-score or average-score fusion.
3. EXPERIMENT RESULTS
Experiments on NTU RGB+D evaluate feature choices and score-fusion strategies, showing that multiply-score fusion and CNN–LSTM fusion improve recognition. The proposed method achieves state-of-the-art performance and wins the depth-video challenge.
- Dataset: NTU RGB+D contains 56,578 samples across 60 classes and supports cross-subject and cross-view evaluation protocols.The samples involve 40 subjects and multiple viewing angles.
- Fusion methods: Multiply-score fusion substantially improves final accuracy compared with average-score and max-score fusion.The experiments evaluate three fusion rules across ten channels.
- Feature evaluation: Feature concatenation does not consistently improve performance and can perform worse than a single distance-between-joints feature in cross-view evaluation.The compared features include relative position, joint distance, and distance between joints and lines.
- Feature evaluation: JTM-xyz performs worse with CNN than joint distance with LSTM, but improves when fused with relative-position features processed by LSTM.This comparison highlights differences between feature representations and model types.
- Fusion methods: CNN–LSTM score fusion greatly improves final results, while the proposed method outperforms Multi-CNN models in both cross-subject and cross-view evaluation.The method also achieves state-of-the-art results against handcrafted-feature and deep-learning approaches.
- Challenge results: 87.40% accuracy earns the proposed method first place in the Large Scale 3D Human Activity Analysis Challenge in Depth Videos.The reported result is presented as evidence of the method’s effectiveness.
4. CONCLUSIONS
The paper concludes that late score fusion between LSTM and CNN models combines temporal and spatial information for skeleton-based action recognition. Multiply-score fusion improves accuracy, producing state-of-the-art results on NTU RGB+D.
- Conclusions: The proposed method combines LSTM and CNN models through late score fusion for skeleton-based action recognition.LSTM exploits temporal information, while CNN mines spatial information.
- Conclusions: Multiply-score fusion improves recognition accuracy substantially in the proposed LSTM–CNN framework.The conclusion describes the method as simple yet effective.
- Conclusions: State-of-the-art results are achieved on the NTU RGB+D dataset.The paper presents these results as verification of the proposed method’s effectiveness.