Source-linked AI summary
Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition
Hong Liu, Juanhui Tu, Mengyuan Liu
TL;DR
Skeleton-based action recognition needs spatial-temporal modeling, while RNN-based methods tend to overemphasize temporal information. The paper proposes a two-stream 3D CNN using separate spatial and temporal encodings, multi-temporal processing, and probability fusion. Experiments on SmartHome and NTU RGB+D show that the method outperforms most RNN-based approaches and supports the complementarity of spatial and temporal information.
Problem
RNN-based action recognition methods tend to overemphasize temporal information, although effective recognition requires combining spatial and temporal information.
Method
The method transforms skeleton sequences into spatial and temporal volumes, processes them with separate 3D CNN streams, and extends each stream into multi-temporal versions.
Results
The method outperforms most RNN-based approaches on the SmartHome and NTU RGB+D datasets; on SmartHome it achieves 79.38% accuracy.
Takeaways & Limitations
The results support using complementary spatial and temporal streams and multi-temporal processing for skeleton-based action recognition.
Abstract
from arXiv · showhide
It remains a challenge to efficiently extract spatialtemporal information from skeleton sequences for 3D human action recognition. Although most recent action recognition methods are based on Recurrent Neural Networks which present outstanding performance, one of the shortcomings of these methods is the tendency to overemphasize the temporal information. Since 3D convolutional neural network(3D CNN) is a powerful tool to simultaneously learn features from both spatial and temporal dimensions through capturing the correlations between three dimensional signals, this paper proposes a novel two-stream model using 3D CNN. To our best knowledge, this is the first application of 3D CNN in skeleton-based action recognition. Our method consists of three stages. First, skeleton joints are mapped into a 3D coordinate space and then encoding the spatial and temporal information, respectively. Second, 3D CNN models are seperately adopted to extract deep features from two streams. Third, to enhance the ability of deep features to capture global relationships, we extend every stream into multitemporal version. Extensive experiments on the SmartHome dataset and the large-scale NTU RGB-D dataset demonstrate that our method outperforms most of RNN-based methods, which verify the complementary property between spatial and temporal information and the robustness to noise.
I. INTRODUCTION
Skeleton-based action recognition benefits from combining spatial and temporal information, but RNN-based methods tend to emphasize temporal evolution. The paper proposes a two-stream 3D CNN that separately encodes and learns spatial and temporal features, then extends them across temporal scales.
- RNN-based action recognition methods tend to overstress temporal information despite the need to combine spatial and temporal cues.
- The proposed model uses two streams to reinforce spatial and temporal information simultaneously.
- Skeleton joints are transformed, mapped into 3D space, and encoded into separate spatial and temporal volumes before feature learning.
- The approach extends each stream into multi-temporal sequences to capture temporal information at multiple scales.
- Related RNN and LSTM methods model temporal evolutions using hierarchical, part-based, or memory-cell architectures.
B. 3D CNN-based Methods
3D CNNs model spatial-temporal correlations from volumetric inputs, motivating their use with encoded skeleton data. The proposed pipeline transforms skeleton sequences into spatial and temporal volumes and uses complementary streams to address reversed temporal actions.
- 3D CNNs abstract spatial and temporal information layer by layer from volume inputs and can capture correlations between three-dimensional signals.
- The pipeline encodes action sequences into spatial and temporal volumes before applying a two-stream 3D CNN and fusing its outputs.
- The proposed two-stream model is designed to mutually compensate streams and improve robustness to noise.
- Skeleton joints are interpolated, mapped into 3D coordinate space, and encoded separately so 3D CNNs can learn from complete volumetric representations.
- Temporal encoding distinguishes actions with similar motion regions but opposite temporal orders, such as standing up and sitting down.
B. Two-Stream 3D CNN Model
Each stream uses the same 3D CNN architecture to learn from spatial or temporal volumes, while their class probabilities are fused for final recognition. The design also uses dropout and padding to reduce overfitting and preserve convolutional dimensions.
- Each stream contains four 3D convolutional layers, four max-pooling layers, and two fully connected layers.The convolutional filter counts are 3, 8, 32, and 64, followed by fully connected layers with 512 and 256 neurons.
- Dropout is inserted to reduce overfitting, while padding after the first three convolutional layers preserves input-output size.
- The architecture contains 910k parameters, substantially fewer than the C3D model.
- The spatial and temporal streams are trained separately and merged during forward propagation for decision making.
- The two streams fuse class-membership probabilities by element-wise multiplication to obtain final predictions.
C. Multi-Temporal Structure
The multi-temporal structure converts original skeleton sequences into multiple temporal versions and processes their spatial and temporal volumes with two-stream 3D CNNs. Their outputs are fused to capture information at larger temporal scales.
- Original skeleton sequences are converted into multi-temporal sequences to capture large-scale temporal information.
- Multi-temporal spatial and temporal volumes are trained separately by two-stream 3D CNNs and fused for recognition.
A. Datasets
The evaluation uses SmartHome and NTU RGB+D, with NTU providing a large, challenging benchmark under cross-subject and cross-view protocols.
- NTU RGB+D: NTU RGB+D contains 56,880 sequences across 60 classes, performed by 40 subjects and captured by three cameras.The dataset includes 4 million frames and noisy skeleton joints.
- NTU RGB+D: NTU RGB+D evaluation uses cross-subject and cross-view protocols, including separate subject groups and camera-based training/testing splits.Cross-view testing uses camera 1, with cameras 2 and 3 for training.
- SmartHome: SmartHome contains 1,620 depth sequences covering six actions performed by nine subjects in five situations.The dataset includes varied contexts such as sitting, standing, and interactions with objects or another person.
- SmartHome: SmartHome skeletons contain substantial noise from occlusions and unconstrained performer poses.This makes it a challenging setting for evaluating skeleton-based action recognition.
B. Implementation Details
The implementation normalizes skeleton coordinates and trains the network with stochastic gradient descent, dropout, and validation-based hyper-parameter optimization.
- Preprocessing: Joint coordinates are translated into a body-centered coordinate system using the middle-of-hip joint as the origin.The mapped 3D coordinate space is set to width and height 50.
- Optimization: Training uses mini-batch stochastic gradient descent with learning rate 0.0005, momentum 0.9, and weight decay 1e-6.Mini-batches contain 32 samples.
- Regularization: The model applies dropout with probability 0.3 and reserves 10% of the initial training set for validation-based hyper-parameter optimization.The implementation uses Keras on one NVIDIA GeForce GTX 1080 with 8G RAM.
C. 3D CNN Architecture Evaluation
The architecture evaluation examines convergence and multi-temporal design choices for the 3D CNN, with training error converging near epoch 250 and all evaluated levels giving the best NTU performance.
- Convergence: Training error for the spatial and temporal streams tends to converge when the NTU RGB+D training epoch reaches 250.Dropout with p = 0.3 is used to address overfitting and obtain the convergence curves.
D. Two-Stream 3D CNN Model Evaluation
The two-stream model combines spatial and temporal streams and performs especially strongly on NTU RGB+D cross-view evaluation and SmartHome comparisons, while reducing confusion between opposite-order actions.
- Two-stream fusion: On NTU RGB+D cross-view evaluation, the two-stream model achieves 5.46% and 5.90% higher performance than the individual streams.The fusion has no obvious effect on SmartHome cross-subject evaluation because that dataset lacks actions with opposite temporal order.
- Two-stream fusion: For the “sitting down” and “standing up” pair, confusion probability drops from 0.25 with the spatial stream to 0.02 with two streams.The reported reduction concerns mutual recognition between these representative action pairs.
- Benchmark comparison: The proposed model achieves 79.38% accuracy on SmartHome, outperforming Synthesized+Pre-trained.This is the best performance reported among the compared methods in the cited SmartHome evaluation.
- Benchmark comparison: The method performs better than Deep RNN, Deep LSTM, and Part-aware LSTM on both cross-subject and cross-view protocols.It also outperforms ST-LSTM for cross-subject evaluation and obtains competitive cross-view results.
- Interpretation: The reported gains are attributed to simultaneous spatial-temporal correlation capture, mutual stream compensation, and multi-scale learning of local patterns and global relationships.The multi-temporal structure is described as learning information at multiple scales.
V. CONCLUSION AND FUTURE WORK
The paper concludes that its two-stream 3D CNN learns spatial-temporal motion details from skeleton sequences while reducing overfitting and parameter requirements. Experiments support its advantage over most RNN-based approaches and the value of multi-temporal modeling.
- The two-stream 3D CNN learns local and global motion details through mutual enhancement between its spatial and temporal streams.
- The improved 3D CNN architecture reduces overfitting and requires fewer training parameters than C3D.
- The method outperforms most state-of-the-art RNN-based approaches for skeleton-based action recognition.
- The multi-temporal version improves the 3D CNN’s ability to capture information at multiple scales.
- Future work will investigate alternative ways to encode skeleton data for more effective 3D CNN training.