Source-linked AI summary
Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks
Hongsong Wang, Liang Wang
TL;DR
Skeleton-based action recognition needs representations beyond handcrafted features and temporal-only RNN modeling because articulated joint configurations also characterize actions. The paper introduces a two-stream RNN with temporal and spatial pathways, graph-to-sequence spatial modeling, and 3D training augmentation, reporting improvements across generic actions, interactions, and gestures.
Problem
Handcrafted features have limited capacity for complex motion patterns, while recent skeleton RNNs focus on temporal dependencies and neglect spatial configurations of articulated skeletons.
Method
A two-stream RNN models temporal dynamics and spatial joint dependencies, using stacked or hierarchical temporal structures, graph-to-joint sequences, late fusion, and 3D transformations during training.
Results
The method reports considerable improvement across generic actions, interaction activities, and gestures, with two-stream models consistently outperforming individual temporal or spatial RNNs.
Takeaways & Limitations
Joint spatial configurations and temporal dynamics are both effective and complementary signals for skeleton-based action recognition.
Abstract
from arXiv · showhide
Recently, skeleton based action recognition gains more popularity due to cost-effective depth sensors coupled with real-time skeleton estimation algorithms. Traditional approaches based on handcrafted features are limited to represent the complexity of motion patterns. Recent methods that use Recurrent Neural Networks (RNN) to handle raw skeletons only focus on the contextual dependency in the temporal domain and neglect the spatial configurations of articulated skeletons. In this paper, we propose a novel two-stream RNN architecture to model both temporal dynamics and spatial configurations for skeleton based action recognition. We explore two different structures for the temporal stream: stacked RNN and hierarchical RNN. Hierarchical RNN is designed according to human body kinematics. We also propose two effective methods to model the spatial structure by converting the spatial graph into a sequence of joints. To improve generalization of our model, we further exploit 3D transformation based data augmentation techniques including rotation and scaling transformation to transform the 3D coordinates of skeletons during training. Experiments on 3D action recognition benchmark datasets show that our method brings a considerable improvement for a variety of actions, i.e., generic actions, interaction activities and gestures.
1. Introduction
Skeleton-based recognition avoids several difficulties of RGB action recognition and can model action dynamics directly, but existing RNN methods neglect spatial joint configurations. The paper addresses this gap with a two-stream RNN that jointly models temporal and spatial information.
- RGB action recognition is challenged by high-dimensional low-quality inputs, illumination changes, occlusion, background clutter, and difficult-to-extract visual clues.
- Skeleton-based recognition can avoid video feature extraction and explicitly model action dynamics, using joint motions as informative action descriptions.
- Handcrafted joint- and body-part-based features paired with simple time-series models are limited in representing the complexity of intrinsic motion patterns.
- Existing skeleton RNN methods primarily model temporal context by concatenating frame skeletons while neglecting spatial dependencies among articulated joints.
- The proposed two-stream RNN combines a temporal stream for joint-coordinate dynamics with a spatial stream that converts skeleton graphs into joint sequences.
- The streams are fused late in an end-to-end trainable network, while stacked and hierarchical temporal RNNs and 3D transformations are explored.
- The method reports state-of-the-art results on NTU RGB+D, SBU Interaction, and ChaLearn across generic actions, interaction activities, and gestures.
2. Related work
Related work spans deep neural networks, RNN-based action recognition, and handcrafted skeleton features. Prior skeleton RNNs largely model temporal motion, motivating a separate spatial pathway for joint dependencies.
- Deep action-recognition models include 3D CNNs for local spatio-temporal information and two-stream CNNs for complementary appearance and motion.
- RNNs have been used to learn video representations through multilayer LSTMs and end-to-end recurrent-convolutional architectures.
- Skeleton action-recognition RNNs include hierarchical RNN, regularized deep LSTM, differential RNN, and part-aware LSTM models.
- These prior skeleton RNN methods model motion dynamics temporally but neglect articulated skeletons' spatial configurations; one later approach extends LSTM into the spatial-temporal domain.
- Earlier skeleton approaches use handcrafted features such as pairwise joint-location differences, 3D location histograms, orientations, geometric relations, and rotation matrices.
3. Overview of RNN
RNNs transform input sequences into output sequences and are naturally suited to sequence classification. LSTM gates address standard RNN difficulties with long-range information storage and access.
- RNNs map an input sequence X to an output sequence Y, unlike feedforward networks that map one input vector or matrix to one output.
- RNN layers can be stacked for deep sequence classification, with a softmax layer placed above the final hidden layer.
- Standard RNNs suffer from vanishing-gradient and error-blowing-up problems that limit long-term information storage and access.
- LSTM addresses these limitations with input, forget, and output gates that regulate remembering, forgetting, and outputting information.
- The LSTM equations define gate activations, cell-state updates, and hidden-state outputs using recurrent weights and biases.
4. Two-stream RNN
The paper uses an end-to-end two-stream RNN to model both temporal dynamics and spatial dependencies in skeleton sequences. Its temporal stream includes stacked and body-kinematics-based hierarchical RNNs, while the spatial stream converts the skeleton graph into joint sequences; training also uses 3D transformations.
- Architecture: The two-stream architecture combines temporal and spatial RNNs, fusing their softmax class posteriors for action recognition.The temporal stream models joint-coordinate dynamics across time, while the spatial stream models dependencies among joints.
- Temporal RNN: The hierarchical temporal RNN models each body part separately before modeling whole-body movement from the part-level outputs.Its two vertical layers correspond to part-level and whole-body temporal modeling, reflecting human body kinematics.
- Spatial RNN: The spatial stream converts the skeleton graph into a joint sequence and represents each joint using coordinates concatenated over a temporal window.The paper proposes chain and traversal sequences as alternative graph-to-sequence conversions.
- Spatial RNN: The spatial RNN can recognize actions from a single frame when its temporal window has size 1, and it does not use body-part hierarchy because skeletons contain relatively few joints.The paper gives 25 joints for the NTU RGB+D dataset as an example.
- 3D Transformation: During training, random 3D transformations augment skeleton sequences, including rotations for viewpoint changes and scaling for differences in performer size.Rotation is applied within selected x- and y-axis ranges, while scaling uses axis-specific factors; these transformations are training-only.
5. Experiments
Experiments evaluate the two-stream RNN across three datasets and compare data augmentation, stream combinations, sequence structures, and network configurations. Results show complementary spatial-temporal modeling, benefits from 3D transformations, and strong performance across action-recognition tasks.
- The model is evaluated on NTU RGB+D, SBU Interaction, and ChaLearn Gesture Recognition datasets.
- Comparison between models: The two-stream RNN consistently outperforms individual temporal and spatial RNNs, confirming complementary spatial and temporal channels.
- Comparison between models: 7.8% improvement occurs with 3D transformation for NTU RGB+D cross-view evaluation, compared with 2.7% for cross-subject evaluation.
- Comparison between structures: Hierarchical RNN generally outperforms stacked RNN, with an average NTU RGB+D improvement of 1.6%.
- Comparison between structures: Traversal sequences outperform chain sequences because traversal better preserves graph spatial relationships by visiting most joints twice.
- Comparison between structures: The default stacked structure is R512-512, while the default hierarchical structure is P128, B512 to balance performance and computational cost.
- Two-stream RNN versus temporal RNN: On SBU Interaction, two-stream RNN leaves one misclassified action pair versus three for temporal RNN and correctly recognizes all pushing samples.
- Two-stream RNN versus temporal RNN: The two-stream RNN improves recognition across individual NTU RGB+D actions, including brushing teeth, shaking head, and walking towards, by more than 8% over temporal RNN.
6. Conclusion
The paper proposes an end-to-end two-stream RNN for skeleton action recognition, modeling temporal dynamics and spatial configurations separately. Experiments report improved performance across generic actions, interaction activities, and gestures.
- The temporal stream models skeleton sequence dynamics using stacked or hierarchical RNN structures.The paper explores two sequence-modeling structures for the temporal channel.
- The spatial stream converts skeleton structure into joint sequences before using an RNN to model spatial dependency.Two methods are devised for converting the skeleton structure into a sequence.
- 3D rotation, scaling, and shear transformations are used during training to improve generalization and prevent overfitting.These augmentations transform skeleton coordinates during training.
- Two-stream RNN outperforms existing state-of-the-art skeleton-based approaches on datasets covering generic actions, interaction activities, and gestures.The evaluated datasets are NTU RGB+D, SBU Interaction, and ChaLearn.