PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates

Jun Liu, Amir Shahroudy, Dong Xu, Alex C. Kot, Gang Wang

arXiv:1706.08276v1cs.CV

TL;DR

Skeleton-based action recognition methods have largely modeled temporal dependencies, leaving spatial relationships among joints insufficiently represented. This paper introduces an ST-LSTM with tree traversal, trust gating, and multimodal fusion, and reports state-of-the-art performance on seven benchmark datasets.

  • Problem

    Existing skeleton-action recognition methods primarily model temporal dependencies, although spatial dependencies among skeletal joints are also discriminative for classification.

  • Method

    The paper proposes an ST-LSTM with tree-structured skeleton traversal, a trust gate for noisy inputs, and multimodal feature fusion within the LSTM unit.

  • Results

    The proposed method achieves state-of-the-art performance on seven benchmark datasets.

  • Takeaways & Limitations

    The framework analyzes skeletal action information across spatial and temporal domains while incorporating structured traversal, input reliability, and complementary visual features.

  • Takeaways & Limitations

    The initial ST-LSTM formulation assumes joints are arranged in a simple chain before the more advanced traversal scheme is introduced.

Abstract

from arXiv · show

Skeleton-based human action recognition has attracted a lot of research attention during the past few years. Recent works attempted to utilize recurrent neural networks to model the temporal dependencies between the 3D positional configurations of human body joints for better analysis of human activities in the skeletal data. The proposed work extends this idea to spatial domain as well as temporal domain to better analyze the hidden sources of action-related information within the human skeleton sequences in both of these domains simultaneously. Based on the pictorial structure of Kinect's skeletal data, an effective tree-structure based traversal framework is also proposed. In order to deal with the noise in the skeletal data, a new gating mechanism within LSTM module is introduced, with which the network can learn the reliability of the sequential data and accordingly adjust the effect of the input data on the updating procedure of the long-term context representation stored in the unit's memory cell. Moreover, we introduce a novel multi-modal feature fusion strategy within the LSTM unit in this paper. The comprehensive experimental results on seven challenging benchmark datasets for human action recognition demonstrate the effectiveness of the proposed method.

1. Introduction

The paper extends recurrent skeleton-action analysis from temporal modeling to joint spatial dependencies, using an ST-LSTM with tree traversal, trust gates, and multimodal fusion. It reports state-of-the-art performance on seven benchmark datasets.

  • 1. Introduction: RNN-based skeleton-action methods mainly model long-term temporal context, while spatial dependencies among skeletal joints are also discriminative.The paper identifies joint-structure modeling as a missing dimension in existing recurrent approaches.
  • 1. Introduction: The proposed ST-LSTM models each joint’s context from previous frames and neighboring joints in the same frame.This extends recurrent analysis across both temporal and spatial domains.
  • 1. Introduction: A tree-structured traversal follows skeletal adjacency and kinematic relationships instead of feeding joints as a simple chain.The traversal is designed to represent the human skeleton’s structured spatial dependencies.
  • 1. Introduction: The trust gate estimates input reliability so the network can adjust when to update, forget, or retain long-term memory.It targets inaccuracies in 3D joint positions from depth sensors.
  • 1. Introduction: A multimodal fusion strategy is introduced inside the ST-LSTM unit, alongside the paper’s reported state-of-the-art performance on seven benchmark datasets.The contribution list identifies feature fusion and seven-dataset evaluation as central additions.

2. Related Work

Prior work used RNNs and LSTMs for skeleton action recognition through hierarchical, differential, regularized, part-based, reconstructive, and manifold-based approaches. This paper instead jointly models spatial and temporal joint dependencies, adds a trust gate, and extends evaluation and analysis beyond its preliminary version.

  • 2. Related Work: Earlier approaches organized recurrent processing hierarchically, by body parts, or through differential memory-state modeling.Examples include hierarchical bidirectional RNNs, differential LSTM, and separate memory sub-cells for body parts.
  • 2. Related Work: Other methods regularized LSTMs for joint co-occurrence, reconstructed skeleton sequences, or used encoded skeleton features to assist RGB-action recognition.These approaches introduced internal dropout, encoder-decoder reconstruction, or feature manifolds for supervised learning.
  • 2. Related Work: Unlike prior methods that concatenate joint inputs, the proposed network explicitly models dependencies across temporal and spatial dimensions.The paper presents this concurrent recurrent analysis as a distinction from earlier approaches.
  • 2. Related Work: A trust gate is added to improve robustness against noisy skeletal input data.The gate is presented as a reliability-oriented extension of the ST-LSTM framework.
  • 2. Related Work: Compared with the preliminary conference version, this paper evaluates seven rather than four benchmark datasets and adds feature fusion, a last-to-first link scheme, and further empirical analysis.The expanded version broadens both the proposed framework and its evaluation.

3. Spatio-Temporal Recurrent Networks

The paper extends LSTM-based skeleton recognition from temporal modeling to concurrent spatial and temporal dependency analysis. It adds tree traversal, trust gating for noisy measurements, and internal multimodal feature fusion.

  • 3. Spatio-Temporal LSTM: The framework uses LSTM memory and gating to update, retain, and expose long-term context representations.The standard unit contains input, forget, and output gates, an internal memory cell, and a hidden state.
  • 3. Spatio-Temporal LSTM: ST-LSTM units model dependencies across frames and neighboring joints by receiving temporal and spatial hidden representations.Each unit processes the current joint input together with the same joint from the previous frame and the previous joint in the current frame.
  • 3. Spatio-Temporal LSTM: Tree traversal preserves skeletal adjacency instead of imposing a semantically weak simple joint chain.The traversal unfolds the tree into a sequence that can discover stronger long-term spatial dependency patterns.
  • 3. Spatio-Temporal LSTM: Trust gates estimate input reliability and regulate whether noisy joint measurements alter the memory cell.The gate compares predicted and actual inputs; unreliable observations are blocked while history is retained more strongly.
  • 3. Spatio-Temporal LSTM: The network fuses geometric and visual features inside the ST-LSTM unit rather than concatenating them at the input.This addresses the dimensional mismatch that can make simple input-level concatenation degrade performance.

4. Experiments

The method is evaluated on seven skeletal-action datasets spanning varied action classes, viewpoints, subjects, sequence lengths, and joint configurations. Experiments include ablations of traversal and trust-gate components, plus geometric–visual feature fusion.

  • Datasets: The proposed method is evaluated on seven benchmark datasets: NTU RGB+D, UT-Kinect, SBU Interaction, SYSU-3D, ChaLearn Gesture, MSR Action3D, and Berkeley MHAD.These datasets differ in scale, action classes, subjects, viewpoints, recording duration, and numbers of provided skeletal joints.
  • Ablation models: The experiments compare a joint-chain ST-LSTM, a tree-traversal ST-LSTM, and a tree-based ST-LSTM augmented with a trust gate.The comparisons isolate the effects of exploiting skeletal tree structure and handling noisy inputs.
  • Feature fusion: The fusion model is evaluated by combining geometric joint coordinates with visual features extracted around corresponding joints.The NTU RGB+D and UT-Kinect datasets are used because appearance information can complement geometry in human–object interactions.
  • Evaluation challenges: The datasets include challenging conditions such as large viewpoint variation, similar motion patterns, two-person interactions, and inaccurate joint measurements.These conditions provide varied settings for testing skeleton-based action recognition.

4.2. Implementation Details

Implementation uses randomized temporal sampling, fixed optimization settings, and two stacked ST-LSTM layers, with evaluation on NTU RGB+D protocols and targeted component analyses. Results examine trust gates, network depth, feature fusion, and early stopping.

  • Implementation details: Each sequence is divided into T equal-length sub-sequences, with one randomly selected frame from each; cross-validated evaluation found T = 20 optimal on NTU RGB+D.Random sampling adds input variation and performed better than uniform frame sampling.
  • Implementation details: Training uses stochastic gradient descent with learning rate 2×10−3, momentum 0.9, decay rate 0.95, unit size d = 128, λ = 0.5, and two ST-LSTM layers.The same parameter settings are adopted across datasets despite differences in joint number, sequence length, and acquisition equipment.
  • NTU RGB+D protocols: On NTU RGB+D, evaluation uses cross-subject and cross-view protocols, with half the subjects or one-third of viewpoints held out for testing, respectively.The results are reported in Table 1.
  • Trust-gate analysis: The tree-based ST-LSTM with trust gate outperforms the deep RNN baseline and benefits from trust-gate handling of noisy or occluded Kinect joints.The trust-gate model improves recognition for most classes and raises side-view accuracy from 76.5% to 81.6%.
  • Network depth: Using one ST-LSTM layer yields 65.5% cross-subject and 77.0% cross-view accuracy, lower than the reported two-layer configuration.The authors attribute this difference to the greater representation power of the stacked network.
  • Feature fusion: Within-unit geometric–visual fusion improves performance over geometric features alone and is superior to the evaluated early- and late-fusion baselines.Visual descriptors are extracted from 80 × 80 RGB patches centered at joints, reduced from 300D to 20 dimensions using PCA.
  • Sensitivity analysis: Adding the trust gate gives better performance for all tested λ values than omitting it.This sensitivity analysis is shown in Figure 9 alongside variation in neuron size d.
  • Early stopping: Recognition improves when a larger portion of the testing video is supplied to the network under early-stopping evaluation.The experiment feeds portions p ∈ {0.1, 0.2, ..., 1.0} of each video using the cross-subject protocol.

4.4. Experiments on the UT-Kinect Dataset

The UT-Kinect experiments evaluate the method under two protocols and assess its multimodal feature fusion. The method performs competitively or better than comparison approaches, while fusion raises accuracy to 98%.

  • The method is evaluated under both LOOCV and half-vs-half protocols on UT-Kinect.Both evaluation protocols are used for the dataset.
  • Under LOOCV, the method achieves better performance than other skeleton-based methods.The comparison is reported in Table 3.
  • Under the half-vs-half protocol, the method reaches 95.0% accuracy, compared with 94.9% for Elastic functional coding.Elastic functional coding is described as an extension of the Lie Group model.
  • 98% accuracy is achieved with the proposed feature fusion, versus 97% using geometric features alone.Simple concatenation of geometric and visual features improves accuracy only slightly, whereas the proposed fusion reaches 98%.

4.5. Experiments on the SBU Interaction Dataset

The SBU Interaction experiments use five-fold cross-validation and traverse two skeletons across spatial steps. Tree traversal and the trust gate improve performance, with the combined model outperforming other skeleton-based methods.

  • The SBU Interaction evaluation uses five-fold cross-validation, with traversal spanning both skeletons in each frame.Average classification accuracy is reported in Table 6.
  • The ST-LSTM with tree traversal and trust gate outperforms all other skeleton-based methods.The result is reported for the combined “ST-LSTM (Tree) + Trust Gate” model.
  • Tree traversal achieves higher accuracy than the joint-chain arrangement because the latter introduces false links between less-related joints.
  • 93.3% accuracy is reached with the trust gate, compared with 88.6% for ST-LSTM (Tree) without it.The trust-gate model also exceeds Hierarchical RNN and is compared with Co-occurrence LSTM at 88.6%.

4.6. Experiments on the SYSU-3D Dataset

The SYSU-3D experiments evaluate the combined model with subject-based training and testing, while also examining skeleton normalization for viewpoint variation. The combined method outperforms baselines, and the trust gate provides about a 3% improvement on noisy data.

  • SYSU-3D uses samples from 20 subjects for training and 20 remaining subjects for testing, with 30-fold cross-validation.Mean accuracy is reported in Table 7.
  • The combined ST-LSTM (Tree) + Trust Gate model outperforms all baseline methods on SYSU-3D.The tree traversal strategy also improves the model’s classification accuracy.
  • The trust gate brings about a 3% improvement on SYSU-3D, where Kinect skeletal joints are noisy.The passage attributes the gate’s purpose to handling noisy data.
  • Skeleton normalization rotates each skeleton so normalized bodies face the same direction, addressing large viewpoint variations.The transformation aligns the shoulder vector with the X axis.

4.7. Experiments on the ChaLearn Gesture Dataset

The ChaLearn Gesture experiments report F1-score on the validation set under the adopted evaluation protocol. The proposed method surpasses the cited state-of-the-art methods, with its design targeting spatial-temporal dependencies and noisy Kinect data.

  • The ChaLearn Gesture evaluation follows the cited protocol and reports F1-score on the validation set.
  • The proposed method surpasses the state-of-the-art methods listed for the ChaLearn Gesture dataset.The comparison is reported in Table 9.
  • The method models both temporal and spatial dependency patterns in skeleton sequences.
  • The trust gate is incorporated to handle noisy Kinect skeleton data and can further improve results.

4.9. Experiments on the Berkeley MHAD Dataset

Experiments on Berkeley MHAD evaluate recognition accuracy, spatial traversal choices, temporal modeling, noise handling, and architectural links. The results support tree-structured spatial modeling and trust-gated spatio-temporal processing.

  • Berkeley MHAD Dataset: 100% accuracy is achieved on Berkeley MHAD using the specified train/test protocol, without preliminary manual smoothing.Training uses 384 sequences from the first seven persons; testing uses 275 sequences from the remaining five.
  • Trust Gate and Noise: Trust-gate experiments compare noisy and manually rectified skeletal inputs to assess how the mechanism responds to input noise.The MSR Action3D study measures trust-gate activation magnitudes, while Berkeley MHAD noise experiments perturb the right-foot joint by about 30 cm.
  • Spatial Sequence Models: Adding more spatial passes improves performance, but Double Joint Chain remains weaker than Tree traversal.The authors suggest that a second pass provides global context from the previous pass, while tree traversal preserves better semantic relations.
  • Spatial Sequence Models: ST-LSTM (Tree) outperforms the Joint Chain and Double Joint Chain models, supporting semantically structured spatial traversal.The tree connects semantically related joints while avoiding false connections between less-related joints.

4.12. Evaluation of Temporal Average, LSTM and ST-LSTM

This evaluation compares temporal averaging, conventional LSTM, and ST-LSTM to test temporal ordering and joint-wise spatial dependencies. ST-LSTM with the trust gate performs best among the compared temporal models, while the trust gate also benefits conventional LSTM.

  • Temporal and Spatial Modeling: ST-LSTM + Trust Gate significantly outperforms LSTM + Trust Gate, demonstrating the value of modeling spatial and temporal dependencies together.The comparison replaces ST-LSTM with an original LSTM that models temporal dynamics without explicitly modeling spatial dependencies.
  • Trust Gate: Adding the trust gate improves LSTM performance, but its gain is smaller than the gain on ST-LSTM.The authors attribute this difference to ST-LSTM having both spatial and temporal context available for estimating input reliability.
  • Temporal Average Baseline: Temporal Average ignores temporal ordering by averaging geometric features across all frames before classification.The averaged feature is passed to a two-layer network and then a softmax classifier.

5. Conclusion

The paper extends recurrent action recognition from temporal modeling to joint spatial and temporal modeling. It combines tree traversal, trust gating, and multi-modal fusion, with experiments across seven benchmark datasets supporting its effectiveness.

  • Conclusion: The proposed ST-LSTM analyzes 3D skeletal-joint locations across both frames and processing steps.This extends RNN-based action recognition beyond temporal dependencies to include spatial structure.
  • Conclusion: Tree traversal follows the body-joint adjacency graph to connect related joints and improve sequence representation.The traversal is designed to better represent skeletal structure and improve network performance.
  • Conclusion: Trust gating improves robustness to noisy input sequences, while multi-modal feature fusion is incorporated into the ST-LSTM framework.Experiments on seven challenging benchmark datasets validate the approach and report better performance than existing state-of-the-art methods.
Loading 1706.08276v1…