Source-linked AI summary
Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition
Jun Liu, Amir Shahroudy, Dong Xu, Gang Wang
TL;DR
3D skeleton action recognition needs models that capture both temporal context and spatial joint dependencies while handling noisy or occluded inputs. The paper introduces ST-LSTM with tree-based traversal and a trust gate, reporting state-of-the-art performance across four benchmark datasets.
Problem
Existing RNN-based methods primarily model temporal context, while spatial joint dependencies and unreliable 3D inputs remain important modeling challenges.
Method
The paper uses ST-LSTM to jointly model spatial and temporal context, traverses skeletons as trees, and applies a trust gate to estimate input reliability.
Results
The method achieves state-of-the-art performance on all evaluated datasets, with accuracy reaching 93.3% on the reported SBU result after adding the trust gate.
Takeaways & Limitations
Jointly modeling skeletal structure and temporal context, while adapting memory updates to input reliability, improves robustness to noisy and occluded 3D skeleton data.
Abstract
from arXiv · showhide
3D action recognition - analysis of human actions based on 3D skeleton data - becomes popular recently due to its succinctness, robustness, and view-invariant representation. Recent attempts on this problem suggested to develop RNN-based learning methods to model the contextual dependency in the temporal domain. In this paper, we extend this idea to spatio-temporal domains to analyze the hidden sources of action-related information within the input data over both domains concurrently. Inspired by the graphical structure of the human skeleton, we further propose a more powerful tree-structure based traversal method. To handle the noise and occlusion in 3D skeleton data, we introduce new gating mechanism within LSTM to learn the reliability of the sequential input data and accordingly adjust its effect on updating the long-term context information stored in the memory cell. Our method achieves state-of-the-art performance on 4 challenging benchmark datasets for 3D human action analysis.
1 Introduction
Existing RNN-based approaches for 3D action recognition primarily model temporal context, while spatial dependencies between joints also provide discriminative information. The paper addresses both domains with ST-LSTM, tree-based skeleton traversal, and a trust gate for unreliable inputs.
- Existing RNN-based 3D action recognition methods mainly represent motion dynamics through long-term temporal context.
- Spatial dependencies between joints are also strong, and joint configurations within video frames can be highly discriminative.
- ST-LSTM extends LSTM learning to temporal and spatial domains concurrently, allowing each joint to receive context from neighboring joints and previous frames.
- A tree-based skeleton traversal models skeletal adjacency and kinematic relationships more naturally than a simple chain of joints.
- The trust gate estimates input reliability at each spatiotemporal step and adjusts how noisy or occluded data affect long-term memory updates.
- The paper reports state-of-the-art performance across all evaluated datasets.
2 Related Work
Prior RNN and LSTM methods organize skeleton information through hierarchical, regularized, differential, or part-based representations. In contrast, this paper jointly models spatial and temporal joint dependencies and adds a trust gate for noisy inputs.
- HBRNN hierarchically divides the skeleton into five joint groups, processes them with separate bidirectional RNNs, and progressively combines body representations.
- Deep LSTM methods use mixed-norm regularization to encourage discriminative joint co-occurrence and internal dropout on LSTM gate activations.
- Differential LSTM adds a gate that tracks memory-state derivatives to discover patterns in salient motion dynamics.
- Part-aware LSTM separates the memory cell into part-based sub-cells for learning long-term context representations independently by body part.
- Unlike these approaches, the proposed framework explicitly models dependencies between joints while recurrently analyzing spatial and temporal domains concurrently.
- The framework also introduces a trust gate intended to improve LSTM robustness to noisy input data.
3 Spatio-Temporal Recurrent Networks
The paper extends recurrent modeling of 3D skeleton actions from temporal dynamics to concurrent spatial and temporal dependencies. It uses tree-structured traversal and a trust gate to preserve skeletal adjacency and regulate unreliable sensor inputs.
- Existing RNN methods primarily model temporal context, although spatial joint configurations also contain discriminative information for 3D action recognition.
- ST-LSTM units receive the current joint, the previous joint’s hidden representation, and the same joint’s hidden representation from the previous frame.Separate spatial and temporal forget gates handle the two incoming contextual channels.
- The model applies recurrent learning concurrently across spatial joints and temporal frames to encode spatio-temporal context.Each unit corresponds to a skeletal joint and receives context from neighboring joints and previous frames.
- Tree-Structure based Traversal: Tree traversal preserves skeletal adjacency and transmits contextual information both top-down and bottom-up, strengthening spatial dependency modeling over a simple chain.The traversal visits each tree connection twice, so nodes receive information from descendants and ancestors.
- Single-joint inputs reduce model parameters and provide weight-sharing regularization, which can improve generalization when training samples are limited.The paper identifies small training sets as common in current 3D action-recognition datasets.
- Spatio-Temporal LSTM with Trust Gates: The trust gate estimates input reliability from contextual prediction and adjusts whether unreliable measurements influence long-term memory updates.It can block the input gate when current joint measurements are corrupted by noise or occlusion.
4 Experiments
Experiments evaluate joint traversal, trust gating, layer stacking, early stopping, and robustness across four 3D action-recognition datasets. The proposed configurations consistently benefit from modeling skeletal structure and noisy-input reliability, with strong reported results across protocols.
- Experimental setup: Experiments compare simple joint-chain, tree-traversal, and tree-traversal-plus-trust-gate ST-LSTM configurations across four benchmark datasets.The datasets are NTU RGB+D, SBU Interaction, UT-Kinect, and Berkeley MHAD.
- NTU RGB+D: Both ST-LSTM (Joint Chain) and ST-LSTM (Tree Traversal) outperform temporal-only Deep RNN and deep LSTM models on NTU RGB+D.The compared temporal-only models concatenate joint features at each frame and ignore spatial dynamics.
- NTU RGB+D: 81.6% accuracy with trust gate exceeds 76.5% without it on NTU RGB+D side-view samples, where skeletal data is less accurate.The authors interpret this as evidence that the trust gate can handle severely noisy data.
5 Conclusion
The paper extends RNN-based 3D action recognition into the spatio-temporal domain and introduces skeleton-aware traversal and reliability gating. Experiments on four datasets validate these contributions and report superior performance over existing state-of-the-art methods.
- The ST-LSTM analyzes each individual joint’s 3D location at each processing step across spatial and temporal domains.
- The skeleton tree traversal algorithm uses body-joint adjacency to arrange related joints together in the input sequence.
- A new gating mechanism improves robustness against noise and occlusion in unreliable 3D input data.
- Experimental results on four evaluated datasets validate the contributions and show superior performance over existing state-of-the-art methods.