Source-linked AI summary
Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning
Chenyang Si, Ya Jing, Wei Wang, Liang Wang, Tieniu Tan
TL;DR
Skeleton-based action recognition still lacks abundant spatial structure information and detailed temporal dynamics in many representations. The paper proposes SR-TSL, combining spatial reasoning with temporal stack learning and clip-based incremental loss, and reports much better results than state-of-the-art methods on SYSU and NTU RGB+D.
Problem
Many skeleton-based action-recognition methods do not model abundant spatial structure information and detailed temporal dynamics features.
Method
SR-TSL combines a residual-GNN spatial reasoning network, multiple skip-clip LSTMs for temporal stack learning, and clip-based incremental loss.
Results
SR-TSL achieves much better results than state-of-the-art methods on the SYSU 3D Human-Object Interaction and NTU RGB+D datasets.
Takeaways & Limitations
The experiments verify the effectiveness of the model for skeleton-based action recognition and provide an approach to long-term sequence optimization.
Abstract
from arXiv · showhide
Skeleton-based action recognition has made great progress recently, but many problems still remain unsolved. For example, most of the previous methods model the representations of skeleton sequences without abundant spatial structure information and detailed temporal dynamics features. In this paper, we propose a novel model with spatial reasoning and temporal stack learning (SR-TSL) for skeleton based action recognition, which consists of a spatial reasoning network (SRN) and a temporal stack learning network (TSLN). The SRN can capture the high-level spatial structural information within each frame by a residual graph neural network, while the TSLN can model the detailed temporal dynamics of skeleton sequences by a composition of multiple skip-clip LSTMs. During training, we propose a clip-based incremental loss to optimize the model. We perform extensive experiments on the SYSU 3D Human-Object Interaction dataset and NTU RGB+D dataset and verify the effectiveness of each network of our model. The comparison results illustrate that our approach achieves much better results than state-of-the-art methods.
1 Introduction
Skeleton-based action recognition remains limited by incomplete spatial structure modeling and insufficiently detailed temporal dynamics. SR-TSL addresses these challenges with spatial reasoning, temporal stack learning, and clip-based incremental optimization.
- Skeleton-based action recognition supports applications including intelligent video surveillance, sports analysis, video retrieval, and improved robot–human interaction.
- Existing methods face difficulty representing coordinated relationships among different body parts and detailed temporal dynamics in skeleton sequences.
- SR-TSL combines a spatial reasoning network using residual graph neural networks with a temporal stack learning network using multiple skip-clip LSTMs.
- The clip-based incremental loss improves temporal stack learning, speeds convergence, and improves recognition performance.
- SR-TSL achieves state-of-the-art results on the SYSU 3D Human-Object Interaction and NTU RGB+D datasets.
2 Related Work
Related work spans handcrafted skeleton-motion descriptors and deep neural networks, especially CNN-, RNN-, and LSTM-based approaches. Graph neural networks provide another family of methods for graph-structured data.
- Skeleton-based action-recognition literature is divided into handcrafted-feature and deep-neural-network classes.
- Handcrafted approaches represent skeleton motion using local occupancy patterns, covariance descriptors, or Lie-group relationships between body parts.
- Deep learning approaches use CNNs and recurrent architectures, including hierarchical RNNs, fully connected LSTMs, and spatial-temporal attention models.
- Graph neural network research includes spectral graph convolutions and spatial convolutions over graph nodes and their neighbors.
3 Overview
The paper reviews graph neural networks, recurrent neural networks, and LSTMs as components relevant to its framework. These models update node or sequence representations to capture structural and temporal dependencies.
- A graph neural network represents graph-structured data as nodes and edges, with node states updated from neighborhood information.
- At each time step, GNNs aggregate messages from neighboring nodes before updating hidden states and computing outputs.
- GNN message, state, and output functions are learned neural networks shared across time steps.
- Standard RNNs capture sequence dependencies but can suffer vanishing and exploding gradients on long-term sequences.
- LSTM architecture uses input, forget, and output gates plus a cell to improve learning of long-term dependencies.
4 Model Architecture
SR-TSL combines spatial reasoning over body parts with temporal stack learning over continuous skeleton clips. Its RGNN encodes spatial relationships, while skip-clip LSTMs and clip-based incremental losses model increasingly rich temporal dynamics.
- Spatial Reasoning Network: The spatial reasoning network encodes each body part with a shared linear layer before modeling inter-part structural relationships using an RGNN.The body is decomposed into parts such as arms, legs, and trunk; RGNN updates operate over these part features.
- Spatial Reasoning Network: The RGNN residual design adds relationship features between body parts to individual part features, producing fused spatial representations.After repeated updates, node-level outputs represent each part’s spatial structural relationships, which are combined into frame-level spatial information.
- Temporal Stack Learning Network: The temporal stack learning network divides a sequence into continuous clips and processes spatial features alongside temporal differences from position and velocity streams.Each clip contains d-frame features, while velocity features are computed as differences between consecutive spatial-feature frames.
- Temporal Stack Learning Network: Three skip-clip LSTM layers use shared LSTM processing across clips, with later clips initialized from previous clip representations to preserve long-term dependencies.The representation of clip m aggregates its detailed dynamics with those of preceding clips, and later clips contain richer temporal dynamics.
- Training and Classification: A clip-based incremental loss weights clips according to their temporal information and is combined into the model’s training loss.The method uses the final aggregated representation and probability vector during testing to predict the sequence class.
5 Experiments
Experiments evaluate SR-TSL on SYSU and NTU under standard protocols, compare it with prior methods and component variants, and analyze convergence, clip length, and RGNN time steps.
- Datasets and Experimental Settings: SR-TSL is evaluated on NTU RGB+D and SYSU using their reported standard evaluation protocols.NTU uses Cross-Subject and Cross-View settings; SYSU uses setting-1 and setting-2.
- Experimental Results: 84.8% and 92.4% are SR-TSL’s best NTU accuracies for Cross-Subject and Cross-View evaluation, respectively.These results outperform VA-LSTM by about 5.4% and 4.8%, and Ensemble TS-LSTM by 10.2% and 11.1%.
- Experimental Results: 80.7% and 81.9% are SR-TSL’s best SYSU accuracies for setting-1 and setting-2, respectively.The results outperform the cited state-of-the-art approach by about 3.8% and 4.4%.
- Model Analysis: Component comparisons on NTU and SYSU show that SRN and TSLN improve performance, with the temporal stack learning network contributing especially strongly.The evaluated variants include FC+LSTM, SRN+LSTM, FC+TSLN, SR-TSL (Position), and SR-TSL (Velocity).
- Model Analysis: The two-stream TSLN learns temporal dynamics from both velocity and position sequences, while increasing clips provides richer temporal information and improves accuracy.The reported analysis also states that SR-TSL speeds convergence and improves performance during learning.
- Model Analysis: Increasing the RGNN time step yields only small gains before performance quickly saturates, while clip-length effects are examined separately.These analyses are reported in Tables 4 and 5.
6 Conclusions
The paper concludes that SR-TSL combines spatial reasoning, temporal stack learning, and clip-based incremental loss for long-term skeleton-based action recognition. Experiments on NTU RGB+D and SYSU verify the model’s effectiveness, while future work considers error analysis and contextual interactions.
- Conclusions: SR-TSL captures high-level within-frame spatial structure and detailed sequence dynamics for long-term skeleton-based action recognition.The spatial reasoning network models spatial structure, while temporal stack learning models temporal dynamics.
- Conclusions: The clip-based incremental loss further improves stack learning and provides an approach to long-term sequence optimization.
- Conclusions: Extensive experiments on NTU RGB+D and SYSU verify the effectiveness of the proposed model.
- Conclusions: Future work will analyze error samples and incorporate contextual information such as interactions.