Source-linked AI summary
Spatial Temporal Transformer Network for Skeleton-based Action Recognition
Chiara Plizzari, Marco Cannici, Matteo Matteucci
TL;DR
Skeleton-based action recognition still lacks effective encoding of latent 3D-skeleton information. The paper proposes ST-TR, which uses spatial and temporal Transformer self-attention in a two-stream network, achieving state-of-the-art performance on NTU-RGB+D 60 and 120 with matching input information and stream setups.
Problem
Skeleton-based action recognition lacks effective context-aware, fine-grained spatial-temporal feature extraction for encoding 3D skeleton information.
Method
ST-TR is a two-stream Transformer-based architecture using SSA for adaptive joint relationships and TSA for temporal correlations.
Results
ST-TR achieves state-of-the-art performance on NTU-RGB+D 60 and 120 against methods using the same input information and stream setup.
Takeaways & Limitations
SSA can replace graph convolution for more flexible spatial representations, while TSA extracts global motion patterns beyond standard convolutional locality.
Abstract
from arXiv · showhide
Skeleton-based human action recognition has achieved a great interest in recent years, as skeleton data has been demonstrated to be robust to illumination changes, body scales, dynamic camera views, and complex background. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network which outperforms state-of-the-art models using the same input data on both NTU-RGB+D 60 and NTU-RGB+D 120.
1 Introduction
The paper targets limitations in skeleton-based action recognition by applying Transformer self-attention to model adaptive spatial and temporal joint relationships. Its two-stream model outperforms ST-GCN and prior state-of-the-art methods using the same input data.
- Skeleton-based action recognition still lacks architectures that extract context-aware, fine-grained spatial-temporal features from 3D skeletons.
- Transformer self-attention is used to model long-range dependencies and address weaknesses associated with convolution-based approaches.
- The proposed two-stream model applies self-attention across both spatial and temporal dimensions.
- SSA dynamically builds action-conditioned links between skeleton joints, independently of the natural human body structure.
- ST-TR outperforms the ST-GCN baseline and previous state-of-the-art methods using the same input data on NTU-RGB+D.
2 Spatial Temporal Transformer Network
ST-TR uses separate self-attention modules to learn spatial relations between joints and temporal correlations across frames. These modules are combined with convolutional and graph-convolutional components in a two-stream architecture.
- 2 Spatial Temporal Transformer Network: ST-TR applies Transformer self-attention separately to spatial and temporal dimensions through SSA and TSA modules.
- 2.1 Spatial Self-Attention (SSA): SSA computes pairwise joint correlations independently within each frame to embed relationships between body parts.
- 2.1 Spatial Self-Attention (SSA): SSA predicts correlation scores dynamically from the skeleton pose rather than using a fixed structure across actions.
- 2.2 Temporal Self-Attention (TSA): TSA compares features of the same joint across frames, allowing the network to learn long-range inter-frame relations beyond standard convolutional locality.
- 2.3 Two-Stream Spatial Temporal Transformer Network: In 2s-ST-TR, the S-TR stream combines SSA with temporal convolution, while the T-TR stream combines TSA with graph convolution.
3 Model Evaluation
The model is evaluated on NTU-60 and NTU-120 using accuracy comparisons, parameter analysis, and separate stream experiments against ST-GCN. The combined streams improve performance over the baseline, while bone information is also examined.
- Datasets: NTU-60 contains 25-joint 3D skeletons across 60 action classes, while NTU-120 extends the benchmark to 120 classes and 113,945 videos.The datasets use cross-subject and cross-view or cross-setup evaluation criteria.
- Single-stream evaluation: S-TR outperforms ST-GCN by 0.7% on NTU-60 X-Sub and 1.3% on X-View.This comparison evaluates the spatial self-attention stream against the graph-convolution baseline.
- Combined-stream evaluation: The combined ST-TR streams achieve 88.7% accuracy on X-Sub and 95.6% on X-View, outperforming ST-GCN by up to 3%.The combination also surpasses other two-stream architectures in the reported comparison.
- Parameter analysis: TSA introduces 41.3 × 10^4 fewer parameters than the corresponding temporal convolution module.The parameter comparison uses Cin = Cout = 256.
- Evaluation protocol: The experiments compare joint-only and joint-plus-bones configurations against methods using matching input information and stream setups.Bone vectors are computed from differences between the 3D coordinates of connected joints and concatenated with joint information.
4 Comparison with State-Of-The-Art
ST-TR is compared with state-of-the-art methods on NTU-60 and NTU-120 under matched joint or joint-plus-bones inputs and stream configurations. It outperforms the corresponding prior methods in each reported comparison.
- NTU-60: On NTU-60, joint-only ST-TR outperforms all compared state-of-the-art models without bones, including 1s-AGCN and SAN.The joint-plus-bones ST-TR configuration likewise outperforms prior two-stream methods using bones, including 2s-AGCN and 2s Shift-GCN.
- NTU-120: On NTU-120, joint-only ST-TR outperforms all compared state-of-the-art methods using the same joint information.The comparisons use matching input information and one- or two-stream architectures for fairness.
5 Conclusions
The paper presents Transformer self-attention as an alternative to graph convolution for skeleton action recognition. SSA models spatial joint relations, TSA models temporal dynamics, and ST-TR achieves state-of-the-art performance with matched inputs and stream setups.
- Conclusion: ST-TR introduces Transformer self-attention in skeleton activity recognition as an alternative to graph convolution.The approach is evaluated on NTU-60 and NTU-120.
- Conclusion: SSA can replace graph convolution with more flexible and dynamic representations of spatial relations.The module operates on relationships between skeleton joints.
- Conclusion: TSA addresses the strict locality of standard convolution and extracts global motion patterns over time.It models temporal dependencies in the skeleton sequence.
- Conclusion: ST-TR achieves state-of-the-art performance on NTU-RGB+D when compared with methods using the same input information and stream setup.The conclusion covers experiments on both NTU-60 and NTU-120.