Source-linked AI summary
Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
Chiara Plizzari, Marco Cannici, Matteo Matteucci
TL;DR
Skeleton-based action recognition still faces challenges in encoding 3D skeleton information, particularly joint-motion patterns and correlations. ST-TR uses spatial and temporal Transformer self-attention in a two-stream network, achieving state-of-the-art results with joint coordinates and on-par results when bones are added.
Problem
Encoding latent 3D skeleton information, especially joint-motion patterns and correlations, remains an open problem in skeleton-based action recognition.
Method
ST-TR is a two-stream Transformer model that applies self-attention across spatial joints and temporal frames.
Results
ST-TR achieves state-of-the-art performance with joint information across the evaluated datasets and on-par performance with state-of-the-art methods when bones are added.
Takeaways & Limitations
Spatial self-attention provides flexible joint representations, while temporal self-attention extracts long-range dependencies across actions.
Takeaways & Limitations
Prior graph-convolution approaches use fixed body topology and local convolutional neighborhoods, which can underestimate correlations between nonadjacent joints.
Abstract
from arXiv · showhide
Skeleton-based Human Activity Recognition has achieved great interest in recent years as skeleton data has demonstrated being robust to illumination changes, body scales, dynamic camera views, and complex background. In particular, Spatial-Temporal Graph Convolutional Networks (ST-GCN) demonstrated to be effective in learning both spatial and temporal dependencies on non-Euclidean data such as skeleton graphs. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem, especially when it comes to extracting effective information from joint motion patterns and their correlations. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network, whose performance is evaluated on three large-scale datasets, NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics Skeleton 400, consistently improving backbone results. Compared with methods that use the same input data, the proposed ST-TR achieves state-of-the-art performance on all datasets when using joints' coordinates as input, and results on-par with state-of-the-art when adding bones information.
1. Introduction
Skeleton action recognition benefits from graph-based spatial-temporal modeling, but fixed topology and local convolutions can miss dynamic and long-range joint correlations. ST-TR addresses these limitations with spatial and temporal self-attention and reports stronger benchmark performance.
- Existing approach: ST-GCNs capture spatial intra-frame and temporal inter-frame information by operating on skeleton bone connections and time connections.
- Limitations: Fixed graph topology can restrict representations because information may only flow along predefined paths.
- Limitations: Convolutional spatial and temporal operations are limited to local neighborhoods, underestimating relevant correlations between unconnected joints such as clapping hands.
- Proposed method: ST-TR uses Transformer self-attention in a two-stream model operating across spatial and temporal dimensions.
- Proposed method: SSA dynamically builds action-conditioned links between joints, while TSA models the dynamics of each joint across time.
- Results: The model outperforms ST-GCN and A-GCN, achieves state-of-the-art performance with joint inputs, and is on-par with state-of-the-art methods when bones are added.
2. Related Works
Related work spans handcrafted features, recurrent and graph-based models for skeleton data, and Transformer self-attention for long-range dependencies. ST-TR applies self-attention directly to skeleton joints rather than coarse temporal clips.
- Skeleton action recognition: Early skeleton-action methods used handcrafted relative 3D rotations and translations, while later deep models exploited different aspects of skeleton data.
- Transformer-based methods: Prior SAN applies self-attention among temporal clips, whereas ST-TR applies it to skeleton joints to model joint-level dependencies.
- Graph neural networks: Graph neural networks generalize convolutional processing from grid data to non-Euclidean domains by transforming node and neighboring features iteratively.
- Transformers: Transformers rely on multi-head self-attention and were introduced to address long-sequence processing and sequential limitations in recurrent architectures.
3. Background
This section introduces ST-GCNs, their fixed-graph and local-convolution limitations, and Transformer self-attention as a non-local alternative for modeling skeleton dependencies.
- ST-GCN: ST-GCNs process skeletons with spatial graph convolutions over bone connections and temporal convolutions over consecutive time frames.The spatial-temporal graph contains intra-skeleton and inter-frame connections.
- ST-GCN limitations: Spatial and temporal convolutions operate on local neighborhoods determined by convolution kernel sizes.This limits the receptive field of the convolutional operations.
- ST-GCN limitations: ST-GCN graph topology is predefined and fixed across layers and actions, potentially restricting representations of skeleton movements.Directed links may also constrain information flow to predefined paths.
- Transformer self-attention: Transformer self-attention compares node pairs and combines value vectors using normalized relevance scores to produce context-enriched embeddings.The operator uses query, key, and value vectors; dividing by √d_k improves gradient stability, while multi-head attention combines multiple parameterized attention operations.
4. Spatial Temporal Transformer Network
ST-TR replaces fixed spatial-temporal graph convolutions with self-attention modules that separately model joint relations within frames and across time. Its two-stream design combines spatial and temporal Transformer processing with complementary convolutional and graph-convolutional features.
- Overview: ST-TR uses Spatial Self-Attention (SSA) and Temporal Self-Attention (TSA) to extract correlations separately along spatial and temporal dimensions.The two modules operate within the broader Spatial Temporal Transformer network.
- Spatial Self-Attention: SSA computes correlations between every pair of joints independently in each frame, producing embeddings that encode body-part relationships.The input is reshaped so self-attention operates separately on each time frame.
- Spatial Self-Attention: SSA dynamically predicts joint relationships from each skeleton pose, allowing the correlation structure to adapt across actions rather than remain fixed.The module behaves like graph convolution on a fully connected graph with pose-dependent kernel values.
- Temporal Self-Attention: TSA studies each joint across all frames, computing inter-frame correlations for the same body joint and capturing relations between temporally distant frames.This can capture dependencies such as those between the first and last frames beyond a standard ST-GCN kernel’s receptive field.
- Two-Stream Architecture: The two-stream architecture uses SSA followed by temporal convolution in S-TR, TSA alongside graph convolution in T-TR, and sums their softmax outputs for prediction.Both streams begin with three residual feature-extraction layers and are trained end-to-end with their corresponding layers.
5. Model Evaluation
The evaluation compares self-attention modules with graph and temporal convolution baselines on NTU-60, measuring accuracy, parameter efficiency, stream combinations, and architectural variants.
- Experimental design: The experiments evaluate separate S-TR and T-TR streams, their combination, and alternative configurations on NTU-60 and related benchmarks.The study also compares module complexity and uses controlled baselines to assess spatial and temporal self-attention.
- Model complexity: SSA uses fewer parameters than GC and AGC, with maximum reductions of 1.1 × 10^5 and 5 × 10^5, respectively.The comparison sets Cin = Cout and reports parameter differences for spatial modules.
- Model complexity: Temporal self-attention reduces parameters relative to temporal convolution by 16.8 × 10^5 at the individual-module level and 16.7 × 10^5 for the T-TR stream versus ST-GCN.The reported temporal convolution uses a 1 × F filter, typically with F = 9, across T = 300 frames.
- Stream comparisons: T-TR is 13.4 × 10^5 parameters lighter than the ST-GCN temporal baseline and improves accuracy by 0.9%.This comparison evaluates self-attention replacing standard temporal convolution.
- Stream combinations: Combining S-TR and T-TR reaches 88.7% accuracy on NTU-60 X-Sub and 95.6% on X-View using joint information only.The combined model outperforms the ST-GCN baseline and other two-stream architectures in the cited comparison.
- Ablations: Applying self-attention from the first layer performs worse than the corresponding later-stage configurations, although it still outperforms the ST-GCN baseline.The authors therefore use high-level features extracted by residual GCN and TCN modules before applying self-attention.
- Ablations: The augmented spatial configuration reaches 94.5%, compared with 94% for S-TR, while reduced temporal attention features prevent improvement on the T-TR stream.The augmentation concatenates convolutional and self-attention features.
6. Comparison with State-Of-The-Art Results
Across NTU-60, NTU-120, and Kinetics, ST-TR outperforms methods using the same joint inputs and remains competitive with leading methods when bones are added.
- NTU-60: On NTU-60, ST-TR with joints only outperforms all compared state-of-the-art models using the same information, including SAN by up to 3%.With bones information, the transformer configurations are also compared against methods using the same input type.
- NTU-120: On NTU-120, the joints-only model outperforms all state-of-the-art methods using the same information.With bones, ST-TR and ST-TR-agcn outperform 2s-AGCN by up to 3% on both X-Sub and X-Set.
- Kinetics: On Kinetics, the joints-only model outperforms ST-GCN by 5% and all previous methods using only joint information.Adding bones yields performance above 2s-AGCN and DGNN and on-par with MS-G3D.
7. Qualitative Results
Spatial Self-Attention maps reveal action-dependent joint relevance and correlations, evolving from sparse local patterns in early layers to global node importance in deeper layers.
- 25×25 attention maps encode predicted correlations between every pair of body joints within the same frame.Rows and columns represent body joints; each element (i, j) represents the correlation between joints i and j.
- Different actions activate different body parts according to the spatial self-attention patterns.Joint-circle radii indicate predicted relevance, while heatmaps show attention scores from the last layer.
- Early layers capture sparse, low-level correlations between body joints.The sparsity of activations highlights these initial local relationships.
- Deeper layers reveal global importance across joints, including especially relevant nodes represented by vertical attention lines.
8. Conclusions
The paper introduces Transformer self-attention as an alternative to graph convolution for skeleton activity recognition. Across three datasets, ST-TR provides flexible spatial and long-range temporal representations while achieving strong benchmark performance, although self-attention-only configurations remain sub-optimal.
- Transformer self-attention is introduced as an alternative to graph convolution in skeleton activity recognition.
- SSA enables more flexible and dynamic spatial representations than graph convolution.The module replaces graph convolution for modeling spatial information.
- TSA extracts long-range dependencies across actions beyond the strict locality of standard convolution.
- ST-TR achieves state-of-the-art performance on NTU-60, NTU-120, and Kinetics with the same joint input and stream setup.With bones information added, its results are on-par with state-of-the-art methods.
- Configurations using only self-attention modules are sub-optimal, motivating future work on a unified Transformer architecture.