Source-linked AI summary
An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, Tieniu Tan
TL;DR
Skeleton-based action recognition requires discriminative spatial and temporal feature extraction, including relationships between the two domains. The paper proposes AGC-LSTM with graph convolutional LSTM layers, key-joint attention, and temporal hierarchy, and reports state-of-the-art results on NTU RGB+D and Northwestern-UCLA. A stated limitation is that sparse finger-joint annotations make subtle hand movements difficult to capture.
Problem
Extracting discriminative spatial and temporal features from skeleton sequences remains challenging, despite the importance of modeling both domains.
Method
AGC-LSTM uses graph convolutional LSTM layers to model spatial-temporal features and their co-occurrence, with key-joint attention and a temporal hierarchical architecture.
Results
The proposed model achieves state-of-the-art results on both the NTU RGB+D and Northwestern-UCLA datasets.
Takeaways & Limitations
AGC-LSTM provides a framework for synchronously learning discriminative spatial and temporal characteristics from skeleton data.
Takeaways & Limitations
On NTU, only two finger joints are annotated, making subtle hand movements difficult to capture and contributing to confusion among similar actions.
Abstract
from arXiv · showhide
Skeleton-based action recognition is an important task that requires the adequate understanding of movement characteristics of a human action from the given skeleton sequence. Recent studies have shown that exploring spatial and temporal features of the skeleton sequence is vital for this task. Nevertheless, how to effectively extract discriminative spatial and temporal features is still a challenging problem. In this paper, we propose a novel Attention Enhanced Graph Convolutional LSTM Network (AGC-LSTM) for human action recognition from skeleton data. The proposed AGC-LSTM can not only capture discriminative features in spatial configuration and temporal dynamics but also explore the co-occurrence relationship between spatial and temporal domains. We also present a temporal hierarchical architecture to increases temporal receptive fields of the top AGC-LSTM layer, which boosts the ability to learn the high-level semantic representation and significantly reduces the computation cost. Furthermore, to select discriminative spatial information, the attention mechanism is employed to enhance information of key joints in each AGC-LSTM layer. Experimental results on two datasets are provided: NTU RGB+D dataset and Northwestern-UCLA dataset. The comparison results demonstrate the effectiveness of our approach and show that our approach outperforms the state-of-the-art methods on both datasets.
1. Introduction
Skeleton-based action recognition must model spatial structure, temporal continuity, and their co-occurrence while extracting discriminative features. AGC-LSTM addresses this with graph convolutional LSTM layers, attention to key joints, and temporal hierarchy, achieving state-of-the-art results on two datasets.
- Motivation: Skeleton sequences exhibit adjacent-joint correlations, temporal continuity across joints and body structure, and spatial-temporal co-occurrence.These characteristics motivate synchronously learning spatial and temporal representations.
- Proposed approach: Graph convolution within AGC-LSTM captures discriminative spatial configuration, temporal dynamics, and their co-occurrence.The graph convolutional operator lets the model process spatially structured skeleton information while modeling temporal behavior.
- Proposed approach: AGC-LSTM is proposed as a graph convolutional LSTM framework for skeleton-based action recognition.The work describes it as the first graph convolutional LSTM attempt for this task.
- Proposed approach: Attention enhances features of key nodes to improve spatiotemporal expressions.The mechanism is applied within the AGC-LSTM framework to select discriminative spatial information.
- Proposed approach: A temporal hierarchical architecture expands high-level spatiotemporal receptive fields while significantly reducing computational cost.The architecture is intended to improve learning of high-level semantic features.
- Results: The model achieves state-of-the-art results on both the NTU RGB+D and Northwestern-UCLA datasets.The paper reports extensive experiments demonstrating the model’s effectiveness.
2. Related Work
Related work spans graph-based models, hand-crafted and deep-learning skeleton representations, and architectures that separately or jointly model spatial and temporal information.
- Graph-based models: Graph-based models include graph neural networks, which combine graph and recurrent neural networks, and graph convolutional networks, which generalize convolution to graph data.GCNs include spectral and spatial variants.
- Skeleton representations: Traditional skeleton methods use hand-crafted representations such as relative 3D rotations and relative 3D geometry between body parts.
- Deep-learning methods: Deep-learning approaches include hierarchical recurrent networks, spatial-temporal attention, view-adaptive models, hierarchical CNNs, and graph convolutional networks.
- Spatial-temporal modeling: Recent methods emphasize discriminative spatial and temporal features, while ST-GCN models spatial characteristics and temporal dynamics through separate graph and convolutional operators.
3. Model Architecture
AGC-LSTM replaces LSTM’s fully connected inner operation with graph convolution to jointly model skeleton spatial structure and temporal dynamics. Its attention mechanism enhances key-joint information, while temporal hierarchy and joint-part processing extend temporal context and support action prediction.
- Graph convolutional modeling: AGC models skeleton sequences with graph convolution over spatially connected joints, addressing LSTM’s limitation of ignoring spatial correlation.Graph convolution operates on graph-structured inputs, hidden states, and cell memories.
- AGC-LSTM unit: The AGC-LSTM unit retains input, forget, and output gates, but computes them with graph convolution so states encode spatial structure and temporal dynamics.The graph-structured cell memory and hidden state are passed through the recurrent unit.
- Spatial attention: The attention network adaptively selects key joints from an intermediate hidden state and enhances their representations without weakening non-focused-node information.The enhanced hidden state is fed to the next AGC-LSTM layer.
- Prediction and joint-part modeling: The model forms global and local features for class scoring, and a hybrid joint-part model is reported to further improve performance.The joint-based, part-level, and hybrid configurations are all evaluated within the proposed framework.
- Input feature representation: Joint features combine position and frame-difference information, with a shared LSTM used to reduce scale variance before AGC-LSTM processing.The linear layer maps joint coordinates to position features, while frame differences provide dynamic information.
- Temporal hierarchical architecture: Three stacked AGC-LSTM layers use temporal average pooling to enlarge the top layer’s temporal receptive field.The temporal hierarchy is inspired by spatial pooling in CNNs.
4.1. Datasets
The experiments use NTU RGB+D and Northwestern-UCLA, two skeleton-action datasets captured with Kinect cameras and covering diverse action categories, subjects, and viewpoints.
- NTU RGB+D: NTU RGB+D contains 60 action classes and 56,880 samples performed by 40 subjects, with 3D skeletons captured by three Kinect v2 cameras.The classes include daily, mutual, and health-related actions.
- Northwestern-UCLA: Northwestern-UCLA contains 1,494 clips across 10 categories, performed by 10 subjects and captured from three Kinect cameras.The dataset includes RGBD and human skeleton data with multiple viewpoints.
- Northwestern-UCLA: Northwestern-UCLA trains on samples from the first two cameras and tests on samples from the remaining camera.This protocol evaluates performance across camera viewpoints.
4.2. Implementation Details
Implementation samples fixed-length skeleton sequences and uses directly connected neighbors with a three-subset graph labeling scheme.
- Input sampling: The input sequence length is T = 100 for NTU and T = 50 for Northwestern-UCLA.Each dataset therefore uses a different fixed temporal input length.
- Graph configuration: Each node uses only directly connected neighbors, setting D = 1, and neighbors are partitioned into K = 3 subsets.The subsets are the root node, centripetal group, and the remaining group described in the implementation passage.
4.3. Results and Comparisons
AGC-LSTM is evaluated against state-of-the-art methods on NTU RGB+D and Northwestern-UCLA, achieving the best reported results on both datasets. Comparisons also examine performance against CNN-, GCN-, and LSTM-related approaches.
- 95.0% and 89.2% are the best reported accuracies for AGC-LSTM under the two NTU evaluation protocols.
- AGC-LSTM outperforms HCN by about 3.9% for cross-view evaluation and 2.7% for cross-subject evaluation.
- AGC-LSTM reaches 93.5% and 87.5% on joint-level evaluation, outperforming ST-GCN by 5.2% and 6.0%.
- AGC-LSTM outperforms PB-GCN by 1.8% and 1.7% across the two reported evaluations.
- AGC-LSTM outperforms the spatial-reasoning and temporal-stack model by 2.6% and 4.4%, respectively.
- 93.3% is the best reported accuracy on Northwestern-UCLA, exceeding TS-LSTM by 4.1%.
4.4. Model Analysis
Model analyses attribute performance gains to temporal hierarchy, graph-convolutional spatial-temporal modeling, attention over key joints, and joint/part feature fusion. Attention visualizations and confusion matrices further examine learned focus and failure cases.
- Architecture Analysis: Temporal hierarchical architecture increases the top-layer temporal receptive field and improves temporal-dynamics representation.With GC-LSTM, it increases accuracies by 2.5% and 4.9% on NTU and 10.9% on Northwestern-UCLA.
- Architecture Analysis: AGC-LSTM adds spatial attention to GC-LSTM to select information from key joints and promote feature representation.
- Attention Visualization: Across three layers, attention for handshaking increasingly focuses on the right elbow, right wrist, and right hand.The right-hand tip and right thumb also receive some attention.
- Architecture Analysis: Fusing part-based and joint-based AGC-LSTM further improves performance.
- Confusion Analysis: AGC-LSTM improves classification of similar Northwestern-UCLA actions such as picking up with one hand versus two hands.
- Failure Case: On NTU, 20% of reading samples are misclassified as writing, while 19% of writing sequences are misclassified as typing on a keyboard.These failures involve subtle hand movements, with only two finger joints marked in the NTU skeleton data.
5. Conclusion and Future Work
The paper concludes that AGC-LSTM achieves state-of-the-art skeleton-based action recognition by jointly modeling spatial, temporal, and cross-domain co-occurrence information. It identifies pose-object relations and combining skeleton sequences with object appearance as future directions.
- AGC-LSTM jointly captures spatial configuration, temporal dynamics, and their co-occurrence relationship for skeleton-based action recognition.
- Attention enhances information from key joints in each AGC-LSTM layer, while temporal hierarchy captures high-level spatiotemporal semantic features.
- The model achieves state-of-the-art results on two challenging benchmarks.
- The paper identifies pose-object relations and combining skeleton sequences with object appearance as directions for overcoming failure-case limitations.