Source-linked AI summary
Memory Attention Networks for Skeleton-based Action Recognition
Chunyu Xie, Ce Li, Baochang Zhang, Chen Chen, Jungong Han, Changqing Zou, Jianzhuang Liu
TL;DR
Skeleton-based action recognition is challenged by complex spatio-temporal joint variations that remain difficult for RNNs. MANs address this with end-to-end temporal attention recalibration followed by CNN-based spatial-temporal modeling, achieving the best results on four benchmark datasets.
Problem
Complex spatio-temporal variations from action speed, jitters, and surrounding distractions remain challenging for RNN-based skeleton action recognition.
Method
MANs use TARM to recalibrate attention across skeleton frames, then STCM applies CNNs to model spatial and temporal information in the calibrated sequences.
Results
MANs consistently achieve the best results on four benchmark datasets: NTU RGB+D, HDM05, SYSU-3D, and UT-Kinect.
Takeaways & Limitations
The unified temporal-then-spatial recalibration framework improves skeleton-based action recognition across the evaluated benchmarks.
Abstract
from arXiv · showhide
Skeleton-based action recognition task is entangled with complex spatio-temporal variations of skeleton joints, and remains challenging for Recurrent Neural Networks (RNNs). In this work, we propose a temporal-then-spatial recalibration scheme to alleviate such complex variations, resulting in an end-to-end Memory Attention Networks (MANs) which consist of a Temporal Attention Recalibration Module (TARM) and a Spatio-Temporal Convolution Module (STCM). Specifically, the TARM is deployed in a residual learning module that employs a novel attention learning network to recalibrate the temporal attention of frames in a skeleton sequence. The STCM treats the attention calibrated skeleton joint sequences as images and leverages the Convolution Neural Networks (CNNs) to further model the spatial and temporal information of skeleton data. These two modules (TARM and STCM) seamlessly form a single network architecture that can be trained in an end-to-end fashion. MANs significantly boost the performance of skeleton-based action recognition and achieve the best results on four challenging benchmark datasets: NTU RGB+D, HDM05, SYSU-3D and UT-Kinect.
1 Introduction
Skeleton action recognition remains difficult because joint sequences exhibit complex spatio-temporal variations, motivating MANs, an end-to-end temporal-then-spatial recalibration framework combining attention, RNNs, and CNNs.
- Research challenge: RNN-based methods struggle with joint variations caused by action speed, jitters, and surrounding distractions.
- Spatial-temporal modeling: STCM treats skeleton sequences as images and uses CNNs to model spatial layout and temporal information.The coordinate channels are represented as a three-channel image before CNN processing.
- Proposed framework: MANs combine memory attention and convolution in an end-to-end framework for modeling complex spatio-temporal joint variations.The framework is presented as a “RNNs + CNNs” architecture for skeleton-based action recognition.
- Temporal recalibration: TARM recalibrates temporal features so informative skeleton frames receive greater attention.
- Reported outcomes: MANs achieve state-of-the-art results on four challenging datasets, with ablation studies evaluating the effectiveness of each unit.
2 Memory Attention Networks
MANs use TARM to extract BiGRU memory information and recalibrate frame-wise temporal attention, then use STCM to model enhanced spatial and temporal skeleton features with CNNs.
- Input representation: The input skeleton data is represented as O = {X, Y, Z} ∈ R^T×N×3, containing x, y, and z coordinates for N joints across T frames.Each coordinate forms a T × N matrix, and X is used to explain TARM.
- Architecture: MANs combine a residual-based Temporal Attention Recalibration Module with a CNN-based Spatio-Temporal Convolution Module in one trainable architecture.TARM recalibrates temporal features, while STCM extracts higher-level spatio-temporal representations from the TARM outputs.
- Temporal Attention Recalibration Module: TARM uses a BiGRU memory cell to summarize sequence information and an attention branch to assign frame-wise weights to the memory features.The memory features FM and attention weights FA are combined through element-wise multiplication, with attention emphasizing representative frames.
- Temporal Attention Recalibration Module: The attention branch aggregates each row of X by average pooling, duplicates the resulting temporal vector K times, and processes it through bottleneck fully connected layers.The bottleneck uses dimensionality reduction with ReLU followed by dimensionality expansion with sigmoid; the resulting attention weights are multiplied element-wise with FM.
- Temporal Attention Recalibration Module: TARM resizes the recalibrated feature with a fully connected layer to obtain eX, and similarly produces eY and eZ for the three coordinate channels.The outputs form eO = {eX, eY, eZ} ∈ R^T×N×3 for subsequent processing.
- Spatio-Temporal Convolution Module: STCM treats eX, eY, and eZ as a three-channel image and applies CNNs to capture spatial joint layout and enhanced spatio-temporal features.The resulting feature representation is passed through a fully connected output and softmax classifier for action prediction.
3 Experiments
MANs is evaluated on four skeleton-action datasets through dataset-specific protocols, parameter studies, ablations, and comparisons with prior RNN- and CNN-based methods. Across these experiments, MANs achieves strong results, with temporal attention and the temporal-then-spatial architecture contributing to performance and flexible complexity–accuracy trade-offs.
- Datasets and implementation: MANs is evaluated on NTU RGB+D, HDM05, SYSU-3D, and UT-Kinect using their standard dataset protocols.NTU RGB+D uses cross-subject and cross-view protocols; HDM05 uses 10-fold cross-validation, SYSU-3D uses 30-fold cross-validation, and UT-Kinect uses LOOCV.
- Experiment analysis: The experiments vary the attention ratio, CNN depth, and attention configuration to examine MANs components and network complexity.The study evaluates α values, MANs variants with 9, 33, or 61 CNN layers, ResNet-18 and DenseNet-161, and ablations without or with alternative temporal attention.
- Experiment analysis: 99.23% accuracy is achieved by MANs (ResNet-18) on HDM05 with α = 16, which is selected for subsequent experiments.The parameter study reports high training efficiency for α = 4, 8, 16, and 32, with α = 16 producing the best reported accuracy.
- Experiment analysis: MANs-9 converges at epoch #26 versus epoch #40 for MANs-9 with other temporal attention on NTU RGB+D.The residual attention module combines recalibrated temporal information with original-input spatial structure through an identity shortcut.
- Results and comparisons: MANs variants outperform state-of-the-art RNN- and CNN-based approaches across the four datasets, supporting the proposed memory attention plus convolution architecture.The comparison includes different MANs parameter amounts and prior approaches such as VA-LSTM, GCA-LSTM, and Clips+CNN+MTLN.
- Results and comparisons: 83.01% cross-subject and 90.66% cross-view accuracy are obtained by MANs-9 on NTU RGB+D, while MANs (DenseNet-161) reaches 93.22% cross-view accuracy.The reported improvements over other methods are 3.44% for cross-subject evaluation and 5.62% for cross-view evaluation.
- Results and comparisons: MANs-61 exceeds the previous best SYSU-3D approach by 9.03%, while MANs-9 and MANs (ResNet-18) reach 100% on UT-Kinect.On HDM05, MANs (ResNet-18) reaches 99.04%; on UT-Kinect, the reported improvement over GCA-LSTM is 1.0%.
4 Conclusion
MANs enhance spatio-temporal features for skeleton-based action recognition through a unified end-to-end framework. Experiments report the best performance on four benchmark datasets.
- MANs use TARM to recalibrate temporal attention to skeleton frames and STCM to model spatial structure and temporal dependence with CNNs.
- The unified MANs framework significantly boosts performance for skeleton-based action recognition.
- MANs consistently perform best on four benchmark datasets and contribute new state-of-the-art results.