Source-linked AI summary
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jiaying Liu
TL;DR
The paper addresses how to extract discriminative spatial-temporal features for skeleton-based action recognition. It proposes an end-to-end LSTM-based model with joint and frame attention, regularized training, and joint optimization. The model demonstrates effectiveness on SBU and NTU, with about 10% accuracy gains on NTU’s Cross-Subject and Cross-View settings.
Problem
Human action recognition requires effective modeling of discriminative spatial and temporal evolutions, while skeleton-based recognition must select informative joints and frames.
Method
An end-to-end LSTM framework uses spatial joint-selection gates, temporal frame attention, regularized cross-entropy, and joint training for skeleton sequences.
Results
The model demonstrates effectiveness against state-of-the-art methods on SBU and NTU, achieving about 10% accuracy gains on NTU Cross-Subject and Cross-View settings.
Takeaways & Limitations
The proposed attention framework automatically emphasizes discriminative joints within frames and important outputs across frames for skeleton-based action recognition.
Abstract
from arXiv · showhide
Human action recognition is an important task in computer vision. Extracting discriminative spatial and temporal features to model the spatial and temporal evolutions of different actions plays a key role in accomplishing this task. In this work, we propose an end-to-end spatial and temporal attention model for human action recognition from skeleton data. We build our model on top of the Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM), which learns to selectively focus on discriminative joints of skeleton within each frame of the inputs and pays different levels of attention to the outputs of different frames. Furthermore, to ensure effective training of the network, we propose a regularized cross-entropy loss to drive the model learning process and develop a joint training strategy accordingly. Experimental results demonstrate the effectiveness of the proposed model,both on the small human action recognition data set of SBU and the currently largest NTU dataset.
1 Introduction
Skeleton-based action recognition models spatial and temporal evolution from 3D joint positions, while this work introduces end-to-end spatial and temporal attention to focus on discriminative joints and frames.
- Human action recognition requires discriminative spatial-temporal features to model how actions evolve over space and time.
- Skeleton data represents people through 3D key-joint coordinates and is robust to variations in location and appearance.
- Skeleton-action methods combine discriminative feature mining from joints with temporal-dynamics modeling.
- The spatial attention module adaptively emphasizes discriminative joints whose importance can change across action stages.
- The temporal attention module assigns content-dependent importance to frames rather than simply skipping non-key frames.
- The proposed LSTM-based framework combines spatial and temporal attention, spatio-temporal regularization, and joint end-to-end training.
2 Related Work
Prior work mines spatial joint relationships and temporal key frames, whereas this paper adapts attention to changing joint and frame importance in skeleton sequences.
- 2.1 Spatial Co-Occurrence Exploration: Actionlet and orderlet methods represent actions through discriminative subsets or combinations of skeleton-joint features.
- 2.1 Spatial Co-Occurrence Exploration: Earlier joint-mining methods fix joint or feature importance after mining, rather than adapting it across frames and sequences.
- 2.2 Temporal Key Frame Exploration: Key-frame approaches seek frames with more discriminative information and can exclude frames that are less relevant to an action.
- 2.2 Temporal Key Frame Exploration: The temporal attention module weights all frames differently instead of skipping frames, preserving their respective discriminative information.
- 2.3 Attention-Based Models: Existing action-recognition attention models primarily use RGB videos, leaving skeleton sequences comparatively underexplored.
3 Overview of RNN and LSTM
RNNs model sequential data using recurrent hidden states, while LSTMs add a memory cell and gates to support long-term temporal modeling and mitigate vanishing gradients.
- At time t, an RNN output depends on the current input x_t and the previous hidden output h_t−1.
- The recurrent structure and internal memory of RNNs facilitate modeling long-term temporal dynamics.
- LSTMs extend RNNs with a memory cell and input, forget, and output gates that control writing, resetting, and reading.
4 Deep LSTM with Spatio-Temporal Attention Model
The proposed end-to-end multi-layered LSTM network integrates spatial and temporal attention to weight discriminative joints and frames in skeleton-based action recognition. Regularized learning and joint training address optimization difficulties in the coupled architecture.
- Architecture: The network combines a main LSTM with spatial and temporal attention subnetworks in an end-to-end architecture.The attention modules assign different importance to joints within frames and to frame outputs across sequences.
- Spatial Attention: Spatial attention assigns normalized weights to joints using current-frame joints and preceding LSTM hidden variables, then modulates each joint input before the main LSTM.Larger gate activations indicate greater importance, and x′t,k = αt,k · xt,k.
- Temporal Attention: Temporal attention assigns content-dependent weights to frame outputs, so sequence-level class scores are computed from weighted contributions across all time steps.The frame-selection gate depends on the current input and preceding hidden variables, while ReLU is used for nonlinear activation.
- Learning Objective: A regularized cross-entropy objective constrains spatial attention, temporal attention, and network weights during learning.The regularizers encourage attention across joints, control temporal attention with an l2 norm, and reduce overfitting with an l1 norm.
- Training: A joint training strategy uses separate pre-training of the attention modules to improve convergence despite mutual influence among the three networks.The procedure trains the spatial and temporal attention modules together with the main LSTM network.
5. Experimental Results
Experiments on SBU and NTU evaluate learned attention visualizations, attention configurations, regularization, and comparisons with state-of-the-art methods. The combined spatio-temporal attention model achieves the strongest configuration performance and about 10% accuracy gains on NTU Cross-Subject and Cross-View settings.
- Datasets and Settings: Experiments use the SBU and NTU datasets, with NTU evaluated under Cross-Subject and Cross-View settings.SBU contains 230 sequences across 8 classes, while NTU contains 56880 sequences across 60 classes and 4 million frames.
- Visualization of the Learned Attentions: The learned spatial attention varies across frames, emphasizing different joints on the two actors during the kicking action.For example, the right person’s left foot, right elbow, and left hand receive large weights, while the left person’s torso and right foot receive large weights.
- Visualization of the Learned Attentions: The differentiated temporal attention peaks when the person on the right lifts his foot to its highest point, identifying that frame as more discriminative.Differentiated temporal attention is defined as △βt = βt − βt−1.
- Effectiveness of the Proposed Attention Models: 5.1% and 6.4% accuracy improvements over LSTM come from the spatial and temporal attention modules, respectively, while STA-LSTM achieves the best performance.The comparison covers SBU, NTU Cross-Subject, and NTU Cross-View datasets.
- Effectiveness of the Proposed Attention Models: The spatial and temporal regularization items improve the performance of their respective attention models.The regularization terms are included in the objective function for the spatial and temporal attention modules.
- Comparisons to Other State-of-the-Art: About 10% accuracy gains are achieved on the NTU dataset for both Cross-Subject and Cross-View settings compared with other state-of-the-art methods.The comparisons are reported in Table 1 for SBU and Table 2 for NTU.
6. Conclusion
The paper presents an end-to-end spatio-temporal attention model for skeleton-based human action recognition. It adaptively weights joints and frames and uses regularized cross-entropy with joint training, achieving performance improvements over other state-of-the-art methods.
- Conclusion: The model combines joint-selection and frame-selection attention modules in an end-to-end LSTM-based framework for skeleton action recognition.Spatial attention assigns different importance to joints, while temporal attention assigns different weights to frames.
- Conclusion: Regularized cross-entropy and a joint training procedure combine spatial and temporal attention during learning.The training design is intended to efficiently integrate both attention types.
- Conclusion: Experimental results demonstrate performance improvement compared with other state-of-the-art methods.