PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks

Jun Liu, Gang Wang, Ling-Yu Duan, Kamila Abdiyeva, Alex C. Kot

arXiv:1707.05740v5cs.CV

TL;DR

Skeleton action recognition requires identifying joints that matter for each action and frame, but original LSTMs lack strong global-context attention. The paper proposes GCA-LSTM with global-context-guided and recurrent attention, plus stepwise training and a two-stream extension, achieving state-of-the-art performance on five challenging datasets.

  • Problem

    Irrelevant joints can introduce noise, while original LSTMs lack strong attention because their context is relatively local rather than global.

  • Method

    GCA-LSTM uses a global context memory cell to guide joint attention, with recurrent refinement, stepwise training, and a two-stream joint-level/body-part-level extension.

  • Results

    The proposed GCA-LSTM outperforms original ST-LSTM by 6.9% under cross-subject and 6.3% under cross-view evaluation on NTU RGB+D.

  • Takeaways & Limitations

    Global context and iterative attention enable the network to selectively emphasize informative joints for skeleton-based action recognition.

Abstract

from arXiv · show

Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, Long Short-Term Memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, Global Context-Aware Attention LSTM (GCA-LSTM), for skeleton based action recognition. This network is capable of selectively focusing on the informative joints in each frame of each skeleton sequence by using a global context memory cell. To further improve the attention capability of our network, we also introduce a recurrent attention mechanism, with which the attention performance of the network can be enhanced progressively. Moreover, we propose a stepwise training scheme in order to train our network effectively. Our approach achieves state-of-the-art performance on five challenging benchmark datasets for skeleton based action recognition.

I. INTRODUCTION

The paper addresses the difficulty of selecting informative skeletal joints when their relevance varies across actions and frames, while original LSTMs lack strong global-context attention. It proposes GCA-LSTM, recurrent attention, stepwise training, and a two-stream extension, reporting state-of-the-art performance on evaluated benchmarks.

  • Motivation: Informative skeletal joints vary across actions and frames, while irrelevant joints can introduce noise that degrades recognition.For example, hand motion is informative for clapping, whereas foot motion may not be.
  • Motivation: Original LSTMs model sequential data but have limited attention capability because their available context is relatively local rather than global.The paper links this limitation to difficulty retaining information from far in the past.
  • Proposed approach: GCA-LSTM introduces a global context memory cell that supports selective attention to informative joints throughout the sequence.Its first LSTM layer encodes the sequence and initializes global context, while the second layer uses that context to generate an attention representation.
  • Proposed approach: The paper further proposes a two-stream GCA-LSTM combining fine-grained joint-level and coarse-grained body-part-level attention.The stated goal is more accurate action recognition.
  • Proposed approach: A recurrent attention mechanism progressively improves attention, and a stepwise training scheme is proposed to train the network more effectively.These mechanisms are presented as extensions of the core GCA-LSTM design.
  • Results: The proposed end-to-end network achieves state-of-the-art performance on the evaluated benchmark datasets.The paper describes extensive evaluation and reports results on five challenging datasets.

II. RELATED WORK

Skeleton-based action recognition research has progressed from hand-crafted feature and classifier methods to RNN/LSTM models. The paper positions GCA-LSTM as an approach that explicitly uses global action context to attend to informative joints.

  • Hand-crafted features: Earlier methods used diverse feature extractors and classifiers for skeleton-based action recognition, including LDSs, Lie-group curves with SVMs, HMMs, actionlet ensembles, part-based features, and tensor representations.These approaches targeted spatial structure, temporal dynamics, intra-class variation, relevant body parts, or high-order joint relationships.
  • RNN and LSTM methods: RNN and LSTM methods model skeleton structure, temporal dynamics, information changes, joint co-occurrences, and body-part awareness.Examples include hierarchical RNNs, differential gating, mixed-norm regularization with in-depth dropout, and part-aware LSTM models.
  • RNN and LSTM methods: RNN and LSTM models have also been applied to skeleton-based action forecasting and detection beyond action recognition.
  • Attention-based methods: Unlike prior RNN/LSTM approaches, GCA-LSTM uses global context to emphasize informative joints across all spatio-temporal steps and generate an attention representation for classification.The paper presents this global-context attention as the distinguishing feature of its method.

C. Attention Mechanism

ST-LSTM models spatial dependencies among joints and temporal dependencies across frames. The paper contrasts prior local-context attention with GCA-LSTM, which uses global context to assess joint informativeness.

  • Attention Mechanism: Prior attention methods measure the next input using the previous LSTM hidden state, whose context is relatively local.The paper argues that global classification requires broader action-sequence information for reliable joint-importance estimation.
  • Attention Mechanism: GCA-LSTM introduces a global context memory cell that measures informativeness at each step and uses it as an informativeness gate inside the LSTM unit.The gate adjusts each input’s contribution when updating the memory cell.
  • Spatio-Temporal LSTM: ST-LSTM arranges joints within each frame as a spatial chain while feeding corresponding joints across frames in the temporal direction.This design models spatial and temporal dependencies simultaneously.
  • Spatio-Temporal LSTM: Each ST-LSTM unit receives the current joint location, the same joint’s previous-frame representation, and the previous joint’s same-frame representation.Its gates separately capture temporal and spatial context, alongside input and output gating.
  • Spatio-Temporal LSTM: The ST-LSTM cell state combines the modulated input with spatial and temporal forget-gated context, while the hidden representation is produced through the output gate.The transition equations define this recurrent update using element-wise products and an affine transformation.

B. Global Context-Aware Attention LSTM

GCA-LSTM addresses the mismatch between local LSTM context and the global information needed to identify informative joints. Its architecture initializes, iteratively refines, and reuses a global context memory for attention and classification.

  • Motivation: Skeleton actions contain informative joints that vary across action sequences and frames, while irrelevant joints may contribute little or introduce noise.The method therefore emphasizes informative joint features and suppresses irrelevant ones.
  • Motivation: LSTM’s stepwise hidden representation provides relatively local context compared with the global action sequence, limiting reliable joint-informativeness assessment.
  • Network overview: GCA-LSTM introduces a global context memory cell that retains action-sequence context and feeds it to each LSTM step to assist attention.The architecture is named Global Context-Aware Attention LSTM.
  • Network overview: The first ST-LSTM layer encodes the skeleton and initializes global context, while the second performs attention over spatio-temporal inputs and refines that context.The first layer receives 3D joint coordinates; the second layer receives hidden representations from the first.
  • Recurrent attention: Multiple recurrent attention iterations refine the global context memory iteratively, after which the refined memory is used for classification.
  • Memory initialization: The initial global context memory can be computed by averaging first-layer hidden representations or by feeding their concatenation to a feed-forward network.The paper reports that these two initialization schemes perform similarly.

3) Performing Attention in the Second ST-LSTM Layer:

The second ST-LSTM layer uses global context to score input informativeness and selectively update its cell state. Recurrent attention refines the global context across iterations, progressively improving action representations.

  • The network computes a normalized informativeness score for each input using the input and the previous iteration’s global context memory.The score is learned at each spatio-temporal step during the n-th attention iteration.
  • The informativeness score acts as a gate in the second ST-LSTM layer.It controls how strongly the unit incorporates the current input during cell-state updates.
  • Informative inputs contribute more new information, whereas irrelevant inputs are suppressed and the unit relies more on historical cell-state information.This update rule is conditioned on informativeness relative to the global context representation.
  • The final spatio-temporal output of the second layer forms the attention representation, which is fed back to refine the global context memory cell.The refined memory cell is used in subsequent attention iterations.
  • Multiple recurrent attention iterations repeatedly identify informative joints and refine the global context, making it more discriminative for action classification.The refinement transformation uses iteration-specific parameters.

5) Classifier:

The classifier predicts the action class from the final refined global context memory cell. Training uses negative log-likelihood and back-propagation, with either direct or stepwise network optimization.

  • The last refined global context memory cell is fed to a softmax classifier to predict the action class label.
  • Negative log-likelihood measures the difference between the true label and prediction, and back-propagation minimizes this loss.
  • The paper describes direct optimization of the whole network and a more advanced stepwise training scheme for GCA-LSTM.

2) Stepwise Training:

Because recurrent attention creates interactions among modules and introduces parameters across iterations, the paper trains GCA-LSTM incrementally. The two-stream extension combines joint-level and body-part-level attention within separate attention streams.

  • Stepwise Training: Frequent module interactions and newly introduced iteration parameters make direct optimization of all network parameters difficult.The interacting modules include the two ST-LSTM layers and the global context memory cell.
  • Stepwise Training: The proposed stepwise scheme incrementally optimizes model parameters across attention iterations.The training procedure is illustrated as direct whole-network training versus stepwise optimization.
  • Stepwise Training: At training step n, only parameters and modules used by iterations 0 through n are optimized, while later steps expand the trained network.Each step trains the corresponding shrunken network until validation error stops decreasing.
  • Two-Stream GCA-LSTM: The two-stream GCA-LSTM jointly uses fine-grained joint-level attention and coarse-grained body-part-level attention.The coarse-grained stream assigns one shared informativeness score to all joints within each of five body parts.
  • Two-Stream GCA-LSTM: The two attention streams maintain separate global context memory cells and second ST-LSTM layers, while sharing the first ST-LSTM encoder.The coarse-grained stream focuses selectively on informative body parts in each frame.

A. Experiments on the NTU RGB+D Dataset

On NTU RGB+D, GCA-LSTM outperforms the original ST-LSTM and other reported skeleton-based methods, while stepwise training improves performance. The two-stream variant further improves over fine-grained attention alone.

  • 6.9% with cross subject and 6.3% with cross view are the reported improvements of GCA-LSTM over original ST-LSTM on NTU RGB+D.
  • GCA-LSTM outperforms other compared skeleton-based methods on the NTU RGB+D experiments.The dataset contains more than 56 thousand samples, 60 action classes, and 40 subjects, with cross-subject and cross-view protocols.
  • Stepwise training improves GCA-LSTM performance compared with direct training on NTU RGB+D.
  • The two-stream GCA-LSTM performs better than GCA-LSTM with fine-grained attention only, and also benefits from stepwise training.

B. Experiments on the SYSU-3D Dataset

On SYSU-3D, GCA-LSTM surpasses prior skeleton-based methods, and its stepwise training scheme is useful. The two-stream extension is also effective for action recognition.

  • GCA-LSTM surpasses state-of-the-art skeleton-based methods on SYSU-3D.
  • The results show that stepwise training is useful for the GCA-LSTM network.
  • The two-stream GCA-LSTM network is effective for action recognition on SYSU-3D.

D. Experiments on the SBU-Kinect Interaction Dataset

On the challenging SBU-Kinect Interaction dataset, GCA-LSTM achieves the best performance among several relevant LSTM-based and other skeleton-action-recognition methods.

  • SBU-Kinect Interaction contains 282 two-person interaction sequences spanning 8 action classes and 6822 frames.
  • The dataset is challenging because Kinect joint coordinates are relatively inaccurate and many sequences involve complicated two-person interactions.
  • GCA-LSTM achieves the best performance among the compared methods on SBU-Kinect Interaction.
  • Evaluation follows the standard 5-fold cross-validation protocol.

G. Evaluation of Parameter Sharing Schemes

The experiments examine parameter sharing, training strategy, context initialization, and attention design. Sharing parameters improves generalization and helps address over-fitting, while stepwise training accelerates convergence and reduces over-fitting relative to direct training.

  • Parameter sharing: Sharing parameters within an attention iteration improves generalization performance.
  • Parameter sharing: Sharing parameters across attention iterations helps handle over-fitting but may limit representation capacity.
  • Training strategy: Stepwise training makes the network converge faster and direct training is more prone to over-fitting.
  • Context initialization: A feed-forward network and averaging hidden representations perform similarly for global-context initialization, but the feed-forward scheme converges faster.
  • Attention design: Using soft attention lowers NTU RGB+D accuracy by about one percentage point compared with using the informativeness score as an LSTM gate.

J. Visualizations

Visualizations on NTU RGB+D show that recurrent attention improves attention performance across iterations, with corresponding accuracy gains for three illustrated actions.

  • Attention performance increases between the two recurrent attention iterations for the visualized actions.
  • The visual analysis evaluates taking a selfie, pointing to something, and kicking another person on NTU RGB+D.
  • Without attention, the three illustrated action classes achieve accuracies of 67.7%, 71.7%, and 81.5%, respectively.
  • One attention iteration raises the three action accuracies to 67.8%, 72.4%, and 83.4%, respectively.
Loading 1707.05740v5…