Source-linked AI summary
Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction
Xiangbo Shu, Liyan Zhang, Guo-Jun Qi, Wei Liu, Jinhui Tang
TL;DR
Human motion prediction must generate future motions from observed skeleton sequences while accounting for both spatial joint coherence and temporal skeleton evolution. The paper proposes SC-RNN, which uses skeleton-joint co-attention, recurrent modeling, and a weighted gram-matrix loss. Experiments demonstrate that SC-RNN outperforms related methods.
Problem
Human motion prediction seeks to generate future motions from observed motions, but existing RNN methods do not jointly model spatial joint coherence and temporal skeleton evolution.
Method
SC-RNN uses skeleton-joint co-attention and an SCA-embedded GRU to model human-skeleton and human-joint motions together in spatiotemporal space.
Results
Experiments on human motion prediction demonstrate that SC-RNN outperforms related methods.
Takeaways & Limitations
The method combines spatial joint coherence and temporal skeleton evolution through a co-attention motion context for human motion prediction.
Abstract
from arXiv · showhide
Human motion prediction aims to generate future motions based on the observed human motions. Witnessing the success of Recurrent Neural Networks (RNN) in modeling the sequential data, recent works utilize RNN to model human-skeleton motion on the observed motion sequence and predict future human motions. However, these methods did not consider the existence of the spatial coherence among joints and the temporal evolution among skeletons, which reflects the crucial characteristics of human motion in spatiotemporal space. To this end, we propose a novel Skeleton-joint Co-attention Recurrent Neural Networks (SC-RNN) to capture the spatial coherence among joints, and the temporal evolution among skeletons simultaneously on a skeleton-joint co-attention feature map in spatiotemporal space. First, a skeleton-joint feature map is constructed as the representation of the observed motion sequence. Second, we design a new Skeleton-joint Co-Attention (SCA) mechanism to dynamically learn a skeleton-joint co-attention feature map of this skeleton-joint feature map, which can refine the useful observed motion information to predict one future motion. Third, a variant of GRU embedded with SCA collaboratively models the human-skeleton motion and human-joint motion in spatiotemporal space by regarding the skeleton-joint co-attention feature map as the motion context. Experimental results on human motion prediction demonstrate the proposed method outperforms the related methods.
1 INTRODUCTION
Human motion prediction uses observed skeleton sequences to generate future motions, but prior RNN-based methods largely model temporal evolution without spatial joint coherence. SC-RNN addresses this gap with skeleton-joint co-attention, recurrent modeling, and a weighted gram-matrix loss.
- Motivation: Human motion prediction generates future skeleton motions from observed motion sequences captured by motion-capture devices.
- Motivation: RNN-based methods model sequential motion over time, but direct recurrent propagation is weak for long-term prediction.
- Motivation: Prior methods overlook spatial coherence among joints, such as coordinated swing directions between limbs during walking.
- Proposed approach: SCA constructs a skeleton-joint feature map and dynamically learns separate skeleton- and joint-attention factors across temporal and spatial spaces.The resulting co-attention map refines observed motion information for predicting a future motion.
- Proposed approach: SC-RNN models human-skeleton and human-joint motions simultaneously in spatiotemporal space using the co-attention map as motion context.Its framework also integrates predicted skeleton and joint motions through a spatial confidence gate.
- Proposed approach: The proposed weighted gram-matrix loss exploits correlations between skeletons to train SC-RNN with structurally consistent predictions.
2 RELATED WORK
Related work spans skeleton-based action recognition and human-skeleton motion prediction, progressing from probabilistic models to recurrent architectures. Existing prediction methods improve temporal modeling but remain limited in handling spatial joint dependencies and noisy or insufficiently informative historical motions.
- 2.1 Skeleton-based Action Recognition: Skeleton-based action recognition uses human-skeleton positions as input, alongside RGB-based approaches that use visual image features.
- 2.1 Skeleton-based Action Recognition: Deep networks, including GRUs and LSTMs, improved skeleton-action representations, while attention methods selected informative joints over irrelevant ones.
- 2.2 Human-Skeleton Motion Prediction: Human-skeleton motion prediction generates future motions from observed motions and supports applications including surveillance, virtual reality, and human-machine interfaces.
- 2.2 Human-Skeleton Motion Prediction: Early prediction methods used latent-variable, Markov, generative, and Gaussian-process dynamical models for sequential nonlinear motion.
- 2.2 Human-Skeleton Motion Prediction: RNN variants such as ERD, residual GRU, and structural RNN improved prediction, but prior approaches overlooked dependencies among spatially neighboring joints.
- 2.2 Human-Skeleton Motion Prediction: Temporal attention methods use all historical motions because relying on the latest motion can introduce noise when it is sudden or inconsistent.
3 METHODOLOGY
SC-RNN predicts future skeleton motion by combining temporal attention over observed skeletons with spatial attention over joints in a co-attention GRU framework. It uses the resulting motion context, spatial confidence gating, and weighted gram-matrix loss to model and regularize predictions.
- Background and Idea: The prediction model uses all historical motions because sudden or noisy last-frame motion can disturb recurrent prediction.
- Skeleton-joint Co-attention: SC-RNN embeds SCA in recurrent networks that simultaneously model skeleton and joint motion using the co-attention map as motion context.
- Background and Idea: Human motion x_t is represented by concatenated 3D angle-axis locations of all K joints, forming the observed sequence S.
- Skeleton-joint Co-attention: SCA computes temporal attention over observed skeletons and spatial attention over joints, producing a skeleton-joint co-attention feature map.
- Skeleton-joint Co-attention: The spatial joint-attention network follows a predefined joint sequence, with traveling-based and surrounding-based orders chosen to reflect body structure.
- Weighted gram-matrix loss: A spatial confidence gate suppresses updated joint states that differ from input joint states, while weighted gram-matrix loss compares predicted and ground-truth motions.
4 EXPERIMENTS
Experiments evaluate SC-RNN on H3.6M using short- and long-term prediction settings, ablations, sequence strategies, quantitative comparisons, and visualizations. SC-RNN generally outperforms related methods, with especially strong long-term and action-specific results.
- Experimental setting: Experiments use H3.6M’s 15 actions, training on two actors and testing on five, with two seconds observed to predict 400ms and 1000ms motions.Each joint is represented by a three-dimensional location vector, and evaluation uses Mean Angle Error.
- Sequence strategy: The traveling-based sequence gives SC-RNN its best performance among the ID-based, traveling-based, and surrounding-based sequence strategies.These strategies are compared in Table 2.
- Ablation studies: SC-RNN outperforms the ablation baselines, while combining skeleton and joint attention in spatiotemporal space performs better than single-attention variants.Both attention-only ablations improve over SC-RNN without SCA, but full SCA performs best.
- Competitive methods: SC-RNN performs better than ERD, LSTM-3LR, Res-GRU, and MHU, including about 0.8 lower MAE than MHU for 1000ms prediction.The comparison covers short- and long-term prediction across all 15 actions on H3.6M.
- Action-specific results: SC-RNN achieves the lowest MAE on most actions and makes significant improvement on greeting, where spatial coherence helps model large arm and leg variation.For posing, all methods exceed 2.5 MAE at 1000ms, while SC-RNN remains comparable to MHU.
- Qualitative results: Visualizations show SC-RNN predictions generally close to ground truth, whereas competing methods exhibit inaccurate details such as leg joints in eating.The figures arrange observed motion, ground truth, and predictions from ERD, LSTM-3LR, Res-GRU, MHU, and SC-RNN.
5 CONCLUSIONS
The paper concludes that SC-RNN captures joint-level spatial coherence and skeleton-level temporal evolution using a co-attention feature map. Its SC-GRU and weighted Gram-matrix loss support spatiotemporal modeling, and experiments show improved performance over related methods.
- Conclusions: SC-RNN uses all useful historical motion information as context while jointly capturing spatial coherence among joints and temporal evolution among skeletons.The model learns a co-attention feature map in spatiotemporal space.
- Conclusions: SC-GRU embeds SCA to model human-joint and human-skeleton motions simultaneously in spatiotemporal space.The architecture integrates these motion representations for prediction.
- Conclusions: A weighted Gram-matrix loss trains the SC-GRU model in spatiotemporal space, and experiments demonstrate that SC-RNN outperforms related methods.