Source-linked AI summary
Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamic
Yongyi Tang, Lin Ma, Wei Liu, Weishi Zheng
TL;DR
Long-term human motion prediction is limited when historical skeletons are represented mainly by the latest recurrent hidden state. The paper summarizes motion context relative to the current pose, uses a modified highway unit and gram matrix loss, and reports stronger long-term prediction while supporting activity-conditioned motion transfer.
Problem
Latest-hidden-state recurrent encoders can lose historical motion information, limiting long-term human motion prediction and weakening motion dynamics.
Method
The model summarizes historical motion context, uses a modified highway unit to filter motionless joints, and minimizes gram matrix loss to enhance dynamics.
Results
The proposed model outperforms existing methods especially for long-term motion prediction and also performs human motion transfer using activity labels.
Takeaways & Limitations
Motion context and dynamic modeling enable reliable long-term human motion prediction and label-controlled motion sequences within the reported experiments.
Takeaways & Limitations
High-uncertainty activities involving different subjects can exceed the evidence available from the observed motion, producing failure cases.
Abstract
from arXiv · showhide
Human motion prediction aims at generating future frames of human motion based on an observed sequence of skeletons. Recent methods employ the latest hidden states of a recurrent neural network (RNN) to encode the historical skeletons, which can only address short-term prediction. In this work, we propose a motion context modeling by summarizing the historical human motion with respect to the current prediction. A modified highway unit (MHU) is proposed for efficiently eliminating motionless joints and estimating next pose given the motion context. Furthermore, we enhance the motion dynamic by minimizing the gram matrix loss for long-term motion prediction. Experimental results show that the proposed model can promisingly forecast the human future movements, which yields superior performances over related state-of-the-art approaches. Moreover, specifying the motion context with the activity labels enables our model to perform human motion transfer.
1 Introduction
The paper targets long-term human motion prediction by modeling historical motion context rather than relying only on the latest recurrent hidden state. It combines motion-aware generation, dynamic enhancement, and activity-conditioned motion transfer.
- Motivation: Existing RNN encoders use the latest hidden state to represent historical skeletons, but information loss can cause mean-pose convergence and weak long-term dynamics.Prior methods also treat all joints equally, although some joints may remain motionless during activities such as walking and eating.
- Motion context modeling: The model summarizes historical skeletons with respect to the current skeleton to capture repeated activity patterns and reduce uncertainty in long-term prediction.This motion context is designed to preserve historical motion evidence beyond the latest recurrent state.
- Motion-aware generation: A modified highway unit uses the last-frame pose and summarized motion context to filter motionless joints and emphasize joints with motion during generation.The proposed unit estimates the next human skeleton while propagating motion information through the prediction process.
- Motion dynamics: A gram matrix loss penalizes mean-pose convergence and error accumulation to produce more consistent temporal motion dynamics.The loss is introduced specifically to enhance dynamic behavior in long-term predictions.
- Motion transfer: Specifying activity labels lets the model transfer motion by generating smooth sequences with multiple activities from an observed motion sequence.This extends prior single-activity prediction toward label-controlled motion manipulation.
2 Related Works
Related work places skeleton-based human motion analysis within computer vision and robotics, using motion captured from devices, videos, and images. Prior approaches include structural models for skeleton-based action recognition.
- Human Motion Analysis: Human motion analysis is established as a key problem in computer vision and robotics.
- Human Motion Analysis: Human motion can be obtained from motion-capture systems, Kinect, videos, and static images for subsequent analysis.
- Human Motion Analysis: Skeleton-based action recognition has used structural models including hierarchical recurrent neural networks and trust gates.
3 Proposed Model
The proposed model predicts long-term human motion by combining motion-context modeling with a modified highway unit and a gram-matrix objective. It directly summarizes historical skeletons relative to the predicted pose, selectively updates informative joints, and models spatial-temporal motion correlations.
- 3.1 Problem Formulation: Human motion prediction generates consecutive future skeletons from an observed sequence of body poses represented in axis-angle form.The prediction target is a sequence of future poses, with each joint rotation represented by a three-dimensional rotation vector.
- 3.1 Problem Formulation: Conventional RNN predictors rely on the last hidden state, but long-term failures indicate that it does not adequately capture historical motion information.The final hidden state can be dominated by the latest input, while historical skeletons remain useful for future prediction.
- Motion Context Modeling: Motion context modeling uses temporal attention to summarize all historical skeleton embeddings with respect to the last predicted skeleton at each prediction step.The model directly accesses historical skeletons, allowing it to repeat observed periodic patterns and retain directional or movement-habit information for aperiodic activities.
- Modified Highway Unit: The recurrent prediction layer combines motion context with the current skeleton through a modified highway unit that focuses on joints containing meaningful motion information.Its gate ranges from 0 to 1 and partially updates joints through element-wise interpolation between the current skeleton and estimated next joints.
- Enhancing Motion Dynamics with Gram Matrix Objective: The gram matrix objective models spatial relations among joints and temporal correlations between consecutive skeletons to enhance dynamic human motion.The objective is introduced because mean square error can encourage predictions near mean poses and does not directly model motion transitions.
4 Experiments
Experiments evaluate the model on H3.6m using mean angle error for short- and long-term prediction, compare variants and prior methods, and test motion transfer. The results show stronger long-term dynamics, best combined prediction performance, and smooth label-conditioned activity transitions, with failures in highly uncertain activities.
- Experimental settings: The experiments use H3.6m motion-capture data and evaluate short- and long-term prediction with mean angle error.The dataset includes 15 activities, while the reported comparisons cover all activities and selected individual activities.
- Experimental settings: The evaluation compares ERD, LSTM-3LR, Res-GRU, MHU-MSE, and MHU-Gram against the proposed framework.MHU-MSE uses conventional mean square loss, whereas MHU-Gram replaces it with gram matrix loss.
- Human motion prediction: The complete model achieves the best performance on both short-term and long-term prediction across the evaluated activities.MHU-MSE performs particularly well from 160ms to 720ms, while MSE-Gram reaches 1.82 mean angle error at 1000ms.
- Human motion prediction: For walking, competing predictions converge toward the mean body pose, whereas the proposed method preserves dynamic walking behavior despite similar objective evaluation to Res-GRU.Figure 3 compares mean pose convergence and highlights the converging motion sequence with dashed boxes.
- Human motion transfer: Motion transfer modifies decoder hidden states with embedded activity labels to generate specified activities from observed motion.The method constructs a context representation by concatenating the hidden state with an embedded action label before decoding.
- Human motion transfer: The model transfers motion between activities with smooth transitions, including walking-to-sitting and the inverse, but fails more often for highly uncertain activities across subjects.The failure cases are attributed to insufficient evidence in the observed information.
5 Conclusion
The model predicts long-term human motion by summarizing historical skeletons and enhancing temporal dynamics, while also supporting action-conditioned motion transfer.
- The model explores motion context and enhanced motion dynamics for long-term human-motion prediction.Motion context summarizes historical skeletons, while gram matrix training loss captures temporal transitions.
- The model performs human motion transfer by conditioning generated motion on action commands.The generated motion types are altered according to the specified action command.