HP-GAN: Probabilistic 3D human motion prediction via GAN

Emad Barsoum; John Kender; Zicheng Liu

Source-linked AI summary

HP-GAN: Probabilistic 3D human motion prediction via GAN

Emad Barsoum, John Kender, Zicheng Liu

arXiv:1711.09561v1Nov 27, 2017cs.CVcs.AIcs.HCcs.NE

TL;DR

Human motion prediction requires modeling multiple plausible futures from past skeletal poses. HP-GAN uses a modified WGAN-GP sequence-to-sequence model with random sampling and a motion-quality assessor, and performs well on Kinect- and MoCap-based datasets.

Problem
The paper addresses the need to predict multiple plausible future human poses from a given past.
Method
HP-GAN combines a sequence-to-sequence pose generator, modified WGAN-GP training with a human-motion and anatomy-aware loss, random vector z sampling, and a motion-quality-assessment model.
Results
Experiments show that the architecture performs well on both NTURGB-D Kinect data and Human3.6M motion-capture data.
Takeaways & Limitations
The model provides multiple possible futures from the same input and quantitatively assesses whether generated skeleton sequences resemble real human motion.
Takeaways & Limitations
Training convergence and stability remain unresolved because the model can diverge after appearing to converge.

Abstract

from arXiv · show

Predicting and understanding human motion dynamics has many applications, such as motion synthesis, augmented reality, security, and autonomous vehicles. Due to the recent success of generative adversarial networks (GAN), there has been much interest in probabilistic estimation and synthetic data generation using deep neural network architectures and learning algorithms. We propose a novel sequence-to-sequence model for probabilistic human motion prediction, trained with a modified version of improved Wasserstein generative adversarial networks (WGAN-GP), in which we use a custom loss function designed for human motion prediction. Our model, which we call HP-GAN, learns a probability density function of future human poses conditioned on previous poses. It predicts multiple sequences of possible future human poses, each from the same input sequence but a different vector z drawn from a random distribution. Furthermore, to quantify the quality of the non-deterministic predictions, we simultaneously train a motion-quality-assessment model that learns the probability that a given skeleton sequence is a real human motion. We test our algorithm on two of the largest skeleton datasets: NTURGB-D and Human3.6M. We train our model on both single and multiple action types. Its predictive power for long-term motion estimation is demonstrated by generating multiple plausible futures of more than 30 frames from just 10 frames of input. We show that most sequences generated from the same input have more than 50\% probabilities of being judged as a real human sequence. We will release all the code used in this paper to Github.

1. Introduction

Human motion prediction must represent uncertain futures rather than a single deterministic outcome. HP-GAN addresses this with a probabilistic sequence-to-sequence model and a separate motion-quality assessment model.

Motivation: Future human motion becomes more uncertain over longer horizons, with some possible outcomes more plausible than others.The same past can lead to alternatives such as turning versus falling or dropping versus successfully throwing a ball.
Approach: HP-GAN predicts multiple plausible future skeleton poses from one observed past using modified WGAN-GP and a human-motion-aware custom loss.The model allows the number of input and future poses to be parameters, with the predicted future length unrestricted.
Approach: Different random vectors z produce different future pose sequences from the same input sequence.The generator is an RNN-derived sequence-to-sequence pose model, while the critic and discriminator use multilayer networks.
Evaluation: A simultaneously trained motion-quality-assessment model estimates the probability that a skeleton sequence is real human motion.This provides a quantitative assessment of non-deterministic predictions.
Evaluation: The model is evaluated on NTURGB-D Kinect skeleton data and Human3.6M motion-capture data.These are described as two large datasets captured with different modalities.

2. Related work on prediction

Prior probabilistic motion-prediction methods commonly rely on non-deep statistical models or action-specific constraints, while recent deep methods are primarily deterministic. The paper positions its approach as deep, probabilistic prediction without statistical motion constraints and adds simultaneous motion-quality assessment.

Probabilistic prediction: Earlier probabilistic approaches commonly use non-deep learning methods such as Markov models, Gaussian process dynamical models, and low-dimensional search structures.These methods include assumptions or constraints such as the Markov property, Gaussian processes, action-specific training, or indexed pose spaces.
Probabilistic prediction: Some prior probabilistic methods predict a single action only or train separately for each action.The cited limitations include a single-action restriction and action-specific Gaussian-process training.
HP-GAN positioning: HP-GAN differs from the reviewed approaches by using deep neural networks for probabilistic motion prediction without statistical models constraining the motion.The paper presents this as a first use of deep neural networks for probabilistic motion prediction, to its knowledge.
Deterministic prediction: Recent deep human-motion prediction methods are primarily deterministic, and L2 objectives can average distinct possible futures into blurred motions.The paper identifies this as a problem because identical previous poses may lead to multiple possible futures.
Comparison with prior methods: Prior RNN and GAN approaches had limitations including sequential sampling, fixed-length outputs, or insufficient temporal dynamics from a single image.The paper contrasts HP-GAN with these constraints in its support for probabilistic future prediction.
Motion quality assessment: Automatic motion-quality evaluation has received relatively little attention and existing techniques were often specialized to particular motion types or editing software.HP-GAN adds a quality-assessment model trained simultaneously with motion prediction.

3. Our model for human motion prediction

HP-GAN models human motion probabilistically by conditioning future pose sequences on past poses and sampling different futures with random vectors. It combines sequence-to-sequence generation, adversarial training, anatomy-aware losses, and a separate discriminator for motion-quality assessment.

Probabilistic prediction: The model learns the conditional distribution P(z|x), where x is an input pose sequence and z is a predicted future sequence.Each pose corresponds to joint locations or joint angles.
Probabilistic prediction: Different random vectors z produce different valid future pose sequences from the same input sequence.The vector z is drawn from a uniform or Gaussian distribution and is incorporated into the sequence-to-sequence network’s encoder states.
HP-GAN architecture: HP-GAN uses a modified WGAN-GP with a sequence-to-sequence generator, a critic for adversarial training, and a discriminator that distinguishes real from fake motion sequences.The critic is trained alternately with the generator, while the discriminator outputs a probability that a sequence is real.
Generator training: The generator loss combines adversarial, consistency, and bone terms to encourage realistic temporal changes and stable bone lengths.Consistency loss smooths predicted poses, while bone loss reduces bone-length differences between predicted and ground-truth skeletons.
Motion-quality assessment: The discriminator quantitatively assesses generated motion by estimating the probability that a skeleton sequence is real, without using that probability to train the generator.Keeping discriminator probabilities out of generator optimization preserves WGAN-GP training for stability and avoidance of model collapse.
Generator training: The pose-gradient loss is especially important on small action subsets because it helps prevent discontinuities at the first predicted pose.For larger datasets such as all 60 NTURGB-D classes, the authors found this loss was not critical.

4. Experiments

Experiments evaluate HP-GAN on NTURGB-D and Human3.6M using absolute 3D joint positions, including noisy Kinect skeletons. The study also adds a discriminator-based procedure to assess the validity of multiple generated futures during training.

Datasets: HP-GAN is evaluated on NTURGB-D and Human3.6M, two large human-motion datasets with Kinect and motion-capture skeleton data.NTURGB-D contains noisy skeletons affected by occlusions, carried objects, and interactions, while Human3.6M provides 3D poses captured by a Vicon MoCap system.
Datasets: NTURGB-D provides 25 joints across 60 action classes and 40 subjects, whereas Human3.6M provides 32-joint skeletons from 11 actors performing 17 action classes.NTURGB-D actions were recorded from three Kinect viewpoints, and Human3.6M angle representations were converted to absolute 3D joint positions.
Data representation: The experiments use 3D joint positions directly, requiring the model to learn relationships among joints to predict valid human poses.This representation provides a shared pipeline across both datasets but has more degrees of freedom than angle-based training.
Preprocessing: Human3.6M clips are split into shorter segments and subsampled by using every other frame so both datasets can follow the same training pipeline.This compensates for Human3.6M having fewer but substantially longer clips.
Motion-quality assessment: Because GAN loss is not strictly monotonic with generated-motion quality, the authors add a discriminator that estimates whether a sequence is valid human motion.During training, they generate N predictions, count k predictions scored above 50%, and track the model with the maximum k after an initial training period.

5. Results

HP-GAN generates multiple plausible futures from the same input, with predictions diverging over longer horizons while remaining continuous at the input boundary. Results also expose dataset- and training-dependent limitations, including discontinuities on action subsets and differing Human3.6M adjustments.

NTURGB-D: Each random z value generates a separate possible future, with early predictions close to ground truth and later poses diverging more.On NTURGB-D, the model predicts 30 poses from 10 input poses when trained across all 60 classes.
NTURGB-D: The generated sequence has no discontinuity between the last input pose and the first predicted pose in the all-class NTURGB-D experiment.
Training diagnostics: The discriminator loss provides a more stable training signal than the generator or critic losses, whose best parameters are difficult to identify without visual inspection.
Limitations: Not all generated poses are smooth when training on a subset of actions, and discontinuities can appear between the input and predicted sequences.The paper states that consistency loss, reduced network capacity, or directly feeding the last input to the first decoder node can mitigate this issue.
Human3.6M: Human3.6M training requires different adjustments: bone loss is more important, and using every other frame increases variation because its actions are slower.The paper attributes the bone-loss difference to Kinect skeleton-length inconsistencies in NTURGB-D.
Quality assessment: For the NTURGB-D “throw” action, the ground-truth sequence receives a 97.8% probability of being a valid human sequence from the discriminator.The discriminator probability is not always accurate, especially at the start of training, and adding it to critic-based generator updates worsened results and stability.

6. Conclusions and future work

HP-GAN provides probabilistic human-motion prediction by generating multiple plausible futures from the same input and assessing their realism. Experiments on Kinect- and MoCap-based datasets support its performance, while convergence measurement and alternative recurrent architectures remain open problems.

HP-GAN predicts multiple plausible future human poses from the same input sequence.The model is presented as a sequence-to-sequence approach for probabilistic motion prediction.
Experiments show that the architecture performs well on both Kinect-based and MoCap-based datasets.The tested datasets use different capture modalities: Kinect sensing and motion capture.
The authors identify the lack of a reliable convergence measure for WGAN-GP training as a limitation.Training may diverge after appearing to have converged if the loop continues.
The study leaves investigation of newer recurrent architectures, including skip-connected RNNs and Part-Aware LSTMs, for future work.The current sequence-to-sequence network uses a standard GRU.