Source-linked AI summary
A Neural Temporal Model for Human Motion Prediction
Anand Gopalakrishnan, Ankur Mali, Dan Kifer, C. Lee Giles, Alexander G. Ororbia
TL;DR
Human motion prediction must produce accurate short-term forecasts and plausible long-term trajectories from brief pose histories. The paper introduces a two-level recurrent architecture, derivative features, and a multi-objective loss, alongside NPSS for long-term evaluation. The models achieve competitive short-term performance and stronger long-term trajectories, while NPSS better reflects long-term sample quality than MSE in the reported user study.
Problem
Human motion prediction requires forecasting 3D pose from brief histories across short-term accuracy and long-term plausible-trajectory evaluation, despite nonlinear and stochastic dynamics.
Method
The paper combines a two-level recurrent architecture with finite-difference derivative features, a multi-objective loss for closed-loop prediction, and the NPSS evaluation metric.
Results
VGRU-d and GRU-d produce better long-term trajectories than MBR-long and VGRU-ac across all four action classes, while short-term models remain competitive at short horizons.
Takeaways & Limitations
NPSS is better equipped than MSE to capture long-term sample-quality differences, and the proposed framework achieves state-of-the-art long-term synthesis with competitive short-term performance.
Takeaways & Limitations
Further studies are needed to validate and strengthen the user-study findings supporting NPSS.
Abstract
from arXiv · showhide
We propose novel neural temporal models for predicting and synthesizing human motion, achieving state-of-the-art in modeling long-term motion trajectories while being competitive with prior work in short-term prediction and requiring significantly less computation. Key aspects of our proposed system include: 1) a novel, two-level processing architecture that aids in generating planned trajectories, 2) a simple set of easily computable features that integrate derivative information, and 3) a novel multi-objective loss function that helps the model to slowly progress from simple next-step prediction to the harder task of multi-step, closed-loop prediction. Our results demonstrate that these innovations improve the modeling of long-term motion trajectories. Finally, we propose a novel metric, called Normalized Power Spectrum Similarity (NPSS), to evaluate the long-term predictive ability of motion synthesis models, complementing the popular mean-squared error (MSE) measure of Euler joint angles over time. We conduct a user study to determine if the proposed NPSS correlates with human evaluation of long-term motion more strongly than MSE and find that it indeed does. We release code and additional results (visualizations) for this paper at: https://github.com/cr7anand/neural_temporal_models
1. Introduction
The paper targets human motion prediction from brief pose histories, addressing both short-term accuracy and long-term plausible trajectories. It combines temporal modeling innovations with a complementary metric for evaluating long-term motion synthesis.
- The task is forecasting 3D human pose from a small set of priming frames using motion-capture data.
- Short-term prediction is typically evaluated with MSE, whereas long-term prediction is mainly judged by visual inspection of plausible trajectories.
- Human motion prediction is difficult because motion is high-dimensional, nonlinear, stochastic, and influenced by intent and high-level semantics.
- The proposed models augment joint angles with finite-difference motion derivatives and use a multi-objective loss to improve closed-loop long-term prediction.
- A two-stage architecture with differentiable backward planning supports trajectory generation while remaining simpler and less computationally expensive than competing alternatives.
- NPSS complements MSE by comparing power spectra, addressing frequency and phase shifts that can make qualitatively similar motions appear dissimilar under MSE.
2. Related Work
Prior work established neural architectures and baselines for short- and long-term human motion prediction, but the field continued to explore how to model motion trajectories effectively. The reviewed approaches include deep recurrent and sequence-to-sequence systems.
- LSTM-3LR uses three layers of 1000 LSTM units, while ERD uses two layers of 1000 LSTM units with nonlinear encoder and decoder networks.
- Earlier work reported neural models that outperformed previous methods in both short-term prediction and long-term qualitative motion.
- Later work proposed simple short-term baselines and a one-layer sequence-to-sequence model with 1024 GRU units and a linear output decoder.
3. A Neural Motion Synthesizer
The paper presents a neural motion-synthesis system that combines a two-level recurrent architecture, derivative-augmented inputs, and a curriculum-style multi-objective loss. It also introduces NPSS to evaluate long-term motion beyond trajectory-level MSE.
- 3.2. Architecture: The VTLN-RNN uses top-level and bottom-level recurrent networks to generate guide vectors for future motion trajectories.The top-level process runs backward from a sampled hidden state and conditions on the action label, while the lower-level process uses the guide vectors with observed data.
- 3.2. Architecture: The proposed two-level process improves model generalization, while the overall model uses fewer parameters than prior alternatives and remains competitive across prediction horizons.The paper reports state-of-the-art long-term motion prediction and competitive short-term prediction on the adopted benchmark.
- 3.3. Incorporating Derivative Information: Motion derivatives are appended to joint-angle features using finite backward differences, with first-, second-, and third-order derivatives computed using h = 1.These features provide local motion information, require no additional model parameters, and are recalculated during closed-loop prediction.
- 3.4. Facilitating Closed-Loop Prediction: The multi-objective loss gradually increases the weight of closed-loop prediction as training progresses from one-step prediction.This addresses the mismatch between ground-truth inputs during training and model-generated inputs during iterative prediction, which can otherwise cause long-term drift.
- 3.5. A Complementary Long-Term Motion Metric: Long-term evaluation is difficult because stochastic motion can diverge from ground-truth trajectories while remaining qualitatively plausible.Frequency shifts and phase shifts can produce large MSE even when predicted motion remains visually similar.
- 3.5. A Complementary Long-Term Motion Metric: NPSS compares normalized power spectra of predicted and ground-truth joint-angle sequences using power-weighted averages of one-dimensional Earth Mover’s Distances.The metric is intended to capture distributional differences in motion spectra and better reflect visual trajectory quality.
4. Experiments
Experiments compare short- and long-term recurrent motion models using motion-derivative inputs and the proposed loss, then assess whether NPSS agrees with human judgments better than MSE. Long-term visual comparisons and a user study support the proposed models and metric.
- Training Setup: Long-term models were trained on single-action data with 50 seed frames and evaluated over 100 predicted frames, corresponding to four seconds.Optimization used RMSprop for 10,000 iterations with mini-batches of 32 and the proposed cost function.
- Training Setup: VGRU-d and GRU-d use motion-derivative inputs and the proposed loss, while VGRU-ac is an auto-conditioning baseline.VGRU-d uses single-layer 512-cell GRUs in both recurrent components; GRU-d uses two 512-unit GRU layers.
- Long-Term Motion Synthesis: GRU-d and VTLN-GRU-d are qualitatively closer to the walking ground-truth sequence than MBR-long and VTLN-GRU-ac across snapshots through 4000 milliseconds.Snapshots are shown at 160, 560, 1000, 2000, and 4000 milliseconds along the prediction time-axis.
- User Study: The user study compared six models using pairwise videos judged by overall skeletal pose and joint-motion dynamics across entire sequences.Twenty sequences were presented for each of four action classes, with samples selected from pairwise model combinations.
- Metric Agreement: A binomial test found that NPSS agreed with human ordering more often than MSE, with p-value 1.7 × 10^-5.The test used H0: p1 ≤ p2 and HA: p1 > p2, where p1 and p2 denote NPSS and MSE agreement probabilities.
5. Results and Discussion
The proposed models perform competitively for short-term prediction and improve long-term motion trajectories, while NPSS better distinguishes long-term sample quality than MSE. User-study results support preferring NPSS for long-horizon evaluation, alongside MSE for holistic assessment.
- Evaluation: The user study supports preferring NPSS over MSE for long-sequence generation quality, while recommending both metrics for models covering short- and long-term prediction.Further studies are needed to validate and strengthen the NPSS findings.
- Metric comparison: Short-term models have lower MSE through 1 second but worse NPSS than long-term models, matching the visual quality of generated samples.This discrepancy illustrates that MSE is less suited to evaluating long-term motion synthesis.
- Long-term prediction: GRU-d and VGRU-d produce better long-term trajectories than MBR-long and VGRU-ac across all four action classes.The comparison is based on NPSS scores.
- Prediction horizons: Across 0-1, 1-2, and 2-4 second horizons, short-term models degrade increasingly relative to long-term models as the prediction horizon grows.The largest NPSS gaps occur in the 2-4 second long-term window, while GRU-d and VGRU-d perform best across actions and time horizons.
- Short-term prediction: VGRU-r1 is competitive with the state-of-the-art short-term MBR model and the zero-velocity baseline on multi-action prediction.The model uses the VTLN-RNN architecture with motion-derivative features and a multi-objective loss.
- Ablation findings: The study finds that the proposed innovations were necessary for improving long-term synthesis performance, with the multi-objective cost playing the most important role.The comparison concerns the architecture, derivative features, and loss-function innovations.
6. Conclusions and Future Work
The paper introduces VTLN-RNN for human motion prediction and synthesis, combining derivative features with a multi-objective loss. It achieves state-of-the-art long-term synthesis, competitive short-term prediction, and introduces NPSS to address limitations of MSE in long-term evaluation.
- Conclusions: VTLN-RNN combines motion-derivative features and a novel multi-objective loss to achieve state-of-the-art long-term motion synthesis.The framework also achieves competitive short-term motion prediction.
- Future work: NPSS is proposed as a metric that addresses key drawbacks of MSE in evaluating long-term motion synthesis.Future work includes incorporating NPSS into optimization and improving multi-action training for long-term synthesis.