PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Prediction Helps 3D Human Motion Modelling

Emre Aksan, Manuel Kaufmann, Otmar Hilliges

arXiv:1910.09070v1cs.CV

TL;DR

Human motion prediction is difficult because existing methods do not explicitly model dependencies among skeletal joints, and common H3.6M evaluation is limited. The paper introduces a structured prediction layer that decomposes pose prediction by joint and connects neural networks along kinematic chains. On AMASS, the layer improves every tested baseline across metrics and input representations, while the study also shows that some H3.6M-leading techniques do not scale well to this larger benchmark.

  • Problem

    Human motion prediction remains difficult because existing work only implicitly models skeletal spatial structure, while H3.6M offers limited scale and diversity for evaluation.

  • Method

    The structured prediction layer decomposes pose prediction into individual joints, propagates parent predictions along kinematic chains, and uses a joint-wise loss decomposition.

  • Results

    On AMASS, the SP-layer benefits every tested baseline model, metric, and input representation, and a simple RNN with SPL achieves state-of-the-art performance.

  • Takeaways & Limitations

    Explicit skeletal structure can improve motion forecasting across base networks, joint-angle representations, and prediction horizons.

  • Takeaways & Limitations

    H3.6M metrics exhibit high variance because of few test samples, and low errors do not always correspond to good qualitative predictions.

Abstract

from arXiv · show

Human motion prediction is a challenging and important task in many computer vision application domains. Existing work only implicitly models the spatial structure of the human skeleton. In this paper, we propose a novel approach that decomposes the prediction into individual joints by means of a structured prediction layer that explicitly models the joint dependencies. This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function. The proposed layer is agnostic to the underlying network and can be used with existing architectures for motion modelling. Prior work typically leverages the H3.6M dataset. We show that some state-of-the-art techniques do not perform well when trained and tested on AMASS, a recently released dataset 14 times the size of H3.6M. Our experiments indicate that the proposed layer increases the performance of motion forecasting irrespective of the base network, joint-angle representation, and prediction horizon. We furthermore show that the layer also improves motion predictions qualitatively. We make code and models publicly available at https://ait.ethz.ch/projects/2019/spl.

1. Introduction

Human motion prediction remains difficult because motion is dynamic, uncertain, and structurally dependent across joints, while common evaluation on H3.6M is limited. The paper introduces AMASS-based evaluation and a structured prediction layer that explicitly models skeletal dependencies.

  • Motivation: Human motion is dynamic, nonlinear, stochastic, and difficult to model because joints are interdependent and activities influence motion sequences.These challenges motivate explicit structural and temporal priors for forecasting.
  • Research gap: Quantitative performance may not yield qualitatively meaningful predictions, while H3.6M’s saturation motivates broader evaluation questions.The paper asks how to measure meaningful pose accuracy and exploit skeletal structure for better predictions.
  • Benchmark: 14x more samples and broader activities make AMASS a larger, more diverse benchmark than H3.6M.The paper also introduces several evaluation metrics for human motion prediction.
  • Method: The structured prediction layer decomposes poses into individual joints and propagates parent predictions through neural networks arranged along kinematic chains.Its joint-wise loss decomposition explicitly models spatial dependencies and the layer can combine with existing architectures.
  • Results: On AMASS, the layer benefits every tested base model, metric, and input representation, including simple architectures that otherwise trail a zero-velocity baseline.The impact is most pronounced on AMASS, and the layer is presented as a way to exploit spatial priors and improve temporal coherence.
  • Contributions: The paper contributes an analysis of existing methods, an AMASS benchmark and protocol, a structure-aware prediction layer, and broad evaluations across base models.The stated contributions cover both evaluation and modeling advances.

2. Related Work

Prior work uses recurrent, latent, quaternion, positional, and structure-aware models for human motion, but evaluation on H3.6M can be misleading. The paper positions its SP-layer as a flexible alternative that improves performance on the larger AMASS benchmark.

  • Recurrent models: Earlier recurrent approaches use RNNs, LSTMs, GRUs, curriculum learning, residual connections, or adversarial training to address temporal drift and long-term prediction.These methods include encoder-recurrent-decoder and sequence-to-sequence formulations.
  • Benchmark findings: Sequence-to-sequence models that perform well on H3.6M fare worse on the larger, more diverse AMASS dataset.Adding the SP-layer boosts their performance, but a simple RNN with the same layer performs better.
  • Structured prediction: Existing structure-aware methods include manually specified spatio-temporal graphs and kinematic-chain hierarchies, but differ in where structure is introduced and which architectures they support.The paper emphasizes that its layer avoids coarse joint subdivision and manual graph definition while interfacing with most existing architectures.
  • Related structure-aware models: Structure-aware architectures also appear in 3D pose estimation through structured latent spaces, distance matrices, skeletal losses, and joint-specific recurrent cells.These approaches provide related precedents for explicitly incorporating skeletal structure into networks.
  • Representations: Motion models parameterize poses with exponential maps, quaternions, or 3D joint positions, with quaternions enabling differentiable forward-kinematics loss computation.The paper instead uses joint angles because they encode symmetries and are bone-length invariant.

3. Method

The structured prediction layer decomposes pose prediction by joint and follows the skeleton’s kinematic hierarchy, explicitly modelling spatial dependencies while remaining compatible with existing architectures. A joint-wise objective and restricted connections further focus learning on structurally relevant information.

  • Structured Prediction Layer: SPL decomposes the predicted pose into individual joints and conditions each prediction on the human skeleton’s kinematic hierarchy.The layer receives a context representation summarizing past frames and predicts the full pose by following parent-child connections.
  • Structured Prediction Layer: Each joint receives information from its immediate parent’s prediction and the shared temporal context, enabling hierarchical spatial factorization.The parent prediction provides explicit local structural information, while the context supplies information from the motion history.
  • Structured Prediction Layer: Separate small networks model individual joints, allowing dedicated joint representations and parent-to-child message passing for local predictions.The paper describes this as integrating a structural prior while saving model capacity through joint-specific representations.
  • Evaluation: Experiments show that SPL improves prediction performance across diverse underlying architectures, settings, and metrics.The layer is designed as an interface component that can be added to existing motion-modelling networks.
  • Structured Prediction Layer: SPL restricts connections using the hierarchy, so hidden-unit gradients are affected by the joint hierarchy rather than being distributed arbitrarily across all joints.Compared with a dense layer, the structured design explicitly sets some connections to zero and narrows gradient flow.
  • Per-joint Loss: The objective preserves joint integrity by calculating the loss for each joint before summing across the motion sequence.The implementation uses MSE, while the formulation can accommodate domain-specific losses such as geodesic distance.

4. Human Motion Modelling

The evaluation uses H3.6M and the larger AMASS dataset with three base architectures and three pose representations. SPL is integrated modularly by replacing the final output layer while preserving the remaining model setup as much as possible.

  • Experimental Setup: Experiments evaluate SPL on two datasets and three underlying architectures using three different pose representations.The evaluation includes H3.6M for comparison with prior work and AMASS for larger-scale motion modelling.
  • Datasets: AMASS contains 8′593 motion sequences drawn from multiple public motion databases and represented with the SMPL model.The dataset is introduced to address the limited size and diversity of H3.6M.
  • Datasets: AMASS uses roughly 90%, 5%, and 5% training, validation, and test splits, respectively, with 3′304 test samples extracted by shifting 2-second windows.The target sequences are 400 ms, or 24 frames, while H3.6M uses 120 test samples across 15 categories.
  • Datasets: H3.6M and AMASS represent the human skeleton with 21 and 15 major joints, respectively, requiring separate SP-layers for each skeleton.The differing joint counts reflect dataset-specific underlying skeletons.
  • Models: The evaluated base models are Seq2seq with exponential maps, QuaterNet with quaternions, and RNN with rotation matrices.For Seq2seq and QuaterNet, the final linear output layer is replaced by SPL while the remaining setup is retained.
  • SP-layer Configuration: Each SPL joint network uses one small hidden layer with 64 or 128 units, and the true kinematic chain performs best among tested hierarchical configurations.Some models benefit from conditioning on all parent joints rather than only the immediate parent.

5. Evaluation on H3.6M Dataset

The H3.6M evaluation compares SP-layer-augmented models with established baselines using the Euler angle metric across activities and prediction horizons. Adding the SP-layer improves Seq2seq substantially and benefits several other models or categories, although H3.6M metrics have important limitations.

  • Evaluation setup: The evaluation measures walking, eating, smoking, and discussion across prediction horizons using the Euler angle metric.Table 1 reports H3.6M results at the given prediction times.
  • Results: Seq2seq augmented with the SP-layer achieves state-of-the-art performance in the walking category.The SP-layer significantly improves Seq2seq performance relative to its baseline.
  • Results: SP-layer augmentation gives QuaterNet the best performance for short-term smoking and discussion motions and generally matches or marginally exceeds vanilla QuaterNet.The comparison follows the original baseline training and evaluation protocols.
  • Results: The SP-layer boosts the RNN model for walking, eating, and smoking, while discussion performance remains similar.The reported gains vary across motion categories.
  • Evaluation limitations: H3.6M evaluation metrics exhibit high variance because of few test samples, and low errors do not always correspond to qualitatively good predictions.The study retains the prior evaluation setting for direct comparability.

6. AMASS: A New Benchmark

AMASS provides a larger, more diverse benchmark and motivates complementary evaluation metrics for human motion forecasting. Experiments show that the SP-layer improves existing models across architectures, representations, metrics, and prediction horizons, with especially strong benefits for RNNs.

  • Benchmark and metrics: AMASS increases task complexity and test reliability through its larger quantity and diversity of motion samples.The evaluation uses AMASS to address limitations of the smaller, less diverse H3.6M benchmark.
  • Benchmark and metrics: Euler-angle evaluation is supplemented with joint-angle difference, positional, and PCK metrics to assess complementary aspects of pose accuracy.The proposed evaluation aggregates metrics through time and uses more test samples, while training remains unchanged.
  • Results: The SP-layer improves models across AMASS settings, including different base models, metrics, input representations, and prediction horizons.The improvement is reported for existing approaches and is most pronounced on the larger and more challenging AMASS dataset.
  • Results: RNN-SPL achieves the best results under different metrics and prediction horizons, while Seq2seq models are generally noncompetitive on AMASS.Seq2seq models are sometimes outperformed by the zero-velocity baseline, whereas SP-layer extensions improve QuaterNet and especially RNN models.
  • Results: 1.26 is the Seq2seq-dropout-SPL improvement reported for the Euler-angle metric at 100 ms, although proposed metrics place it marginally below RNN-SPL.This contrast illustrates why the complementary metrics are useful: Euler-angle results alone do not identify the same best-performing model.
  • Results: Residual connections improve short-term predictions across datasets and pose representations.All trained models performed better with the residual connection.
  • Ablation Study: The ablation study identifies output decomposition and per-joint loss as the main sources of the SP-layer’s performance boost.Per-joint loss alone helps on AMASS, while the true kinematic-chain hierarchy generally performs best except on the positional metric.

7. Conclusion

The paper introduces SPL to encode human skeletal structure in neural motion models and reports broad benefits across baselines, metrics, and representations on AMASS.

  • SPL explicitly incorporates prior knowledge of skeletal structure into neural networks by decomposing poses into individual joints.
  • On AMASS, SPL improves every baseline model across every evaluated metric and input representation.
  • The simple RNN augmented with SPL achieves state-of-the-art performance on the AMASS benchmark.
  • The authors introduce AMASS and several metrics alongside SPL for evaluating motion prediction.

8. Appendix

The appendix details SPL implementations across RNN, Seq2seq, and QuaterNet models, including their architectures, representations, training objectives, and evaluation settings. It reports longer-term results, qualitative examples, PCK plots, and ablation studies.

  • Architecture: SPL replaces standard dense layers with joint-wise networks that follow the skeleton’s kinematic chain, using 64 or 128 hidden units per joint.Joint output size is 3, 4, or 9 for exponential-map, quaternion, or rotation-matrix representations, respectively.
  • Training and evaluation: The models use 2-second seed sequences and 400-millisecond targets, with SPL variants optimizing the proposed joint-wise loss and validation using joint-angle early stopping.The early-stopping metric differs from the training objective for all models.
  • RNN-SPL: RNN-SPL uses rotation matrices, teacher forcing, joint-wise l2 loss, input dropout, and a single 1024-unit LSTM for autoregressive prediction.The dropout is intended to improve robustness to exposure bias in the absence of sampling-based training.
  • Seq2seq-SPL and QuaterNet-SPL: Seq2seq-SPL and QuaterNet-SPL retain sequence-to-sequence encoders and decoders, while using exponential-map or quaternion pose representations and joint-structured prediction.QuaterNet-SPL uses quaternion multiplication for residual velocity and explicitly normalizes predictions to enforce valid rotations.
  • Results: The proposed SP-layer consistently improves long-term performance, with RNN-SPL performing best across different metrics while QuaterNet-SPL remains competitive.The appendix also includes qualitative comparisons, PCK plots for 100–400 ms horizons, and full H3.6M and AMASS ablations.
Loading 1910.09070v1…