PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction

Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, Qi Tian

arXiv:2003.08802v1cs.CVcs.LGstat.ML

TL;DR

3D skeleton-based motion prediction must capture relations among body components across different scales. DMGNN learns adaptive, layer-dynamic multiscale graphs with an encoder-decoder architecture, and experiments on Human 3.6M and CMU Mocap report stronger short- and long-term prediction than most state-of-the-art methods.

  • Problem

    Existing motion-prediction methods rarely exploit comprehensive relations across body-component scales, although these relations carry important motion information.

  • Method

    DMGNN uses adaptive, layer-dynamic multiscale graphs, MGCUs for multiscale feature extraction and fusion, and a graph-based GRU decoder for pose generation.

  • Results

    DMGNN outperforms most state-of-the-art methods for short- and long-term motion prediction on Human 3.6M and CMU Mocap.

  • Takeaways & Limitations

    The learned multiscale graphs provide an interpretable representation of body relations for action-category-agnostic motion prediction.

Abstract

from arXiv · show

We propose novel dynamic multiscale graph neural networks (DMGNN) to predict 3D skeleton-based human motions. The core idea of DMGNN is to use a multiscale graph to comprehensively model the internal relations of a human body for motion feature learning. This multiscale graph is adaptive during training and dynamic across network layers. Based on this graph, we propose a multiscale graph computational unit (MGCU) to extract features at individual scales and fuse features across scales. The entire model is action-category-agnostic and follows an encoder-decoder framework. The encoder consists of a sequence of MGCUs to learn motion features. The decoder uses a proposed graph-based gate recurrent unit to generate future poses. Extensive experiments show that the proposed DMGNN outperforms state-of-the-art methods in both short and long-term predictions on the datasets of Human 3.6M and CMU Mocap. We further investigate the learned multiscale graphs for the interpretability. The codes could be downloaded from https://github.com/limaosen0/DMGNN.

1. Introduction

The paper addresses limited modeling of relations among body components by introducing dynamic multiscale graphs and an encoder-decoder DMGNN for action-category-agnostic motion prediction. The model extracts and fuses multiscale features, generates poses with a graph-based GRU, and is evaluated on two motion datasets.

  • Motivation: Existing motion-prediction methods rarely model relations across body-component scales, despite their importance for prediction.Prior graph methods model joint pairs or fixed body parts but do not capture functional groups, movement coordination, and relations comprehensively.
  • Multiscale representation: A multiscale graph represents body components at multiple scales using single-scale and cross-scale sub-graphs.Single-scale graphs form a body-skeleton pyramid, while bipartite cross-scale graphs connect components between adjacent scales.
  • Dynamic graph learning: The multiscale graph is initialized from physical connections, adaptively tuned during training, and dynamically changed across MGCUs.Cross-scale graphs are constructed adaptively from input motions to reflect discriminative motion patterns.
  • Architecture: DMGNN uses cascaded MGCUs in an encoder to extract and fuse features across scales, followed by a graph-based GRU decoder for pose generation.The decoder also uses trainable graphs, residual connections, and multiple motion-difference orders.
  • Evaluation: Experiments on Human 3.6M and CMU Mocap report that DMGNN outperforms most state-of-the-art methods for short- and long-term prediction.The paper reports improvements in both effectiveness and efficiency and visualizes learned graphs for interpretability.

3. Problem Formulation

The task is to generate future 3D skeleton poses from observed historical poses. DMGNN represents the body at joint, low-level-part, and high-level-part scales to exploit richer body relations in end-to-end prediction.

  • Prediction task: Given historical 3D skeleton poses, motion prediction generates future poses close to the target sequence.The observed and future poses contain M joints with three-dimensional features.
  • Multiscale body representation: DMGNN models the body with three scales: body joints, low-level parts, and high-level parts.Spatially nearby joints are merged into coarser components using human-prior structure.

4. Key Components

DMGNN builds motion features with adaptive multiscale graphs, extracting features within scales and fusing information across scales. Its components include MGCUs, cross-scale graph inference, a graph-based GRU, and difference operators for richer motion dynamics.

  • Multiscale graph computational unit: An MGCU extracts features at individual scales with single-scale graph convolution blocks and fuses them across scales with cross-scale fusion blocks.The single-scale blocks use graph and temporal convolutions, while cross-scale blocks convert features between scales.
  • Single-scale graph convolution block: Single-scale graphs start from physical skeleton connections and are adaptively tuned during training to capture flexible body relations.Each graph is associated with a scale whose body-components define its nodes.
  • Cross-scale fusion block: Cross-scale graphs are bipartite relations that connect components between scales, such as a coarse arm node and fine hand or elbow nodes.Their edge weights are inferred from temporally processed component features using relative-feature aggregation, inner products, and softmax normalization.
  • Cross-scale fusion block: Cross-scale fusion lets coarse body-part features absorb detailed joint information before feeding the fused representation into the next single-scale block.Fusion is described from joint scale s1 to part scale s2 and can be defined in the reverse direction similarly.
  • Graph-based GRU: The graph-based GRU propagates hidden-state information with a trainable graph while generating the state for the next frame.The cell receives online 3D skeleton information and the current hidden state, then applies graph convolution for state propagation.
  • Difference operator: The difference operator computes higher-order pose differences, with β = 2 representing positions, velocities, and accelerations.Zero padding handles boundary conditions after differences are computed.

5. DMGNN Framework

DMGNN uses a multiscale graph-based encoder to extract motion semantics and a graph-based recurrent decoder to sequentially generate future poses. It trains the model end-to-end with an ℓ1 prediction loss.

  • 5. DMGNN Framework: The encoder initializes three body scales from joint differences and uses cascaded MGCUs to extract spatio-temporal motion features.The input concatenates zeroth-, first-, and second-order differences; coarser components are formed by averaging corresponding joint clusters.
  • 5. DMGNN Framework: Each MGCU has its own trained multiscale graph, allowing graph topology to change across encoder layers before scale features are weighted and summed.Temporal average pooling then produces the historical motion state used to initialize the decoder.
  • 5. DMGNN Framework: The decoder’s graph-based GRU updates motion states from three orders of motion differences, then an output function predicts pose displacements.Residual connections are used to stabilize prediction.
  • 5. DMGNN Framework: Predicted displacements are added to the input pose to generate the next frame sequentially.The decoder uses MLPs as the output function, with its initial hidden state supplied by the encoder.
  • 5. DMGNN Framework: The model is trained with an ℓ1 loss over predicted and ground-truth future poses using end-to-end stochastic gradient descent.The paper states that ℓ1 loss provides stable gradients for large-loss joints and more precise predictions than ℓ2 loss in experiments.

6. Experiments

Experiments evaluate DMGNN on Human 3.6M and CMU Mocap using mean angle error, qualitative samples, efficiency, and ablations. DMGNN generally performs strongly across short- and long-term prediction, while graph scale, fusion depth, motion differences, and encoder depth affect results.

  • Experimental setup: Experiments use Human 3.6M and CMU Mocap, with mean angle error measuring generated-versus-ground-truth motions in angle space.Qualitative predicted samples are also evaluated.
  • Short-term prediction: DMGNN achieves the lowest errors on Eating and Smoking, competitive results on Walking and Discussion, and the best performance on most remaining Human 3.6M actions.The complete model also outperforms its fixed-graph, no-G-GRU, and single-scale variants.
  • Long-term prediction: DMGNN outperforms competitors on Eating and Discussion at 560 ms and remains competitive in other reported long-term Human 3.6M cases.Long-term prediction covers 560 ms and 1000 ms horizons.
  • CMU Mocap: On CMU Mocap, DMGNN significantly outperforms state-of-the-art methods for Basketball, Basketball Signal, Running, and Walking, with competitive results on other actions.The evaluation covers eight action classes across future 1000 ms.
  • Efficiency: DMGNN generates 400 ms of motion in 29.18 ms and has the shortest reported running time for both 400 ms and 1000 ms predictions.The efficiency comparison is performed on Human 3.6M.
  • Ablation and interpretability: Two scales outperform the joint-only scale, whereas overly abstract scales hurt prediction; four MGCUs balance low error and running efficiency.Two CS-FBs and 0-, 1-, and 2-order motion differences also yield the best reported ablation settings.
  • Ablation and interpretability: Dynamic cross-scale graphs capture nearby components in early fusion and more global, action-related effects in later fusion.The learned graphs differ across actions, and the second cross-scale graph is more informative for action classification.

7. Conclusion

The paper concludes that DMGNN represents human bodies with dynamic multiscale graphs and combines them with an encoder-decoder architecture for 3D skeleton-based motion prediction. Reported results show advantages over most state-of-the-art methods in both prediction quality and efficiency.

  • DMGNN uses dynamic multiscale graphs, MGCUs in the encoder, and a graph-based GRU in the decoder for 3D skeleton-based motion prediction.
  • The reported model outperforms most state-of-the-art methods for short- and long-term prediction in effectiveness and efficiency.

8. Detailed Architecture

The architecture combines single-scale graph convolutions and cross-scale fusion in the encoder with a graph-based recurrent decoder. Its graph operations learn component features, cross-scale relations, and sequential pose displacements.

  • Encoder: The encoder combines SS-GCBs at multiple scales with CS-FBs across scales inside four cascaded MGCUs.The first two MGCUs use both blocks, while the last two use only SS-GCBs.
  • Single-scale graph convolution block: Each SS-GCB applies temporal convolution followed by graph convolution to extract features for body components at a given scale.
  • Cross-scale fusion block: CS-FB aggregates relative features across scales and uses inner products with softmax to infer normalized cross-scale adjacency weights.The resulting graph models effects from components in one scale to components in another.
  • Graph-based recurrent decoder: The G-GRU receives historical motion state and online 3D skeleton information, then uses graph convolution to propagate motion information into the next state.Its hidden dimension is 256.
  • Graph-based recurrent decoder: At each decoding step, the G-GRU and an MLP output function model motion displacement between consecutive frames from the hidden state and current input.

9. Quantitative Comparison with more Baselines

An expanded comparison against additional baselines evaluates short-term prediction on four representative Human 3.6M actions. DMGNN outperforms state-of-the-art methods on most actions.

  • DMGNN outperforms the expanded set of state-of-the-art methods on most short-term prediction actions in Human 3.6M.The comparison reports mean angle errors for four representative actions.

10. Coarser Body-scales in Ablation Studies

The ablation study adds two coarser body scales, s4 and s5, to the effective three-scale configuration. These scales represent larger body components by averaging input features and connecting the resulting nodes.

  • The ablation initializes s4 and s5 as two coarser body scales alongside the three scales used in DMGNN.The three effective scales are s1, s2, and s3.
  • s4 represents left-body, head-and-torso, and right-body as three graph nodes initialized by averaged input features.Two edges connect head-and-torso with left-body and right-body.
  • s5 represents upper-body and lower-body as two graph nodes initialized by averaged input features.An edge connects the two body components.

11. Effects of Numbers and Positions of CS-FBs

This ablation studies how many CS-FBs to use, where to place them, and whether they aggregate relative information. The strongest variant uses two relative-information CS-FBs in the first two MGCUs, while later fusion and excessive capacity are less effective.

  • The experiment varies one to four CS-FBs across four MGCUs and records average prediction MAEs for each configuration.Table 17 distinguishes CS-FB number, position, and MAE with or without relative information.
  • Two CS-FBs with relative information aggregation at the 1st and 2nd MGCUs produce the most precise predictions among the tested variants.The comparison uses average prediction MAEs on H3.6M short-term motion prediction across 400 ms.
  • Aggregating global relative information in CS-FBs yields lower MAEs than omitting relative-information aggregation.
  • Fusing multiscale features at the first few MGCUs outperforms fusing them at the last MGCUs.
  • Using only one CS-FB may limit comprehensive pattern learning, whereas too many CS-FBs increase network capacity and can lead to overfitting.

12. More Generated Motion Samples

Additional qualitative examples compare DMGNN with baseline models on long-term Human 3.6M and CMU Mocap predictions. Across posing, waiting, basketball, and washing-window actions, the passages report smaller or more stable DMGNN prediction errors than the baselines.

  • Human 3.6M: For Human 3.6M posing, DMGNN models stretched bodies and arms, while baselines show discontinuities or larger errors by 400 ms.The comparison uses Res-sup., CSM, and Traj-GCN over a 1000 ms prediction.
  • The section presents predicted samples from four models for Human 3.6M actions and three models for CMU Mocap actions.The figures cover posing, waiting, basketball, and washing window.
  • Human 3.6M: For Human 3.6M waiting, baselines lose movement or develop large errors by 320 ms, whereas DMGNN avoids the reported unreasonable 1000 ms poses.
  • CMU Mocap: For CMU Mocap basketball, CSM and Traj-GCN develop rising errors after 320 ms and unreasonable 1000 ms postures, while DMGNN has smaller short- and long-term errors.The examples focus on running legs and swaying arms.
  • CMU Mocap: For CMU Mocap washing window, baseline discontinuities or missed left-arm motion accompany large 1000 ms errors, while DMGNN has smaller short- and long-term errors.
Loading 2003.08802v1…