PaperlayerView exact revision on arXiv

Source-linked AI summary

MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction

Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, Guiqing Li

arXiv:2108.07152v2cs.CV

TL;DR

Accurate long-term human-motion prediction is difficult because future motion is stochastic and aperiodic, and pose dependencies are challenging to model. MSR-GCN combines graph convolutions with recursively abstracted multi-scale poses, bidirectional scale processing, residual learning, and intermediate supervision. It outperforms state-of-the-art methods on Human3.6M and CMU Mocap benchmarks.

  • Problem

    Stochastic and aperiodic future motion makes accurate long-term prediction difficult, particularly because human-pose spatiotemporal dependencies must be modeled.

  • Method

    MSR-GCN uses an end-to-end multi-scale residual GCN architecture with fine-to-coarse and coarse-to-fine feature extraction, multi-scale decoding, and intermediate supervision.

  • Results

    MSR-GCN outperforms state-of-the-art methods on the Human3.6M and CMU Mocap benchmark datasets.

  • Takeaways & Limitations

    Multi-scale residual graph convolution provides a strong approach for predicting future human motion across short- and long-term benchmark settings.

Abstract

from arXiv · show

Human motion prediction is a challenging task due to the stochasticity and aperiodicity of future poses. Recently, graph convolutional network has been proven to be very effective to learn dynamic relations among pose joints, which is helpful for pose prediction. On the other hand, one can abstract a human pose recursively to obtain a set of poses at multiple scales. With the increase of the abstraction level, the motion of the pose becomes more stable, which benefits pose prediction too. In this paper, we propose a novel Multi-Scale Residual Graph Convolution Network (MSR-GCN) for human pose prediction task in the manner of end-to-end. The GCNs are used to extract features from fine to coarse scale and then from coarse to fine scale. The extracted features at each scale are then combined and decoded to obtain the residuals between the input and target poses. Intermediate supervisions are imposed on all the predicted poses, which enforces the network to learn more representative features. Our proposed approach is evaluated on two standard benchmark datasets, i.e., the Human3.6M dataset and the CMU Mocap dataset. Experimental results demonstrate that our method outperforms the state-of-the-art approaches. Code and pre-trained models are available at https://github.com/Droliven/MSRGCN.

1. Introduction

Human motion prediction remains difficult because future poses are stochastic and aperiodic, while existing methods struggle to model spatiotemporal and kinematic dependencies. MSR-GCN addresses this with multi-scale residual GCNs, bidirectional scale processing, and intermediate supervision.

  • Stochastic and aperiodic future motion makes accurate long-term prediction from arbitrary observed poses challenging.
  • Prior CNN, RNN, and GAN approaches neglect inner-frame kinematic dependencies between body joints.
  • GCNs dynamically learn relations between pose joints, while recursive joint grouping represents human poses from fine to coarse scales.
  • MSR-GCN uses descending and ascending GCN paths to extract features from fine to coarse and then coarse to fine scales.
  • The method differs from prior multi-scale work by using a U-Net-like encoder-decoder with intermediate losses instead of a recurrent decoder.
  • Intermediate supervision at each scale encourages more representative features and benefits high-quality future prediction.

2. Related work

Prior human-motion methods use CNNs, RNNs, GANs, and graph networks, but each has limitations in representing pose structure or maintaining stable predictions. Multi-scale graph approaches address hierarchical structure, with MSR-GCN extracting features separately at different scales.

  • CNN-based methods treat pose sequences as spatial-temporal matrices, but pose data lacks the repeated structure that benefits image convolutions.
  • RNN-based methods can model temporal tasks but often suffer discontinuities, error accumulation, and training instability.
  • Existing methods commonly neglect inner-frame kinematic dependencies between body joints.
  • GCNs suit human-motion prediction because human skeletons have graph structure and GCNs support non-grid data.
  • A prior multi-scale graph unit fuses features across scales, whereas this work uses GCNs at different scales to extract features separately.

3. Methodology

MSR-GCN predicts future poses by learning residuals with graph convolutions across recursively abstracted pose scales. Its descending and ascending paths, multi-scale decoding, residual connections, and intermediate losses support coarse-to-fine prediction.

  • The prediction task maps observed pose sequences to future pose sequences, with MSR-GCN learning this mapping end to end.
  • The input sequence is padded by repeating its last pose, and the network predicts residuals between the padded input and ground-truth sequence.
  • 3.1. Basic GCNs: Graph convolution models spatial dependencies among pose joints using adjacency matrices and trainable feature transformations.
  • 3.1. Basic GCNs: The basic architecture uses a start GCN, 6 residual GCNs, and an end GCN, with a global skip connection for residual prediction.
  • 3.1. Basic GCNs: DCT transformations are omitted because directly computing global residuals is effective and computationally more efficient.
  • 3.2. Multi-scale Residual GCNs: Human poses are recursively abstracted into four levels with 22, 12, 7, and 4 joints, producing progressively more stable coarse-scale motion.
  • 3.2. Multi-scale Residual GCNs: Four descending and four ascending GCN blocks extract features across scales, while four end GCNs decode concatenated features into poses.
  • 3.2. Multi-scale Residual GCNs: Intermediate L2 losses supervise decoded poses at all scales, and residual connections make the network learn residuals between input and ground-truth poses.

4. Experiments

Experiments evaluate MSR-GCN on Human3.6M and CMU Mocap using MPJPE and comparisons with established baselines, including short- and long-term prediction. Results show stronger overall performance, with ablations supporting residual prediction, multi-scale architecture, intermediate supervision, and graph convolutions.

  • Evaluation metric: MPJPE in millimeters is used to measure prediction error against corresponding ground-truth joint positions.The metric averages Euclidean errors over joints and frames.
  • Datasets and settings: MSR-GCN is evaluated on Human3.6M and CMU Mocap for 400ms short-term and 1000ms long-term prediction.Human3.6M contains 15 action categories, while CMU Mocap contains 8.
  • Results on Human3.6M: On Human3.6M, MSR-GCN outperforms the compared methods overall, with the advantage becoming clearer as forecast time increases.The comparisons include short-term and long-term prediction and average errors across actions.
  • Results on CMU Mocap: On CMU Mocap, MSR-GCN achieves the best average performance at all short-term forecast times and the best long-term results for four action categories.For remaining long-term actions, its errors are second best and close to the best results.
  • Ablation analysis: A global residual improves both Traj-GCN and MSR-GCN, while MSR-GCN without global residual still outperforms baseline variants without residuals.The experiments report that inferring residuals is easier than directly predicting target poses.
  • Ablation analysis: The multi-scale strategy yields a 3.15 performance gain versus 0.55 for DCT, while ablations support intermediate supervision, residual GCNs, and the default joint grouping.Across most cases, the full MSR-GCN configuration outperforms reduced-scale, no-intermediate-loss, fully connected, and alternative-grouping variants.

5. Conclusion

MSR-GCN predicts 25 future frames from 10 observed poses using multi-scale residual graph convolutions and intermediate supervision. It outperforms state-of-the-art methods on Human3.6M and CMU Mocap.

  • MSR-GCN predicts 25 future frames from 10 observed historical poses.
  • The model uses multi-scale graph convolutions, residual connections, and losses at all scales for intermediate supervision.
  • MSR-GCN outperforms state-of-the-art methods on the Human3.6M and CMU Mocap benchmark datasets.
  • The authors identify further exploration of multi-scale grouping manners as future work.

A. Loss Function

MSR-GCN optimizes an ℓ2 objective computed across all pose scales, summing scale-specific losses with equal weighting.

  • The loss compares predicted and ground-truth joint positions across training sequences, scales, and frames using an ℓ2 objective.
  • Losses are calculated at all S scales and added together to optimize the proposed model.
  • The optimization objective uses network parameters P and scale weights λ, with λ set to 1 for every scale.

B. Model Structure

MSR-GCN is built from specialized GCN groups arranged around descending and ascending multi-scale paths, with linear sampling layers and repeated graph-convolution blocks.

  • The architecture contains Start GCNs, Descending and Ascending GCNs, and End GCNs.
  • Each Graph Convolution Layer combines graph convolution, batch normalization, tanh activation, and dropout with rate 0.1.
  • Each GCN contains two Graph Convolution Layers, while linear layers perform downsampling and upsampling.

C. Different Multi-Scale Grouping Manners

For CMU Mocap, MSR-GCN uses a default 25-12-7-4 joint grouping across progressively coarser scales and compares it with alternative groupings.

  • The default CMU grouping has 25 joints at the finest level, followed by 12, 7, and 4 joints at coarser levels.
  • Alternative experiments include three random 25-12-7-4 groupings and a manually specified 25-10-5-3 grouping.
  • Table 9 is identified as the detailed architecture of MSR-GCN.

D. More Results

MSR-GCN is compared with Traj-GCN using error bars, different forecast times, and Traj-GCN’s evaluation protocol. Across these comparisons, MSR-GCN achieves lower prediction errors and better long-term motion handling.

  • MSR-GCN reports lower errors and smaller variances than Traj-GCN across both benchmark datasets.The comparison uses five training runs with random seeds.
  • 58.37±0.43 and 37.52±0.48 are MSR-GCN’s average prediction errors on Human3.6M and CMU, respectively.
  • MSR-GCN performs better than Traj-GCN at different forecast times, including challenging long-term motion prediction.
  • Using Traj-GCN’s evaluation method with randomly selected samples per action, MSR-GCN also outperforms Traj-GCN.
Loading 2108.07152v2…