PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Trajectory Dependencies for Human Motion Prediction

Wei Mao, Miaomiao Liu, Mathieu Salzmann, Hongdong Li

arXiv:1908.05436v3cs.CV

TL;DR

Human motion prediction models based on RNNs can accumulate errors and create discontinuities, while feed-forward alternatives require manually chosen temporal and spatial dependency structures. The paper uses DCT trajectory encoding with graph convolutions that learn connectivity, achieving state-of-the-art benchmark results and favoring 3D representations, with geodesic loss unavailable in 3D space.

  • Problem

    RNN motion predictors suffer from accumulated errors and discontinuities, while convolutional temporal and spatial modeling depends on manually chosen filter or graph structures.

  • Method

    The method predicts future DCT coefficients in trajectory space and uses graph convolutional networks with learnable connectivity to encode spatial joint dependencies.

  • Results

    The approach achieves state-of-the-art results on standard human motion prediction benchmarks, with experiments favoring direct 3D-space prediction over ambiguous angle representations.

  • Takeaways & Limitations

    The experiments support using compact DCT-based temporal encoding and learned spatial connectivity, and argue that motion prediction should be performed in 3D space.

  • Takeaways & Limitations

    The geodesic loss used by prior work does not apply to 3D space.

Abstract

from arXiv · show

Human motion prediction, i.e., forecasting future body poses given observed pose sequence, has typically been tackled with recurrent neural networks (RNNs). However, as evidenced by prior work, the resulted RNN models suffer from prediction errors accumulation, leading to undesired discontinuities in motion prediction. In this paper, we propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. In this context, we then propose to encode temporal information by working in trajectory space, instead of the traditionally-used pose space. This alleviates us from manually defining the range of temporal dependencies (or temporal convolutional filter size, as done in previous work). Moreover, spatial dependency of human pose is encoded by treating a human pose as a generic graph (rather than a human skeletal kinematic tree) formed by links between every pair of body joints. Instead of using a pre-defined graph structure, we design a new graph convolutional network to learn graph connectivity automatically. This allows the network to capture long range dependencies beyond that of human kinematic tree. We evaluate our approach on several standard benchmark datasets for motion prediction, including Human3.6M, the CMU motion capture dataset and 3DPW. Our experiments clearly demonstrate that the proposed approach achieves state of the art performance, and is applicable to both angle-based and position-based pose representations. The code is available at https://github.com/wei-mao-2019/LearnTrajDep

1. Introduction

The paper targets error accumulation, discontinuities, and manually bounded temporal or spatial dependencies in human motion prediction. It proposes DCT-based trajectory encoding and graph convolutions with learned connectivity, reporting state-of-the-art results.

  • Motivation: RNN motion predictors can accumulate errors during generation and produce discontinuities between observed and predicted frames.These issues are associated with feeding current estimates into subsequent steps and frame-by-frame regression that does not encourage global smoothness.
  • Motivation: Feed-forward temporal modeling with convolutions depends strongly on the chosen temporal filter size.The paper introduces trajectory-space encoding to remove this manual dependency.
  • Contributions: The method represents each joint trajectory with DCT bases and predicts future DCT coefficients from those of the observed poses.The representation applies to both angle-based poses and 3D joint positions.
  • Contributions: Skeleton-based spatial modeling cannot capture dependencies across different limbs, while large spatial filters impose another manually chosen size.The paper uses graph convolutions to address these limitations.
  • Contributions: Learned graph connectivity lets the network model joint dependencies beyond the kinematic tree or an arbitrarily selected convolutional kernel.The authors identify learnable temporal encoding and learnable graph convolution as the paper’s two contributions.
  • Contributions: Experiments on standard human motion prediction benchmarks report state-of-the-art results in all cases.The introduction presents these experiments as evidence for the benefits of the proposed approach.

2. Related Work

Prior motion-prediction work is dominated by RNNs and manually designed spatio-temporal structures. The paper positions learned graph connectivity as a way to provide more flexible modeling of long-range joint interactions.

  • RNN-based human motion prediction: RNNs became the de facto approach for human motion prediction because of their success on sequence-to-sequence prediction.Early models included encoder-recurrent-decoder architectures, and error accumulation was already observed in this line of work.
  • RNN-based human motion prediction: Structural-RNN models manually design spatio-temporal graphs, limiting discovery of long-range interactions between different limbs.This motivates more flexible alternatives to fixed graph structures.
  • Graph-based modeling: The proposed architecture aggregates node information through learned adjacency matrices rather than relying only on a predefined graph.Figure 2 depicts this learned connectivity within the graph-convolutional architecture.
  • Graph-based modeling: A kinematic-tree graph is a straightforward option for motion prediction, but it does not automatically determine broader joint neighborhoods.The related-work discussion contrasts fixed graph construction with self-attention-based neighborhood selection.

3. Our Approach

The approach predicts future motion in DCT trajectory space and uses a graph convolutional network with learned joint connectivity to model temporal and spatial structure. It supports angle- and coordinate-based poses and achieves state-of-the-art results on benchmark motion-prediction tasks.

  • 3.1. DCT-based Temporal Encoding: DCT trajectory encoding represents each joint’s temporal variation with basis coefficients and can discard high frequencies to compactly capture smooth human motion.Using all coefficients is lossless, whereas truncating high frequencies can help prevent jittery motion.
  • 3.1. DCT-based Temporal Encoding: The model reformulates motion prediction as learning future DCT coefficients from observed pose trajectories rather than directly mapping observed poses to future poses.The observed sequence is extended by replicating its last pose, and the network learns residuals between the resulting and true future DCT representations.
  • 3.2. Graph Convolutional Layer: The spatial model treats the body as a fully connected graph and learns weighted adjacency matrices, allowing dependencies beyond a fixed kinematic tree or convolutional filter size.Each graph convolutional layer computes H^(p+1) = σ(A^(p)H^(p)W^(p)), with trainable adjacency and weight matrices.
  • 3.3. Network Architecture: The resulting residual graph convolutional network combines temporal encoding and spatial modeling through stacked graph-convolutional layers and residual blocks.The experimental network contains 12 residual blocks, with additional graph-convolutional layers for temporal encoding and decoding residual DCT coefficients.

4.1. Datasets

The evaluation uses Human3.6M, CMU-Mocap, and 3DPW under standard short- and long-term prediction settings, reporting both angle and 3D errors. Across these benchmarks, the approach generally outperforms baselines, with direct 3D training often yielding the strongest results.

  • Human3.6M contains 15 actions performed by seven actors and uses a 32-joint skeleton; evaluation follows prior preprocessing and tests subject 5.
  • The evaluation reports Euclidean joint-angle error and MPJPE in millimeters for 3D joint-position predictions.
  • Short-term prediction uses 10 observed frames to forecast 10 future frames, while long-term prediction forecasts 25 future frames.These correspond to 400 milliseconds for short-term prediction and 1 second for long-term prediction.
  • On H3.6M short-term angle prediction, the approach outperforms all baselines on average, with qualitative predictions closer to ground truth across three actions.
  • For H3.6M long-term prediction, the approach achieves the best 3D results by a large margin, particularly when trained directly on 3D coordinates.In angle space, it is best at 500ms but has higher error than at 1000ms, whose comparison is complicated by angle-representation ambiguity.
  • Across CMU-Mocap and 3DPW, the method consistently outperforms baselines for short- and long-term prediction, with the best results obtained using direct 3D representations.
  • Ablation study: Ablations show that padding and residual connections improve accuracy, DCT is especially beneficial in 3D space, and learned graph connectivity outperforms predefined connectivity.

5. Conclusion

The paper presents a feed-forward motion-prediction approach that combines DCT-based temporal encoding with GCNs having learnable connectivity. It reports state-of-the-art benchmark performance and concludes that 3D-space evaluation and training are more reliable than angle-space alternatives.

  • The model jointly encodes temporal information with the DCT and spatial structure with GCNs using learnable connectivity.
  • The approach achieves state-of-the-art results on standard human motion-prediction benchmarks.
  • Angle-space evaluation is unreliable because different angle sets can correspond to the same 3D pose.
  • Models trained on 3D coordinates consistently outperform models trained on angles in the reported experiments.

1. Datasets

Experiments use Human3.6M, CMU-Mocap, and 3DPW with angle-based and 3D-coordinate pose representations.

  • The experiments provide additional dataset details before evaluating motion prediction across these standard benchmarks.
  • Human3.6M uses 48-dimensional exponential-map angle vectors and 66-dimensional 3D-coordinate vectors after removing specified global and stationary components.
  • CMU-Mocap uses 64-dimensional exponential-map angle vectors and 75-dimensional 3D-coordinate vectors, without a validation set because training data are limited.
  • 3DPW uses 24 joints, yielding 72-dimensional angle vectors and 69-dimensional 3D-coordinate vectors after removing global translation.

2. Visualizing the Results on H3.6M in Video

Video results compare the proposed approach with state-of-the-art methods on periodic and aperiodic Human3.6M actions, including cases where angle-space and 3D-space errors differ.

  • The approach shows better performance than state-of-the-art methods on periodic walking and aperiodic eating and direction actions.
  • Figure 1 compares ground truth, two prior methods, and the proposed results for CMU-Mocap basketball motion, highlighting improved leg and arm predictions.
  • Additional visualizations examine cases with large angle-representation errors but small 3D-space errors.These cases support the existence of ambiguities in angle space for human motion prediction.

3. Visualizing the Results on CMU-Mocap

Qualitative CMU-Mocap visualizations cover basketball, basketball signal, and directing-traffic actions, with the proposed approach outperforming state-of-the-art methods.

  • The proposed approach outperforms state-of-the-art methods on qualitative 3D pose predictions for basketball, basketball signal, and directing-traffic actions.

4. Number of DCT Coefficients

DCT coefficient experiments test temporal compression for predicting 25 future frames from 10 observed frames, showing that smooth 3D trajectories need fewer coefficients than discontinuous angle trajectories.

  • 4.1. Using Fewer Coefficients: Figures 2–4 and the coefficient analysis provide visual and trajectory-level comparisons of prediction behavior and reconstruction quality.
  • 4.1. Using Fewer Coefficients: 35 DCT coefficients reconstruct the 35-frame trajectory losslessly, while 10 coefficients reconstruct it with very low error because the joint trajectory is smooth in 3D space.
  • 4.1. Using Fewer Coefficients: The experiment observes 10 frames and predicts 25 future frames using DCT coefficients of the full 35-frame sequence.Inputs are formed by replicating the last observed frame 25 times before the DCT transform.
  • 4.1. Using Fewer Coefficients: In 3D coordinates, 10 DCT coefficients already produce very small prediction error at both 160ms and 560ms.
  • 4.1. Using Fewer Coefficients: Angle error generally decreases as the number of DCT coefficients increases, unlike the stronger compression observed for 3D-coordinate prediction.
  • 4.1. Using Fewer Coefficients: Large discontinuities in angle trajectories make reconstruction with fewer DCT coefficients lossy.
  • 4.2. Results on H3.6M: The model uses 15 coefficients for short-term and 30 for long-term 3D prediction, versus 20 and 35 for angle-based prediction.

5. Ablation Study Details

The ablation compares a fully connected network with graph-based motion prediction and examines how DCT coefficients affect temporal reconstruction and prediction errors. The fully connected alternative uses the same trajectory encoding process but does not learn a better representation than the graph network.

  • DCT Coefficients: Figures 6 and 7 compare average angle and 3D prediction errors across four actions at 160 ms and 560 ms using different numbers of DCT coefficients.
  • Fully-connected Network: The fully connected baseline flattens DCT coefficients of the padded past sequence and learns residuals between past and future temporal encodings.It uses two fully connected layers with residual connections for encoding and decoding the DCT-coefficient residual.
  • Fully-connected Network: The fully connected network replaces graph convolutional layers with two fully connected layers while retaining DCT-based temporal encoding.
  • DCT Coefficients: Using fewer DCT coefficients can produce nonsmooth reconstructed trajectories with large jumps, whereas all 35 coefficients reconstruct the trajectory losslessly.
  • Fully-connected Network: The fully connected structure cannot learn a better representation than the graph network in the reported ablation study.

6. Mean Pose Problem

The paper addresses the mean-pose behavior that can arise in RNN-based motion prediction when past information vanishes during long propagation paths. By encoding the whole trajectory directly rather than relying on an RNN, the proposed method retains motion farther from the mean pose over long predictions.

  • Mean Pose Problem: RNN-based temporal encoding can produce the mean pose when past information vanishes during long propagation paths.
  • Mean Pose Problem: Directly encoding the whole trajectory avoids reliance on RNNs and inherently prevents loss of past information.
  • Mean Pose Problem: In a four-second Phoning prediction, the RNN baseline converges to the mean pose while the proposed method's legs continue moving.
Loading 1908.05436v3…