PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-Temporal Gating-Adjacency GCN for Human Motion Prediction

Chongyang Zhong, Lei Hu, Zihao Zhang, Yongjing Ye, Shihong Xia

arXiv:2203.01474v3cs.CV

TL;DR

Human motion prediction must represent spatio-temporal dependencies that vary across actions, while decoupled modeling and fixed adjacency leave these relationships difficult to capture. GAGCN blends candidate adjacency matrices with gating and balances and fuses spatial and temporal features. The authors report state-of-the-art performance on three benchmarks for short- and long-term prediction.

  • Problem

    Decoupled spatio-temporal modeling misses cross-dependencies, and stable adjacency can generalize poorly as joint and frame relationships vary across actions.

    Source: §3.2 · p. 4

  • Method

    GAGCN uses input-dependent gating to blend candidate adjacency matrices, balances spatial and temporal modeling, and fuses their features to learn cross-dependencies.

    Source: §1 · p. 2

  • Results

    Experiments on Human3.6M, AMASS [23], and 3DPW show state-of-the-art performance for short- and long-term motion prediction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that GAGCN can extract cross-dependencies in multi-action motion and achieve state-of-the-art results on widely used benchmarks.

    Source: §5 · p. 8

    The spatio-temporal modeling weights are adjusted manually; the authors identify automatic balancing as future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Predicting future motion based on historical motion sequence is a fundamental problem in computer vision, and it has wide applications in autonomous driving and robotics. Some recent works have shown that Graph Convolutional Networks(GCN) are instrumental in modeling the relationship between different joints. However, considering the variants and diverse action types in human motion data, the cross-dependency of the spatio-temporal relationships will be difficult to depict due to the decoupled modeling strategy, which may also exacerbate the problem of insufficient generalization. Therefore, we propose the Spatio-Temporal Gating-Adjacency GCN(GAGCN) to learn the complex spatio-temporal dependencies over diverse action types. Specifically, we adopt gating networks to enhance the generalization of GCN via the trainable adaptive adjacency matrix obtained by blending the candidate spatio-temporal adjacency matrices. Moreover, GAGCN addresses the cross-dependency of space and time by balancing the weights of spatio-temporal modeling and fusing the decoupled spatio-temporal features. Extensive experiments on Human 3.6M, AMASS, and 3DPW demonstrate that GAGCN achieves state-of-the-art performance in both short-term and long-term predictions.

1. Introduction

Human motion prediction must handle long-term error accumulation and complex spatial-temporal dependencies that vary across actions. GAGCN combines adaptive adjacency with balanced and fused spatio-temporal modeling, and is reported to outperform state-of-the-art methods on short- and long-term prediction.

  • Earlier RNN predictors face error accumulation on long sequences, while modeling complex spatio-temporal relationships remains difficult.Convolution-based sequence-to-sequence approaches reduce long-term error to some extent, but the problem remains [3] [17].Source: §1 · p. 1
  • Changing inter-joint and inter-frame relationships across motion variants and action types make stable adjacency matrices poorly generalize across multi-action motions.Source: §1 · p. 1
  • Directly concatenating decoupled spatial and temporal features does not fully capture their cross-dependency.Source: §1 · p. 2
  • GAGCN uses input-dependent gating coefficients to blend trainable candidate adjacency matrices, adapting joint and frame relationships across motions.Source: §1 · p. 2
  • GAGCN outperforms state-of-the-art methods on short- and long-term motion prediction across Human3.6M, AMASS [23], and 3DPW.Source: §1 · p. 2
  • The authors describe gating-based adaptive adjacency as a way to enhance GCN generalization across multi-action motions.Source: §1 · p. 2
  • GAGCN balances spatial and temporal modeling and fuses their features to learn expressive embeddings of spatio-temporal cross-dependencies.Source: §1 · p. 2

2. Related Works

Prior motion-prediction work includes recurrent, convolutional, attention-based, and graph-convolutional approaches, with challenges in long-term prediction and modeling cross-dependencies. GAGCN builds on gating and adaptive adjacency to address changing motion relationships and fuse spatial and temporal features.

  • RNN methods model temporal correlations but can accumulate error over long predictions, while sequence-to-sequence convolutional and attention-based approaches model whole sequences or joint/frame dependencies.Source: §2 · p. 2
  • Prior work often decouples spatial and temporal modeling and concatenates the features, although spatial dependencies can couple with the global temporal trajectory.Source: §2 · p. 2
  • Earlier spatio-temporal methods include SRNN, which uses RNNs on joint and edge nodes, and STGCN, which combines spatial and temporal connections in one adjacency matrix but uses a constant matrix.Source: §2 · p. 2
  • The network overview places GAGCN as the historical-motion encoder and TCN as the decoder.Source: Figure 2 · p. 3
  • GAGCN adjusts the number of candidate adjacency matrices to balance spatial and temporal modeling, then fuses their features with a Kronecker product.Source: §2 · p. 3
  • Inspired by mixture-of-experts gating, GAGCN blends candidate adjacency matrices according to input motion to adapt to diverse motion variants and action types.Source: §2 · p. 3

3. Our Method

GAGCN uses input-dependent gating to blend candidate spatial and temporal adjacency matrices, then balances and fuses their features to model cross-dependencies. Its encoder feeds a TCN decoder, with training losses matched to the pose representation.

3. Our Method

  • The encoder uses six GAGCN layers to extract spatio-temporal features, which a TCN decoder uses to predict the future sequence.The overall model uses an encoder-decoder structure.Source: §3 · p. 3, §3.2 · p. 5

3.1. Review of GCN

  • A trainable adjacency matrix lets a GCN model skeletal connections and implicit dependencies between joints.The layer applies the adjacency matrix to input features before a learned transformation.Source: §3.1 · p. 4

3.2. Spatio-Temporal Gating-Adjacency GCN

  • GAGCN gates candidate adjacency matrices to form adaptive spatial and temporal relationships suited to motion variations and action types.The gating network uses preceding-layer features to produce blending coefficients for the candidate matrices.Source: §3.2 · p. 4
  • Adjusting the candidate-matrix counts balances spatial and temporal modeling, while a Kronecker product fuses their features to capture cross-dependencies.The spatial matrix represents joint-to-joint dependencies, and the temporal matrix represents frame-to-frame dependencies in the historical sequence.Source: §3.2 · pp. 4–5

3.3. Training

  • Training is end-to-end and supervised, using MPJPE loss for 3D joint coordinates and MAE loss for angle-based representations.The losses aggregate prediction errors across joints and future frames.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate GAGCN on Human 3.6M, AMASS, and 3DPW, with benchmark comparisons, component ablations, and qualitative visualizations. The reported results show advantages across prediction horizons and dataset or action-type generalization tests.

4.1. Datasets and Baselines

  • The evaluation uses Human 3.6M, AMASS, and 3DPW, comparing MPJPE and MAE angle errors against selected baselines under specified training and testing splits.For 3DPW, models trained on AMASS are tested on the 3DPW dataset.Source: §4.1 · p. 5, §4.2 · p. 5

4.2. Comparisons with the State-of-the-art Methods

  • On Human 3.6M, GAGCN outperforms all baselines on average across time horizons, with larger gains on difficult actions and advantages at 1000ms.The comparison covers MPJPE for short- and long-term prediction across 14 action types.Source: Table 1 · p. 6, §4.2 · p. 6
  • On Human 3.6M, GAGCN also achieves state-of-the-art results using angle-based representation, measured by average MAE angle error.The comparison notes that Res-GRU has no long-term prediction results.Source: §4.2 · p. 6, Table 2 · p. 7
  • On AMASS-BMLrub and 3DPW, GAGCN outperforms the compared methods, supporting its generalization across datasets.The AMASS model is tested on BMLrub and then evaluated on 3DPW.Source: §4.2 · p. 6, Table 3 · p. 7, Table 4 · p. 7

4.3. Ablation Study

  • In the unseen-action test, GAGCN improves results for Walking Together over the stable-adjacency GCN, and its results are close to the seen-action setting.The ablation compares GAGCN with S&TGCN under otherwise identical experimental settings.Source: §4.3 · p. 7, Table 5 · p. 7
  • The balancing-block ablations indicate that gating both spatially and temporally helps model dependencies, while excessive candidate matrices can increase complexity and cause under-fitting.The study also reports that the relative weight of spatial and temporal modeling affects prediction accuracy.Source: §4.3 · p. 7, Table 6 · p. 7

4.4. Qualitative Evaluation

  • Predictions closely match ground truth across horizons for periodic actions and remain close in the long term for non-periodic actions, with some limb errors.The examples include Walking, Walking Together, Discussions, and Posing.Source: §4.4 · pp. 7–8, Figure 3 · p. 8
  • Average spatial blending coefficients differ across action types, while similar motions such as Walking and Walking Together have similarly distributed coefficients.The visualized averages are computed from 16 sequences per action type; temporal coefficient visualizations are in the supplementary material.Source: §4.4 · p. 8, Figure 4 · p. 8

5. Conclusion and Future Work

GAGCN combines adaptive adjacency, balanced spatio-temporal modeling, and feature fusion to capture cross-dependencies and achieve state-of-the-art results on several benchmark datasets. Future work will examine automatic balancing and more efficient generalization strategies.

5. Conclusion and Future Work

  • GAGCN achieves state-of-the-art results on several widely used benchmark datasets for multi-action motion prediction.Source: §5 · p. 8
  • Gating blends candidate adjacency matrices into an adaptive matrix, while adjusting their number balances spatio-temporal modeling and feature fusion captures cross-dependencies.Source: §5 · p. 8
  • Future work will study automatic balancing of spatio-temporal modeling weights and more efficient approaches to improve GCN generalization.Source: §5 · p. 8

Related papers

Loading 2203.01474v3…