PaperlayerView exact revision on arXiv

Source-linked AI summary

A Spatio-temporal Transformer for 3D Human Motion Prediction

Emre Aksan, Manuel Kaufmann, Peng Cao, Otmar Hilliges

arXiv:2004.08692v3cs.CV

TL;DR

3D human motion prediction must learn plausible articulation and dynamics while avoiding long-horizon degradation. The paper introduces an autoregressive ST-Transformer with decoupled temporal and spatial self-attention, reporting accurate short-term predictions and plausible long-term motion, while aperiodic horizons remain limited to a few seconds.

  • Problem

    The problem is learning plausible future 3D human poses from past motion while capturing latent articulation priors and underlying dynamics.

  • Method

    The ST-Transformer explicitly learns spatio-temporal representations through decoupled temporal and spatial self-attention over joints and time.

  • Results

    The model outperforms state-of-the-art models on short-term horizons and produces convincing long-term predictions up to 20 seconds for periodic motions.

  • Takeaways & Limitations

    Direct access to past information lets the model capture temporal and structural dependencies explicitly while mitigating long-term dependency issues in autoregressive architectures.

  • Takeaways & Limitations

    For aperiodic motion types, the prediction horizon is limited to a few seconds because training uses relatively short 2-second windows.

Abstract

from arXiv · show

We propose a novel Transformer-based architecture for the task of generative modelling of 3D human motion. Previous work commonly relies on RNN-based models considering shorter forecast horizons reaching a stationary and often implausible state quickly. Recent studies show that implicit temporal representations in the frequency domain are also effective in making predictions for a predetermined horizon. Our focus lies on learning spatio-temporal representations autoregressively and hence generation of plausible future developments over both short and long term. The proposed model learns high dimensional embeddings for skeletal joints and how to compose a temporally coherent pose via a decoupled temporal and spatial self-attention mechanism. Our dual attention concept allows the model to access current and past information directly and to capture both the structural and the temporal dependencies explicitly. We show empirically that this effectively learns the underlying motion dynamics and reduces error accumulation over time observed in auto-regressive models. Our model is able to make accurate short-term predictions and generate plausible motion sequences over long horizons. We make our code publicly available at https://github.com/eth-ait/motion-transformer.

1. Introduction

3D human motion prediction requires learning both natural pose articulation and underlying dynamics from a seed sequence. The ST-Transformer explicitly decouples temporal and spatial attention to model these dependencies and generate plausible short- and long-term motion.

  • Problem: 3D human motion modelling predicts future poses from a past horizon while learning natural articulation and motion dynamics.The task is framed as generative modelling conditioned on a seed sequence.
  • Problem: Autoregressive models can accumulate error over time and collapse to non-plausible poses because predictions are factorized into step-wise conditionals.This issue is associated with exposure bias caused by discrepancies between data and model distributions.
  • Prior approaches: DCT-based approaches encode joint trajectories as frequency coefficients and produce predictions for a predetermined horizon.This implicit temporal representation mitigates some autoregressive failure cases but constrains output length.
  • Approach: The ST-Transformer learns spatio-temporal representations explicitly without propagating a fixed-size hidden state or using fixed DCT encodings.Its representation decouples temporal and spatial dimensions for inherently spatio-temporal motion prediction.
  • Approach: Temporal attention follows each joint across past steps, while spatial attention relates other joints at the same time step.The next pose summarizes current joint information and previous time steps as a weighted combination.
  • Results: The ST-Transformer outperforms state-of-the-art models on short-term horizons and produces convincing long-term predictions up to 20 seconds for periodic motions.The dual self-attention accesses past information directly and is reported to mitigate error accumulation.

2. Related Work

Prior 3D motion models use recurrent, convolutional, or frequency-domain representations, each with structural, temporal, horizon, or long-term dependency constraints. The ST-Transformer interleaves explicit temporal and spatial attention to support autoregressive generation of arbitrarily long sequences.

  • Recurrent and convolutional models: Autoregressive RNN and CNN sequence-to-sequence models use training strategies such as prediction exposure, noise, or adversarial losses to address exposure bias.These methods aim to reduce discrepancies between training inputs and model-generated inputs.
  • Recurrent models: RNNs dominate 3D motion modelling, but fixed-size hidden states make maintaining long-term dependencies difficult.Earlier approaches also commonly model vectorized poses without explicitly representing structural dependencies.
  • Implicit temporal models: DCT-based models represent joint trajectories with frequency coefficients and generate complete outputs from temporal windows.Their predetermined output length limits full generative modelling of arbitrarily long sequences.
  • ST-Transformer: The ST-Transformer learns spatio-temporal representations directly from data and models temporal dependencies autoregressively.Unlike DCT-based pipelines, it does not require windowed inputs or fixed temporal encodings.
  • ST-Transformer: Its computation blocks interleave temporal and spatial aggregation, passing combined information between successive layers.This differs from approaches that model temporal information first and capture spatial structure afterward.
  • Conclusion: Self-attention is presented as effective for learning motion dynamics and designing a versatile mechanism that is effective and easy to train.The paper positions this approach alongside prior uses of decomposed attention across dimensions in other domains.

3. Method

The ST-Transformer predicts future poses by combining explicit temporal dependencies within each joint with spatial dependencies across joints at the same timestep. Its autoregressive architecture stacks decoupled attention blocks, projects embeddings back to joint space, and trains with per-joint rotation-matrix loss.

  • Architecture: The model uses a decoupled spatio-temporal attention mechanism and a fully autoregressive design.Temporal and spatial attention are computed separately while predictions are generated recursively.
  • Embedding and processing: Input joints are projected into D-dimensional embeddings, augmented with sinusoidal positional encodings, and processed by L = 8 stacked attention layers.Dropout is applied before the embeddings enter the stacked attention blocks.
  • Temporal and spatial attention: Each joint’s temporal embedding attends to its past instances, while spatial attention aggregates information from other joints at the current timestep.The two blocks explicitly model temporal dependencies within joints and inter-joint structural dependencies.
  • Aggregation: Temporal and spatial summaries are summed, passed through a pointwise feedforward network, and used to iteratively refine joint embeddings.Each attention layer applies dropout and layer normalization after the feedforward network.
  • Prediction and inference: The final embeddings are projected into joint space, combined with a residual connection from the previous pose, and optimized using per-joint ℓ2 distance on rotation matrices.At test time, the model predicts autoregressively using a sliding temporal window equal to the seed-sequence length.

4. Experiments

Experiments evaluate the ST-Transformer on AMASS and H3.6M using short-term prediction metrics, long-term power-spectrum metrics, qualitative sequences, attention visualizations, and ablations. The model achieves strong short-term performance, plausible long-term generations, and interpretable spatio-temporal attention, while aperiodic motions remain limited to shorter horizons.

  • AMASS: The ST-Transformer achieves state-of-the-art performance across all short-term AMASS metrics.The evaluation compares vanilla Transformer, RNN-based, and DCT-based architectures; LTD-Attention remains competitive at 400 ms.
  • Long-Term: Over predictions up to 15 seconds, the ST-Transformer remains closer to real-data power-spectrum statistics than all baselines.The comparison uses distribution-based metrics because direct ground-truth comparison becomes problematic at longer horizons.
  • Long-Term: Higher PS Entropy and lower symmetric PS KLD indicate less collapse to static poses and prediction distributions closer to the data distribution.The entropy difference from baselines becomes more pronounced as the prediction horizon increases.
  • H3.6M: On H3.6M, the model is competitive and often achieves state-of-the-art, although the benchmark has high variance and marginal improvements.H3.6M is roughly 14 times smaller than AMASS and contains only a few test sequences.
  • AMASS: Compared with RNN-based autoregressive baselines, the model predicts more accurately short term and generates more plausible longer-term motion.The authors attribute this to mitigating error accumulation over time.
  • Qualitative Evaluation: In qualitative walking predictions, the ST-Transformer maintains motion for 20 seconds, whereas the vanilla Transformer collapses around 15 seconds and LTD-Attention before 10 seconds.RNN-SPL loses variation within 5 seconds.
  • Limitations: The model performs well on periodic motions for long horizons, but aperiodic motions are limited to a few seconds because training uses short 2-second windows.The reported horizon still exceeds previously reported horizons significantly.
  • Attention Analysis: Attention heads show diverse patterns, attend across long temporal horizons, and capture joint dependencies beyond the kinematic chain.Examples include attention from predicted knees and elbows to spine, collar, hip, and knee joints.

5. Conclusion

The ST-Transformer explicitly models intra- and inter-joint dependencies with decoupled temporal and spatial attention. It supports short- and long-term prediction, including sequences up to 20 seconds for periodic motions.

  • ST-Transformer performs generative modeling of 3D human motion.
  • Decoupled temporal and spatial attention explicitly learns intra- and inter-joint dependencies.
  • The model synthesizes motion up to 20 seconds for periodic motion types such as locomotion.

Supplementary Material for

The supplementary material provides implementation details and additional evidence for the ST-Transformer. It covers experiments, multi-head attention, attention visualizations, ablations, comparisons with naive 2D attention, and further analyses.

  • The supplement gives additional implementation details for the proposed ST-Transformer.
  • It provides further experimental evidence of the model’s performance.
  • The material includes multi-head attention details, attention visualizations, ablations, and comparisons with naive 2D attention.

A. Experimental Details

The experiments use specific optimization, representation, attention, and training settings across H3.6M and AMASS. Rotation matrices perform best, while the selected attention parameterization improves AMASS Euler error over alternatives.

  • Training protocol: Training uses early stopping on joint angle error, full seed-and-target sequences, and dataset-specific temporal windows.The H3.6M window is 75 frames; AMASS uses 120-frame sequences because of memory limitations.
  • Implementation: The feed-forward subnetwork follows each attention block, and the experiments use Adam with gradient clipping and dropout.The joint embedding size is D, with D = 128 for AMASS and H3.6M feed-forward projections returning to D.
  • Data representations: Rotation matrix representations achieved the best performance across the evaluated models.Predictions are projected to the nearest valid rotation matrix in SO(3) before metric computation.
  • Weight sharing: At 400ms on AMASS, the current attention design achieved 0.490 Euler error, versus 0.511 with separate and 0.504 with shared projection weights.
  • Attention mechanism: Temporal attention operates over time steps, while spatial attention operates over joints, using separate temporal projections and shared spatial key and value projections.
  • Attention visualization: Attention weights adapt at every step, retaining focus on relevant past frames and quasistatic joints such as the hips or spine.

D. Hyper-parameters

Hyper-parameter studies show that performance depends on the number of attention heads, temporal window length, and layers. More heads, longer sequences, and deeper models generally improve the learned representations within the tested settings.

  • Attention heads: 8 attention heads produced the best performance on AMASS, while 2 heads remained reasonable.The comparison indicates benefits from using more than one spatio-temporal configuration.
  • Temporal window: Performance improved as the temporal attention window increased in the tested seed-length settings.Attention masks also showed access to poses from the beginning of the sequence.
  • Number of layers: Three layers already reached reasonable AMASS performance, while additional layers tuned the learned representations better.The authors also interpret layers as message-passing steps that update available representations.

E. Additional Ablation on 2D Attention

The ablation compares decoupled spatio-temporal attention with computationally matched 2D-attention configurations. The decoupled mechanism consistently outperforms the strongest plain 2D-attention settings.

  • The ablation evaluates computational efficiency using stacked attention layers, context-frame window size, and batch size.
  • Computationally matched 2D-attention configurations perform best when balancing batch size, window size, and attention layers.Maximizing one hyper-parameter is usually detrimental, while trade-offs between the three produce the strongest 2D-attention results.
  • Our decoupled ST-attention outperforms the best 2D-attention configurations under comparable computational-efficiency settings.The comparison includes the best trade-off configurations and the best ST-attention configuration.

F. Evaluation of LTD on AMASS

The LTD baseline is evaluated on AMASS using adapted data handling and settings for its 400 ms input-output protocol. The section also reports the ablation context for the 2D-attention comparison.

  • The reported LTD-10-10 AMASS results use the best results obtained after hyper-parameter tuning.
  • LTD is evaluated on AMASS using 24 input frames and 24 predicted frames, corresponding to 400 ms at AMASS’s 60 Hz sampling rate.
  • The LTD configuration uses 35 DCT coefficients because they produced slightly better results than 48 coefficients.
  • Table 6 organizes attention configurations by stacked layers, context-frame window size, and batch size.

G. Evaluation of LTD-Attention on AMASS

LTD-Attention is evaluated on AMASS with longer seeds and predictions to reflect the dataset’s frame rate. The selected setup performs better than directly predicting 24 frames.

  • LTD-Attention receives 120 frames of context and predicts 60 frames, representing 2 seconds of input and 1 second of output.
  • Predicting 60 frames from 120-frame seeds performs better than predicting 24 frames directly on AMASS.
  • The AMASS LTD-Attention setup uses a learning rate of 0.005, batch size 128, and 45 DCT coefficients.

H. Power Spectrum Metrics

Power Spectrum metrics evaluate very long motion predictions when ground-truth sequences are shorter than the forecast horizon. They compare frequency-domain distributions rather than individual prediction-target pairs.

  • Power Spectrum metrics are used for predictions lasting up to 15–20 seconds, where pairwise MSE is often unavailable because ground-truth targets are shorter.
  • The adapted protocol converts angle representations to 3D joint positions and computes each coordinate’s spectrum as PS(x_f) = ||FFT(x_f)||^2.The feature sequence x_f represents one coordinate of one joint over time.
  • PS KLD compares the real-data power-spectrum distribution with each non-overlapping one-second prediction window.The reference distribution is computed from 20,000 randomly sampled one-second test sequences.
  • The symmetric PS KLD measures distributional discrepancy rather than pairwise alignment between ground-truth and predicted sequences.It combines KLD(G || P_t) and KLD(P_t || G).
Loading 2004.08692v3…