PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Use Diffusion Probabilistic Models for 3D Motion Prediction?

Hyemin Ahn, Esteve Valls Mascaro, Dongheui Lee

arXiv:2302.14503v1cs.CVcs.AI

TL;DR

The paper asks whether diffusion probabilistic models can predict future 3D human motion from observed motion. It evaluates diffusion models on deterministic and stochastic prediction, finding a balance between likelihood and diversity while noting limits against existing state-of-the-art methods.

  • Problem

    The paper investigates whether diffusion probabilistic models can predict future 3D human motion from observed poses, a task evaluated on Human3.6M and HumanEva-I.

    Source: §I · p. 1

  • Method

    The authors use Transformer-based diffusion denoisers for motion prediction and obtain deterministic predictions by excluding randomness during denoising.

    Source: C. Transformer-based Motion Denoiser · p. 4, §V · p. 6

  • Results

    Diffusion is not superior to the state-of-the-art overall, but performs better than other Transformer-based approaches for long-term 1000ms deterministic prediction and balances stochastic likelihood and diversity.

    Source: §V · p. 6

  • Takeaways & Limitations

    A single trained diffusion model can serve deterministic and stochastic prediction, with stochastic samples balancing diversity against likelihood.

    Source: §V · p. 6

    A reverse process with 1000 denoising steps can make sampling slow, limiting suitability when prediction latency matters.

    Source: B. Diffusion Probabilistic Models · p. 3

Abstract

from arXiv · show

After many researchers observed fruitfulness from the recent diffusion probabilistic model, its effectiveness in image generation is actively studied these days. In this paper, our objective is to evaluate the potential of diffusion probabilistic models for 3D human motion-related tasks. To this end, this paper presents a study of employing diffusion probabilistic models to predict future 3D human motion(s) from the previously observed motion. Based on the Human 3.6M and HumanEva-I datasets, our results show that diffusion probabilistic models are competitive for both single (deterministic) and multiple (stochastic) 3D motion prediction tasks, after finishing a single training process. In addition, we find out that diffusion probabilistic models can offer an attractive compromise, since they can strike the right balance between the likelihood and diversity of the predicted future motions. Our code is publicly available on the project website: https://sites.google.com/view/diffusion-motion-prediction.

I. INTRODUCTION

The introduction frames 3D motion prediction as forecasting pose sequences from observed poses and asks whether diffusion can serve both deterministic and stochastic tasks. It reports that diffusion balances likelihood and diversity after one training procedure, while not fully replacing existing state-of-the-art methods.

I. INTRODUCTION

  • The task is to predict future 3D pose-skeleton sequences from observed poses, evaluated on Human3.6M and HumanEva-I.Source: §I · p. 1
  • Deterministic prediction targets one likely future and is commonly evaluated by L2 distance, while stochastic prediction evaluates sample likelihood and diversity.Source: §I · p. 1
  • Diffusion models support deterministic and stochastic prediction after one training procedure, but do not fully replace existing state-of-the-art methods.The reported attraction is a balance between predicted-motion likelihood and diversity.Source: §I · p. 1, Figure 1 · p. 1
  • Compared with, diffusion samples are more likely to occur without diverging into out-of-context motion, an issue noted in.Source: §I · p. 2

A. 3D Motion Prediction

Prior 3D motion-prediction work includes deterministic models and generative approaches for sampling diverse futures, with likelihood often based on the closest sample to ground truth. The section also introduces diffusion models, their sampling-speed drawback, and their previously unexplored use for 3D motion prediction.

A. 3D Motion Prediction

  • Deterministic methods include handcrafted spatiotemporal graphs, DCT-GCN, and Transformer models that process spatial and temporal information.DCT-GCN's best result may require separate short- and long-term training.Source: A. 3D Motion Prediction · p. 2
  • Prior stochastic approaches use VAEs or GANs, while later work adds a post-hoc model to increase diversity among samples.Source: A. 3D Motion Prediction · p. 2
  • Minimum-distance likelihood can label a set plausible when only its closest sample matches ground truth, even if other samples are contextually implausible.The paper therefore also evaluates mean and standard deviation of sample-to-ground-truth distances.Source: A. 3D Motion Prediction · p. 2

B. Diffusion Probabilistic Models

  • Diffusion gradually adds noise in a forward process and learns to denoise in reverse, but many reverse steps can make sampling slow.The section presents 3D motion prediction as a diffusion application not previously attempted to the authors' knowledge.Source: B. Diffusion Probabilistic Models · pp. 2–3

III. METHOD

Diffusion probabilistic models learn a data distribution through gradual noise injection and reverse denoising. The paper extends this framework with conditioning for motion prediction.

A. Preliminaries

  • The forward process gradually adds Gaussian noise to a data point over K steps, ending at a standard normal distribution.The transitions follow a Markov chain, and the noise level is set by β_k.Source: A. Preliminaries · p. 3
  • The reverse process starts from standard normal noise and iteratively denoises it to recover a data sample.DDPM parameterizes the reverse mean and samples each reverse transition.Source: A. Preliminaries · p. 3
  • Training uses a simplified denoising loss, with the diffusion step k randomly sampled for each training process.For conditional diffusion, the observed sequence is added as conditioning information to the denoising model and loss.Source: A. Preliminaries · p. 3

B. Problem Formulation

The task is to predict future 3D pose sequences from observed poses. The method uses conditional diffusion and supports both stochastic and deterministic prediction.

B. Problem Formulation

  • A motion prediction input is T observed 3D poses, and the output is a sequence of L future poses.Each pose is a D-dimensional vector, where D = 3n and n is the number of joints.Source: B. Problem Formulation · p. 3
  • The method uses CSDI to generate future poses conditioned on the observed sequence, training a denoiser to predict the noise in the noised future poses.The denoiser is trained by minimizing a noise-prediction loss.Source: B. Problem Formulation · p. 3
  • The denoiser processes spatial and temporal information either in series or in parallel, and its positional encoding includes a learnable diffusion-step vector.The figure identifies pose-parameter dimension d, time t, and Transformer layers.Source: Figure 2 · p. 4
  • For deterministic prediction, the method sets the initial future-noise sample and reverse-process noise to zero, removing randomness from sampling.The reverse process otherwise repeats for K steps.Source: B. Problem Formulation · p. 4

C. Transformer-based Motion Denoiser

The paper compares Transformer denoisers that process spatial and temporal information in series or in parallel. It reports that Transformer-based denoisers processing both types of information are most effective.

C. Transformer-based Motion Denoiser

  • Transformer denoisers that process both spatial and temporal information were found empirically to be most effective.The authors report that RNNs and Transformers are candidate architectures for processing time-series data.Source: C. Transformer-based Motion Denoiser · p. 4
  • The series design applies spatial and temporal Transformer layers in sequence, with positional encodings for pose-parameter and time dimensions.A learnable positional encoding represents the diffusion step.Source: C. Transformer-based Motion Denoiser · p. 4
  • The parallel design processes spatial and temporal information in separate Transformer layers, then combines their outputs with a 1 × 1 convolution.The resulting output is used to estimate the denoising prediction.Source: C. Transformer-based Motion Denoiser · p. 4
  • The authors did not use an encoder-decoder structure because the Transformer- and RNN-based variants they tried were not effective.Their encoder-decoder setup would encode observed poses and decode the denoising prediction from those features and the noisy future poses.Source: C. Transformer-based Motion Denoiser · p. 4

D. Implementation Details

The denoiser uses eight attention heads and 512-dimensional query, key, and value vectors, and is trained for 50,000 iterations.

D. Implementation Details

  • Each denoiser uses eight attention heads with 512-dimensional query, key, and value vectors, and each Transformer layer is a single-layer encoder.Training uses batch size 512, Adam, and a learning rate of 0.0001 for 50,000 iterations.Source: D. Implementation Details · p. 4
  • The diffusion step ranges from 0 to 20, with linearly scheduled noise levels β_k from 0.001 to 0.333.Source: D. Implementation Details · p. 4

A. Dataset and Metric

The experiments use Human3.6M for deterministic prediction and Human3.6M plus HumanEva-I for stochastic prediction, with distinct observation and prediction horizons. Evaluation pairs Euler-angle MSE with likelihood, diversity, and ground-truth proximity metrics.

Dataset

  • Deterministic experiments use 50 observed and 25 predicted frames at 25 fps on Human3.6M; stochastic experiments use 25 observed and 100 predicted frames at 50 fps on Human3.6M and HumanEva-I.Stochastic predictions are evaluated using 50 samples.Source: A. Dataset and Metric · p. 5

Metrics

  • Euler-angle MSE evaluates deterministic prediction, while stochastic evaluation uses metrics for likelihood and diversity, including additional measures of sample proximity to ground truth.The additional metrics are aDE, sDE, aFDE, and sFDE.Source: A. Dataset and Metric · p. 5
  • APD measures pairwise prediction diversity; mDE, aDE, and sDE summarize full-motion distances to ground truth, while mFDE, aFDE, and sFDE summarize final-pose distances.The m metrics report minima, a metrics averages, and s metrics standard deviations.Source: A. Dataset and Metric · p. 5

B. Quantitative Results

The diffusion denoisers are not state of the art for deterministic prediction, but compare favorably with transformer models at long horizons and with VAEs on stochastic metrics. Their stochastic results also perform well on measures of sample proximity to ground truth, though a classifier-based evaluation was unavailable.

Deterministic Prediction

  • At 1000ms, the diffusion models outperform other transformer-based models for deterministic prediction, although DCT-GCN has the best overall performance.Among the diffusion approaches, the series denoiser outperforms the parallel denoiser.Source: B. Quantitative Results · p. 5, Table I · p. 5

Stochastic Prediction

  • Prior diversity-focused methods, perform best in APD and are generally better in mDE and mFDE, while the diffusion approaches generally perform better on aDE, sDE, aFDE, and sFDE.The parallel denoiser outperforms the series denoiser on the latter measures.Source: B. Quantitative Results · p. 5
  • The authors could not use the classifier-based quality and context metrics from because its pretrained classifiers were not openly released.Source: B. Quantitative Results · p. 5
  • The diffusion denoisers generally outperform VAEs [29] implemented by on diversity and likelihood, especially on HumanEva-I.Source: B. Quantitative Results · p. 5

C. Qualitative Results

The qualitative examples contrast deterministic and stochastic predictions for observed “smoking” and “walking” motions. Deterministic predictions resemble the ground truth, while stochastic samples vary and retain the observed action context.

Qualitative Results

V. CONCLUSION

The paper evaluates transformer-based diffusion models for deterministic and stochastic 3D motion prediction, with deterministic use possible after training by removing randomness from denoising. Results show a trade-off between diversity and likelihood, while sampling latency remains a concern.

  • Two transformer-based diffusion models process motion’s spatial and temporal information in series or parallel, and can predict deterministically after training by removing denoising randomness.Source: §V · p. 6
  • Deterministic results were not state-of-the-art overall, but 1000ms prediction outperformed other transformer-based approaches.Source: §V · p. 6
  • The diffusion models balance diversity and likelihood, assessed with additional metrics for the mean and standard deviation of distances to plausible motions.Source: §V · p. 6
  • Sampling uses K = 20 denoising processes, which may cause high latency and motivates future work on efficient sampling for real-time applications.Source: §V · p. 6
Loading 2302.14503v1…