PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanMAC: Masked Motion Completion for Human Motion Prediction

Ling-Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia, Tongliang Liu

arXiv:2302.03665v4cs.CVcs.AI

TL;DR

Human motion prediction seeks future poses that are continuous, diverse, and realistic, while prior encoding-decoding methods face burdensome losses and training and struggle with category switches. HumanMAC treats prediction as masked completion, using diffusion and observation-conditioned denoising. The paper reports state-of-the-art quantitative and qualitative comparisons, while noting that its 100-step DDIM sampling limits real-time use.

  • Problem

    Human motion prediction must generate continuous, diverse, realistic future poses, while prior encoding-decoding methods face complicated loss constraints, multi-stage training, and difficulty switching motion categories.

    Source: §1 · pp. 1–2

  • Method

    HumanMAC trains a motion diffusion model and uses masked completion during inference to condition denoising-based predictions on observed motions.

    Source: §1 · p. 2

  • Results

    HumanMAC achieves state-of-the-art performance in both quantitative and qualitative comparisons.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The authors present masked completion as a new perspective for human motion prediction and a basis for follow-up research.

    Source: §7 · p. 9

    The 100-step DDIM sampling procedure limits HumanMAC as a real-time system.

    Source: §6 · p. 9

Abstract

from arXiv · show

Human motion prediction is a classical problem in computer vision and computer graphics, which has a wide range of practical applications. Previous effects achieve great empirical performance based on an encoding-decoding style. The methods of this style work by first encoding previous motions to latent representations and then decoding the latent representations into predicted motions. However, in practice, they are still unsatisfactory due to several issues, including complicated loss constraints, cumbersome training processes, and scarce switch of different categories of motions in prediction. In this paper, to address the above issues, we jump out of the foregoing style and propose a novel framework from a new perspective. Specifically, our framework works in a masked completion fashion. In the training stage, we learn a motion diffusion model that generates motions from random noise. In the inference stage, with a denoising procedure, we make motion prediction conditioning on observed motions to output more continuous and controllable predictions. The proposed framework enjoys promising algorithmic properties, which only needs one loss in optimization and is trained in an end-to-end manner. Additionally, it accomplishes the switch of different categories of motions effectively, which is significant in realistic tasks, e.g., the animation task. Comprehensive experiments on benchmarks confirm the superiority of the proposed framework. The project page is available at https://lhchen.top/Human-MAC.

1. Introduction

Human motion prediction requires future poses that are continuous, diverse, and realistic, while prior encoding-decoding methods face burdensome losses and training and struggle with motion-category switches. HumanMAC reframes prediction as masked completion and reports state-of-the-art quantitative and qualitative comparisons.

1. Introduction

  • Human motion prediction must produce future pose sequences that are continuous, diverse, and realistic despite the problem's practical complexity.Applications include autonomous driving and healthcare.Source: §1 · p. 1
  • Prior encoding-decoding methods rely on multiple loss constraints and multi-stage training, and struggle to switch motion categories rarely seen in observations.Balancing losses requires carefully designed hyperparameters, while multi-stage pipelines may require additional engineering tuning.Source: §1 · pp. 1–2
  • HumanMAC trains a motion diffusion model to generate integrated observed and predicted motions from noise, then uses DCT-Completion to condition denoising on observations.At each inference denoising step, it combines denoised and noisy observed-motion spectra using a mask.Source: Figure 1 · p. 1, §1 · p. 2
  • HumanMAC uses one loss and end-to-end training, and can produce category switches even when they are rare or absent from observed sequences.The authors attribute the switch ability to holistic modeling of integrated observed and predicted motions and continuity between them.Source: §1 · p. 2
  • The authors discuss three issues in prior paradigms and provide benchmark experiments, qualitative comparisons, and ablation studies.They describe the framework as a simple, strong baseline for future research.Source: §1 · p. 2

2. Related Work

Prior human motion prediction methods use encoding-decoding pipelines, multiple losses, and often multi-stage training. The paper introduces diffusion for motion prediction and positions its end-to-end approach against earlier diffusion work that predicts in latent space.

2.1. Human Motion Prediction

  • Prior human motion prediction methods use encoding-decoding, multiple loss constraints, and often multi-stage training, whereas the proposed model uses one loss and end-to-end training.The paper links multi-stage training to dependence on encoder and decoder pre-training quality.Source: §2.1 · p. 2

2.2. Denoising Diffusion Models

  • Denoising diffusion models learn a reverse process from random noise to the data distribution, and the paper applies this approach to human motion prediction.The passage describes generated results as diverse and high-quality.Source: §2.2 · pp. 2–3
  • Earlier diffusion work includes text-, audio-, and music-driven motion synthesis, while BeLFusion predicts motions via latent-space diffusion.The paper contrasts its end-to-end framework with BeLFusion's multi-stage training and dependence on encoder-decoder pre-training.Source: §2.2 · p. 3

3. Preliminaries

The problem formulation represents observed motion as H frames of joint coordinates and asks for the next F frames. The method uses a truncated DCT representation, retaining low-frequency components to reduce computation.

3.1. Problem Formulation

  • Human motion prediction takes H observed frames of joint coordinates and predicts the following F frames.Each frame contains the coordinates of J joints.Source: §3.1 · p. 3

3.2. Discrete Cosine Transform

  • The method projects the full motion sequence into DCT coefficients and reconstructs it with the inverse transform.It retains the first L rows of the DCT basis, discarding high-frequency components to reduce computational cost.Source: §3.2 · p. 3

4. Methodology

HumanMAC trains a motion diffusion model with a single end-to-end noise-prediction loss, then uses DCT-Completion to condition inference on observed motions. Its masked completion procedure combines observed and predicted motion to produce continuous predictions and support motion-category switches.

4.1. Model Training

  • HumanMAC trains a diffusion model on full-motion DCT spectra by predicting sampled noise, using the noise-prediction loss as its only end-to-end training objective.The model predicts noise at each diffusion timestep.Source: §4.1 · p. 3

4.2. DCT-Completion in Inference

  • DCT-Completion uses a trained diffusion model to condition predictions on observed motions without retraining, addressing the vanilla procedure's inability to use observations.The vanilla procedure generates motions through T denoising steps but cannot use observed motions, making predictions uncontrollable.Source: §4.2 · p. 3
  • At each reverse step, DCT-Completion combines the inverse-DCT noisy observation with the inverse-DCT denoised prediction using a temporal mask.The observation is padded with its final frame and projected into the DCT domain before noise is added; the prediction branch denoises the noisy prediction spectrum.Source: §4.2 · pp. 3–4
  • The resulting motion is continuous across observed, predicted, and target frames, and the model can switch motion categories, such as Sitting to Standing.Inference can include final target motions to obtain controllable predictions.Source: §4.2 · p. 4

5. Experiments

HumanMAC is evaluated on Human3.6M, HumanEva-I, CMU-MoCap, and AMASS, with comparisons covering prediction quality, motion diversity, controllability, and generalization. The reported results include strong benchmark comparisons and qualitative abilities, while ablations examine design choices and long-horizon prediction.

5.1. Experimental Setup

  • Experiments use the standard Human3.6M and HumanEva-I partitions, comparing against 11 named methods quantitatively and DLow and GSPS visually.HumanMAC uses a 1000-step diffusion model and 100-step DDIM sampling; the reported visual comparisons show 10 stochastic predictions per frame.Source: §5.1 · pp. 4–6, §5.2 · p. 6

5.2. Comparison with the State-of-the-Arts

  • Human3.6M results are state of the art on ADE and FDE; HumanEva-I results are state of the art on other metrics and competitive on MMFDE.The authors describe a trade-off between diversity and authenticity, and report that their predictions have fewer failure examples than baselines that appear more diverse.Source: Table 1 · p. 5, §5.2 · p. 6

5.3. Motion Switch Ability

  • With free-training DCT-Completion, HumanMAC smoothly switches observed motions to target categories, including rare or unseen motions and large-gap transfers such as Sitting to Standing.The authors also report natural transitions when body-facing directions differ substantially.Source: §5.3 · p. 6, Figure 4 · p. 7

5.4. Part-body Controllable Prediction

  • Masking supports control of any body part without retraining: fixing lower-body motion yields diverse upper-body predictions while keeping lower-body error limited.The examples include large-angle turns and standing-to-bending transitions; the authors contrast this with baselines that require body-part disentanglement and retraining.Source: §5.4 · p. 6, Figure 5 · p. 7

5.5. Ablation Study

  • Ablations select L=10 and 20 for Human3.6M and HumanEva-I, respectively, and noise-network depths of 8 and 4 layers; Cosine scheduling performs best.The main results use 1000 noising and 100 sampling steps; reducing both by a factor of 10 damages authenticity and produces jerky motions, while diversity and authenticity trade off.Source: Table 2 · p. 6, Table 3 · p. 6, §5.5 · pp. 7–8

5.6. Zero-shot Motion Prediction

  • Training on Human3.6M and predicting from AMASS observations produces reasonable, diverse zero-shot results, including vivid Kneeling motion unseen in Human3.6M.The authors present this as evidence of HumanMAC's generalization ability.Source: §5.6 · p. 8, Figure 6 · p. 8

5.7. Evaluation on Both CMU-MoCap and AMASS Datasets

  • HumanMAC achieves superior reported performance against VAE-based and diffusion-based models on both CMU-MoCap and AMASS.The comparison uses tuned baselines, and results are reported in Table 6.Source: §5.7 · p. 8, Table 6 · p. 9

5.8. Ablation Study on the DCT Design; 5.9. Long Time Series Prediction

  • DCT improves prediction accuracy and smoothness, while recurrent long-horizon predictions remain more diverse than baselines that become oversmoothed or nearly static.The authors attribute the DCT effect to explicitly modeling low- and high-frequency signals; the long-time prediction approach treats the final H-frame predictions as observations for further recurrent prediction.Source: §5.8 · p. 8, §5.9 · p. 8, Table 7 · p. 9

6. Limitation and Discussion

HumanMAC’s 100-step DDIM sampling limits it from being a real-time system, and the authors identify diversity evaluation as an unresolved issue.

6. Limitation and Discussion

  • 100 DDIM sampling steps limit HumanMAC from being a real-time system.The authors plan to explore methods such as DPM-solver++ to reduce sampling steps.Source: §6 · p. 9
  • The authors plan to test HumanMAC on larger-scale datasets and explore zero-shot transfer for human motion prediction.They also plan to further simplify the network architecture and completion algorithm.Source: §6 · p. 9
  • The authors describe prior diversity metrics as questionable because they can reward failure cases.They identify developing a more reasonable diversity metric as future work.Source: §6 · p. 9

7. Conclusion

The conclusion presents masked completion as a new framework for human motion prediction, while TransLinear is the noise-prediction network used in the method.

7. Conclusion

  • The authors report state-of-the-art quantitative and qualitative performance for their masked-completion framework.They present the framework as a new perspective on human motion prediction and suggest it may inspire follow-up research.Source: §7 · p. 9

A. Noise Prediction Network TransLinear

  • TransLinear maps the DCT-spectrum input and output through linear layers, with stacked blocks combining Transformer encoders and linear operations.Each block uses skip connections and FiLM modules modulated by the first K frames’ DCT spectrum and the diffusion-time embedding; the modulating motion is padded to full length.Source: A. Noise Prediction Network TransLinear · p. 13
  • Figure 9 depicts TransLinear as N blocks with skip connections followed by a linear layer, taking the diffusion-step DCT spectrum as input.Source: Figure 9 · p. 13

B. Implementation Details

The implementation section specifies the evaluation setup and training configuration, and reports visual comparisons and additional motion-switch cases.

B. Implementation Details

  • The authors make HumanMAC’s source code public to support reproducibility.Source: B. Implementation Details · p. 13
  • HumanMAC is evaluated on Human3.6M and HumanEva-I using established dataset partitions and prediction horizons of 2 seconds and 1 second, respectively.Human3.6M uses 25 observed frames to predict 100; HumanEva-I uses 15 to predict 60.Source: B. Implementation Details · p. 13
  • Training uses 1000 noising steps and 100 DDIM sampling steps, with a cosine variance scheduler and dataset-specific TransLinear configurations.The implementation also specifies 500 epochs, batch size 64, Adam, and dataset-specific DCT lengths and modulation ratios.Source: B. Implementation Details · p. 13
  • HumanMAC’s predicted-motion diversity is described as more reasonable than DLow and GSPS because the baselines show motions violating physical constraints.Figure 10 compares motion sequences and end poses; its red arrows and dashed boxes mark failure cases.Source: B. Implementation Details · p. 13, Figure 10 · p. 14

D. Motion Switch

  • Additional motion-switch examples include Sitting-to-Walking and turning, with upper- and lower-body changes described as natural across large distribution gaps.The authors report smooth switching even when transferring between motions with a large distribution gap.Source: D. Motion Switch · p. 15

E. Part-body Controllable Prediction Results

HumanMAC supports controllable predictions for arbitrary body parts, including individual limbs and the torso.

E. Part-body Controllable Prediction Results

  • HumanMAC supports controllable predictions for the right leg, left leg, right arm, left arm, and torso.The authors state that this capability can facilitate controllable automatic animation synthesis.Source: E. Part-body Controllable Prediction… · p. 15

F. Ablation Study on Network Architecture

The skip connection improves motion authenticity, while HumanMAC also supports motion switching and part-body control at inference time without dedicated modeling.

F. Ablation Study on Network Architecture

G. Comparison on Motion Switch and Part-body Controllable Ability

  • HumanMAC supports motion switching and part-body controllable prediction at inference without specific modeling.DLow and GSPS require specific training stages for part-body control; HumanMAC supports control of any body part.Source: G. Comparison on Motion Switch and… · p. 15

H. Supplementary Results of Zero-shot Motion Prediction

Supplementary results show zero-shot predictions of motions not seen in Human3.6M and provide visualizations of motion control. Engineering optimization parallelizes evaluation and substantially speeds it up.

H. Supplementary Results of Zero-shot Motion Prediction

I. Engineering Optimization for Evaluation

  • ∼1k× overall evaluation speedup is reported after parallelizing model inference and metric calculation; metric calculation alone achieves ∼6× speedup.The overall comparison is shown in Table 11; Table 10 presents the metric-calculation simulation.Source: §I · p. 15, Table 11 · p. 19, Table 10 · p. 19
  • The supplementary visualizations show DCT-Completion motion transfer and end poses for ten examples of part-controlled motions.The part-control examples cover left and right legs, left and right arms, and the torso.Source: Figure 11 · p. 16, Figure 12 · p. 17
  • Figure 14 compares engineering optimization with the unoptimized evaluation process and indicates a significant speed improvement.Source: Figure 14 · p. 19

Related papers

Loading 2302.03665v4…