PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation

Chengqun Yang, Liang Xu, Yanping Li, Fulong Liu, Jingnan Gao, Weili Zeng, Yichao Yan

arXiv:2608.23279v1cs.CV

TL;DR

Existing motion representations and generators provide limited information preservation or fine-grained body-part control. DeMoDiff jointly redesigns representation and generation with a joint-wise spatial-temporal VAE and masked autoregressive diffusion, and reports strong reconstruction, generation, and editing results on HumanML3D and KIT-ML.

  • Problem

    VQ representations lose information, holistic continuous representations hinder part-level flexibility, and existing diffusion-based generators have limited fine-grained control over individual body parts.

    Source: §I · p. 1

  • Method

    DeMoDiff uses a spatial-temporal VAE that encodes each body joint and a spatial-temporal autoregressive diffusion generator with masking and attention.

    Source: §I · pp. 1–2, B. Spatial-Temporal Autoregressive… · p. 2

  • Results

    The model reports improved reconstruction and generation metrics, superior performance in several motion tasks especially joint-level editing, and strong temporal and spatial editing capabilities.

    Source: §I · p. 2, D. Temporal and Spatial Editing · p. 6

  • Takeaways & Limitations

    DeMoDiff provides a unified framework for motion generation with temporal and spatial latent sampling and flexible temporal and spatial editing.

    Source: §V · p. 6

Abstract

from arXiv · show

Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/

I. INTRODUCTION

DeMoDiff addresses limitations in motion representation and generation by decoupling spatial and temporal structure. It combines a joint-wise spatial-temporal VAE with masked autoregressive diffusion to support generation and fine-grained editing.

  • Existing motion-generation methods rely on VQ-based or latent-based representations, while diffusion and autoregressive diffusion architectures still provide limited individual-body-part controllability.Source: §I · p. 1
  • DeMoDiff encodes each body joint with a lightweight spatial-temporal VAE instead of compressing whole-body motion into one latent space.The decoupled representation is intended to improve representation extraction and part-level controllability.Source: §I · p. 1
  • Its autoregressive diffusion generator combines spatial-temporal masking and attention to provide motion generation and controllable editability.Source: §I · pp. 1–2
  • Experiments on HumanML3D and KIT-ML evaluate DeMoDiff for text-driven motion generation and motion editing, with analysis and visualizations supporting its effectiveness.Source: §I · pp. 1–2
  • The method reports improved reconstruction and generation metrics from continuous 2D latent motion representation.Source: §I · p. 2
  • The framework supports temporal and spatial editing without editing-specific fine-tuning, including joint-level editing among its reported motion-related tasks.Source: §I · p. 2

II. RELATED WORK

Prior work represents motion with discrete VQ tokens or continuous latents, but these approaches face limitations in information preservation, long-horizon modeling, and fine-grained joint control.

  • Text-driven human motion generation must map natural-language descriptions to realistic 3D motions while handling fine-grained semantics, body-part control, and elaborate prompts.Source: §II · p. 2
  • VQ-based methods encode motion as discrete tokens for autoregressive synthesis, but sequential generation inherently impedes modeling long-horizon temporal relationships.Source: §II · p. 2
  • Bidirectional masked generation models predict masked tokens simultaneously, improving context modeling capability over sequential token generation.Source: §II · p. 2
  • VQ-based approaches nevertheless suffer from inevitable information loss during discrete motion representation.Source: §II · p. 2
  • Related representations also divide or split motion along body-part or joint dimensions to improve joint encoding and spatial relationships.Source: §II · p. 2

III. THE DEMODIFF MODEL

DeMoDiff represents motion as a two-dimensional temporal-joint structure and uses a spatial-temporal VAE to encode and decode continuous motion latents.

  • The model architecture comprises a spatial-temporal VAE for motion representation and a spatial-temporal autoregressive diffusion architecture for continuous human-motion generation.Source: §III · p. 2
  • The VAE organizes motion sequences across temporal and joint dimensions as a 2D map before processing them with convolutional networks.Source: A. Spatial-Temporal VAE · p. 2
  • A 2D convolutional encoder maps 2D motion features to latent-distribution parameters, which are decoded into motion sequences.Source: A. Spatial-Temporal VAE · p. 2
  • The VAE objective combines reconstruction quality with latent-space regularization through KL and joint-velocity losses.The KL and joint-velocity terms are weighted by hyperparameters.Source: A. Spatial-Temporal VAE · p. 2

B. Spatial-Temporal Autoregressive Diffusion.

Spatial-Temporal Autoregressive Diffusion predicts masked motion latents using text conditioning and three complementary attention mechanisms. During inference, it iteratively generates latent representations that are decoded into motion.

Architecture

  • Text-conditioned AdaLN transformer processing supplies diffusion-head conditions for predicting masked 2D motion latents.CLIP extracts text features, which are injected through AdaLN; the diffusion head is a small MLP.Source: B. Spatial-Temporal Autoregressive… · pp. 2–3

Attention

  • Spatial-temporal attention preserves joint-time relationships, while spatial and temporal attention separately model within-frame joint relations and within-joint temporal relations.The three modules are applied sequentially in the integrated version selected for better performance.Source: B. Spatial-Temporal Autoregressive… · p. 3

Masking and training

  • Training randomly masks the 2D latent sequence using a sampled mask ratio, with BERT-style remasking replacing selected latents by mask tokens, noise, or unchanged values.The mask ratio is generated from τ sampled uniformly from (0, 1), and remasking uses 80%, 10%, and 10% probabilities respectively.Source: B. Spatial-Temporal Autoregressive… · p. 3

Inference

  • During inference, all latent vectors begin as mask latents, and transformer-conditioned latents are passed to diffusion MLPs for iterative prediction.The predicted latent representations are subsequently decoded into the motion sequence.Source: B. Spatial-Temporal Autoregressive… · p. 3, Figure 2 · p. 3

A. Datasets.

The evaluation uses HumanML3D and KIT-ML for text-to-motion experiments, with established train, validation, and test splits. Tables report generation metrics and motion-compression errors, while visualizations compare generated sequences and compression quality.

Datasets

  • HumanML3D contains 14,616 motions and 44,970 texts, while KIT-ML contains 3,911 motions and 6,278 texts.The datasets use distinct training, validation, and testing splits for evaluation.Source: A. Datasets · p. 3
  • HumanML3D uses 23,384/1,460/4,383 samples and KIT-ML uses 4,888/300/830 samples for training, validation, and testing.These splits follow prior approaches.Source: A. Datasets · p. 3

Evaluation

  • Table I evaluates HumanML3D and KIT-ML using averages over 20 repetitions with 95% confidence intervals across generation metrics.Metric arrows indicate whether higher, lower, or closer-to-real values are preferred.Source: Table I · p. 4
  • Table II evaluates HumanML3D motion compression using MPJPE, PAMPJPE, and acceleration error, with the first two measured in millimeters.These metrics assess reconstruction-related motion quality.Source: Table II · p. 4

Motion features

  • The motion features have 263 dimensions for HumanML3D and 251 dimensions for KIT-ML, including global motion features and local joint information.The representations include 22 joints for HumanML3D and 21 joints for KIT-ML.Source: A. Datasets · p. 4

B. Evaluation Metrics.

The evaluation follows standard procedures and compares DeMoDiff’s reconstruction, generation, and visualization performance against prior methods. Results indicate strong reconstruction and competitive text-to-motion generation, with improved motion similarity and global positional alignment.

  • Evaluation follows standard procedures established by previous research, with metric details provided in the supplementary material.Source: B. Evaluation Metrics · p. 4
  • DeMoDiff achieves highly competitive text-to-motion generation performance on both HumanML3D and KIT-ML.The comparison includes previous diffusion-based and continuous regression-based methods.Source: B. Evaluation Metrics · p. 4
  • DeMoDiff achieves better text-motion alignment through higher R-precision and lower MM-Dist.Source: B. Evaluation Metrics · p. 4
  • DeMoDiff’s reconstruction accuracy significantly surpasses previous methods on HumanML3D.Source: B. Evaluation Metrics · p. 5
  • Visualizations show improved motion similarity and global positional alignment compared with other compression approaches.Figure 4 compares dark-blue ground truth with light-blue predictions without additional alignment processing.Source: B. Evaluation Metrics · p. 5, Figure 4 · p. 5

C. Ablation Study.

The ablation study examines the spatial-temporal VAE, latent dimension, and attention mechanisms. Results show that larger latent dimensions improve reconstruction, while integrating all three attention modules improves generation performance.

  • The ablations separately evaluate Spatial-Temporal VAE choices and Spatial-Temporal Autoregressive Diffusion attention choices.Source: C. Ablation Study · p. 5

Spatial-Temporal VAE

  • Larger VAE latent dimensions improve reconstruction metrics by providing greater representational capacity for detailed input features.Source: C. Ablation Study · p. 5

Spatial-Temporal Autoregressive Diffusion

  • Integrating temporal-spatial, temporal, and spatial attention produces a significant performance improvement in human motion generation.Source: C. Ablation Study · p. 5

D. Temporal and Spatial Editing.

DeMoDiff supports temporal and spatial motion editing through bidirectional attention and spatial-temporal masking. Edited regions are selectively masked and regenerated using unmasked latents and textual instructions.

V. CONCLUSION

DeMoDiff combines spatial-temporal VAE representation with spatial-temporal autoregressive diffusion to generate and edit motion in temporal and spatial latent spaces. The framework demonstrates competitive generation with greater flexibility.

  • DeMoDiff directly predicts temporal and spatial motion latents through a spatiotemporally decoupled autoregressive diffusion model.Source: §V · p. 6
  • The framework enables sampling and editing in both temporal and spatial latent spaces.Source: §V · p. 6
  • DeMoDiff demonstrates competitive motion generation while providing greater flexibility.Source: §V · p. 6

Related papers

Loading 2608.23279v1…