Source-linked AI summary
Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation
Chengqun Yang, Liang Xu, Yanping Li, Fulong Liu, Jingnan Gao, Weili Zeng, Yichao Yan
TL;DR
Existing motion representations and generators provide limited information preservation or fine-grained body-part control. DeMoDiff jointly redesigns representation and generation with a joint-wise spatial-temporal VAE and masked autoregressive diffusion, and reports strong reconstruction, generation, and editing results on HumanML3D and KIT-ML.
Problem
VQ representations lose information, holistic continuous representations hinder part-level flexibility, and existing diffusion-based generators have limited fine-grained control over individual body parts.
Method
DeMoDiff uses a spatial-temporal VAE that encodes each body joint and a spatial-temporal autoregressive diffusion generator with masking and attention.
Results
The model reports improved reconstruction and generation metrics, superior performance in several motion tasks especially joint-level editing, and strong temporal and spatial editing capabilities.
Takeaways & Limitations
DeMoDiff provides a unified framework for motion generation with temporal and spatial latent sampling and flexible temporal and spatial editing.
Abstract
from arXiv · showhide
Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/
I. INTRODUCTION
DeMoDiff addresses limitations in motion representation and generation by decoupling spatial and temporal structure. It combines a joint-wise spatial-temporal VAE with masked autoregressive diffusion to support generation and fine-grained editing.
- Existing motion-generation methods rely on VQ-based or latent-based representations, while diffusion and autoregressive diffusion architectures still provide limited individual-body-part controllability.
- DeMoDiff encodes each body joint with a lightweight spatial-temporal VAE instead of compressing whole-body motion into one latent space.The decoupled representation is intended to improve representation extraction and part-level controllability.
- Its autoregressive diffusion generator combines spatial-temporal masking and attention to provide motion generation and controllable editability.
- Experiments on HumanML3D and KIT-ML evaluate DeMoDiff for text-driven motion generation and motion editing, with analysis and visualizations supporting its effectiveness.
- The method reports improved reconstruction and generation metrics from continuous 2D latent motion representation.
- The framework supports temporal and spatial editing without editing-specific fine-tuning, including joint-level editing among its reported motion-related tasks.
II. RELATED WORK
Prior work represents motion with discrete VQ tokens or continuous latents, but these approaches face limitations in information preservation, long-horizon modeling, and fine-grained joint control.
- Text-driven human motion generation must map natural-language descriptions to realistic 3D motions while handling fine-grained semantics, body-part control, and elaborate prompts.
- VQ-based methods encode motion as discrete tokens for autoregressive synthesis, but sequential generation inherently impedes modeling long-horizon temporal relationships.
- Bidirectional masked generation models predict masked tokens simultaneously, improving context modeling capability over sequential token generation.
- VQ-based approaches nevertheless suffer from inevitable information loss during discrete motion representation.
- Related representations also divide or split motion along body-part or joint dimensions to improve joint encoding and spatial relationships.
III. THE DEMODIFF MODEL
DeMoDiff represents motion as a two-dimensional temporal-joint structure and uses a spatial-temporal VAE to encode and decode continuous motion latents.
- The model architecture comprises a spatial-temporal VAE for motion representation and a spatial-temporal autoregressive diffusion architecture for continuous human-motion generation.
- The VAE organizes motion sequences across temporal and joint dimensions as a 2D map before processing them with convolutional networks.
- A 2D convolutional encoder maps 2D motion features to latent-distribution parameters, which are decoded into motion sequences.
- The VAE objective combines reconstruction quality with latent-space regularization through KL and joint-velocity losses.The KL and joint-velocity terms are weighted by hyperparameters.
B. Spatial-Temporal Autoregressive Diffusion.
Spatial-Temporal Autoregressive Diffusion predicts masked motion latents using text conditioning and three complementary attention mechanisms. During inference, it iteratively generates latent representations that are decoded into motion.
- Architecture: Text-conditioned AdaLN transformer processing supplies diffusion-head conditions for predicting masked 2D motion latents.CLIP extracts text features, which are injected through AdaLN; the diffusion head is a small MLP.
- Attention: Spatial-temporal attention preserves joint-time relationships, while spatial and temporal attention separately model within-frame joint relations and within-joint temporal relations.The three modules are applied sequentially in the integrated version selected for better performance.
- Masking and training: Training randomly masks the 2D latent sequence using a sampled mask ratio, with BERT-style remasking replacing selected latents by mask tokens, noise, or unchanged values.The mask ratio is generated from τ sampled uniformly from (0, 1), and remasking uses 80%, 10%, and 10% probabilities respectively.
- Inference: During inference, all latent vectors begin as mask latents, and transformer-conditioned latents are passed to diffusion MLPs for iterative prediction.The predicted latent representations are subsequently decoded into the motion sequence.
A. Datasets.
The evaluation uses HumanML3D and KIT-ML for text-to-motion experiments, with established train, validation, and test splits. Tables report generation metrics and motion-compression errors, while visualizations compare generated sequences and compression quality.
- Datasets: HumanML3D contains 14,616 motions and 44,970 texts, while KIT-ML contains 3,911 motions and 6,278 texts.The datasets use distinct training, validation, and testing splits for evaluation.
- Datasets: HumanML3D uses 23,384/1,460/4,383 samples and KIT-ML uses 4,888/300/830 samples for training, validation, and testing.These splits follow prior approaches.
- Evaluation: Table I evaluates HumanML3D and KIT-ML using averages over 20 repetitions with 95% confidence intervals across generation metrics.Metric arrows indicate whether higher, lower, or closer-to-real values are preferred.
- Evaluation: Table II evaluates HumanML3D motion compression using MPJPE, PAMPJPE, and acceleration error, with the first two measured in millimeters.These metrics assess reconstruction-related motion quality.
- Motion features: The motion features have 263 dimensions for HumanML3D and 251 dimensions for KIT-ML, including global motion features and local joint information.The representations include 22 joints for HumanML3D and 21 joints for KIT-ML.
B. Evaluation Metrics.
The evaluation follows standard procedures and compares DeMoDiff’s reconstruction, generation, and visualization performance against prior methods. Results indicate strong reconstruction and competitive text-to-motion generation, with improved motion similarity and global positional alignment.
- Evaluation follows standard procedures established by previous research, with metric details provided in the supplementary material.
- DeMoDiff achieves highly competitive text-to-motion generation performance on both HumanML3D and KIT-ML.The comparison includes previous diffusion-based and continuous regression-based methods.
- DeMoDiff achieves better text-motion alignment through higher R-precision and lower MM-Dist.
- DeMoDiff’s reconstruction accuracy significantly surpasses previous methods on HumanML3D.
- Visualizations show improved motion similarity and global positional alignment compared with other compression approaches.Figure 4 compares dark-blue ground truth with light-blue predictions without additional alignment processing.
C. Ablation Study.
The ablation study examines the spatial-temporal VAE, latent dimension, and attention mechanisms. Results show that larger latent dimensions improve reconstruction, while integrating all three attention modules improves generation performance.
- The ablations separately evaluate Spatial-Temporal VAE choices and Spatial-Temporal Autoregressive Diffusion attention choices.
- Spatial-Temporal VAE: Larger VAE latent dimensions improve reconstruction metrics by providing greater representational capacity for detailed input features.
- Spatial-Temporal Autoregressive Diffusion: Integrating temporal-spatial, temporal, and spatial attention produces a significant performance improvement in human motion generation.
D. Temporal and Spatial Editing.
DeMoDiff supports temporal and spatial motion editing through bidirectional attention and spatial-temporal masking. Edited regions are selectively masked and regenerated using unmasked latents and textual instructions.
- Bidirectional attention enables both temporal and spatial editing without editing-specific fine-tuning.
- Figure 5 marks edited regions in green while light-to-dark blue indicates the motion sequence order.
- Temporal editing masks latents at the temporal locations to be changed and conditions generation on unmasked latents and editing instructions.
V. CONCLUSION
DeMoDiff combines spatial-temporal VAE representation with spatial-temporal autoregressive diffusion to generate and edit motion in temporal and spatial latent spaces. The framework demonstrates competitive generation with greater flexibility.
- DeMoDiff directly predicts temporal and spatial motion latents through a spatiotemporally decoupled autoregressive diffusion model.
- The framework enables sampling and editing in both temporal and spatial latent spaces.
- DeMoDiff demonstrates competitive motion generation while providing greater flexibility.