PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Motion Diffusion Models for Autoregressive Motion Generation

Qing Yu, Akihisa Watanabe, Kent Fujiwara

arXiv:2602.22594v1cs.CV

TL;DR

Existing motion-generation approaches trade off diffusion models' generation quality against autoregressive models' temporal causality and stability. CMDM combines causal diffusion with motion-language-aligned latent representations and frame-wise sampling, achieving superior motion fidelity, semantic alignment, and efficiency across HumanML3D and SnapMoGen. The paper also identifies remaining limits for abstract text, extremely long sequences, and multi-character scenarios.

  • Problem

    Existing diffusion models lack temporal causality for online generation, while autoregressive models can accumulate errors and become unstable for long-horizon synthesis.

  • Method

    CMDM uses MAC-VAE for semantically aligned causal latents, Causal-DiT for autoregressive diffusion denoising, and frame-wise sampling with causal uncertainty.

  • Results

    CMDM consistently outperforms state-of-the-art diffusion and autoregressive models in semantic fidelity and temporal smoothness while reducing inference latency by an order of magnitude.

  • Takeaways & Limitations

    CMDM provides a unified formulation for high-fidelity, semantically aligned, efficient, streaming, and long-horizon motion generation.

  • Takeaways & Limitations

    Performance may be limited for highly abstract or ambiguous text, extremely long sequences may accumulate minor temporal artifacts, and the framework focuses on single-person motion.

Abstract

from arXiv · show

Recent advances in motion diffusion models have substantially improved the realism of human motion synthesis. However, existing approaches either rely on full-sequence diffusion models with bidirectional generation, which limits temporal causality and real-time applicability, or autoregressive models that suffer from instability and cumulative errors. In this work, we present Causal Motion Diffusion Models (CMDM), a unified framework for autoregressive motion generation based on a causal diffusion transformer that operates in a semantically aligned latent space. CMDM builds upon a Motion-Language-Aligned Causal VAE (MAC-VAE), which encodes motion sequences into temporally causal latent representations. On top of this latent representation, an autoregressive diffusion transformer is trained using causal diffusion forcing to perform temporally ordered denoising across motion frames. To achieve fast inference, we introduce a frame-wise sampling schedule with causal uncertainty, where each subsequent frame is predicted from partially denoised previous frames. The resulting framework supports high-quality text-to-motion generation, streaming synthesis, and long-horizon motion generation at interactive rates. Experiments on HumanML3D and SnapMoGen demonstrate that CMDM outperforms existing diffusion and autoregressive models in both semantic fidelity and temporal smoothness, while substantially reducing inference latency.

1. Introduction

CMDM addresses the tension between diffusion models' motion quality and autoregressive models' temporal causality by combining causal diffusion with semantically aligned latent representations. Its frame-wise causal sampling supports efficient generation, and evaluations report stronger semantic fidelity and temporal smoothness with substantially lower latency.

  • Existing diffusion models improve motion quality but use bidirectional full-sequence denoising, breaking temporal causality and preventing online generation.
  • Autoregressive models preserve causal consistency and online generation but can accumulate errors, destabilizing and slowing long-horizon synthesis.
  • CMDM combines causal autoregression and diffusion denoising in a motion-language-aligned latent space using the MAC-VAE and Causal-DiT.MAC-VAE produces temporally causal latent representations, while Causal-DiT uses causal self-attention so each frame depends only on preceding frames.
  • CMDM trains with independent frame noise levels and causal dependencies, then samples frames from partially denoised preceding frames through causal diffusion forcing.This frame-wise sampling schedule is designed to reduce the need for fully autoregressive denoising.
  • CMDM outperforms state-of-the-art diffusion and autoregressive models on semantic fidelity and temporal smoothness while reducing inference latency by an order of magnitude.The evaluations cover HumanML3D and SnapMoGen.

2. Related Works

Prior work spans motion-language alignment, diffusion-based motion synthesis, and autoregressive modeling. CMDM differs by combining causal diffusion in a motion-language-aligned latent space with frame-wise sampling for streaming generation.

  • Motion-language methods learn joint motion-text representations, extending beyond single-frame alignment toward temporal motion semantics.
  • Diffusion-based methods have established strong benchmarks for motion realism and diversity across motion-space and latent-space formulations.
  • Autoregressive motion models predict future frames from past context but face exposure bias, cumulative errors, or instability from teacher forcing and large diffusion heads.
  • CMDM introduces causal diffusion within a motion-language-aligned latent space and a frame-wise sampling schedule for high-quality streaming generation.

3. Method

CMDM integrates causal latent encoding, causal diffusion denoising, and frame-wise sampling to generate semantically aligned motion while preserving temporal order. Its MAC-VAE and Causal-DiT operate causally, while causal diffusion forcing and streaming inference reduce exposure bias and support efficient generation.

  • MAC-VAE: MAC-VAE encodes motion into semantically aligned, temporally causal latent representations using a causal encoder–decoder and motion-language supervision.The latent representation uses 4× temporal downsampling, encoding T frames into T/4 temporal steps.
  • MAC-VAE: MAC-VAE combines reconstruction, Kullback–Leibler divergence, and motion-language alignment objectives to structure the latent space.The alignment loss includes feature similarity and relative distance-matrix consistency between motion and motion-language embeddings.
  • Causal Diffusion Forcing: Causal diffusion forcing assigns independent noise levels to motion frames while causal attention restricts each prediction to past representations.This trains temporally ordered denoising transitions rather than jointly denoising the full sequence.
  • Causal-DiT: Causal-DiT performs diffusion denoising with causal self-attention and text cross-attention, combining autoregressive temporal dependencies with language conditioning.Causal masking prevents future-frame access, while cross-attention conditions frame-level latents on word-level text embeddings.
  • Inference and Streaming Generation: Frame-wise sampling progressively denoises each frame and reuses partially denoised frames as context, mitigating exposure bias and lowering inference cost.The schedule begins denoising subsequent frames before the current frame is fully denoised, enabling continuous low-latency generation.

4. Experiments

Experiments on HumanML3D and SnapMoGen evaluate CMDM’s text-to-motion quality, long-horizon continuity, ablations, and computational efficiency. CMDM achieves strong semantic and temporal performance while enabling substantially faster generation.

  • Experimental Setup: HumanML3D contains 14,616 motion clips with 44,970 descriptions, while SnapMoGen contains 20,450 clips, 122K captions, and about 43.7 hours of data.SnapMoGen emphasizes expressive, temporally continuous long-horizon activities.
  • Text-to-Motion Results: CMDM w/ FSS achieves HumanML3D R-Precision of 0.588/0.778/0.860, FID of 0.068, and CLIP-Score of 0.685.These results are reported as the best overall performance for the listed combination of metrics, with FID described as second lowest.
  • Text-to-Motion Results: On SnapMoGen, CMDM w/ FSS achieves the best overall results across evaluation metrics, including the lowest FID and high CLIP-Score.The result is reported as state-of-the-art performance across VQ-, diffusion-, and autoregressive-based methods.
  • Long-Horizon Motion Generation: CMDM generates continuous, seamless, and realistic long-horizon motion with smooth transitions, supporting streaming and continuous text-to-motion generation at real-time speed.Qualitative comparisons report fewer incorrect actions and unnatural transitions than FlowMDM and MARDM.
  • Computational Efficiency: CMDM uses 114M parameters and reaches 28 fps with standard autoregressive sampling and up to 125 fps with frame-wise sampling, versus MARDM’s 20 fps and MotionStreamer’s 11 fps.The comparison measures 6-second sequences on an NVIDIA A100 over 100 repetitions.
  • Ablation Studies: Ablations show that causal latent modeling, semantic alignment, causal diffusion, positional components, and frame-wise sampling jointly improve motion quality, semantic consistency, and temporal stability.The FSS analysis identifies smaller uncertainty scale L=5 as producing smoother transitions and lower AUJ.

5. Limitations

CMDM’s performance is bounded by pretrained motion-language alignment, extremely long-sequence generation, and its focus on single-person motion. The authors identify these as directions for improving robustness and scope.

  • Limitations: Pretrained motion-language alignment may limit CMDM on highly abstract or ambiguous text descriptions.The causal latent encoder relies on alignment quality from models such as Part-TMR.
  • Limitations: Frame-wise sampling may still accumulate minor temporal artifacts when generating extremely long sequences, such as sequences lasting several minutes.The authors suggest motion-aware feedback or adaptive re-anchoring as possible improvements.
  • Limitations: CMDM focuses primarily on single-person motion and has not yet been extended to interactive or multi-character scenarios.The authors identify interactive and multi-character generation as future work.

6. Conclusion

CMDM combines diffusion realism and stability with autoregressive temporal causality and efficiency. Experiments show superior fidelity, semantic alignment, and efficiency against existing diffusion and autoregressive models.

  • CMDM combines diffusion models’ realism and stability with autoregressive generation’s temporal causality and efficiency.
  • MAC-VAE provides semantically grounded causal latent encoding for motion generation.
  • Causal-DiT performs temporally ordered diffusion denoising within CMDM.
  • The frame-wise sampling schedule enables real-time streaming generation.
  • Experiments on HumanML3D and SnapMoGen demonstrate superior motion fidelity, semantic alignment, and efficiency compared with existing models.

A. Implementation Details

CMDM uses causal latent encoding, causal diffusion forcing, and frame-wise sampling to generate motion efficiently. Evaluations cover compact representations, compositional generation, latency, and long-horizon synthesis.

  • Architecture: MAC-VAE uses causal convolutional and residual layers to encode temporally causal motion latents.The implementation uses seven causal convolutional layers and two causal ResNet blocks with left padding.
  • Architecture: Frame-level motion–language contrastive alignment supplies semantic supervision for MAC-VAE.Part-TMR frame features are aligned with corresponding text features rather than aggregated through a class token.
  • Training: Causal diffusion forcing independently perturbs frames while preserving causal dependencies for temporally ordered denoising.Each frame receives an independent noise level during training, while Causal-DiT conditions on preceding latent frames and text.
  • Inference: Frame-Wise Sampling Schedule inference begins denoising frame t+1 at step K−L of frame t, using partially denoised frames as guidance.The reported setting uses K=50 diffusion steps and uncertainty scale L=2.
  • Evaluation: On BABEL, CMDM achieves the best overall performance across subsequence and transition metrics for long-horizon motion synthesis.Subsequence metrics assess within-segment quality and diversity, while transition metrics assess continuity and smoothness between segments.
  • Evaluation: CMDM outperforms compositional baselines across semantic and realism metrics while reducing latency for token generation.In the reported latency test, CMDM with AR requires approximately 150 ms for the first token, versus 210 ms for MARDM and 360 ms for MotionStreamer.

B.5. Ablation Studies

Ablations examine MAC-VAE representations, motion–language alignment, Causal-DiT size, and text encoders. Compact latent settings and word-level DistilBERT achieve strong quality and semantic alignment.

  • MAC-VAE: MAC-VAE with latent dimension 64 and temporal downsampling 1/4 balances reconstruction fidelity and generation quality.It achieves reconstruction FID=0.000 and MPJPE=0.012, with generation R-Top1=0.588, FID=0.068, and MM-Dist=2.620.
  • MAC-VAE: Larger temporal downsampling reduces temporal resolution and slightly degrades generation metrics.The reported degradation is attributed to information loss at larger downsampling rates such as r=1/8.
  • Motion-Language Models: Part-TMR provides the strongest motion–language alignment among the evaluated backbones.It reports reconstruction FID=0.000, MPJPE=0.012, and the highest R-Precision of 0.588.
  • Causal-DiT Size: The medium Causal-DiT configuration balances quality and efficiency, while scaling to 304M parameters yields marginal gains.The 38M model reports R-Precision 0.588 and FID 0.068; the 304M model reports R-Precision=0.590 and FID=0.042.
  • Text Encoder: DistilBERT achieves the best overall text-encoder performance with word-level embeddings.It obtains the highest R-Precision of 0.588 and lowest FID of 0.068.

C. Additional Qualitative results

Qualitative comparisons show CMDM generating semantically accurate, temporally coherent motion across long-horizon and text-to-motion settings. Compared methods exhibit incorrect actions, unstable transitions, or less faithful textual details.

  • Long-Horizon Generation: CMDM avoids content drift and skeleton flipping in qualitative long-horizon comparisons.Previous methods are reported to suffer from static poses, incorrect transitions, or inconsistent actions.
  • Text-to-Motion Generation: CMDM more faithfully reflects fine-grained textual semantics while preserving natural body articulation.Examples include arm rotations, leg movements, and walking direction.
  • Text-to-Motion Generation: On SnapMoGen, CMDM generates more realistic motions from raw text prompts without LLM-based augmentation.

D. Sample Code

The paper provides supplementary qualitative comparisons and releases training and evaluation code for CMDM. The visualizations cover long-horizon and text-to-motion generation on HumanML3D and SnapMoGen.

  • Sample Code: The authors will release code for training and evaluating CMDM on HumanML3D.The repository includes training code and directs readers to its README for details.
  • Long-Horizon Motion Generation: Figures 4 and 5 compare CMDM with previous methods for long-horizon motion generation on HumanML3D and SnapMoGen.The long sequences are split into caption-aligned shorter segments for visualization, while complete sequences are available in supplementary videos.
  • Text-to-Motion Generation: On HumanML3D, CMDM’s qualitative text-to-motion results better capture fine-grained textual semantics and natural body articulation than previous methods.The paper refers readers to supplementary videos for clearer visualization.
  • Text-to-Motion Generation: On SnapMoGen, CMDM achieves strong text-to-motion generation quality using raw text prompts without LLM-based augmentation.The comparison is presented in Figure 7, with supplementary videos provided for visualization.
Loading 2602.22594v1…