Source-linked AI summary
MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model
Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, Yansong Tang
TL;DR
Existing spatial-temporal control methods for text-conditioned motion generation are not runtime-efficient enough for real-time use. MotionLCM addresses this with latent consistency distillation and a latent-space motion ControlNet, achieving high-quality controllable motion generation with real-time efficiency.
Problem
Existing diffusion and spatial-temporal control methods for text-to-motion generation require substantial sampling time, limiting real-time controllable motion generation.
Method
MotionLCM distills a motion latent diffusion model into a one-step or few-step latent consistency model and adds a motion ControlNet with motion-space supervision.
Results
MotionLCM achieves a balance among generation quality, controlling capability, and real-time efficiency in controllable motion generation.
Takeaways & Limitations
MotionLCM extends controllable motion generation to a real-time level while maintaining high-quality generation and effective control with text and control signals.
Abstract
from arXiv · showhide
This work introduces MotionLCM, extending controllable motion generation to a real-time level. Existing methods for spatial-temporal control in text-conditioned motion generation suffer from significant runtime inefficiency. To address this issue, we first propose the motion latent consistency model (MotionLCM) for motion generation, building on the motion latent diffusion model. By adopting one-step (or few-step) inference, we further improve the runtime efficiency of the motion latent diffusion model for motion generation. To ensure effective controllability, we incorporate a motion ControlNet within the latent space of MotionLCM and enable explicit control signals (i.e., initial motions) in the vanilla motion space to further provide supervision for the training process. By employing these techniques, our approach can generate human motions with text and control signals in real-time. Experimental results demonstrate the remarkable generation and controlling capabilities of MotionLCM while maintaining real-time runtime efficiency.
1 Introduction
MotionLCM targets real-time controllable text-to-motion generation by addressing the runtime costs of diffusion sampling and spatial-temporal control. It combines latent consistency distillation with latent-space motion control to balance generation quality, controllability, and efficiency.
- Motivation: Existing diffusion-based motion generators require substantial inference time, while spatial-temporal control methods remain far from real-time applications.MDM and MLD require approximately 24s and 0.2s per high-quality motion sequence, while OmniControl requires approximately 81s per sequence.
- MotionLCM: MotionLCM applies consistency distillation to the motion latent diffusion model, enabling one-step or few-step motion generation at real-time speed.The method introduces consistency distillation into motion generation and targets real-time inference through latent consistency distillation.
- Controllability: MotionLCM introduces a motion ControlNet in latent space to incorporate spatial-temporal control signals such as initial motions.Because latent representations lack explicit motion semantics, the method decodes predicted latents through a frozen VAE decoder to provide explicit motion-space control supervision during training.
- Outcome: The method is designed to balance generation quality, controlling capability, and real-time efficiency in controllable motion generation.The paper reports that MotionLCM achieves this balance and demonstrates high-quality controllable motion generation at a real-time level.
2 Related Work
Human-motion generation methods are organized by their input conditions, including unconditional synthesis, multimodal conditions, and user-defined trajectories. Recent work has explored diverse generative models, with diffusion-based methods improving motion-generation performance and diversity.
- Input Conditions: Human-motion generation can use no condition, multimodal conditions, or user-defined trajectories.Multimodal conditions include action labels, textual descriptions, audio, and music.
- Input Conditions: Textual descriptions are one major multimodal input for generating diverse, natural, and high-quality human motions.The related-work discussion lists textual descriptions alongside action labels, audio, and music as conditioning signals.
- Generative Models: Many generative models have been explored for human-motion generation, and diffusion-based models have recently improved generation performance and diversity.
3 Method
MotionLCM distills a pre-trained motion latent diffusion model into a consistency model for one-step or few-step generation, then adds latent-space motion control with explicit supervision from initial motions.
- 3.1 Preliminaries: Consistency models learn self-consistent mappings from noisy points on an ODE trajectory to its origin, enabling one-step or few-step generation.The model enforces consistent outputs for points on the same trajectory and can be trained by distilling a pre-trained diffusion model.
- 3.2 MotionLCM: Motion Latent Consistency Model: The distillation objective compares online-network and target-network predictions, with the target network updated using exponential moving average parameters.The teacher network is the pre-trained MLD model, while the online and target networks are initialized from it.
- 3.2 MotionLCM: Motion Latent Consistency Model: MotionLCM compresses motions with a pre-trained VAE and distills the motion latent diffusion model MLD to target one-step or 2–4-step inference.The latent representation is decoded back into motion sequences after consistency-based generation.
- 3.3 Controllable Motion Generation in Latent Space: MotionLCM introduces a latent-space motion ControlNet initialized from a trainable copy of MotionLCM to incorporate initial-pose trajectory controls.A trajectory encoder processes the control-joint trajectories and supplies features to the motion ControlNet during denoising.
- 3.3 Controllable Motion Generation in Latent Space: The training objective combines reconstruction and control losses so explicit vanilla-space control signals provide additional supervision for motion generation.The weight λ balances the reconstruction and control terms, and the paper reports improved motion-control performance from this supervision.
4 Experiments
Experiments evaluate MotionLCM on HumanML3D for text-to-motion and controllable motion generation, using quality, control, diversity, matching, and inference-time measures. MotionLCM achieves strong generation and control performance while substantially improving runtime efficiency, with ablations examining training and control settings.
- Experimental setup: Evaluation uses HumanML3D and reports inference time, FID, diversity, condition matching, and control-related metrics.The text-to-motion evaluation repeats each metric 20 times and reports a 95% confidence interval.
- Text-to-motion comparison: 30ms one-step MotionLCM surpasses all state-of-the-art models on text-conditional motion synthesis.The comparison is conducted on HumanML3D using the suggested metrics.
- Controllable motion generation: 1929× faster than OmniControl and 13× faster than MLD, one-step MotionLCM maintains higher fidelity and improves motion-control performance.Qualitative results also show better adherence to initial-pose control and textual descriptions than OmniControl.
- Controllable motion generation: MotionLCM outperforms OmniControl and MLD in generation quality, control performance, and inference speed.The motion-control comparison includes latent-space and motion-space control supervision settings.
- Ablation studies: Dynamic training guidance w ∈[5, 15], larger EMA rates, increasing skipping intervals, and Huber loss improve one-step distillation performance, while excessively large ranges or k = 50 can hurt results.The ablation identifies Huber loss as more robust than L2 loss.
- Ablation studies: Introducing ControlNet improves control metrics, while larger control-loss weights improve control but reduce generation quality; λ = 1 balances both.A fixed training control ratio τ = 0.25 performs better than a dynamic ratio in the reported comparison.
5 Conclusion
MotionLCM is presented as an efficient controllable motion-generation framework that balances runtime efficiency with generation quality. Its latent-space Motion ControlNet provides control under given conditions, while the lack of explicit temporal modeling in MLD's VAE remains a limitation.
- Conclusion: MotionLCM combines latent consistency distillation with latent-space Motion ControlNet manipulation for efficient controllable motion generation.The framework is designed to balance runtime efficiency and generation quality while using given control conditions.
- Conclusion: The MLD VAE lacks explicit temporal modeling, limiting MotionLCM's ability to achieve a good temporal explanation.The authors identify a more explainable compression architecture as future work for efficient motion control.
A.1 Comparison to other ODE Solvers
MotionLCM is compared with DDIM, DPM, and DPM++ to assess latent consistency distillation. It outperforms these baselines while requiring only one forward pass, reducing sampling time and memory costs under CFG.
- ODE solver comparison: One forward pass lets MotionLCM save time and memory costs that baseline solvers incur through additional peak memory per CFG sampling step.The cited comparison specifically contrasts MotionLCM with DDIM, DPM, and DPM++ under CFG.
- ODE solver comparison: MotionLCM notably outperforms DDIM, DPM, and DPM++ on HumanML3D with testing CFG scale w = 7.5.The comparison is reported as quantitative evidence for latent consistency distillation.
A.2 Impact of different testing CFGs
Testing-CFG ablations examine how inference-step count affects MotionLCM performance and runtime. More steps initially improve performance, but further increases add time without meaningful gains.
- Testing CFG ablation: Increasing inference steps continuously improves performance under different testing CFGs.The comparison is presented as an ablation over testing CFG settings.
- Testing CFG ablation: Further increasing inference steps yields comparable performance while significantly increasing time cost.This identifies a runtime-efficiency trade-off in selecting the inference-step count.
B More Qualitative Results
This section presents additional qualitative MotionLCM results for both text-to-motion generation and motion control, shown in Figures 8 and 9.
- Figure 8 presents additional MotionLCM generation results for the text-to-motion task.
- Figure 9 presents additional MotionLCM visualization results for the motion control task.
C Metric Definitions
The evaluation defines runtime, motion quality, condition matching, control error, and motion diversity metrics. These metrics quantify efficiency, generation fidelity, text-motion alignment, control accuracy, and variation across generated motions.
- Time cost: Average Inference Time per Sentence (AITS) measures inference efficiency in seconds on HumanML3D with batch size 1, excluding loading time.
- Motion quality: FID measures the distributional difference between generated and real motions using a dataset-specific feature extractor.
- Motion diversity: Diversity measures variation across generated motions by averaging distances between feature vectors from two randomly sampled subsets.
- Motion diversity: MultiModality measures variation among motions generated for the same textual descriptions by averaging feature-vector distances across sampled descriptions and motion subsets.
- Condition matching: R-Precision reports text-motion Top-1/2/3 matching accuracy, while MM Dist measures mean distance between generated motions and texts.
- Control error: Trajectory error and Location error count unsuccessful trajectories and joints using a 50cm distance threshold, while Average error is the mean control-joint distance.