Source-linked AI summary
Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer
Chenyang Gu, Mingyuan Zhang, Haozhe Xie, Zhongang Cai, Lei Yang, Ziwei Liu
TL;DR
Motion generation must reconcile semantic conditioning with fine-grained kinematic control. The paper addresses this with a three-stage Perception–Planning–Control framework centered on MoTok, which separates semantic token planning from diffusion-based reconstruction; on HumanML3D, it improves controllability and fidelity over MaskControl with one-sixth the tokens.
Problem
Integrating fine-grained, time-varying kinematic constraints into semantically conditioned motion generation remains a central challenge.
Method
A Perception–Planning–Control framework uses MoTok to produce compact semantic tokens, plans them with AR or DDM generators, and applies coarse-to-fine kinematic conditioning during planning and diffusion decoding.
Results
Trajectory error falls from 0.72 cm to 0.08 cm and FID from 0.083 to 0.029 versus MaskControl on HumanML3D, using one-sixth of the tokens.
Takeaways & Limitations
Separating semantic abstraction from low-level reconstruction supports compact tokens while retaining controllability and motion fidelity under kinematic constraints.
Abstract
from arXiv · showhide
Prior motion generation largely follows two paradigms: continuous diffusion models that excel at kinematic control, and discrete token-based generators that are effective for semantic conditioning. To combine their strengths, we propose a three-stage framework comprising condition feature extraction (Perception), discrete token generation (Planning), and diffusion-based motion synthesis (Control). Central to this framework is MoTok, a diffusion-based discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction by delegating motion recovery to a diffusion decoder, enabling compact single-layer tokens while preserving motion fidelity. For kinematic conditions, coarse constraints guide token generation during planning, while fine-grained constraints are enforced during control through diffusion-based optimization. This design prevents kinematic details from disrupting semantic token planning. On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens, reducing trajectory error from 0.72 cm to 0.08 cm and FID from 0.083 to 0.029. Unlike prior methods that degrade under stronger kinematic constraints, ours improves fidelity, reducing FID from 0.033 to 0.014.
1 Introduction
The paper proposes a Perception–Planning–Control framework that combines semantic token planning with diffusion-based motion reconstruction and fine-grained kinematic control. MoTok enables compact tokens by separating semantic abstraction from low-level reconstruction, improving controllability and fidelity on HumanML3D.
- Fine-grained, time-varying kinematic control remains difficult to integrate with semantic motion generation while preserving semantic intent.
- Existing tokenizers entangle semantics with motion details, requiring high token rates or hierarchical codes and complicating controllable generation.
- The Perception–Planning–Control paradigm encodes heterogeneous conditions, predicts discrete motion tokens with AR or DDM planners, and synthesizes motion through diffusion-based decoding.
- MoTok uses a single-layer codebook and delegates motion recovery to a diffusion decoder, reducing downstream token burden without requiring tokens to encode fine-grained kinematics.
- Kinematic signals guide token generation coarsely during planning and impose fine-grained constraints during diffusion control, preventing low-level details from disrupting semantic planning.
- 0.08 cm trajectory error and 0.029 FID are achieved versus 0.72 cm and 0.083 for MaskControl, using one-sixth of the tokens on HumanML3D.
2 Related Work
Related work spans continuous and discrete approaches to motion generation, while Fig. 2 situates MoTok as a bridge between compact token planning and diffusion-based reconstruction.
- Motion generation research progressed from classical unconditional methods to learned generative models and text- or action-conditioned approaches.
- MoTok factorizes motion into compact discrete tokens and diffusion-based reconstruction by decoding tokens into per-frame conditioning.
- The unified framework supports both discrete diffusion and autoregressive planners while integrating global and local conditions through a generator-agnostic interface.
- Discrete text-to-motion methods frame motion as a language-like sequence and use VQ-based tokenizers for text–motion translation.
3 Our Approach
The approach compresses motion into discrete tokens for semantic planning, then uses diffusion-based reconstruction to recover fine-grained motion while supporting global and local conditions.
- Motion representation: A motion sequence θ1:T is compressed into N discrete tokens, with compression ratio ρ = T/N measuring token reduction.The tokenization objective is high-quality generation under aggressive compression, reducing downstream sequence-modeling burden.
- Conditional generation: Global conditions provide sequence-level guidance, while local conditions align with the motion timeline to specify kinematic control signals.Text is encoded as a global feature, whereas trajectories are encoded into token-aligned local features for text-and-trajectory generation.
- MoTok: MoTok factorizes motion into a compact discrete code sequence and a diffusion decoder for fine-grained reconstruction.The tokenizer uses a convolutional encoder, vector quantizer, convolutional decoder, and conditional diffusion model.
- MoTok: The tokenizer encodes motion through temporal downsampling, vector quantization, and assignment of latent vectors to codebook entries.The encoder produces h1:N, the quantizer uses a codebook of size K = 1024, and the result is z1:N with quantized latents q1:N.
- Diffusion reconstruction: Diffusion reconstruction upsamples quantized latents into per-frame conditioning and predicts clean motion through conditional reverse diffusion.The decoder produces s1:T, which conditions a denoiser that predicts x̂0 from noisy motion xt at timestep t.
- Conditional generation: Token-space planning supports discrete diffusion and autoregressive generators through a shared conditioning interface.The interface places global conditioning at a dedicated position and incorporates local features at motion-token positions while preserving temporal alignment.
4 Experiments
Experiments evaluate MoTok on HumanML3D and KIT-ML for controllable and text-to-motion generation, including quantitative comparisons, qualitative trajectory alignment, and tokenizer and conditioning ablations.
- Experimental setup: HumanML3D and KIT-ML provide paired text–motion benchmarks for evaluating motion generation.
- Experimental setup: MoTok uses CLIP-based global text features and token-aligned local trajectory features for conditional planning.Trajectory features are downsampled to match the token length.
- Experimental setup: During inference, conditional diffusion decodes discrete tokens into continuous motion and applies auxiliary denoising objectives for trajectory control.
- Text and trajectory control: 0.08 cm trajectory error and 0.029 FID improve over MaskControl’s 0.72 cm and 0.083 while using one-sixth of the tokens.
- Text and trajectory control: MoTok improves fidelity under stronger trajectory constraints, while prior methods degrade as more joints are controlled.
- Text and trajectory control: MoTok follows provided trajectories more faithfully while producing smooth motion; MaskControl’s alignment is limited by tokenizer capacity.
- Text-to-motion generation: On HumanML3D, MoTok-DDM-2 reaches FID 0.033, while MoTok-DDM-4 reaches 0.039 versus MoMask’s 0.045 using one-sixth of the tokens.
- Text-to-motion generation: On KIT-ML, MoTok achieves the lowest FID of 0.144 versus the strongest baseline’s 0.155.
5 Conclusion
The work bridges semantic conditioning and kinematic control through a Perception–Planning–Control framework with MoTok diffusion decoding. Experiments show improved reconstruction and semantic usability under compact token representations.
- The method reduces trajectory error from 0.72 cm to 0.08 cm while using one-sixth of the tokens in controllable HumanML3D generation.The reported comparison is against MaskControl and covers text-and-trajectory controllable motion generation.
- The framework upgrades decoding into a Control stage that incorporates fine-grained kinematic conditions after token planning.This division lets planning focus on high-level conditions while diffusion-based decoding handles motion recovery and control.
- MoTok serves as a bridge between motion and semantic spaces by recovering high-frequency details and preserving low-frequency motion characteristics.These properties support both faithful reconstruction and mapping textual semantics to discrete motion tokens.
- With identical discrete tokens, MoTok’s diffusion decoder consistently improves reconstruction and downstream text-to-motion performance over a conventional convolutional decoder.The two-stage protocol freezes the encoder and codebook before decoder training, isolating decoder capability.
A.2 Better semantic usability of discrete tokens
MoTok tokens provide stronger semantic usability than standard VQ-VAE tokens, while diffusion decoding improves detail recovery without changing the discrete token information.
- MoTok’s diffusion-based decoder recovers motion details more faithfully under the same token information.A two-stage protocol freezes the encoder and codebook, preventing gains from being attributed to increased token capacity.
- MoTok tokens outperform standard VQ-VAE tokens across nearly all motion-to-text captioning metrics.The comparison uses MotionGPT and evaluates semantic information carried by the tokenizers.
- The captioning results indicate that MoTok tokens encode richer and more usable semantic information for text–motion alignment.The reported benefit also reduces the burden on the downstream planner.
- A plausible explanation is that diffusion decoding shifts high-frequency detail recovery away from discrete tokens, preserving more low-frequency motion structure.This explanation is presented as plausible rather than established causality.
B From Richer Conditions to Broader Tasks: Generality of the Unified Paradigm
The unified conditioning framework distinguishes global conditions from local conditions to support different types of motion-generation inputs.
- Text features are treated as global conditions, whereas kinematic signals are treated as local conditions.The framework also explores additional condition variants and usages to demonstrate generality and flexibility.
B.1 Inject Semantic Condition via Cross Attention
Cross-attention improves text–motion alignment relative to in-context conditioning but lowers FID, while adding global kinematic hints brings almost no improvement.
- Cross-attention yields a clear improvement in R-Precision compared with the in-context variant.The result indicates stronger text–motion alignment under the alternative text-injection strategy.
- Table 6 reports controllable HumanML3D results using Trajectory Error, Localization Error, and Average Error, with larger or smaller values preferred by metric.The table also marks whether closer-to-real-data values are better.
- The cross-attention improvement in R-Precision comes with a noticeable drop in FID.The results suggest a trade-off between text–motion alignment and generation fidelity across conditioning strategies.
- Adding global kinematic conditions brings almost no improvement across the evaluations.The setting retains the original local injection pathway while adding globally extracted hint features.
B.3 Motion Editing as a new condition-driven task
Motion editing uses a source motion as a local condition alongside a text prompt, with source-motion features injected as position-wise embeddings. The framework also reports efficiency and CFG-scale behavior during inference.
- Motion Editing: Motion editing combines a source motion with a text prompt describing the desired modification.The source motion is treated as a local condition, similarly to trajectory control.
- Motion Editing: Source-motion features are extracted by a MoTok-like motion encoder and used as position-wise embeddings.Text features are injected using the same mechanism as in the other tasks.
- Efficiency Comparison: 2.63s generates one text-and-trajectory-conditioned sequence on H100, compared with 32.79s for MaskControl.
- CFG-Scale Ablation: Moderate CFG guidance best balances semantic alignment and motion realism, whereas excessive guidance can harm fidelity.Across variants, performance improves, reaches an optimum, and then degrades as CFG scale becomes overly large.
D.1 Evaluation Metrics
The evaluation covers text-to-motion, motion-to-text, and controllable motion generation using semantic, distributional, captioning, diversity, and spatial-control metrics.
- Text-to-Motion: Text-to-motion evaluation measures realism, text–motion alignment, diversity, and multi-modality.The metrics include FID, R-Precision, Multimodal Distance, Diversity, and Multi-Modality.
- Motion-to-Text: Motion-to-text evaluation uses captioning metrics and R-Precision to assess generated-caption quality and motion–caption alignment.Reported captioning metrics include BLEU@1/4, ROUGE-L, CIDEr, and BERTScore.
- Controllable Motion Generation: Controllable motion generation evaluates whether generated motions satisfy prescribed spatial control signals.Metrics include Foot skating ratio, Trajectory error, Location error, and Average keyframe error.
- Distributional Metrics: FID measures distance between real and generated feature distributions using their means and covariances.The real and generated distributions are represented by (µr, Σr) and (µg, Σg), respectively.
- Spatial-Level Metrics: Keyframe location error is the Euclidean distance between generated and ground-truth 3D positions at controlled-joint keyframes.The metric compares positions at a set of keyframes K.
- Spatial-Level Metrics: Location error is the fraction of keyframes whose position error exceeds threshold τ.The indicator-based calculation counts keyframes with e_k > τ.
- Spatial-Level Metrics: Trajectory error marks a trajectory unsuccessful when any keyframe violates the threshold and reports the failure ratio over test samples.
- Spatial-Level Metrics: Foot skating ratio measures undesired horizontal foot motion during frames classified as foot-contact frames.A small threshold ε detects sliding during contact.
D.3 Details for Motion-to-Text
For motion-to-text, MoTok compresses motion into discrete tokens that condition an autoregressive caption decoder. The reported HumanML3D results indicate improved captioning performance over a conventional VQ tokenizer.
- Task Formulation: Motion-to-text generates a natural-language caption y from an input motion sequence x.The task is formulated as conditional language generation.
- Task Formulation: MoTok encodes the motion into a discrete token sequence z before caption generation.The token count depends on the temporal downsampling rate.
- Task Formulation: An autoregressive language decoder generates each caption token conditioned on the MoTok sequence and previous caption tokens.
- Task Formulation: The captioner operates in compact token space while preserving semantically relevant motion content.
- Implementation Details: The captioner uses a standard autoregressive transformer decoder conditioned on pretrained MoTok tokens.The tokenizer remains fixed during captioning training, with teacher forcing used to optimize cross-entropy loss.
- Quantitative Results: Replacing a conventional VQ tokenizer with MoTok consistently improves motion-to-text captioning performance on HumanML3D.The reported comparison is summarized in Table 8.