PaperlayerView exact revision on arXiv

Source-linked AI summary

TLControl: Trajectory and Language Control for Human Motion Synthesis

Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, Lingjie Liu

arXiv:2311.17135v4cs.CVcs.GR

TL;DR

TLControl addresses the challenge of generating human motion that simultaneously follows language semantics and precise, partial joint trajectories. It combines a body-part VQ-VAE, a trajectory-and-language-conditioned MTT, and test-time latent optimization; experiments report superior trajectory accuracy and time efficiency versus existing methods, supporting interactive high-quality animation generation.

  • Problem

    Existing motion-generation methods often provide either language control or trajectory control, leaving precise multi-joint synthesis from partial trajectories insufficiently supported.

  • Method

    TLControl learns a body-part-structured VQ-VAE space, predicts coarse motion latents with an MTT conditioned on language and partial trajectories, and refines them through test-time optimization.

  • Results

    TLControl reports superior trajectory accuracy and time efficiency compared with existing methods while producing motions aligned with language semantics and user trajectories.

  • Takeaways & Limitations

    The framework supports interactive generation and modification of high-quality human animations from flexible trajectory specifications, including analytical shapes, hand drawings, and environment settings.

  • Takeaways & Limitations

    The method assumes that language and trajectory controls do not conflict; handling conflicting inputs remains outside the paper’s scope.

Abstract

from arXiv · show

Controllable human motion synthesis is essential for applications in AR/VR, gaming and embodied AI. Existing methods often focus solely on either language or full trajectory control, lacking precision in synthesizing motions aligned with user-specified trajectories, especially for multi-joint control. To address these issues, we present TLControl, a novel method for realistic human motion synthesis, incorporating both low-level Trajectory and high-level Language semantics controls, through the integration of neural-based and optimization-based techniques. Specifically, we begin with training a VQ-VAE for a compact and well-structured latent motion space organized by body parts. We then propose a Masked Trajectories Transformer (MTT) for predicting a motion distribution conditioned on language and trajectory. Once trained, we use MTT to sample initial motion predictions given user-specified partial trajectories and text descriptions as conditioning. Finally, we introduce a test-time optimization to refine these coarse predictions for precise trajectory control, which offers flexibility by allowing users to specify various optimization goals and ensures high runtime efficiency. Comprehensive experiments show that TLControl significantly outperforms the state-of-the-art in trajectory accuracy and time efficiency, making it practical for interactive and high-quality animation generation.

1 Introduction

TLControl targets realistic human motion synthesis with both high-level language semantics and precise low-level trajectory control. It combines structured motion representations, neural prediction, and optimization to support accurate, efficient, and flexible animation generation.

  • 1 Introduction: The method addresses the need to combine semantic text control with flexible, precise spatial control over arbitrary joints and trajectories.Existing approaches commonly support language or trajectory control separately, while inverse kinematics generally requires complete end-effector trajectories and lacks language semantics.
  • 1 Introduction: TLControl combines a body-part-structured VQ-VAE, a Masked Trajectory Transformer, and optimization-based refinement for partial trajectory and language-conditioned synthesis.The transformer provides coarse predictions, which optimization refines to align with user-specified trajectories.
  • 1 Introduction: The framework supports flexible optimization goals while maintaining high runtime efficiency compared with state-of-the-art methods.Its design is intended for interactive generation and modification of high-quality animations within a brief runtime.
  • 1 Introduction: TLControl generates high-quality motion aligned with language semantics while accurately following low-level trajectories and outperforming existing methods in trajectory accuracy and time efficiency.Users can specify trajectories through analytical-shape parameters, 3D hand drawings, or environment settings, enabling interactive animation generation.

2 Related Work

Related work spans unconditional and conditional human motion generation, including language, trajectory, and other multimodal controls. Prior trajectory methods provide spatial guidance but remain limited in joint coverage, dimensionality, or integration with language semantics.

  • 2 Related Work: Conditional motion generation incorporates text, action labels, partial motion, control signals, music, and images as diverse inputs.The literature review covers unconditional generation and conditional approaches, with particular attention to language-conditioned and trajectory-controlled synthesis.
  • 2 Related Work: Language-conditioned methods have advanced text-to-motion synthesis, while trajectory-controlled methods address spatial guidance through joint or path constraints.These strands represent complementary forms of conditional human motion generation.
  • 2 Related Work: GMD controls only 2D pelvis positions, limiting its flexibility for practical multi-joint motion control.The cited work uses a two-stage diffusion strategy for spatial constraints but remains restricted in the controlled body region and dimensionality.
  • 2 Related Work: OmniControl introduces flexible spatial control over different joints using analytic spatial and realism guidance, but the surrounding literature motivates broader integration of trajectory and language controls.The related-work discussion contrasts language-only, trajectory-only, and more flexible spatial-control approaches.

3 Method

TLControl learns a compact body-part motion representation, predicts coarse full-body latent motions from language and partial trajectories, and refines them through test-time optimization. This staged design supports incomplete user controls while preserving motion structure and semantic coherence.

  • 3 Method: The method assumes feasible partial trajectories and no conflict between language and trajectory controls when generating full-body motion.At each frame, users may specify positions for a subset of joints, potentially changing the controlled joints over time.
  • 3 Method: TLControl first learns a part-based VQ-VAE representation, then trains an MTT for coarse completion, and finally optimizes the latent motion to match control trajectories.The framework separates representation learning, conditional prediction, and trajectory refinement into successive stages.
  • 3.1 Part-based VQ-VAE: The part-based VQ-VAE disaggregates motion into six groups—head, left arm, right arm, left leg, right leg, and root—to learn compact, structured motion priors.Separate encoders and codebooks represent the joint groups, while a decoder combines their quantized features to reconstruct full-body motion.
  • 3.1 Part-based VQ-VAE: Each group is encoded into temporally downsampled features and quantized by selecting the nearest codebook vector.The quantization step maps each feature q_t,k to the closest code c_i,k, while training uses quantization, commitment, and reconstruction losses.
  • 3.2 Masked Trajectory Transformer: The MTT uses text embeddings and randomly masked joint trajectories to predict a distribution over latent code indices for incomplete user specifications.Masking trains the network to model relationships among motion, joint trajectories, and textual information when complete trajectories are unavailable.
  • 3.3 Test-Time Optimization: At test time, optimization minimizes projected joint-position error against the supplied trajectories, starting from the MTT prediction and decoding the refined latent motion.The projection function converts the decoded pose into global positions for the controlled joints.
  • 3.3 Test-Time Optimization: The compact body-part embedding enables efficient refinement while preserving desired controllability and semantic coherence with the text description.The optimization framework is presented as flexible because users can specify different control goals.

4 Experiment

TLControl is evaluated for motion quality, controllability, accuracy, efficiency, latent-space structure, and optimization behavior across HumanML3D and KIT-ML. Experiments show flexible control from diverse inputs, accurate multi-joint synthesis, strong comparisons with prior methods, and a runtime–accuracy trade-off during optimization.

  • Controllable Human Motion Generation: The method supports separate language and joint-trajectory control, simultaneous multi-joint control, 3D hand-drawn trajectories, and environment-based foot placements.Qualitative experiments use analytical shapes, 3D hand drawings, and environment settings as control trajectories.
  • Comparison with SOTAs: TLControl consistently surpasses competing methods on control-accuracy metrics and remains within a 50 cm range of intended signals on HumanML3D.It also outperforms alternatives on KIT-ML for pelvis-only and averaged individual-joint control.
  • Comparison with SOTAs: TLControl handles all-six-joint control effectively, whereas Omnicontrol struggles with multi-joint inputs while producing realistic motions.The comparison includes qualitative results in Figures 5, 7, and 8.
  • Comparison with SOTAs: TLControl consistently outperforms existing methods in runtime, attributed to its structured compact latent space and efficient optimization framework.Runtime is measured as average processing time per HumanML3D test sample on an RTX3080Ti GPU.
  • Ablation Study: The learned part-based latent space preserves language semantics while capturing diverse motions under each text condition.t-SNE visualizations use ten randomly selected text descriptions, and Figure 10 varies body-part latent codes.
  • Ablation Study: Stricter optimization accuracy lowers joint error but increases optimization time, motivating the fixed accuracy criterion of 1E-6.The study reports average errors across all six controlled joints under varying optimization criteria.

5 Conclusion

TLControl combines joint trajectories and language inputs through a structured VQ-VAE latent space, a Masked Trajectory Transformer, and optimization-based refinement. The paper concludes that this framework enables fast, high-fidelity interactive motion generation while leaving conflicting language and trajectory inputs unresolved.

  • 5 Conclusion: TLControl combines joint trajectories and language inputs with a morphology-aware VQ-VAE latent space, a Masked Trajectory Transformer, and efficient optimization.The optimization framework allows users to specify various optimization goals.
  • 5 Conclusion: The framework enables users to quickly generate high-fidelity human motion interactively.This conclusion is supported by the paper’s extensive experiments.
  • 5 Conclusion: Conflicting language and trajectory inputs remain outside the paper’s scope, which assumes that the two controls do not conflict.The authors identify conflict detection and user guidance as future research directions.

Supplementary Material

The supplementary material provides additional ablations, control experiments, runtime analyses, implementation details, and qualitative results. It also includes a video with further visual comparisons.

  • Supplementary Material: The supplementary material covers ablations, multi-joint and incomplete-trajectory experiments, runtime analyses, IK comparisons, single-modality results, implementation details, and additional qualitative results.These materials are organized across Sections A–H.
  • Supplementary Material: A supplementary video provides additional visual results and qualitative comparisons with other methods.The authors encourage readers to view it for more comparisons and results.

A Ablation Study on Part-based VQ-VAE

The ablation compares a body-part-disentangled VQ-VAE with an unsplit counterpart, showing that the part-based embedding improves motion processing efficiency and supports real-time optimization.

  • A Ablation Study on Part-based VQ-VAE: The part-based VQ-VAE encodes motions into a more compact latent space than the unsplit design.Both models use identical full-body feature dimensions for runtime optimization, supporting a controlled comparison.
  • A Ablation Study on Part-based VQ-VAE: 27.3% lower average motion-synthesis processing time is achieved with the part-based embedding than with the unsplit VQ-VAE embedding.The comparison covers various control strategies and attributes the gain to the optimized structure of the part-based embedding.

B Multi-Joint Control Results

TLControl evaluates multi-joint control, trajectory incompleteness, and runtime across joint combinations, demonstrating adaptable motion generation while exposing higher cost for complex hand control.

  • B Multi-Joint Control Results: Adding control trajectories produces increasingly detailed motion representations that support realistic multi-joint results.The method is evaluated on the HumanML3D test set across different joint groups.
  • B Multi-Joint Control Results: Performance degrades gracefully as control trajectories become more incomplete under increasing random masking.With full joint trajectories, previous methods [62] report FID 2.614 and R-precision 0.606.
  • B Multi-Joint Control Results: Optimization using trajectories for both hands is the most time-consuming among the tested joint-control strategies.The runtime comparison records average batch-processing time for different combinations of joint trajectories.

E Ablation with IK Based Solution

The IK ablation compares joint-level adjustment with TLControl’s latent optimization and situates the method alongside single-modality generation examples.

  • E Ablation with IK Based Solution: IK is limited in preserving whole-body motion semantics when adjusting individual joints.TLControl instead optimizes latent features from decoded motions, while the comparison controls the root and left hand under different mask rates.
  • E Ablation with IK Based Solution: For incomplete trajectories, IK cannot guide frames without trajectory observations, causing discrepancies from the intended control goals.The comparison evaluates unoptimized decoded motions, IK-optimized motions, and latent-feature optimization.
  • E Ablation with IK Based Solution: TLControl also supports motion generation with language alone or trajectories alone, in addition to its usual combined conditioning.Figure 4 provides qualitative examples for the two single-modality settings.

G Implementation Details

The implementation combines convolutional residual VQ-VAE components, CLIP-based text processing, grouped trajectory tokens, scheduled masking, and L-BFGS runtime optimization.

  • G Implementation Details: The part-based VQ-VAE uses 1D convolutions, residual blocks, ReLU activations, and temporal downsampling by a factor of 4.Each joint encoder uses a 126-vector codebook with 126-dimensional codes.
  • G Implementation Details: The masked trajectory transformer processes text with frozen CLIP-ViT-B/32 features and groups every four closest waypoints into one temporal token.The text representation is 512-dimensional, and trajectory tokenization matches the VQ-VAE downsampling rate.
  • G Implementation Details: Masking uses continuous-trajectory or joint-level strategies, with continuous masking increasing from 0% to 75%.Each batch iteration has a 50% probability of selecting either masking mode.
  • G Implementation Details: Runtime refinement uses Limited-Memory BFGS with a 0.1 learning rate, 1E-6 precision target, and at most 1000 iterations.The optimizer maintains an update history of 200 and uses the strong_wolfe line-search condition.

H More Qualitative Comparisons on Controllable Motion Synthesis

Figure 5 presents additional qualitative results for TLControl, with further demonstrations and method comparisons available in the supplementary video.

  • Figure 5 provides additional qualitative examples of TLControl’s controllable motion synthesis.
  • The supplementary video contains more demonstrations and comparisons with other methods.
  • TLControl combines accurate trajectory adherence, reduced runtime, and motions that reflect text-prompt semantics.
Loading 2311.17135v4…