PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Controlled Coherent Motion Generation

Yifei Liu, Qiong Cao, Hongwei Yi, Huaiguang Jiang, Changxing Ding

arXiv:2609.11439v1cs.CV

TL;DR

Simultaneous control of 3D-avatar motion by text, speech, and trajectories is difficult because aligned multimodal data are scarce and existing combinations can produce mismatched movements. MOCO uses decoupled diffusion denoising with spatial body-part assembly, and experiments report state-of-the-art performance on multimodal motion generation. Its benchmark is controlled rather than open-ended, and fixed body-part assignments can be suboptimal when controls overlap.

  • Problem

    Simultaneous text, speech, and trajectory control lacks abundant aligned multimodal training data, while existing combination strategies can produce mismatched or imbalanced motions.

  • Method

    MOCO independently generates modality-specific motions at each diffusion denoising step and assembles body parts according to predefined spatial rules.

  • Results

    MOCO delivers state-of-the-art qualitative and quantitative performance and outperforms existing baselines on text-to-motion and speech-to-gesture evaluation.

  • Takeaways & Limitations

    Decoupled denoising enables coherent whole-body motion aligned with multiple conditions without requiring aligned multimodal data.

  • Takeaways & Limitations

    Fixed body-part assignments can be suboptimal for conflicting controls, and the benchmark does not claim generalization to arbitrary text–audio pairs.

Abstract

from arXiv · show

It is natural for humans to walk and talk simultaneously. This paper tackles the challenge of replicating such natural behaviors in 3D avatar motion generation driven by concurrent multimodal inputs, such as a text description of a man walking alongside speech audio. Existing methods, constrained by the scarcity of aligned multimodal data, typically combine motions from individual modalities sequentially or through weighted sums. However, they often result in mismatched or unrealistic movements. To overcome these limitations, we propose MOCO, a novel diffusion-based framework capable of processing multiple simultaneous inputs, including speech audio, text descriptions, and trajectory data, to generate coherent and lifelike motions without requiring aligned multimodal data. Our key innovation lies in decoupling the motion generation process. During each denoising step, the diffusion model independently generates motions for each modality from the input noise and assembles the body parts according to predefined spatial rules. The resulting combined motion is then diffused and serves as the input noise for the subsequent denoising step. This iterative approach enables each modality to refine its contribution within the context of the overall motion, progressively harmonizing movements across modalities. Consequently, the generated motions become increasingly natural and fluid with each iteration, achieving coherent and synchronized behaviors. We evaluate our approach using a purpose-built multimodal benchmark. Experimental results demonstrate that MOCO outperforms existing baselines, advancing the field of multimodal motion generation for 3D avatars.

1 Introduction

MOCO addresses simultaneous text, speech, and trajectory control for realistic 3D-avatar motion without requiring aligned multimodal training data. It uses multimodal inputs to generate coherent whole-body movements and outperforms existing baselines on text-to-motion and speech-to-gesture metrics.

  • MOCO generates realistic, coherent holistic body motion from concurrent text, speech audio, and trajectory controls, including facial expressions and hand gestures.The framework is illustrated through temporally aligned speech gestures and lower-body movements.
  • Scarce aligned multimodal data makes simultaneous control difficult, while weighted-sum and pseudo-label approaches have limitations.Weighted sums can be imbalanced by standing-heavy speech-to-gesture data, and speech-script pseudo-labels limit generalization to diverse motion descriptions.
  • The paper introduces a multimodal benchmark with 1,000 test clips combining movement descriptions and speech audio from multiple speakers.The benchmark uses 40 fundamental text descriptions and 694 audio clips from eight speakers.
  • MOCO outperforms existing baselines on both text-to-motion and speech-to-gesture metrics.The evaluation compares the proposed method with baseline approaches across both modality-specific metric families.

2 Related Work

Prior motion-generation research covers many input modalities and uses diffusion, multimodal fusion, and spatial or temporal composition. However, existing multimodal methods rely heavily on aligned training data or can suppress one condition when combining modality-specific predictions.

  • Motion-generation research spans action labels, text, speech, music, scene context, spatial signals, and another person’s motion.These input conditions motivate expanding beyond single-modality control.
  • Existing multimodal motion models depend heavily on aligned training data, limiting novel combinations such as text with audio or audio with trajectory.Weighted-sum methods can favor speech conditions, while pseudo-label methods introduce a transcript-to-motion-description domain gap.
  • Diffusion methods denoise either directly in motion space or in a VAE latent space, trading editability and controllability against efficiency.MDM operates in original motion space, whereas MLD denoises in VAE latent space; this paper selects MDM for its editability.
  • Motion composition methods combine body parts spatially, sequence actions temporally, or address both axes during diffusion denoising.SINC focuses on spatial composition, TEACH, T2LM, and FlowMDM on temporal composition, and STMC unifies both.

3 Method

MOCO uses diffusion-based denoisers to generate and combine motion streams from text, audio, and trajectory conditions. Its decoupled denoising strategy assigns modalities to body regions and repeatedly re-diffuses the combined motion for coherent generation.

  • 3.1 Preliminary: Motion Diffusion Model: MOCO models motion with diffusion, iteratively predicting clean samples and computing the preceding noisy state until denoising completes.The framework uses MDM as its diffusion-based foundation and classifier-free guidance for conditioning.
  • 3.2 Data Representation and Model Architecture: Four transformer denoisers separately handle text-to-motion, speech-to-gesture, trajectory-to-velocity, and speech-to-details generation.The model represents body pose, facial expressions, hand movements, text, audio, and trajectory data in modality-specific forms.
  • 3.3 Multi-Modal Decoupled Denoising: MOCO approximates joint multimodal denoising by independently generating text-conditioned lower-body and audio-conditioned upper-body motion streams.This factorization follows the observation that speech primarily guides head and arm gestures, while text mainly guides walking and stance-related motion.
  • 3.3 Multi-Modal Decoupled Denoising: At each denoising step, the text- and audio-driven predictions are diffused and assembled with a binary body-part mask into the next body-motion state.The mask assigns lower-body dimensions to the text stream and upper-body dimensions to the audio stream.
  • 3.4 Trajectory and Motion Planning: Trajectory control predicts pelvis linear and angular velocities from trajectory data and applies loss guidance during denoising.The velocity representation contains x- and y-direction pelvis velocities plus angular velocity around the vertical axis.
  • 3.4 Trajectory and Motion Planning: MOCO extends to asynchronous conditions by using motion planning to decompose complex text and audio inputs into timed motion units.The supplied method overview identifies synchronous conditions as signals occurring within the same interval, while the extension addresses more complex timing.

4 Experiments

MOCO is evaluated on multimodal motion generation using text-to-motion and speech-to-gesture metrics, baseline comparisons, ablations, qualitative samples, and user studies. Results show strong concurrent condition alignment, while body-part assignment and iterative combination are important for coherent motion.

  • Result Analysis: MOCO delivers competitive performance across both text-to-motion and speech-to-gesture metrics simultaneously.The comparison evaluates MOCO against Weighted Sum, Pseudo-Text, SynTalker, and adapted STMC baselines.
  • Result Analysis: Weighted Sum and Pseudo-Text perform well on speech-to-gesture metrics but poorly on text-to-motion metrics, revealing limited concurrent multimodal handling.
  • Result Analysis: MOCO produces more natural and better-synchronized motion than adapted STMC, supported by quantitative metrics, naturalness measures, and a user study.
  • Ablation Study: MOCO’s original head-and-arms audio mask with text-controlled lower body outperforms variants that include the spine or legs in the audio-controlled region.Including the spine worsens text-to-motion performance, while including legs and spine significantly degrades text-to-motion metrics and Beat Consistency.
  • Ablation Study: Sharing weights between modalities worsens metrics including R1 and FID-A, suggesting that modality-specific representations are important for accurate and realistic motion.
  • Ablation Study: Combining body parts only at the final denoising step causes temporal discontinuities and reduced body coherence, whereas per-step combination yields more natural synchronized motion.The discontinuities are reflected by high Transition Smoothness Ratio values and are supported by user-study and qualitative evidence.
  • Qualitative Analysis: Qualitative samples show speech-driven upper-body gestures synchronized with jogging, walking, sitting, and other lower-body motions, although foot sliding appears during standing transitions.
  • User Study: MOCO is rated higher than Pseudo-Text and Weighted Sum, and higher than Combine Once in body coherence and temporal fluidity.In a 51-participant comparison with STMC, MOCO is preferred for audio synchronization and naturalness with statistical significance.

5 Discussion and Limitations

MOCO’s fixed spatial assignment can fail when text and audio require control of overlapping body regions. Its benchmark also targets controlled compositional consistency rather than open-ended semantic coverage.

  • Limitations: Fixed body-part assignment can be suboptimal when control signals conflict, such as text instructions for arm movements while audio exclusively controls the upper body.MOCO permits users to configure body-part control through the motion timeline, allowing text to drive the upper body when needed.
  • Limitations: The benchmark focuses on controlled compositional consistency, so it does not claim generalization to arbitrary text–audio pairs.The authors identify adaptive body-part assignment or residual blending as directions beyond fixed spatial rules.

6 Conclusion

MOCO generates coherent holistic motion from concurrent text, speech, and trajectory controls through spatially decoupled denoising. Its task-scoped analysis finds complementary lower- and upper-body motion distributions between text-driven and speech-driven inputs.

  • 6 Conclusion: MOCO uses decoupled denoising to independently generate modality-specific motions and assemble them under predefined spatial rules.The framework targets realistic, coherent whole-body motion from text, speech audio, and trajectory controls.
  • 6 Conclusion: The analysis evaluates 1,000 text-driven sequences and 5,000 BEAT2 speech-driven sequences using normalized body-part activity statistics.The text-driven samples use 40 locomotion- and posture-oriented descriptions.
  • 6 Conclusion: 62.0% of text-driven motion variance is lower-body, compared with 85.1% upper-body variance in BEAT2 speech gestures.Temporal variance is computed by body region and normalized within each sequence.

B Theoretical Analysis for Decoupled Denoising

The theoretical analysis approximates joint denoising by assigning lower-body generation primarily to text and upper-body generation primarily to audio. This factorization relies on assumptions about modality relevance and the information carried across adjacent diffusion states.

  • B Theoretical Analysis for Decoupled Denoising: MOCO approximates joint denoising as the product of text-conditioned lower-body and audio-conditioned upper-body distributions.The current motion b_t contains upper- and lower-body components used at each denoising step.
  • B Theoretical Analysis for Decoupled Denoising: The derivation assumes the current diffusion state sufficiently captures the previous state, allowing lower-body history to be omitted when estimating upper-body motion.The justification is the proximity and strong correlation of adjacent diffusion steps.
  • B Theoretical Analysis for Decoupled Denoising: Text primarily influences lower-body actions, whereas audio predominantly affects upper-body gestures and facial expressions.The factorization excludes audio from lower-body conditioning and text from upper-body conditioning.
  • B Theoretical Analysis for Decoupled Denoising: The motion-planning prompt specifies objectives, output formats, timeline rules, and examples for converting text and audio conditions into motion schedules.The dialogue is augmented with step-by-step reasoning instructions and produces timed body-part assignments.
  • B Theoretical Analysis for Decoupled Denoising: LLM-generated timelines can contain duration-calculation errors, such as miscalculating the second audio segment.The paper states these errors can be resolved through code refinement.

D Conflict Resolution and Flexibility in Multi-Modal Control

MOCO resolves overlapping multimodal controls through a predefined timeline and allows users to reconfigure which modality controls specific body parts. In the example, speech takes precedence by default, while a revised timeline lets text control the arms during overlap.

  • D Conflict Resolution and Flexibility in Multi-Modal Control: Users can reconfigure modality priorities through timeline adjustments to control specific upper-body motions explicitly.This flexibility changes the default assignment of audio to upper body and text to lower body.
  • D Conflict Resolution and Flexibility in Multi-Modal Control: When text commands “waves hands” overlap with speech, the default timeline gives speech precedence and suppresses the text-controlled waving.The overlap occurs from 2.578 to 8.0 seconds.
  • D Conflict Resolution and Flexibility in Multi-Modal Control: During a revised overlap from 7.0 to 11.0 seconds, the text command “wave hands” controls the arms because the condition controlling fewer body parts takes precedence.Speech spans 2.578 to 15.254 seconds, while the text command controls both arms.

E Limitations of Weighted Sum in Multi-Modal Motion Generation

The weighted-sum strategy produces modality-dependent mismatches because speech-driven updates adjust motion more aggressively than text-driven updates. These unequal responses can distort body-part control and motivate MOCO’s decoupled denoising design.

  • E Limitations of Weighted Sum in Multi-Modal Motion Generation: Weighted Sum performs well on speech-to-gesture metrics but poorly on text-to-motion metrics because its modality updates are imbalanced.The experiment computes each model’s perceived mismatch against motion updated by the other modality.
  • E Limitations of Weighted Sum in Multi-Modal Motion Generation: Speech-to-gesture updates compute larger motion differences than text-to-motion updates, indicating more aggressive adjustment to speech conditions.The comparison covers whole-body, arm, and leg differences across the illustrated update scenarios.
  • E Limitations of Weighted Sum in Multi-Modal Motion Generation: For the “sitting” condition, speech-model differences are larger in the legs than the arms, unlike the expected upper-body emphasis of speech gestures.Under “standing,” leg differences are smaller than arm differences, aligning with expectations.
  • E Limitations of Weighted Sum in Multi-Modal Motion Generation: These observations expose weighted-sum limitations in multimodal motion generation and support the validity of decoupled denoising.The passage frames this as a design rationale for the proposed approach.

F Computational Complexity

MOCO is evaluated against per-step composition and stronger diffusion backbones, with emphasis on efficiency and concurrent-control quality. The framework is reported as lightweight and fast while producing natural, synchronized motion.

  • F Computational Complexity: The framework is overall lightweight and sufficiently fast, while GT2V has the longest inference time because guidance improves prediction accuracy.GS2G and GS2D are larger due to additional cross-attention parameters, whereas GT2V is the most lightweight module.
  • F Computational Complexity: MOCO generates a 35-second demo video comprising nine clips in 3.72 seconds, indicating potential for real-time applications.
  • F Computational Complexity: 0.69 s/seq makes MOCO faster than STMC at 0.75 s/seq and the StableMoFusion variant at 0.86 s/seq under concurrent multimodal control.This comparison is reported as preserving MOCO’s naturalness and synchronization advantages without additional inference cost.
  • G.1 Comparison with STMC and Stronger Backbones: MOCO achieves better speech-to-gesture fidelity, boundary smoothness, and MotionCritic scores, although STMC is slightly stronger on pure text-to-motion fidelity.
  • G.1 Comparison with STMC and Stronger Backbones: MOCO produces the most natural and best-synchronized motion under joint text-and-speech control, corroborated by quantitative metrics and a user study.

G.2 Evaluation of Trajectory Control

Trajectory control is evaluated across guidance, dedicated spatial control, and post-hoc optimization, alongside single-modality and perceptual comparisons. L-BFGS gives the best location-oriented trade-off, while MOCO’s joint-control quality is supported by human preference results.

  • G.2 Evaluation of Trajectory Control: Post-hoc L-BFGS on GT2V most effectively reduces location error and modestly improves orientation, so GT2V+L-BFGS remains the default for accuracy and efficiency.CFG can worsen both metrics, while OC+L-BFGS achieves the best orientation at the cost of larger location error and additional inference time.
  • G.2 Evaluation of Trajectory Control: MOCO’s absolute orientation accuracy remains unsatisfactory because its temporally stitchable motion representation accumulates trajectory error.
  • Single-Modality Evaluation: MOCO performs comparably to MDM on HumanML3D text-to-motion and competitively with state-of-the-art methods on BEAT2 speech-to-gesture.
  • G.5 Extended Pairwise-Preference Study against STMC.: MOCO is preferred over STMC for audio synchronization and naturalness, with significant margins of 47.1% vs. 39.4% and 46.1% vs. 39.8%, respectively.Text-alignment preference favors MOCO but is not statistically significant: 45.9% vs. 42.0%, p = 0.11.

H Details of Multi-Modal Benchmark

The multimodal benchmark contains 1,000 automatically generated clips combining curated lower-body movement descriptions with speech audio from multiple speakers. Randomized text and audio timing creates concurrent multimodal conditions.

  • H Details of Multi-Modal Benchmark: The benchmark contains 1,000 test clips, each combining two text descriptions and two audio clips.
  • H Details of Multi-Modal Benchmark: The text set contains 40 manually curated descriptions focused on lower-body movements associated with speech delivery or conversation.Examples include walking, turning, sitting, kneeling, and lying down.
  • H Details of Multi-Modal Benchmark: Audio clips are selected from BEAT2 using eight speakers with IDs below 10 and segmented with a voice activity detector.
  • H Details of Multi-Modal Benchmark: Each clip randomly selects two text descriptions with random durations and two neighboring audio clips, with random start times allowing either modality to begin first.This produces four multimodal intervals for each generated sequence.
Loading 2609.11439v1…