PaperlayerView exact revision on arXiv

Source-linked AI summary

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo

arXiv:2609.04250v1cs.SDcs.CVeess.AS

TL;DR

Spoken dialogue and co-speech motion are typically generated by separate cascaded models, preventing joint optimisation and requiring a second inference pass. Motion-Omni makes facial and full-body motion native outputs of a spoken-dialogue model through hidden-state conditioning and joint training, and Motion-Omni-Q7 matches the teacher cascade on motion quality while responding 5.4× faster.

  • Problem

    Cascaded speech-and-motion systems require a second inference pass and prevent motion objectives from updating speech or dialogue parameters.

  • Method

    Motion-Omni jointly trains an LLM, Speech Generator, and Motion Generator so facial, hand, upper-body, and lower-body motion is generated from speech-producing hidden states.

  • Results

    Motion-Omni-Q7 matches the same-audio teacher cascade on motion quality, responds 5.4× faster, and reaches a 2.62% word error rate, lowest among compared omni-modal systems.

  • Takeaways & Limitations

    The framework supports end-to-end spoken dialogue with synchronized full-body motion while removing a separate audio-to-motion inference stage.

  • Takeaways & Limitations

    Motion quality is bounded by the LOM codebook and teacher-generated pseudo-labels; the model is English-only, non-streaming, and does not condition on explicit speaker identity or emotion.

Abstract

from arXiv · show

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.

1 INTRODUCTION

Motion-Omni addresses the structural limits of cascaded speech-and-motion generation by making full-body co-speech motion a native spoken-dialogue output. Its jointly trained framework and scalable supervision pipeline preserve dialogue ability while improving alignment and inference efficiency.

  • Motivation: Cascaded systems require a second full inference pass and prevent motion supervision from updating speech or dialogue parameters.The paper frames removing both costs as its central research question.
  • Approach: Joint optimisation bridges heterogeneous speech and motion rates while allowing motion supervision to preserve timing cues and speech supervision to retain dialogue ability.The architecture and training procedure must manage rate mismatch and interference between shared objectives.
  • Approach: Motion-Omni natively generates facial expression, hand, upper-body, and lower-body motion from the states that produce speech.Its four-component architecture uses shared representations rather than a rendered waveform as an intermediate input.
  • Results: Motion-Omni-Q7 matches the same-audio teacher cascade on motion quality, responds 5.4× faster, and achieves 2.62% word error rate, lowest among compared omni-modal systems.The result combines motion-quality parity, lower latency, and strong spoken output.
  • Data and evaluation: 422,856 paired samples provide 1,402 hours of quality-ranked motion supervision from a scalable, model-agnostic pseudo-labeling pipeline.The pipeline supports replaceable motion teachers and consistent-voice speech responses.
  • Data and evaluation: The paper releases SwDA-500 and a public evaluation protocol covering matched audio, rendering, automatic metrics, human evaluation, and latency.The protocol targets stochastic open-ended full-body spoken dialogue.

2 RELATED WORK

Related work spans audio-conditioned motion, integrated speech-and-gesture synthesis, spoken dialogue, and open-ended systems that jointly emit speech and articulated motion. Motion-Omni extends these lines by targeting native facial and full-body motion generation within spoken dialogue.

  • Positioning: Motion-Omni is positioned at the intersection of all four research lines: audio-conditioned motion, integrated synthesis, spoken dialogue, and dialogue systems with articulated motion.Its framework is presented as a unified extension of these areas.
  • Audio-conditioned motion: Co-speech motion systems synthesize body motion from supplied speech audio, while full-body models use diffusion, discrete, state-space, or flow-matching representations.The cited line includes TalkShow, Listen, Denoise, Action, EMAGE, MambaTalk, and GestureLSM.
  • Integrated synthesis: Integrated speech-and-gesture work jointly predicts speech acoustics and gesture from prescribed text scripts using neural TTS, diffusion, or parallel prediction heads.These systems apply multimodal co-learning to scripted rather than open-ended dialogue settings.
  • Spoken dialogue models: Spoken dialogue models equip language models with speech input and output through audio-token vocabularies, speech-unit decoders, and flow-matching decoders.The cited systems include SpeechGPT, LLaMA-Omni, GLM-4-Voice, CosyVoice, and Qwen2.5-Omni.
  • Spoken dialogue models: Standalone spoken dialogue models do not produce body motion.This separates their capabilities from integrated spoken-motion systems.
  • Open-ended spoken motion: Open-ended joint systems generate combinations of response text, speech tokens, body or hand motion, and facial animation, but prior systems omit native facial output or generate it post hoc.SOLAMI generates facial animation with an audio-to-face model, while U-Mind does not specify native facial-expression output.

3 METHOD

The method combines a modular speech-and-motion architecture with hidden-state conditioning, staged joint training, and quality-ranked teacher supervision. Motion decoders operate at motion rate while receiving speech-generation representations and interpolated speech-token queries.

  • System overview: Motion-Omni-Q7 uses frozen Whisper-large-v3 features, a speech projector, a Qwen2.5-7B-Instruct backbone, a Speech Generator, and four per-part Motion Generator decoders.The four motion parts are face, hands, upper body, and lower body.
  • System overview: The Speech Generator emits GLM-4-Voice discrete speech units at 12.5 Hz, while motion decoders emit LOM VQ codes at 30 Hz.The decoders separately model face, hands, upper body, and lower body.
  • Motion conditioning: Motion decoders condition on Speech Generator hidden states as keys and values plus learned speech-token queries interpolated from 12.5 Hz to 30 Hz.All four decoders share the conditioning streams but have separate parameters.
  • Motion conditioning: Conditioning on hidden states rather than decoded waveforms removes the separate audio-to-motion inference stage while retaining acoustic timing and response semantics.Speech-token queries identify what is being said, while hidden states also carry speaker timbre.
  • Training objective: The method weights per-part cross-entropy so normalized motion-code predictions receive equal importance per underlying SMPL-X and FLAME feature dimension.The per-part weights sum to one.
  • Training: Training proceeds through ASR, TTS, TTS-with-Motion curriculum, and joint-mixture stages that progressively co-adapt the speech and motion pathways.The curriculum exposes Stage 3 to progressively larger quality quantiles, and joint training is needed after a frozen-Speech-Generator pilot plateaued.
  • Data construction: A replaceable motion teacher supplies four per-part VQ-code streams, while reconstruction error and beat correlation rank samples for curriculum selection.The resulting data construction pipeline supports quality-ranked supervision and Stage 4 pool selection.

4 EXPERIMENTS

Experiments evaluate Motion-Omni-Q7 on speech quality, motion quality, human preference, spoken-dialogue ability, and latency using SwDA-500 and Seed-TTS-Eval. The model preserves dialogue performance, performs strongly against cascades, and avoids the latency cost of a separate motion-inference stage.

  • Spoken-dialogue ability: Motion-Omni-Q7 reaches 47.63 Overall on VoiceBench, above LLaMA-Omni, Ex-Omni, Mini-Omni2, and Moshi.VoiceBench covers open-ended dialogue, factual QA, reasoning, instruction-following, and safety.
  • Speech quality: 2.62% WER on Seed-TTS-Eval is the lowest among the compared omni-modal LLMs.WER is computed on 1,088 English samples using Whisper-large-v3; dedicated speech-only TTS systems can report lower WER.
  • Automatic motion evaluation: Motion-Omni-Q7 places first or second on seven of eight SwDA-500 metrics and leads non-teacher cascades in beat correlation, diversity, facial metrics, and lip-sync.It records beat correlation 7.59 and diversity 13.67; teacher-reference cascades are the rows that surpass it on some comparisons.
  • Human evaluation: Against MO-audio + EMAGE, Motion-Omni-Q7 has a pooled margin of +25, while against MO-audio + LOM the outcome is 25/27/23 and ties are most frequent.The comparisons hold speech fixed, isolating the motion pathway; the authors do not treat smaller margins as reliable preferences.
  • Latency: 4.32 s response time and RTF = 0.78 make Motion-Omni-Q7 faster than real time, while MO-audio + LOM takes 23.35 s and is 5.4× slower.Measurements are offline response-generation times on SwDA-500 using a single GPU.
  • Latency-quality trade-off: MO-audio + EMAGE matches Motion-Omni-Q7’s response time but gives up 0.27 BC, whereas MO-audio + LOM exceeds its BC by 0.08 at 5.4× the response time.The joint comparison places response time and beat correlation on two axes.

5 CONCLUSION

Motion-Omni jointly generates intelligible speech and synchronised full-body co-speech motion, while its reference instance achieves strong motion, lip-synchronisation, and speech results without inference-time motion-teacher cascades.

  • Motion-Omni-Q7 records the lowest word error rate among compared omni-modal systems and the highest beat correlation and diversity among non-teacher systems on SwDA-500.It also achieves better rendered-video lip synchronisation than the evaluated EMAGE and MambaTalk cascades.
  • Within about 1% of the same-audio teacher cascade on reference-free motion metrics, Motion-Omni-Q7 removes the separate audio-to-motion stage.The paper attributes the 5.4× response-time difference to removing that stage.
  • The framework combines jointly generated speech and motion with a model-agnostic pseudo-labeling pipeline and reproducible evaluation setup.All components are swappable, enabling additional instances with stronger backbones or teachers.

LIMITATIONS

The reported system is bounded by its motion representation, training data, language and interaction setting, and by limitations in the current evaluation evidence.

  • Motion quality is bounded by the LOM VQ-VAE codebook and teacher-generated pseudo-labels, so motion outside that distribution cannot be expressed.Only the Motion-Omni-Q7 instance was trained; stronger backbones, larger teachers, and continuous motion heads remain future work.
  • The model uses entirely English training data and does not condition on explicit speaker identity or emotion, restricting transfer across languages, cultures, and speaker-specific styles.The avatar uses a single fixed voice and generic body mesh rather than imitating a specific person.
  • Motion-Omni-Q7 ingests the user utterance fully before responding, making it an offline response generator rather than a streaming interaction model.Low-latency streaming behavior requires a different design and evaluation protocol.
  • Existing automatic motion metrics are imperfect proxies for perceived naturalness, while the four-annotator human preference comparison is exploratory.The absence of a larger non-author human study remains a limitation.

A.4 STAGE-WISE TRAINING HYPERPARAMETERS

The appendix specifies the four-stage training configuration, hardware, model scale, objective composition, and task-specific supervision masks for Motion-Omni-Q7.

  • Stage 4 starts from the best Stage 3d checkpoint, with the complete stage-wise settings organized by rows and training stages by columns.
  • Full four-stage training takes approximately 960 GPU-hours on 4 GPUs using DeepSpeed ZeRO-2, BF16 mixed precision, gradient checkpointing, and effective batch size 128.
  • Motion-Omni-Q7 has approximately 8.3B parameters, dominated by the 7.6B Qwen2.5-7B-Instruct LLM backbone.The Speech Generator contributes approximately 0.5B parameters and the Motion Generator approximately 150M.
  • The per-batch objective combines LLM, speech-unit, and four per-part motion cross-entropies, with motion weights proportional to feature dimensions.The four motion parts are face, hand, upper body, and lower body.
  • Task masks assign LLM-only supervision to ASR and text-to-text samples, while TTS, TTSM, and S2SM add speech supervision and TTSM/S2SM add motion supervision.

B.1 WEIGHTED VQ-VAE RECONSTRUCTION ERROR

The weighted VQ-VAE reconstruction error measures how faithfully frozen LOM codebooks represent motion, while beat correlation measures speech-motion rhythmic alignment; together they filter supervision quality.

  • The weighted VQ-VAE reconstruction error measures fidelity between continuous motion and motion reconstructed through the frozen LOM VQ-VAE codebooks.It aggregates per-part mean absolute errors using feature-dimension weighting.
  • A high Lvq1 indicates that VQ quantisation poorly represents the intended motion and can inject noisy label supervision.
  • Low Lvq1 alone is insufficient because faithfully represented motion may still be poorly aligned with the driving speech.Beat correlation supplies the complementary speech-motion coupling criterion.
  • Beat correlation extracts upper-body kinematic beats and audio onsets, then scores their forward audio-to-motion temporal proximity with σ = 9 frames.The bidirectional diagnostic is exploratory rather than a validated perceptual metric: it agrees with 46% of human rhythm verdicts versus 40% for standard BC, with neither separating from chance on the small set.
  • The combined quality score robustly maps each metric’s 5th percentile to 0 and 95th percentile to 1, clips outside values, and inverts Lvq1 so higher values indicate higher quality.Samples missing either metric are excluded from ranking and curriculum participation.

D HUMAN EVALUATION PROTOCOL

The protocol evaluates conversational motion through blinded paired comparisons of rendered video and audio. It separately assesses rhythm, semantic alignment, and body naturalness across matched system arms, with ties allowed and majority-vote verdicts.

  • Evaluation design: 25 clips per comparison arm are drawn from SwDA-500, covering short and long responses while excluding degenerate responses and missing motion files.The evaluation uses four comparison arms and 100 total pairs.
  • Evaluation design: The four arms compare Qwen2.5-Omni cascades and Motion-Omni audio with LOM or EMAGE motion, isolating motion in the same-audio arms.B1 and B2 differ in both speech and motion from Motion-Omni-Q7; B3 and B4 share its audio.
  • Rating procedure: Four annotators independently rate each pair on rhythm, semantic alignment, and body naturalness using rendered video and audio.System assignment is randomised, and per-pair verdicts use majority vote.
  • Rating dimensions: Rhythm judgments examine timing with stressed words, pauses, and speech rate, while semantic alignment requires gestures that convey spoken content.Generic rhythmic waving does not count as semantic alignment, and ties are allowed when neither side shows clear semantic gestures.
  • Rating dimensions: Body naturalness judgments penalize mechanical artefacts and posture problems while considering relaxed conversational motion rather than amplitude alone.Annotators check jitter, interpenetration, joint pops, foot sliding, frozen poses, exaggerated presentation stances, and natural idle motion.

F SPEECH NATURALNESS PROXY

The study uses UTMOSv2 as an automatic proxy for speech naturalness on SwDA-500, while noting that it is not a listening test or a like-for-like competitive comparison. Motion-Omni-Q7 scores 3.77, below Qwen3-TTS and CosyVoice 3 but above several listed systems.

  • Metric and scope: UTMOSv2 is reported as a lightweight speech-naturalness check because WER measures intelligibility but does not replace a listening test.The reference systems are dedicated TTS models that synthesise supplied sentences rather than plan dialogue.
  • Results: Motion-Omni-Q7 scores 3.77, above GLM-TTS, VoxCPM1.5, F5-TTS, and CosyVoice, but below Qwen3-TTS at 4.05 and CosyVoice 3 at 3.92.The comparison is presented as contextual rather than a fully like-for-like ranking.

G FUSION AND CURRICULUM ABLATIONS

The ablations examine motion-conditioning fusion and curriculum training under controlled settings. Speech Generator context and gated fusion improve the reported beat-correlation comparison, while FGD decreases across curriculum stages and remains lowest after joint fine-tuning.

  • Fusion ablation: Speech Generator context is more useful than direct LLM context for the same TQGF operator, and gated fusion is modestly better than plain cross-attention.These are scoped comparisons under the fixed Stage 3a setup and do not isolate every internal TQGF component.
  • Curriculum ablation: FGD decreases through all curriculum stages and remains lowest after Stage 4 joint fine-tuning.The analysis uses the same LOM latent encoder and teacher-reference definition as the main motion evaluation.
  • Quality–latency view: Figure 2 compares motion quality against response latency, with cascades shown as circles, Motion-Omni as a star, and RTF = 1 marking real-time response.Systems nearer the top-left are better, and values below RTF = 1 are faster than real time.

H NEGATIVE RESULT: VIDEO LLM-AS-JUDGE CALIBRATION

The video LLM-as-judge calibration does not provide reliable motion evaluation, so the paper removes it from the main protocol. Low correlations, systematic judge biases, and weak pairwise agreement motivate retaining human A/B/tie ratings instead.

  • Absolute scoring: The best absolute-scoring iteration reaches only r = +0.32 for speech quality, r = +0.18 for body naturalness, and r = +0.17 for multi-turn coherence.R1 and R2 remain at r ≤0, while average pilot-rubric correlation reaches only +0.05.
  • Failure modes: The calibration is limited by a 26-clip × 1–5 scale, judge ceilings, systematic over-rating of large-amplitude motion, and human disagreement about semantic alignment.The reported per-dimension correlation confidence interval is roughly ±0.4.
  • Pairwise ranking: Pairwise judging agrees with human aggregate rankings on 4/13 pairs, with Cohen’s κ = 0.079, and performs worse than the 5/13 absolute-score baseline.Because the clips share audio, pairwise judgments are dominated by body naturalness, where the judge has its largest systematic bias.
  • Protocol decision: The main evaluation removes the video LLM-as-judge and relies on human R1–R3 A/B/tie ratings.The authors plan to release calibration data, prompts, and inter-annotator-agreement tooling.
  • Visual materials: Figures 3 and 4 use released rendered clips, with Figure 3 selecting temporally separated, mutually distinct poses and Figure 4 matching timestamps across motion sources.Figure 4 keeps the response audio and rendering pipeline fixed so the motion stream is the only variable.
Loading 2609.04250v1…