PaperlayerView exact revision on arXiv

Source-linked AI summary

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Yizhao Li, Pusen Gao, Ming Wang, Shaojie Shen, Shuo Yang, Hao Xu

arXiv:2609.39575v1cs.ROcs.AI

TL;DR

Full-body humanoid co-speech generation must coordinate speech prosody, linguistic content, and robot embodiment, while paired robot-space training data and broad evaluation are not consistently available. ECHO-G jointly conditions a rectified-flow generator on audio and timed transcripts; benchmark comparisons support direct robot-space generation and joint audio–text conditioning for the evaluated co-speech characteristics.

  • Problem

    Full-body humanoid co-speech generation must coordinate acoustic cues and linguistic content while respecting embodiment, and paired audio–text–robot data with broad evaluation are not consistently available.

    Source: §I · pp. 1–2

  • Method

    ECHO-G uses rectified-flow SGDiT to generate full-body robot motion from speech audio and timed transcripts, preserving frame-level acoustic and token-level linguistic conditions.

    Source: §I · p. 2

  • Results

    Benchmark comparisons support direct robot-space generation and joint audio–text conditioning for the reported co-speech characteristics, with direct generation requiring less processing time.

    Source: B. Quantitative Results · p. 6, §V · p. 7

  • Takeaways & Limitations

    The released dataset, benchmark, and code support reproducible research on humanoid co-speech generation, and the framework is demonstrated on a physical humanoid.

    Source: §I · p. 2, §VI · p. 8

    The system requires complete speech audio and timed transcripts and does not yet support causal streaming generation.

    Source: §V · p. 8

Abstract

from arXiv · show

Generating full-body co-speech motion for humanoid robots requires coordinating speech prosody, linguistic content, and embodiment-specific motion. To this end, we present ECHO-G, a framework jointly conditioned on speech audio and timed transcripts. Its Speech-Grounded Diffusion Transformer (SGDiT) combines frame-aligned acoustic features with token-level linguistic context, preserving their distinct granularities. Trained with rectified flow matching, it models one-to-many utterance-motion relationships directly in robot space. To support training and evaluation, we introduce a BEAT2-derived audio-text-robot dataset and a benchmark covering co-speech characteristics, robot-motion quality, and runtime efficiency. Comparative evaluation supports direct robot-space generation over the evaluated human-motion generation and retargeting pipelines, while modality ablations highlight the benefits of joint audio-text conditioning. We further demonstrate deployment on a physical humanoid robot. A complementary video-rating study also favors joint conditioning over the alternatives. The dataset and training, inference, and evaluation code are available through our project page.

I. INTRODUCTION

ECHO-G addresses the need for full-body humanoid motion that reflects both how speech is spoken and what it conveys, while respecting robot embodiment. It combines audio and timed transcripts in a robot-space generator and provides paired data and a benchmark for evaluation.

Motivation

  • Existing humanoid co-speech work leaves a need for full-body robot generation that combines dense acoustic cues with token-level language and paired audio–text–robot data and evaluation.Human co-speech methods primarily generate human-motion representations, while existing public releases do not consistently provide paired training data and a benchmark spanning co-speech characteristics, robot-motion quality, and runtime.Source: §I · p. 2

Framework

  • ECHO-G uses SGDiT to generate full-body robot motion from speech audio and timed transcripts, preserving frame-level acoustic and token-level linguistic features in a rectified-flow model.The model supports sampling different motions for the same utterance, and a fixed pretrained whole-body tracker executes generated joint-position references.Source: §I · p. 2

Dataset and benchmark

  • The authors construct and release a BEAT2-derived robot-space dataset and training, inference, and evaluation code, with a benchmark spanning co-speech characteristics, robot-motion quality, and runtime.The dataset is produced through retargeting and embodiment-specific quality filtering.Source: §I · p. 2

Evaluation and deployment

  • The framework is demonstrated on a physical humanoid robot.The paper also describes pipeline comparisons, modality ablations, video ratings, and physical demonstrations as complementary evaluation evidence.Source: §I · p. 2

II. RELATED WORK

Prior work spans whole-body motion generation, robot co-speech gestures, and human co-speech synthesis and evaluation. ECHO-G combines frame-aligned audio and token-level transcript features for robot-space generation, followed by whole-body tracking.

A. Humanoid Whole-Body Motion Generation

  • Robot co-speech methods include transcript-conditioned upper-body gestures, speech generation with generic prompts, and audio–text robot-native generation; ECHO-G instead targets full-body robot-space motion with distinct frame- and token-level conditions.Earlier approaches variously omit utterance-specific audio–text conditioning, focus on upper-body motion, or do not fully specify multimodal temporal organization.Source: A. Humanoid Whole-Body Motion Generation · p. 2

B. Holistic Human Co-Speech Motion Generation

  • Holistic human co-speech methods synthesize human-motion representations, while evaluation commonly considers distributional fidelity, motion variation, and speech–motion alignment.The benchmark adapts these evaluation dimensions to robot motion and adds robot-motion quality and runtime measures.Source: B. Holistic Human Co-Speech Motion… · p. 2

III. METHOD

  • ECHO-G combines speech audio and timed transcripts in a rectified-flow model, then sends generated joint-position references to a fixed whole-body motion tracker for execution.The architecture combines frame-aligned acoustic features with noisy motion frames and uses global and temporally biased text attention.Source: §III · p. 3, Figure 2 · p. 3

A. Problem Formulation and Motion Representation

ECHO-G models full-body robot motion as a conditional distribution given speech audio and a word-timed transcript. Each frame is represented by 39 features, and absolute root translation is omitted to make the target invariant to global position offsets.

A. Problem Formulation and Motion Representation

  • ECHO-G models a conditional distribution over full-body robot-motion sequences given speech audio and a word-timed transcript.Source: A. Problem Formulation and Motion… · p. 3
  • Each 39-feature frame contains base orientation, inter-frame yaw increment, yaw-aligned local base velocity, and 29 ordered robot joint angles.Absolute root translation is omitted, making the learning target invariant to global position offsets.Source: A. Problem Formulation and Motion… · p. 3
  • Motion features are normalized using feature-wise means and standard deviations from the training split, then denormalized before evaluation or execution.Source: A. Problem Formulation and Motion… · p. 3

B. Audio–Text Conditioning

ECHO-G preserves the distinct frame-level and token-level organization of acoustic and transcript conditions. SGDiT combines them with motion features through acoustic conditioning and global–local transcript cross-attention.

B. Audio–Text Conditioning

  • Acoustic features are interpolated to motion frames, while transcript tokens retain token-level organization and approximate timed intervals.Separate projections map the acoustic and linguistic sequences to a shared hidden dimension.Source: B. Audio–Text Conditioning · p. 3

C. Speech-Grounded Diffusion Transformer

  • Each motion token combines a projected motion frame, its aligned acoustic condition, and a positional embedding before bidirectional temporal self-attention.The self-attention exchanges information across the acoustically conditioned motion sequence.Source: C. Speech-Grounded Diffusion Transformer · p. 3
  • Transcript cross-attention provides global content-based access and a local path that adds a preference for temporally nearby tokens.The local path adds a Gaussian temporal prior to shared content scores; both paths use the same transcript features.Source: C. Speech-Grounded Diffusion Transformer · p. 4
  • The global and local attention weights are mixed using temporal support, reducing the local contribution when a frame is distant from offset-adjusted token centers.The figure shows global weights, the clipped Gaussian log-prior, and the resulting local weights.Source: C. Speech-Grounded Diffusion Transformer · p. 4, Figure 3 · p. 4

D. Training Objective

SGDiT is trained with rectified flow matching, with an additional temporal term that supervises adjacent-frame differences in the predicted velocity.

D. Training Objective

  • Training samples Gaussian noise and a sequence-level flow time, then interpolates the normalized motion toward the noise to define a target velocity.The interpolated motion and target velocity are used for rectified flow matching.Source: D. Training Objective · p. 4
  • The predicted velocity is supervised by matching both the target flow and its adjacent-frame differences.The temporal term is weighted by λtemp.Source: D. Training Objective · p. 4
  • The temporal loss uses only adjacent pairs of valid frames, while the mean-squared errors are averaged over feature dimensions and valid frames.∆τ denotes first differences along the motion-frame axis.Source: D. Training Objective · p. 4

E. Inference and Tracking

At inference, SGDiT generates a motion sequence whose length follows the input clip duration at the motion frame rate, then converts the final state into robot-motion references for execution alongside speech.

E. Inference and Tracking

  • The output length is set by clip duration at the motion frame rate, and generation starts from standard Gaussian noise before integrating the velocity field from flow time 0 to 1.The model uses K uniform Euler steps while keeping the condition fixed.Source: E. Inference and Tracking · p. 4
  • The final generated state is denormalized into robot-motion references, whose joint angles are supplied to the fixed SONIC tracker alongside speech playback.The references are provided as joint-position references.Source: E. Inference and Tracking · p. 4
  • In the selected out-of-dataset frames, joint audio–text conditioning shows broader arm extensions than text-only or audio-only conditioning.The compared rows use matched frame indices; unimodal outputs generally keep the hands closer to the torso.Source: Figure 4 · p. 5

IV. EXPERIMENTS AND RESULTS

The dataset is constructed from BEAT2 recordings retargeted to Unitree G1 robot motion and filtered for motion quality and audio–motion duration consistency.

IV. EXPERIMENTS AND RESULTS

  • The BEAT2-derived dataset contains utterance clips retargeted to the 29-DoF Unitree G1, with robot motion kept at native 30 fps.The 22,192 utterance-level clips retain audio, transcripts, SMPL-X motion, and word timestamps before filtering.Source: 1) Datasets and Preprocessing · p. 5
  • Robot-motion quality and cross-modal consistency checks filter pairs, including a requirement that audio and motion durations differ by no more than one motion frame.Quality checks cover foot contact, self-collision, joint continuity and limits, smoothness, and severe high-frequency artifacts.Source: 1) Datasets and Preprocessing · p. 5
  • The speaker-held-out split uses three English speakers for validation and yields 14,987 training clips and 3,242 validation clips after filtering.The remaining speakers are used for training.Source: 1) Datasets and Preprocessing · p. 5

2) Evaluation Metrics:

The evaluation uses separate measures for co-speech characteristics, robot-motion quality, and inference efficiency. It compares eligible predictions with references and measures end-to-end processing through the robot-motion reference.

2) Evaluation Metrics:

  • Eligible methods use a shared heldout split and common temporal prefixes for prediction–reference comparisons; EMAGE and GestureLSM use publicly released pretrained checkpoints.Eligibility depends on each method’s native input and output-length requirements.Source: 2) Evaluation Metrics · p. 5, 4) Baselines and Ablations · p. 6
  • FGD, Div, MM, and BA evaluate co-speech characteristics, with Div and BA reported as absolute gaps to matched reference statistics.FGD measures distributional discrepancy; MM averages pairwise distances among 20 samples per input for stochastic generators, and BA matches speech onsets to detected upper-body motion beats.Source: 2) Evaluation Metrics · p. 5
  • Robot-motion quality includes the absolute gap in mean body jerk, foot-ground error, and contact sliding speed.Jerk is estimated from third-order finite differences of world-space body positions; foot-ground error measures sole-proxy distance from the ground plane.Source: 2) Evaluation Metrics · p. 5

4) Baselines and Ablations:

  • Inference time covers audio and transcript input through the robot-motion reference, while peak RAM increase is measured above the preload idle baseline.Processing time is divided by the total number of output frames.Source: 4) Baselines and Ablations · p. 6

3) Implementation Details:

The implementation uses pretrained acoustic and linguistic encoders in SGDiT, with distinct conditioning pathways and rectified-flow training. Quantitative comparisons favor direct robot-space generation on the reported co-speech metrics and processing time, while joint conditioning leads the reported co-speech measures.

3) Implementation Details:

  • SGDiT uses frozen wav2vec 2.0 large XLSR acoustic features and Qwen3.5-4B contextual token embeddings, with separately learned global–local transcript attention parameters.The model has 12 transformer blocks, a hidden dimension of 768, and is trained for 63,000 optimizer steps with AdamW and EMA.Source: 3) Implementation Details · p. 6
  • Human-Retargeted generates normalized 136-D human motion before a frozen pretrained VAE maps samples to 39-D robot references; Audio-only and Text-only are trained separately in robot space.Physical deployment and efficiency evaluation use a Jetson AGX Orin; inference uses eight Euler steps and batch size one after warm-up.Source: 3) Implementation Details · p. 6

B. Quantitative Results

  • ECHO-G achieves the best results on all four co-speech metrics and the lowest processing time per output frame; it improves all three robot-motion quality metrics over EMAGE+GMR and GestureLSM+GMR.Human-Retargeted has a smaller Jerk gap, lower foot-ground error, and less contact sliding; the authors support direct robot-space generation for co-speech modeling with reduced processing time.Source: B. Quantitative Results · p. 6, Table I · p. 7
  • Joint audio–text conditioning has the lowest FGD, ∆Div, and ∆BA and highest MM, outperforming both unimodal variants on reported co-speech metrics.Audio-only has the smallest Jerk gap and contact sliding speed, and matches joint conditioning in foot-ground error at displayed precision.Source: B. Quantitative Results · p. 6, Table II · p. 7

C. Qualitative Results

Qualitative examples outside BEAT2 contrast how audio-only, text-only, and joint conditioning shape gesture timing and content. The paper also reports a physical-robot execution and video ratings that favor joint conditioning across the overall score and three criteria.

C. Qualitative Results

  • Outside BEAT2, audio-only motion responds to rhythm but has smaller gestures around salient phrases, while text-only gestures relate to content but align less consistently with audio.In the illustrated example, joint conditioning combines speech-responsive timing with more expansive content-related gestures and fluid transitions.Source: C. Qualitative Results · p. 7

D. Real-Robot Deployment

  • A Unitree G1 executes generated joint-position references through a fixed SONIC motion tracker while the corresponding speech plays.The representative trial uses an independently prepared utterance outside BEAT2.Source: D. Real-Robot Deployment · p. 7

E. User Study

  • The rating study uses four videos of the same utterance per trial, with hidden method identities and randomized positions.Participants completed nine trials, three randomly selected from each criterion-specific pool of 33 utterances.Source: E. User Study · p. 7
  • Joint audio–text conditioning receives the highest overall mean rating and the highest mean ratings for human-likeness, rhythm matching, and motion quality.The study involved 45 participants rating G1 kinematic renderings on a five-point scale; overall is the equally weighted mean of the three criteria.Source: Table III · p. 7, E. User Study · p. 7

V. DISCUSSION AND LIMITATIONS

The discussion reports benefits from direct robot-space modeling and joint audio–text conditioning, while noting that expressive gains do not consistently align with robot-motion consistency. The conclusion identifies instruction-aware gestures and causal streaming as further development targets.

V. DISCUSSION AND LIMITATIONS

  • Benchmark comparisons support direct robot-space modeling and joint audio–text conditioning, with the direct-generation pipeline also requiring less processing time.Source: §V · p. 7
  • Co-speech gains do not consistently translate into smaller Jerk gaps or better foot-contact measures, leaving expressive behavior and robot-motion consistency to be improved together.Source: §V · p. 8
  • Limited examples of explicit deictic or instructional gestures may constrain instruction-aware generation when an utterance calls for a specific semantic motion.Source: §V · p. 8
  • The current system requires complete speech audio and timed transcripts and does not yet support causal streaming generation.Source: §V · p. 8

VI. CONCLUSION

  • The conclusion presents quantitative evaluation, video ratings, and physical demonstrations as complementary evidence, alongside released dataset, benchmark, and code for reproducible research.Source: §VI · p. 8
  • Future work targets gesture expressiveness with robot-motion consistency, richer instruction-aware semantic-gesture data, and causal streaming generation.Source: §VI · p. 8
Loading 2609.39575v1…