PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

Zifan Wang, Ziang Ren, Pengyang Shi, Zirui Wang, Chenghuai Lin, Tianze Wang, Zekun Qi, Liangliang Zhao, He Wang, Li Yi

arXiv:2608.28693v1cs.ROcs.CV

TL;DR

Humanoid robots need synchronized, semantically meaningful gestures, but progress is constrained by scarce semantic data, modality eclipse, and physical-safety challenges. RoboGesture co-designs robot-specific data, hierarchical raw-audio alignment, streaming motion generation, and MPC safety filtering; experiments report safer, more rhythmic, and more semantically appropriate responses on a physical humanoid.

  • Problem

    Humanoid co-speech gesture generation is limited by scarce semantically rich data, audio-insensitive historical-motion shortcuts, and the need for safe physical execution.

  • Method

    RoboGesture combines a 300-plus-category robot gesture dataset, hierarchical semantic-acoustic alignment, streaming Conditional Flow Matching generation, Anti-Inertia CFG Masking, and MPC safety filtering.

  • Results

    Experiments on a physical humanoid report safer, more rhythmic, and more semantically appropriate responses than existing baselines.

  • Takeaways & Limitations

    The framework supports an end-to-end listen–respond–gesture loop for real-time, semantically aligned, safety-aware humanoid interaction.

  • Takeaways & Limitations

    The system targets speech-aligned expressive motion rather than locomotion, object manipulation, or scene-level decision making.

Abstract

from arXiv · show

Enabling humanoid robots to respond to human speech with synchronized and semantically meaningful gestures is fundamental to natural human-robot interaction. However, this task faces three critical barriers: the scarcity of semantically rich datasets, the "modality eclipse" where models ignore audio cues in favor of kinematic inertia, and the sim-to-real gap regarding physical safety. We propose RoboGesture, a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system in which the robot listens, responds, and gestures in real time. We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale collision-free, robot-specific audio-motion pairs. Our architecture features a Hierarchical Semantic-Acoustic Aligner that extracts multi-granular prosodic and semantic cues directly from raw audio tokens. These cues drive a Streaming Conditional Motion Generator based on a diffusion transformer with conditional flow matching. To ensure high responsiveness, we introduce Anti-Inertia CFG Masking, which prevents the model from collapsing into repetitive historical patterns by compelling it to proactively mine control signals from the audio modality. Finally, an MPC-based safety filter ensures real-time, collision-free execution on physical hardware. Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.

1 Introduction

RoboGesture targets real-time humanoid interaction by addressing limited semantically rich data, weak audio–motion alignment, and safety requirements. It combines robot-specific data synthesis, hierarchical audio alignment, streaming generation, and physical-robot validation.

  • Humanoid robots need synchronized, semantically meaningful speech and body motion for natural assistive, educational, and collaborative interaction.
  • The work addresses scarce semantically rich audio-motion data, compromised semantic and temporal coherence, and challenges in online generation.
  • RoboGesture provides a robot-centric, data-model co-designed framework for real-time, semantically aligned, and safety-aware humanoid gesture synthesis.
  • Its data pipeline retargets expressive human motions to humanoid kinematics and filters potential self-collisions, producing robot-specific training pairs.
  • The hierarchical aligner separates raw audio into rhythmic and semantic cues, including anticipatory signals that synchronize movement with speech beat and intent.
  • Anti-inertia masking makes the streaming diffusion generator mine audio controls instead of relying excessively on historical kinematic patterns.
  • Experiments evaluate motion quality, synchrony, naturalness, and safety, with physical-humanoid deployment showing stronger semantic and rhythmic alignment than existing baselines.

2 Related Works

Related work spans virtual-avatar interaction, rule-based robot controllers, and deep co-speech gesture generation. RoboGesture is positioned against modality decoupling, sparse semantic data, text dependence, and limited streaming support.

  • Virtual-avatar systems provide immersive social interaction but leave a significant domain gap to physical humanoid robots.
  • Rule-based robot controllers are interpretable but struggle with open-ended social behavior and often decouple speech from gesture.
  • Modern co-speech gesture methods use large audio-gesture datasets and diverse strategies to connect gestures with linguistic content.
  • Existing approaches respectively show weaknesses in semantic consistency, fine-grained alignment, real-time streaming, and semantic coverage because of text dependence or sparse annotations.

3 Method

RoboGesture uses a streaming-to-streaming architecture that aligns raw audio, generates continuous robot motion, and filters execution for safety. Its two-stage training separates representation learning from joint motion generation while combating historical-motion bias.

  • System architecture: The system processes incoming audio chunks into real-time motions and integrates with streaming language models and audio feeds.
  • System architecture: The method comprises a hierarchical semantic-acoustic aligner, streaming conditional motion generator, and MPC-based kinematic safety filter.
  • Semantic-acoustic alignment: The aligner uses hierarchical audio tokens and multi-task learning to provide low-level rhythmic signals and high-level semantic labels for motion control.
  • Motion generation: The continuous DiT generator with Conditional Flow Matching avoids discrete-token precision limits while modeling expressive motion in a continuous state space.
  • Motion generation: Cross-attention fuses multi-scale acoustic cues with historical motion, while FiLM establishes global emotional and semantic tone.
  • Safety and training: An MPC safety filter solves a real-time constrained optimization problem enforcing collision, velocity, position-tracking, and smoothing requirements.
  • Training: Stage 1 pre-trains the aligner on 1,000 hours of semantic-annotated data using beat and semantic supervision before jointly training the generator.
  • Training: Anti-Inertia CFG Masking randomly removes past-motion conditioning with 15% probability, encouraging audio-driven cold starts during joint training.

4 Automatic Semi-Synthetic Data Generation

The automatic semi-synthetic pipeline expands semantic gesture data while keeping motions robot-specific, executable, smooth, and collision-free. RoboGesture combines a 300-plus-category dataset with language-guided scenario, gesture, audio, blending, and safety processing.

  • Data motivation: Existing audio-motion resources provide rhythm but contain sparse explicit semantic gestures, motivating larger semantically enriched data.
  • Pipeline design: The proposed pipeline generates humanoid-native speech-motion pairs with smooth, collision-free trajectories.
  • Robot feasibility: Human motion transfer must preserve human-likeness while ensuring that robot motions remain collision-free and executable.
  • Robot feasibility: Enhanced GMR and Dex-Retargeting transfer body and hand kinematics to the G1, followed by MPC-based collision cleaning.
  • Dataset: RoboGesture contains over 300 gesture categories recorded with marker-based capture and manually annotated with descriptions and contextual meanings.
  • Dataset: Retargeted gestures are optimized and replayed on the physical robot to verify reachability and control fidelity.
  • Pipeline design: The five-stage pipeline uses LLM scenario generation, gesture tagging, multimodal synthesis, temporal blending, and safety optimization.

5 Experiments

RoboGesture is evaluated on standard co-speech generation, human pairwise judgments, ablations, and dataset benchmarks, with emphasis on motion quality, synchronization, semantic accuracy, and physical safety. Across these evaluations, it generally outperforms baselines while its components contribute distinct capabilities.

  • Quantitative Evaluation: RoboGesture achieves state-of-the-art FGD, BC, and MSE on BEAT, with near-zero Collision Rate of 0.88%/0.13%.The results indicate strong distributional fidelity, speech-motion synchronization, structural reconstruction, and physical plausibility.
  • Human Evaluation: Our method consistently outperforms all baselines in Physical & Hand Consistency, Rhythmic Alignment, Semantic Accuracy, and Overall Preference.Participants judged its motions safer, more synchronized, contextually appropriate, natural, and human-like.
  • Ablation Study: Removing the semi-synthetic data causes a catastrophic drop in Semantic Action and Hand Detail scores, reducing outputs to mainly rhythmic beat motions.Training with approximately 250 hours of the subset also leaves expressiveness and semantic richness noticeably sub-optimal.
  • Ablation Study: Removing context motion causes severe discontinuities, while removing FiLM, semantic classification, or CFG impairs expressiveness, semantic accuracy, or cross-modal integration.Without CFG, the model tends to infer motions solely from past states rather than balancing audio and motion features.
  • Ablation Study: Removing the Kinetic-Aware Loss weakens hand-movement constraints, while disabling the post-processing filter slightly penalizes HN and compromises motion smoothness.The ablations therefore associate data scale, architectural modules, and refinement losses with complementary aspects of generation quality.

6 Conclusion

RoboGesture presents a robot-centric framework that co-designs data, modeling, and control for real-time humanoid interaction through synchronized speech and expressive gestures. Experiments on a humanoid robot show more semantically appropriate and safety-aware responses than baselines.

  • Conclusion: RoboGesture integrates data curation, model architecture, and safety mechanisms into an end-to-end framework for real-time multimodal humanoid interaction.Its contributions include robot-specific data synthesis and a continuous audio-driven motion policy designed to preserve synchronization and avoid repetitive historical motion patterns.
  • Conclusion: Experiments on a humanoid robot demonstrate more semantically appropriate and safety-aware responses than baselines.

Supplementary Material

The supplementary material motivates robot-space learning by relating physical humanoid interaction to safety, latency, semantic timing, and hand fidelity. It contrasts these requirements with human-space pipelines and explains the resulting deployment trade-offs.

  • Requirements of Physical Humanoid Interaction: Physical humanoid interaction requires kinematic safety, low-latency streaming, temporal semantic alignment, and high-fidelity hand modeling.These requirements distinguish physical and social interaction from purely visual avatar animation.
  • Robot-space Learning: Human-space generation followed by online retargeting introduces an avatar-to-humanoid gap, latency amplification, self-collisions, and numerical jerk.The stated causes include mismatched joint limits and link lengths, iterative optimization during retargeting, and models that ignore robot physical volume.
  • Robot-space Learning: RoboGesture internalizes retargeting during offline preprocessing, enabling optimization-based refinement of robot joint trajectories, including high-dimensional hand articulations.This avoids imposing the same stringent temporal constraints on real-time inference.
  • Robot-space Learning: Because robot-space co-speech generation is largely unexplored, the paper benchmarks against state-of-the-art human-space models to assess suitability for physical humanoid targets.Collision rate and executability are treated as primary objectives rather than only virtual-animation quality.

A.3 Structural Breakdown of Methodological Contributions

RoboGesture organizes its contributions across task, system, and method levels to support real-time, semantically aligned, safety-aware humanoid interaction. Its deployment pipeline couples expressive language generation, streaming speech-to-motion processing, and robot execution.

  • Task-level: The task is defined as streaming, semantically aligned co-speech interaction on physical humanoid hardware, emphasizing robot-feasible actions and low latency.
  • System-level: Data-model-control co-design links a robot-centric, physically feasible dataset with a transformer generator and direct-command execution that bypasses online retargeting.
  • Method-level: The method addresses modality eclipse with anti-inertia training, semantic-acoustic alignment, and safety-aware motion generation.
  • Deployment pipeline: The deployment pipeline uses ASR, a LoRA-tuned Qwen module, and TTS before streaming audio tokenization, alignment, motion generation, MPC filtering, and humanoid execution.
  • Language interaction: Emotion/action tags and colloquial responses provide high-level cues that make downstream speech socially expressive and improve coupling among language, prosody, and gesture.

B.3 Runtime Latency Breakdown

The deployment reports module-level first-output latency for streaming interaction rather than strict end-to-end latency. Motion inference is faster than real time, while practical delay depends mainly on speech processing and chunk availability.

  • Measurement basis: Streaming evaluation prioritizes each module’s first usable output, and the reported values are not strict end-to-end latency measurements.Response time also depends on overlap, chunk availability, and communication overhead.
  • Language pipeline: 0.5 s effective ASR latency is set by the 0.5 s endpoint threshold rather than the 0.157 s utterance-finalization computation.The effective latency is max(0.157s, T_idle) ≈0.5 s.
  • Language pipeline: 0.142 s average TTFT enables the warm-start LoRA-tuned Qwen3-4B-Instruct language module to provide an interactive response cue.
  • Language pipeline: 0.414 s is the average time for streaming TTS to produce the first emotional speech chunk.
  • Speech-to-motion pipeline: 0.161 s is the average latency to the first streaming token from the Mimi audio tokenizer under batch size 1.This determines when downstream motion processing can begin.
  • Speech-to-motion pipeline: 249.78 ms per 1-second motion chunk corresponds to 120.11 fps, so motion generation runs faster than real time once a valid chunk is available.The first valid chunk still depends on upstream speech and tokenization rates.
  • System interpretation: The system-level interaction delay is primarily governed by the speech pipeline and streaming chunk availability rather than the motion model.
  • Scope and safety: The deployment scope is stationary upper-body gesturing; across 200 trials, it recorded 0 falls and 0 emergency stops.Locomotion-coupled gestures remain outside the stated scope.

C.2 Audio-Shuffle and Gradient-Magnitude Analysis

The diagnostics examine whether streaming generation responds to incoming audio or remains dominated by motion history. They show that semantic-acoustic conditioning, Anti-Inertia CFG masking, and delayed history injection reduce this bias while preserving continuity.

  • Audio-shuffle analysis: Audio-shuffle FGD rises from 0.624 to 9.151 for the Full model (∆= +8.53), while Only-DiT changes by only ∆= +0.11 and w/o CFG by ∆= −0.33.The larger Full-model degradation indicates stronger dependence on the input audio under this diagnostic.
  • Gradient-magnitude analysis: The Full model has the lowest G_hist/G_audio ratio, 0.55, and the highest absolute audio gradient, G_audio = 0.245; Only-DiT reaches 4.88.
  • Injection-order ablation: Reversing history injection reduces semantic action accuracy from 7.175 to 5.376, hand detail from 7.203 to 5.136, and beat matching from 7.529 to 6.100.
  • Main-paper ablations: Removing semantic-acoustic components or Anti-Inertia CFG masking harms semantic action accuracy, hand detail, naturalness, and beat alignment.
  • Interpretation: The combined design balances responsiveness to new semantic-acoustic cues with continuity from historical motion, producing the best overall perceptual and semantic performance.
  • Diagnostic limitation: The perturbation metric measures conditioning sensitivity, not whether each generated response is semantically correct.

D.1 Data Composition

RoboGesture combines diverse gesture sources with robot-specific retargeting, feasibility filtering, and physical validation to construct a broad, executable dataset and collision-aware processing pipeline.

  • Dataset composition: SeG contains gestures that humanoid robots cannot faithfully reproduce because some rely heavily on limited head mobility.This motivates filtering and supplementing existing gesture sources.
  • Dataset composition: Over 300 high-quality gesture classes combine SeG, EgoGesture, full-body motion capture, online sources, and user surveys.The dataset expands beyond hand-only or limited daily-gesture coverage.
  • Robot-specific validation: Each gesture is retargeted to the humanoid, optimized for physical feasibility, and replayed on the real robot to verify reachability and control fidelity.The resulting motions preserve human-like expressiveness while remaining executable on hardware.
  • MPC-based collision filter: The MPC-based collision filter prevents self-collision and smooths generated trajectories during both data preparation and model execution.Its per-frame streaming operation supports offline dataset curation and real-time inference.
  • MPC-based collision filter: The filter formulates velocity, trajectory-tracking, and smoothing costs in a convex quadratic program solved with OSQP.Collision checking uses selected body-pair groups to reduce computation.
  • Evaluation: RoboGesture achieves FGD 0.845, while directly trained and retargeted baselines remain nearly identical and substantially worse.For example, DiffSHEG changes from FGD 2.195 to 2.232 after retargeting, and SG from 2.781 to 3.015.
  • Evaluation: The semantic head reaches 88.72% train and 64.71% test top-1 accuracy on the 300-class gesture-semantic classification task.This head provides auxiliary supervision while the generator operates in a continuous latent space.

E.5 Text Conditioning and Diversity Analysis

The study compares RoboGesture’s semantic aligner with text conditioning and interprets diversity alongside contextual appropriateness rather than treating global diversity as sufficient.

  • Text conditioning: RoboGesture’s aligner produces more reasonable semantic gestures than Qwen- and T5-based text-conditioning variants on 100 SemanticBEAT videos.The comparison counts identifiable semantic gestures and judges their contextual reasonableness.
  • Diversity analysis: Global DIV is not highest for RoboGesture, but the authors argue that lower, semantically grounded diversity avoids contextually inappropriate gestures.SG achieves the highest DIV while receiving a low reasonable ratio.
  • Semantic head evaluation: Table 13 reports accuracy for the aligner’s 300-class semantic head under strict and semantic-equivalent scoring.Semantic-equivalent scoring credits semantically interchangeable classes.
  • Text conditioning: Table 14 reports identifiable semantic gestures and contextual-reasonableness ratios for text-conditioning ablations and baseline methods.Qwen and T5 replace the aligner in the ablation.

F.1 Scope and Limitations

RoboGesture is scoped to upper-body, speech-aligned gesture generation and remains constrained by motion continuity, semi-synthetic data diversity, system-level latency, and deployment safeguards.

  • Interaction scope: The system targets upper-body co-speech gestures rather than locomotion, object manipulation, or scene-level decision making.The authors describe this as a scoped step toward broader humanoid social intelligence.
  • History-conditioned generation: Anti-Inertia CFG masking reduces but does not eliminate history dominance, so generation still relies on motion-continuity priors.The system must balance responsiveness to new audio with temporal smoothness across chunks.
  • Data and generalization: The model remains influenced by the diversity of its robot-space semi-synthetic training distribution.The authors identify richer real-world interactive recordings as a likely route to broader robustness.
  • System latency: The motion model runs faster than real time once chunked audio is available, but overall delay is dominated by language generation, speech synthesis, and streaming.Reducing system-level latency remains a future deployment direction.
  • Real-world deployment: The safety filter improves executability but is not a substitute for conservative bounds, monitoring, and emergency-stop mechanisms.Physical deployment requires a broader system-level safety stack.
  • Expressive social interaction: The authors recommend transparent communication about the generated nature and intended scope of more expressive robot behavior.Expressiveness can shape user expectations more strongly than purely functional systems.
Loading 2608.28693v1…