PaperlayerView exact revision on arXiv

Source-linked AI summary

GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots

Bosong Ding, Xianglin Zhang, Miao Xin, Murat Kirtay, Giacomo Spigler

arXiv:2609.38400v1cs.ROcs.AIcs.HCcs.LG

TL;DR

Robots need co-speech gestures that fit their available workspace, since the same speech can support different motions and workspace-unaware gestures may violate physical limits. GestAdapt conditions generation on speech, motion history, and a prescribed workspace, then retargets motions for robot execution. In the robot study, its motions ranked first in 69.7% of comparisons under shared workspace-constrained retargeting and execution.

  • Problem

    Workspace-unaware co-speech generation may produce gestures inappropriate for a robot’s surroundings or joint limits, despite the same speech supporting different natural motions.

    Source: §I · pp. 1–2

  • Method

    GestAdapt learns from six co-speech corpora and generates motion conditioned on speech, recent motion, and a wrist workspace, with analytic guidance and embodiment-specific retargeting.

    Source: §I · p. 2

  • Results

    GestAdapt ranked first in 69.7% of robot-study trials, compared with 19.7% for Ours w/o Workspace and 10.6% for ground truth under shared constrained processing.

    Source: C. User Study · p. 7

  • Takeaways & Limitations

    The findings support incorporating the target workspace during gesture generation rather than imposing it only after generating unconstrained trajectories.

    Source: §I · p. 2, §VI · p. 7

    External-baseline comparisons are not information-matched, so they assess conditional motion quality and workspace containment rather than superiority under identical input information.

    Source: B. Quantitative Results · p. 6

Abstract

from arXiv · show

Co-speech gestures for robots must adapt not only to speech and embodiment, but also to the workspace available for performing the motion. Since the same speech can be accompanied by different gestures, a robot can respond to workspace constraints, e.g., gestures for speech next to a wall. In these scenarios, the robot should gesture in a suitable motion rather than simply correcting an unconstrained one. To achieve this goal, we present GestAdapt, a workspace-conditioned framework that conditions co-speech gesture generation on a prescribed wrist workspace. The GestAdapt framework learns from six complementary co-speech corpora through a shared motion representation and supports retargeting to different robot embodiments. Quantitative evaluation shows that generated motions remain close to the real-motion distribution while respecting the workspace. In a user study, gestures generated under modified workspace constraints receive a mean quality score of 3.24/5, above our no-workspace variant (2.43/5) and below the reference motions (3.68/5). In a real robot evaluation, all compared motions are retargeted to the Reachy2 humanoid robot under identical workspace constraints. Motions generated with our framework rank first in 69.7\% of comparisons, higher than our no-workspace variant baseline and retargeted ground-truth motions constrained afterward. Overall, the results support adapting gestures to the available workspace during generation, rather than modifying unconstrained trajectories afterward to satisfy workspace constraints, potentially compromising gesture naturalness.

I. INTRODUCTION

Co-speech gestures must fit the robot’s available space as well as its speech and embodiment. GestAdapt conditions generation on the workspace and evaluates whether this produces plausible, adaptable motions under shared constrained robot execution.

I. INTRODUCTION

  • Workspace-unaware gestures may be inappropriate for the surroundings or joint limits, reducing motion quality or risking unsafe execution.The same speech can support different natural gestures, allowing the robot to adapt its motion to available space rather than simply compressing or correcting it.Source: §I · pp. 1–2
  • GestAdapt conditions robot co-speech gesture generation on a prescribed workspace, alongside speech and motion history, then retargets motions to the target embodiment.It combines six corpora, uses wrist-motion envelopes for spatial supervision, and applies analytic guidance to reduce residual wrist violations.Source: §I · p. 2
  • GestAdapt’s motions remain plausible and smooth under new spatial constraints, and participants prefer workspace-conditioned generation to an unconditioned generator and retargeted human motion.All compared robot motions use the same retargeting and constrained-execution pipeline.Source: §I · p. 2

II. RELATED WORK

Prior co-speech work addresses motion quality and robot execution, while spatially conditioned generation has emerged in human and virtual-character settings. GestAdapt connects workspace-conditioned gesture generation with robot embodiment and execution.

A. Co-Speech Gesture Generation

  • Co-speech corpora differ in acquisition setups, representations, and gesture distributions, and most prior gesture models are developed within a particular corpus.GestAdapt unifies multiple corpora in a shared motion space to use their complementary motion and spatial variation.Source: A. Co-Speech Gesture Generation · p. 2

B. Gesture Generation for Robots

  • Robot gesture studies address embodiment transfer, speech alignment, and physical execution; GestAdapt instead asks how gestures should change with the robot’s available space.It conditions generation on the target workspace before embodiment-specific retargeting and constrained execution.Source: B. Gesture Generation for Robots · p. 2

C. Physical and Spatial Constraints in Motion Generation

  • Spatially conditioned gesture work includes object-aware interactions and geometry-conditioned synthesis, while GestAdapt uses wrist-envelope supervision and evaluates motions after robot retargeting under shared constraints.This approach does not require scene-annotated gesture recordings or a full scene representation.Source: C. Physical and Spatial Constraints in… · p. 2

III. UNIFIED DATASET

The unified dataset combines six complementary speech-motion corpora and maps their incompatible motion formats into a shared representation. The representation preserves geometry for retargeting across robot embodiments.

A. Combining Multiple Co-Speech Corpora

  • Six speech-motion datasets provide complementary posture, gesture-scale, and workspace variation, expanding training coverage despite distinct spatial support.Their motion distributions overlap, while their kinematic and spatial support remain distinct; Fig. 2 visualizes motion distributions and wrist-motion envelopes.Source: A. Combining Multiple Co-Speech Corpora · pp. 2–3, Figure 2 · p. 3

B. Robot-Centered Unified Motion Representation

  • The datasets’ incompatible skeletons and rotation conventions are mapped through native-hierarchy forward kinematics to anatomically corresponding landmarks aligned on a common skeleton.The resulting 123-dimensional representation at 15 Hz uses seven torso and arm directions, palm axes, and finger directions in each palm frame.Source: B. Robot-Centered Unified Motion… · p. 3, Figure 3 · p. 3
  • Reconstruction from the shared representation introduces less than 0.6° average palm-orientation error on BEAT, Trinity, and ZEGGS, while retargeted joint-trajectory differences remain below 1° across evaluated source-robot combinations.Wrist positions remain effectively unchanged after reconstruction.Source: B. Robot-Centered Unified Motion… · p. 3

C. Spatial Representation and Dataset Construction

GestAdapt derives workspace conditions from wrist-motion envelopes because the source corpora lack scene geometry and free-space annotations. The resulting wrist box controls gesture extent, not a complete collision-free region.

C. Spatial Representation and Dataset Construction

  • Wrist trajectories define the task-level spatial condition because they capture gesture extent while leaving elbow and shoulder configurations flexible.Applying a wrist box to the whole upper body could impose unnecessary constraints because proximal links have different feasible regions.Source: C. Spatial Representation and Dataset… · p. 3
  • The wrist box constrains wrist placement rather than defining a complete collision-free region; retargeting applies soft workspace constraints and configured hard physical constraints.The upstream policy determines the safe movement region.Source: C. Spatial Representation and Dataset… · p. 3
  • Each segment’s smallest torso-centered, axis-aligned box contains both wrist trajectories, with its six boundaries normalized by a 69 cm canonical arm length.The box specifies gesture extent without specifying which hand moves or its trajectory and configuration.Source: C. Spatial Representation and Dataset… · p. 4
  • Training examples use four observed frames and 30 target frames, with four additional future frames used only to check envelope stability; unstable or stationary/silent windows are discarded.The procedure retains 378,438 of 799,996 candidate windows and splits by complete recording into 359,088 training and 19,350 held-out examples.Source: C. Spatial Representation and Dataset… · p. 4

IV. METHOD

GestAdapt generates future gestures from speech, recent motion history, and a specified workspace box, then retargets them under embodiment-specific execution constraints. The workspace specifies the desired region for wrist motion, while leaving gesture type and trajectory unspecified.

IV. METHOD

  • GestAdapt conditions diffusion-based future-motion generation on speech, recent motion history, and a workspace box, then retargets the motion under embodiment-specific execution constraints.The workspace lets the gesture itself change to fit the available space.Source: §IV · p. 4

A. Problem Formulation

  • Motion is represented at 15 Hz, with four history frames and a 30-frame window.Source: A. Problem Formulation · p. 4
  • The workspace is a normalized box specifying the desired wrist-motion region; it constrains spatial extent without prescribing hand assignment, trajectory, or gesture type.Selecting the box from scene or task information is outside the stated scope.Source: A. Problem Formulation · p. 4

B. Workspace-Conditioned Diffusion Transformer Model

A diffusion Transformer generates multiple plausible gestures conditioned on speech, motion history, and workspace. Six workspace-boundary tokens enter through cross-attention, while separate output heads specialize body/palm and finger motion.

B. Workspace-Conditioned Diffusion Transformer Model

  • The diffusion Transformer generates multiple plausible motions, using stochastic denoising to preserve the one-to-many nature of co-speech gestures.Source: B. Workspace-Conditioned Diffusion… · p. 4
  • The denoiser combines noisy future motion, four observed history frames, frame-aligned speech features, and the diffusion timestep.The audio features are extracted from 16 kHz audio; the encoder produces a feature for each of the 34 motion frames.Source: B. Workspace-Conditioned Diffusion… · p. 4
  • Each Transformer block injects six workspace-boundary tokens through residual cross-attention, letting temporal motion features attend to workspace boundaries throughout denoising.The normalized motion features provide queries, and workspace tokens provide keys and values.Source: B. Workspace-Conditioned Diffusion… · p. 4
  • Separate output heads model the 33-dimensional body/palm component and 90-dimensional finger articulation while retaining a shared Transformer backbone.The split reflects the body/palm component’s role in arm configuration and wrist/palm geometry, where workspace adaptation is primarily expressed.Source: B. Workspace-Conditioned Diffusion… · p. 4
  • Both branches use a dimension-normalized DDPM noise-prediction objective, and the reported representation ablation improves motion quality and workspace compliance over the 126D holistic design.Training uses 500 diffusion steps; inference uses the full 500-step ancestral sampler.Source: B. Workspace-Conditioned Diffusion… · p. 4

C. Analytic Workspace Guidance

Analytic sampling-time guidance supplements workspace conditioning to reduce residual wrist-boundary violations. It uses differentiable wrist positions from forward kinematics and a workspace-violation cost to adjust denoising.

C. Analytic Workspace Guidance

  • Sampling-time analytic guidance addresses residual wrist-boundary crossings that can remain despite workspace conditioning, without adding a training objective.Source: C. Analytic Workspace Guidance · p. 4
  • At each reverse step, guidance reconstructs a clean-motion estimate and obtains wrist positions differentiably through forward kinematics before evaluating violations across future frames and spatial axes.Source: C. Analytic Workspace Guidance · p. 4
  • Guidance is applied for reverse steps k ≤ 250 and is zero otherwise; G = 0 keeps workspace conditioning but reduces residual violations without guaranteeing hard feasibility.The framework overview describes this analytic guidance as further correcting wrist motion toward the requested workspace.Source: C. Analytic Workspace Guidance · p. 5, Figure 4 · p. 5

D. GMR-Based Robot Retargeting and Constrained Execution

Generated motion is mapped to a target robot using embodiment-specific calibration and inverse kinematics, then tracked with soft workspace constraints and hard physical constraints. Generation-stage wrist containment is evaluated separately from robot-stage tracking and execution feasibility.

D. GMR-Based Robot Retargeting and Constrained Execution

  • GMR maps generated upper-arm and forearm directions and palm orientations into the robot frame, then solves for joint configurations using the robot kinematic model, Jacobians, and joint limits.Source: D. GMR-Based Robot Retargeting and… · p. 5
  • The requested workspace is registered in the robot base frame and converted from normalized generation coordinates to a metric box using embodiment-specific scale.Source: D. GMR-Based Robot Retargeting and… · p. 5
  • The execution optimization adds soft workspace constraints to the existing tracking problem, with slack variables trading off tracking accuracy and workspace adherence.Joint ranges, per-frame motion limits, and configured self-collision constraints remain hard constraints.Source: D. GMR-Based Robot Retargeting and… · p. 5
  • Generation changes the gesture in response to target space, whereas GMR resolves it for a particular robot; wrist containment and execution feasibility are evaluated separately.Source: D. GMR-Based Robot Retargeting and… · p. 5

V. EXPERIMENTS

The experiments assess motion-distribution similarity, speech-motion synchronization, wrist smoothness, and workspace compliance. The full workspace-aware pipeline improves FGD and wrist violations over the no-workspace control, while external-baseline comparisons are not fully information-matched.

A. Metrics

  • Four metrics assess distributional similarity (FGD), speech-motion synchronization (BC), wrist-trajectory smoothness (jerk), and maximum wrist distance outside the prescribed workspace (violation).Lower FGD indicates closer proximity to the held-out real-gesture distribution.Source: A. Metrics · pp. 5–6

B. Quantitative Results

  • FGD falls from 6.933 to 5.019 and mean maximum workspace violation from 9.815 cm to 1.095 cm with the full workspace-aware pipeline versus Ours w/o Workspace.Table I reports quantitative results over five runs.Source: B. Quantitative Results · p. 6, Table I · p. 6
  • BC is interpreted only as a rhythmic-alignment diagnostic, not as a measure of perceptual gesture quality.The authors complement objective metrics with human evaluation of naturalness, expressiveness, and speech appropriateness.Source: B. Quantitative Results · p. 6
  • External-baseline comparisons are not fully information-matched because GestAdapt receives each test sequence’s reference workspace while external methods retain their original input modalities.Ours w/o Workspace is the controlled comparison, sharing the generator, training data, and motion representation.Source: B. Quantitative Results · p. 6

C. User Study

The user studies evaluate overall gesture quality under changed workspace conditions, first for generated human-skeleton motion and then after robot retargeting. Workspace-conditioned generation scores above the no-workspace variant in both settings, including under shared robot execution constraints.

Study design

  • Thirty participants rate overall gesture quality on 12 randomly selected eight-second clips, considering naturalness, expressiveness, and appropriateness to speech.The studies evaluate generated human-skeleton motion and motion after retargeting and constrained execution on a physical robot.Source: C. User Study · p. 6

Unseen-workspace generation

  • Under unseen-workspace generation, GestAdapt scores 3.24/5, compared with 2.43/5 for Ours w/o Workspace and 3.68/5 for ground truth.Three of four generation windows use a workspace different from their original one.Source: C. User Study · p. 7, Table II · p. 7

Robot evaluation

  • GestAdapt is ranked first in 69.7% of robot trials, versus 19.7% for Ours w/o Workspace and 10.6% for ground truth, after identical workspace-constrained retargeting and execution.Mean ranks are 1.37, 2.11, and 2.53, respectively.Source: C. User Study · p. 7, Table II · p. 7

D. Ablation Studies

The ablations examine multi-corpus training, analytic denoising guidance, and motion representation. The conclusion supports conditioning generation on the target workspace, while noting wrist-box and whole-body-collision limitations.

D. Ablation Studies

  • Six-corpus training yields 1.095 cm mean maximum violation, compared with 2.176–7.507 cm for single-corpus experts under unfamiliar target boxes.Single-corpus experts tend to preserve corpus-specific motion patterns at the expense of workspace compliance.Source: D. Ablation Studies · p. 7
  • At G = 96, analytic guidance reduces mean maximum violation from 1.272 cm to 1.095 cm with negligible FGD and jerk changes; G = 128 destabilizes sampling.The authors attribute workspace adaptation primarily to the learned generator, with guidance correcting residual violations.Source: D. Ablation Studies · p. 7
  • The factorized 123D representation improves FGD from 5.942 to 5.019 and violation from 1.679 to 1.095 cm versus a holistic 126D representation.The factorized design represents palm axes explicitly, expresses finger directions in the palm frame, and uses separate body/palm and finger output heads.Source: D. Ablation Studies · p. 7

VI. CONCLUSION

  • GestAdapt combines six co-speech datasets with wrist-envelope supervision and conditions generation on speech, recent motion, and the target region.The conclusion supports incorporating target workspace during generation rather than only through downstream correction.Source: §VI · p. 7
  • Current limitations include generating wrist boxes and lacking whole-body collision guarantees.The conclusion identifies scene-level geometry and whole-body collision constraints as future work.Source: §VI · p. 7

Related papers

Loading 2609.38400v1…