Source-linked AI summary
Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots
Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, Geehyuk Lee
TL;DR
Existing robot gesture systems rely on expert-authored speech–gesture associations, motivating a method that can learn co-speech gestures from human behavior. The paper trains an end-to-end text-to-pose encoder–decoder on 52 h of TED talks and evaluates its generated gestures. The model produces several gesture types, receives favorable subjective ratings relative to baselines, and runs on a NAO robot in real time.
Problem
Existing robot systems use rule-based speech–gesture associations that require expert knowledge and substantial human labor to construct.
Method
An end-to-end sequence-to-sequence model encodes speech text and decodes frame-by-frame upper-body poses, trained from human TED-talk demonstrations without prior gesture-mapping rules.
Results
The model generated iconic, metaphoric, deictic, and beat gestures, and was rated better than baselines on anthropomorphism, likeability, and speech-gesture correlation.
Takeaways & Limitations
The learned system produces varied co-speech gestures for speech text and was demonstrated on a humanoid robot operating in real time.
Takeaways & Limitations
The study uses speech text rather than audio, so generated gestures and speech audio cannot be tightly coupled.
Abstract
from arXiv · showhide
Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.
I. INTRODUCTION
The paper replaces labor-intensive, expert-designed speech–gesture rules with an end-to-end model that learns upper-body co-speech gestures from human demonstrations. It contributes a TED-based dataset, a sequence-to-sequence generator, and a real-time humanoid-robot prototype.
- Motivation: Existing robots use manually crafted gestures that require substantial labor to associate gestures with speech words and cannot perform gestures omitted during design.The approach targets the limits of expert-authored gesture inventories and mappings.
- Approach: The proposed system maps natural-language speech to temporal sequences of upper-body poses using an encoder–decoder architecture.It learns from human behavior without imposing prior knowledge about gesticulation or speech–gesture mappings.
- Contributions: The model generates iconic, metaphoric, deictic, and beat gestures for previously unseen speech text.The authors describe these gesture types as outputs learned from human demonstrations.
- Contributions: The TED Gesture Dataset contains 52 h of human gesture videos with speech transcripts covering varied speakers and topics.The dataset is introduced as a large-scale resource for co-speech gesture studies.
- Contributions: A robot prototype produces the generated gesture motions while speaking in real time.The implementation investigates conversion from 2D to 3D poses and alignment of speech audio with motions.
II. RELATED WORKS
Prior work mainly associates speech words with predefined gestures or learns from demonstrations, while this paper uses end-to-end learning for the ambiguous mapping from speech text to co-speech motion. It also distinguishes speech content from audio as gesture-generation contexts.
- Automatic Co-Speech Gesture Generation: Speech content is treated as the primary context for co-speech gesture generation, while audio is treated as secondary.The paper focuses on text because expressive audio is often unavailable from conventional robot text-to-speech systems.
- Automatic Co-Speech Gesture Generation: Rule-based systems associate selected words with predefined gestures, but constructing those association rules requires significant human labor.A NAO example uses deictic gestures for “you” and metaphoric gestures for “every.”
- Automatic Co-Speech Gesture Generation: The study does not use audio to generate gestures, although it investigates synchronization between synthesized speech audio and generated gestures in the robot prototype.This separates gesture generation from later speech–motion synchronization.
- End-to-End Learning for Motion Generation: The proposed end-to-end approach addresses a text-to-motion mapping whose connections are weaker and more ambiguous than mappings in related motion-generation studies.The authors present it as the first end-to-end study generating co-speech gestures from speech text.
III. TED GESTURE DATASET
The TED Gesture Dataset addresses the shortage of diverse co-speech gesture data by using publicly available TED talks with transcripts and naturally varied speakers and content. Its scale and format are intended to support learning speech-to-gesture mappings.
- Dataset Motivation: Existing co-speech gesture datasets are few, smaller than 3 h, and limited in speech-content and speaker diversity.The paper also notes that laboratory tracking suits and acting can make gestures unnatural.
- Dataset Design: The TED Gesture Dataset was collected from TED talks to provide a larger and more varied source of co-speech gestures.TED recordings are publicly available and contain speeches by many people on diverse topics.
- Dataset Motivation: The dataset is presented as large enough to learn the mapping from speech to gestures because the number of available TED videos continues to grow.This positions the collection as expandable for future learning-based studies.
- Dataset Design: TED talks are considered suitable because prepared speeches are expected to contain proper hand gestures.The dataset is designed around naturally occurring public speaking rather than laboratory performance.
- Dataset Design: Transcripts, flat backgrounds, and steady shots support automated data collection and pose annotation.The paper describes these properties as favorable for extracting and annotating human poses with computer vision.
A. Collection and Annotation
The TED Gesture Dataset was collected through automated video and shot processing, with additional filtering to remove pose and segmentation errors. Conservative selection yielded relatively few shots of interest.
- Collection: 1,295 TED videos with English transcripts and phrase timestamps were collected for pose extraction and gesture annotation.Human poses were extracted frame by frame with OpenPose before shot segmentation and filtering.
- Annotation: Automated shot segmentation detected sudden motion or color changes, followed by rule-based filtering of selected shots.The filtering pipeline targeted shots suitable for gesture learning from TED talks.
- Annotation: Missing joints and jittering poses were removed to reduce errors introduced by automated pose estimation and shot segmentation.The authors note that some errors were inevitable during automation.
- Dataset scope: The ratio of shots of interest was quite low because selection was conservative to avoid samples that could mislead learning.The dataset was made publicly available.
A. Data Representation and Preprocessing
Speech is represented with pretrained word embeddings, while upper-body gestures are normalized and compressed into PCA pose vectors. The sequence-to-sequence model maps speech words to pose sequences using recurrent encoding, attention, and continuous decoding.
- Speech representation: Each speech word is represented using a 300-dimensional pretrained GloVe embedding, with a zero vector for unknown words.Words begin as one-hot dictionary indices and are converted to compact embeddings.
- Pose representation: Upper-body poses use eight OpenPose joint positions, normalized at the neck with unit shoulder length and reduced to 10 PCA dimensions.The 10 principal components explain 94.8% of training-set variance.
- Network architecture: A bidirectional recurrent encoder processes words sequentially, while a recurrent decoder generates gesture motions from the encoded speech context.Soft attention lets the decoder focus on specific words during pose generation.
- Pose representation: PCA organizes upper-body pose variation by principal components, with the mean pose represented by the zero vector.Different component values correspond to different displayed pose changes.
- Network architecture: The decoder consumes n previous poses and produces m successive poses so concatenated inferences remain smooth over long speech.Generating limited motion segments avoids training recurrent networks directly on sequences hundreds of steps long.
C. Training
Training combines pose reconstruction with continuity and variance terms, using empirically fixed loss weights and tens of thousands of word-pose sequence pairs. The trained network generated varied gesture types for speech texts outside its training set.
- Objective: The loss includes mean squared error, a continuity term for successive poses, and a variance term that encourages dynamic motion.The variance term is defined as the negative variance of poses.
- Objective: α=0.01 and β=1 were empirically fixed so the three loss terms had similar orders of magnitude.These parameters control the weights of the loss terms.
- Optimization: 34,469 word-pose sequence pairs were used for training with Adam at a 0.0001 learning rate and batch size 64.Gradients were clipped to (-5, 5), and dropout of 0.1 was applied to the first GRU layers.
- Optimization: 560 epochs were required until the loss stopped decreasing, with training taking about 22 hours on an NVIDIA GTX 1080.The reported training duration reflects the authors’ implementation setting.
- Generated gestures: The trained network generated iconic, metaphoric, deictic, and beat gestures for speech texts not present in the training set.Examples include action depiction, abstract concepts, and pointing gestures.
V. EVALUATION
The evaluation used subjective judgments because value-level pose differences do not adequately measure generative gesture quality. The proposed method was compared with ground truth and random, nearest-neighbor, and manually designed baselines.
- Evaluation design: Subjective evaluation was used because different gestures can be equally appropriate despite large position-level differences from the original motion.The authors give widening arms, raising a hand, and holding something as alternative valid gestures for one concept.
- Baselines: The proposed method was compared against ground truth, random selection, nearest-neighbor retrieval, and manually designed gestures.The baselines were intended to provide competitive human-like or context-fitting motion comparisons.
- Baselines: Ground truth used OpenPose-extracted human poses from TED speakers, with playback speed adjusted to synthesized speech duration.The evaluation treated extracted human poses as the ground-truth reference.
- Baselines: Nearest-neighbor retrieval matched smaller text chunks using BLEU and concatenated their associated pose sequences.Interpolation was used between consecutive chunks to address motion discontinuities.
- Baselines: Manual gestures used beat motions and common metaphoric gestures designed to fit different speech contexts.Different gesture sequences were used across sentences to increase variability and reduce random effects.
B. Participants and Procedure
The evaluation used 46 valid participants who viewed clips of the proposed and baseline methods and rated gestures across three indexes using questionnaires.
- Participants: 46 valid participants evaluated gesture videos after 18 of 64 recruits were excluded for failed attention checks, vague answers, or inconsistent responses.Participants were recruited from Amazon Mechanical Turk; their ages ranged from 23 to 70, with a mean age of 37.
- Procedure: Participants viewed the proposed method, ground truth, and three baselines using speech clips sampled randomly from the TED test set.Each subject evaluated methods for two sentences, and transcripts were provided before evaluation without subtitles during the videos.
- Measures: The questionnaire measured anthropomorphism, likeability, and speech-gesture correlation across multiple questions for each index.The first two indexes came from the Godspeed questionnaires, while the authors designed the speech-gesture correlation index.
- Measures: Responses used five-point Likert scales with randomized question order and randomly flipped scale directions.The evaluation included questionnaire items listed in Table II.
C. Results
The proposed method achieved the second-best evaluation results overall, with significant advantages over random but no significant differences from neural-network or manual baselines.
- Reliability: Cronbach’s α values were 0.93 for anthropomorphism, 0.94 for likeability, and 0.74 for speech-gesture correlation, all exceeding 0.6.These values were used to assess internal consistency of the three indexes.
- Overall results: The proposed method ranked second-best behind ground truth across anthropomorphism, likeability, and speech-gesture correlation.The neural-network and manual methods produced similar results, while random was worst across all indexes.
- Statistical analysis: Anthropomorphism and speech-gesture correlation showed significant method effects, whereas likeability did not.ANOVA reported F(4, 330) = 2.74, p = 0.03 and F(4, 330) = 5.45, p < 0.01 for the significant indexes; likeability had p = 0.11.
- Pairwise comparisons: The proposed method rated significantly higher than random for anthropomorphism, likeability, and speech-gesture correlation.Fisher’s Least Significant Difference tests gave p = 0.009, 0.019, and <0.001, respectively.
VI. ROBOT PROTOTYPE
The robot prototype converts generated 2D poses into 3D poses, retargets them to a NAO humanoid robot, and synchronizes gesture playback with synthesized speech.
- 3D pose estimation: A neural network converted 2D poses to 3D poses because existing estimators failed under TED-dataset environment mismatches.The task was simplified by front-facing subjects and less dynamic co-speech motion than sports movements.
- Robot retargeting: The estimated 3D poses were retargeted to a NAO robot by copying joint angles because the robot and stick figures shared upper-body joint configurations.NAO provided 12 upper-body degrees of freedom.
- Real-time execution: Speech was synthesized with Google TTS, text was split into inference chunks, and generated pose sequences were played with speech of the same duration.Chunk size was determined using motion frame rates from the training dataset.
- Pipeline: The prototype pipeline transforms generated 2D poses into 3D poses and then retargets those poses to a humanoid robot.This procedure bridges the model’s stick-figure output and physical robot execution.
- Performance: 0.14 s was sufficient for seven network inferences for a 25-word sentence on a CPU.The robot generated gestures without observable differences from the corresponding 2D poses and could support other human-like joint configurations.
VII. DISCUSSION AND LIMITATIONS
The model generated varied, human-like gestures that correlated with speech, but excessive motion and text-only input limited naturalness and synchronization.
- Findings: The proposed method outperformed baselines on anthropomorphism, likeability, and speech-gesture correlation in subjective evaluation.Participants associated positive impressions with human-like movement and motions that flowed with speech content.
- Limitations: A motion-variance loss sometimes produced excessive, jerky, or fast gestures that participants disliked.One participant preferred a few clear gestures over incessant gesturing.
- Limitations: Because the model used speech text but not audio, gestures and speech could not be tightly coupled.Participants reported that gestures were sometimes faster than speech; the authors identify joint gesture-audio generation as an extension.