Source-linked AI summary
Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, Geehyuk Lee
TL;DR
Generating convincing co-speech gestures is difficult because gestures vary across modalities and speakers, while objective perceptual evaluation remains underdeveloped. The paper proposes a temporally synchronized end-to-end model using speech text, audio, and speaker identity, alongside the Fréchet gesture distance. The method performs better than previous end-to-end models in objective and subjective evaluations, supports style variation, and works with synthesized audio.
Problem
Gesture generation must account for speech content, rhythm, and speaker variability, while objective perceptual metrics for comparing generated gestures remain unavailable or inadequate.
Method
The paper uses a temporally synchronized end-to-end model combining text for speech content, audio for rhythm, and speaker identity for style, and introduces the Fréchet gesture distance.
Results
The proposed generation method showed better results than previous methods in objective and subjective evaluations, while generating varied styles and working with synthesized audio.
Takeaways & Limitations
Speaker identity and the learned style embedding enable different gesture styles for the same speech, extending gesture generation beyond a single fixed output style.
Takeaways & Limitations
The model offers limited control over specific gestures, and the FGD feature extractor may fail to capture subtle motion.
Abstract
from arXiv · showhide
For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human--agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. By incorporating a multimodal context and an adversarial training scheme, the proposed model outputs gestures that are human-like and that match with speech content and rhythm. We also introduce a new quantitative evaluation metric for gesture generation models. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models. We further confirm that our model is able to work with synthesized audio in a scenario where contexts are constrained, and show that different gesture styles can be generated for the same speech by specifying different speaker identities in the style embedding space that is learned from videos of various speakers. All the code and data is available at https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context.
1 INTRODUCTION
The paper addresses real-time co-speech gesture generation for embodied agents, where gestures support understanding and interaction but remain difficult to model from human demonstrations. It proposes a trimodal end-to-end model and a new evaluation metric to generate and assess more human-like gestures.
- Motivation: Co-speech gestures help listeners understand speech and improve intimacy, persuasion, and credibility in human–agent interaction.The paper focuses on upper-body gestures that occur with speech in virtual avatars and social robots.
- Motivation: Automatically generating gestures in real time is difficult because machines must model speech, gestures, and their relationship.Rule-based methods require substantial human effort, while data-driven methods require complex models and large datasets.
- Research gap: End-to-end methods use raw gesture data but previously considered only speech audio or text, despite gesture variability across and within speakers.The paper argues that multiple speech-related modalities are needed for meaningful, human-like gestures.
- Proposed approach: The proposed model combines text for speech content, audio for speech rhythm, and speaker identity for style variations in a temporally synchronized encoder–decoder.A learned style embedding allows different gesture styles for the same speech by sampling different points associated with speaker identity.
- Evaluation: The paper introduces the Fréchet gesture distance and validates it against human judgements as an objective evaluation metric for gesture generation.The metric adapts the Fréchet inception distance concept to a latent gesture feature space because no general pretrained gesture feature extractor is available.
2 RELATED WORK
Prior gesture-generation systems used rules or end-to-end models, but often restricted speech inputs or style coverage. The proposed architecture combines text, audio, speaker identity, synchronized processing, seed poses, and adversarial training.
- End-to-end Gesture Generation Methods: Rule-based systems require substantial human effort, while end-to-end approaches learn directly from raw gesture data without predefined gesture units.End-to-end regression increases expressive capacity by removing restrictions imposed by handcrafted units and rules.
- End-to-end Gesture Generation Methods: Previous end-to-end models mainly used either speech text or audio, although both semantics and acoustics are important for co-speech gestures.The paper addresses this limitation by using text and audio together.
- Learning Styles of Gestures: Earlier style-generation methods trained separate models for each speaker or style, requiring substantial data and covering only a few individual styles.The proposed work instead learns a manipulable style embedding space from speaker identities.
- Processing Multimodal Data: The model treats gesture generation as translating speech context into pose sequences with separate encoders for text, audio, and speaker identity and a gesture decoder.Inputs are temporally synchronized so the sequential model can generate poses frame by frame.
- Processing Multimodal Data: The generator uses nearby text and audio context because speech and gestures are temporally synchronized, while speaker identity remains constant during short-term synthesis.Seed poses provide continuity between consecutive syntheses.
- Adversarial Scheme: Adversarial training uses a discriminator to distinguish real human gestures from generated gestures, encouraging more realistic outputs.The discriminator produces time-step decisions that are aggregated into a final real-or-generated classification.
4 TRAINING WITH “IN-THE-WILD” VIDEOS
The model is trained on an expanded TED gesture dataset with aligned text, audio, and poses, using reconstruction, adversarial, and style-focused losses. Training data are split by video and speaker, and synthesis is fast enough for real-time use.
- Dataset: The dataset contains 1,766 TED videos with extracted poses, speech audio, and transcribed English speech text from various speakers.Its speaker diversity supports learning individual gesture styles.
- Dataset: 97 hours of clearly visible upper-body gesture footage were resampled at 15 frames per second into 34-frame training samples.The initial four frames served as seed poses, and the model generated the remaining 30 poses over two seconds.
- Dataset: Training, validation, and test sets were divided at the video level with no speaker-ID overlap between splits.The test set was used for qualitative results and human evaluation.
- Training Loss Function: The generator and encoders minimize Huber loss between human and generated pose sequences, while the discriminator is trained with the adversarial loss.The Huber objective combines L1- and L2-like behavior through a once-differentiable smooth loss.
- Training Loss Function: Diversity regularization discourages speaker-style features from being ignored and guides different style embeddings toward different generated gestures.A KL-divergence term also keeps the assumed Gaussian style embedding space from becoming too sparse.
- Synthesis: A 30-pose synthesis takes 10 ms on an NVIDIA RTX 2080 Ti GPU and 80 ms on an Intel i7-5930K CPU.The authors describe the model as lightweight enough for real-time synthesis.
5 OBJECTIVE EVALUATION METRIC
The paper introduces Fréchet Gesture Distance (FGD), an objective metric that compares latent feature distributions of human and generated gestures, and validates it on synthetic disturbances. FGD often tracks perceptual plausibility better than joint-coordinate error, but struggles with temporal noise and speech–gesture matching.
- 5.1 Fréchet Gesture Distance: FGD adapts the image-generation FID concept by comparing Gaussian statistics of latent features from real and generated gesture sequences.Because no general pretrained gesture feature extractor exists, the authors train an autoencoder and use its encoder to obtain latent features.
- 5.1 Fréchet Gesture Distance: The feature extractor is trained unsupervised on about 175 minutes of frontalized Human3.6M motion-capture data from seven actors across 17 scenarios.The data include scenarios such as discussions and purchases involving co-speech gestures.
- 5.2 Experiment with Synthetic Noisy Data: For Gaussian and Salt&Pepper noise, both FGD and MAEJ increase with disturbance, but FGD assigns larger distances to perceptually less plausible Salt&Pepper poses.Gaussian-noise samples remain more human-like, whereas Salt&Pepper noise can produce unrealistic poses such as a displaced neck.
- 5.2 Experiment with Synthetic Noisy Data: Both metrics give relatively low values to temporal noise, although the resulting discontinuous motion is perceptually unnatural.The FGD feature extractor does not discriminate these sequences sufficiently because the autoencoder tends to remove temporal noise during reconstruction.
- 5.2 Experiment with Synthetic Noisy Data: For multiplicative eigenpose transformations, FGD reports a much larger distance at ζ = 0.0 than at ζ = 2.0, unlike MAEJ’s similar distances.The authors consider this reasonable because exaggerated poses are perceptually preferable to repeated mean poses when gesture diversity is considered.
- 5.2 Experiment with Synthetic Noisy Data: For mismatched speech–gesture samples, both metrics increase with mismatch, but FGD rises less because it evaluates gesture distributions without access to input speech.The synthetic dataset also includes disturbances affecting content and speech rhythms.
6 USER STUDY TO VALIDATE EVALUATION METRICS
A user study compared objective gesture metrics with human judgments using noise models and trained gesture-generation models. FGD aligned better with human judgments than MAE-based metrics and provided a more informative training signal, though its agreement remained below human agreement.
- User-study design: Participants evaluated gesture videos through pairwise preference comparisons rather than Likert-scale ratings.The design reduced cognitive load because co-speech gestures are subtle.
- User-study design: Thirty 5–10-second speech samples were selected from the TED test dataset after excluding four faulty pose samples.Excluded cases involved object manipulation, sitting, or podium occlusion.
- User-study design: 22 subjects contributed analyzed responses in the noise-model session after six participants failed the attention check.The participants included 13 male and 9 female subjects and were 36.9 ± 11.5 years old.
- Metric validation: FGD showed greater agreement with human judgments than MAE of joint coordinates or acceleration across all evaluated questions.The comparison covered both synthetic noise models and trained gesture-generation models.
- Learning-curve validation: FGD decreased as generated-gesture distributions became more similar to reference distributions during training, whereas MAEJ remained flat.The lowest MAEJ occurred at Epoch 6, when static mean poses appeared across speech contexts; subsequent model comparisons therefore used FGD.
- User-study findings: Human preferences for gesture dynamics were mixed, indicating that gesture style should be adapted to user preference.Most participants preferred gestures fitting speech words and audio, while opinions differed on dynamic or exaggerated motion.
7 EXPERIMENTS AND HUMAN EVALUATION
Experiments compare the proposed trimodal model with prior gesture generators using qualitative, objective, and human evaluations. The model produces dynamic, speech-related gestures, performs best on FGD, and supports modality-dependent style and synthesized-audio behavior, although FGD does not disentangle quality from diversity.
- Qualitative Results: The proposed model generates metaphoric, iconic, deictic, and beat gestures aligned with speech content, while remaining still during silence.Examples include gestures for “civil rights,” “30 million,” “great leadership,” “I,” and “to the point.”
- Comparisons with State-of-the-Art Models: Compared with attentional Seq2Seq, Speech2Gesture, and joint embedding, the proposed model generates larger and more dynamic motions for the same speech.Joint embedding produced static poses, attentional Seq2Seq showed slow motion and discontinuities, and Speech2Gesture tended toward monotonous beat gestures.
- Comparisons with State-of-the-Art Models: The proposed model performed best in terms of FGD and achieved better objective and subjective results than the compared methods.Human evaluation used pairwise comparisons ranked with the Bradley–Terry model; differences from Speech2Gesture were not distinct for human-likeness and speech–gesture match.
- Ablation Study: Removing text, audio, or speaker identity reduced performance, while removing adversarial or regularization terms worsened FGD and encouraged static poses.Excluding speaker identity degraded FGD the most among modality ablations.
- Ablation Study: FGD measures both motion quality and diversity, so its covariance-related statistics cannot concretely disentangle these factors.The authors hypothesize that covariance is more related to diversity than quality and report a covariance trace of 244 for the full model.
- Incorporating Synthesized Audio: The model works with synthesized audio across voices, prosody, speed, and pauses, and different speaker identities produce different gesture styles for the same speech.Fast speech produces rapid motion, inserted pauses produce static poses, and style vectors correspond to different temporal motion variances.
8 CONCLUSIONS AND LIMITATIONS
The paper concludes that its trimodal architecture generates upper-body gestures matching speech content and audio, while enabling style variation through speaker-based embeddings. It also identifies limited controllability, metric shortcomings, and upper-body-only motion as areas for improvement.
- The model generates upper-body gestures from speech text, audio, and speaker ID using a temporally synchronized architecture.
- Generated gestures match speech text and audio, while sampling style vectors enables different gesture styles.
- The proposed Fréchet gesture distance was validated with synthetic noisy data and agreement with human judgments.
- Users cannot impose specific gesture constraints, FGD may miss subtle motion, and the model covers upper-body rather than whole-body behavior.
A DETAILED ARCHITECTURES
The appendix describes the detailed architectures of the model’s encoders, generator, discriminator, and Fréchet gesture distance feature extractor.
- Figure 12 details the audio encoder, text encoder, speaker embedding, gesture generator, and discriminator architectures.
- Figure 13 details the feature extractor architecture used for the Fréchet gesture distance.
B MODELS IN HUMAN EVALUATION
This appendix section identifies the models used in the human evaluation and references the detailed architectures of the generation and evaluation components.
- Table 4 lists the models used in the human evaluation.
- Figure 12 provides detailed architectures for the encoders, gesture generator, and discriminator.
- Figure 13 provides the detailed architecture of the autoencoder used for Fréchet gesture distance.