PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Review of Data-Driven Co-Speech Gesture Generation

Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Eje Henter, Michael Neff

arXiv:2301.05339v4cs.GRcs.CVcs.HCcs.LG

TL;DR

Co-speech gesture generation aims to produce natural, communicative movement for embodied systems despite gesture’s diverse functions and idiosyncratic motion. This review synthesizes theory, classical and deep-learning methods, datasets, and input modalities, then surveys progress and remaining challenges. It concludes that the field has advanced substantially but still needs better data, evaluation, motion quality, semantic grounding, stochastic modeling, and control.

  • Problem

    Gesture generation must model idiosyncratic, non-periodic motion serving diverse communicative functions, while existing systems often rely on limited audio or text inputs.

  • Method

    The review organizes prior work by input modality and covers gesture theory, rule-based and statistical synthesis, deep generative models, datasets, applications, and research challenges.

  • Results

    The survey finds substantial progress across audio-, text-, and control-driven generation, but the field remains young with significant unresolved challenges.

  • Takeaways & Limitations

    Future gesture systems need better datasets and evaluation, higher-quality and more meaningful motion, richer context grounding, stochasticity handling, output control, and broader interaction support.

  • Takeaways & Limitations

    Motion-only approaches can assess statistical human-likeness but not contextual appropriateness, speech synchronization, semantic relevance, or environmental grounding.

Abstract

from arXiv · show

Gestures that accompany speech are an essential part of natural and efficient embodied human communication. The automatic generation of such co-speech gestures is a long-standing problem in computer animation and is considered an enabling technology in film, games, virtual social spaces, and for interaction with social robots. The problem is made challenging by the idiosyncratic and non-periodic nature of human co-speech gesture motion, and by the great diversity of communicative functions that gestures encompass. Gesture generation has seen surging interest recently, owing to the emergence of more and larger datasets of human gesture motion, combined with strides in deep-learning-based generative models, that benefit from the growing availability of data. This review article summarizes co-speech gesture generation research, with a particular focus on deep generative models. First, we articulate the theory describing human gesticulation and how it complements speech. Next, we briefly discuss rule-based and classical statistical gesture synthesis, before delving into deep learning approaches. We employ the choice of input modalities as an organizing principle, examining systems that generate gestures from audio, text, and non-linguistic input. We also chronicle the evolution of the related training data sets in terms of size, diversity, motion quality, and collection method. Finally, we identify key research challenges in gesture generation, including data availability and quality; producing human-like motion; grounding the gesture in the co-occurring speech in interaction with other speakers, and in the environment; performing gesture evaluation; and integration of gesture synthesis into applications. We highlight recent approaches to tackling the various key challenges, as well as the limitations of these approaches, and point toward areas of future development.

1. Introduction

The introduction explains why gesture matters for communication and applications, then frames gesture synthesis as mapping communicative content to behavior. It positions the review as a deep-learning-focused survey organized around input modalities, datasets, and open challenges.

  • Why gesture matters: Gestures make artificial agents and robots appear more alive, communicate functional information, and convey social information.Examples include pointing, emblems, and iconic or metaphoric gestures.
  • Why gesture matters: Gesture complements speech by providing an additional, more direct motor code for spatial, object-related, and other information that symbolic language represents less naturally.Its iconicity can make ideas easier to comprehend and remember.
  • Why gesture matters: Nonverbal communication supplies social cues, and speech-only interaction can become more impersonal, contentious, and prone to deadlock than face-to-face interaction.The cited negotiation study reported greater praise and interpersonal communication face-to-face, but more blame, disagreement, and deadlock with speech only.
  • Why gesture matters: Gestures support learning through observation, production, and teacher adaptation, with pedagogical-agent gestures improving knowledge transfer, retention, and agent persona.A meta-analysis of twenty experiments did not find a reduction in cognitive load.
  • Review scope: Gesture synthesis is formulated as mapping a representation of communicative content to pose frames or lexicalized gesture representations, usually from audio and/or text.The introduction notes that this input choice has limitations.
  • Review scope: The review covers gesture theory, rule-based and statistical methods, deep-learning systems organized by input modality, datasets, and research challenges.Its contributions include discussion of audio, text, and non-linguistic inputs and potential research directions.

2. Human gesticulation

Human gestures are diverse, speech-linked movements whose forms range from conversational gestures to fully lexicalized sign languages. Their idiosyncrasy, timing, structure, and communicative-intent relationship create central challenges for synthesis.

  • Gesture foundations: Manual gestures are non-verbal, non-manipulative hand or arm movements occurring during speech that complement speech in timing, meaning, and communicative function.They can resolve references, illustrate concepts, and express personality, emotion, and motivation.
  • Gesture categories: Kendon’s Continuum orders gesture categories by increasing lexicalization, from conversational gestures dependent on speech to sign languages with formal syntactic structure.The absence of comparable structure in coverbal gesticulation contributes to its idiosyncrasy.
  • Gesture categories: McNeill describes gesture dimensionally as iconic, metaphoric, deictic, and beat, with individual gestures potentially combining multiple dimensions.Iconic gestures depict concrete properties, metaphoric gestures represent abstractions, deictic gestures point, and beats align with speech rhythm.
  • Gesture structure: Kendon’s hierarchy represents gesture units as sequences of phrases and phases, including preparations, strokes, holds, and retractions.The stroke carries the main meaning, while pre- and poststroke holds synchronize gesture with speech.
  • Synthesis challenges: Gesture synthesis systems generally derive motion from audio, text, or both rather than modeling communicative intent, which can make generated gestures redundant with those channels.The review links this limitation to the growth-point view that gesture and language emerge from common communicative intent.
  • Synthesis challenges: Gesture is idiosyncratic: different people can gesture differently, and the same person can vary gestures while delivering the same text.

3. Approaches for gesture synthesis

Gesture synthesis approaches span rule-based, statistical, and learning-based systems. The field has progressed toward deep learning, trading greater naturalness and continuity against communicative efficacy.

  • Method categories: Gesture generation methods are categorized as rule-based or data-driven, with data-driven systems divided into statistical and learning-based approaches.Rule-based systems use designed heuristics, statistical systems model gesture distributions, and learning-based systems optimize nonlinear speech-to-gesture mappings.
  • Generation mechanisms: Rule-based and statistical systems generally select or retrieve gesture labels and clips, whereas deep-learning systems synthesize motion frame by frame.
  • Field development: Early intent-driven approaches prioritized communicative efficacy but used less-natural predefined animations, while statistical methods improved naturalness with some loss of efficacy.
  • Field development: Modern deep-learning approaches generate continuous, fairly natural gestures but are significantly less communicative than earlier approaches.
  • Hybrid systems: Hybrid systems combine rule-based and deep-learning methods to pursue gestures that are both natural and semantically meaningful.

4. Rule-based approaches

Rule-based systems use engineered knowledge and predefined or procedurally generated motion to coordinate gestures with speech and other behaviors. They can achieve high-quality, semantically controlled animation, but their rules and motion inventories limit scalability and diversity.

  • Early systems: Animated Conversation initiated context-appropriate gestures, facial movements, and intonation between multiple human-like agents through dialogue generation and planning.
  • Interactive systems: Ymir added multimodal user input and action scheduling for synchronized hand animation, but its real-time output remained limited.
  • Behavior planning: BEAT transformed typed text into synchronized speech, gesture, facial animation, and intonation using contextual information and handcrafted heuristics.
  • Motion realization: Model-based systems such as Max generated gesture trajectories with non-uniform cubic B-Splines satisfying velocity and position constraints.
  • Behavior languages: BML unified several incompatible behavior-markup languages by connecting intent planning, behavior planning, and behavior realization.
  • Strengths and limitations: Rule-based systems can synchronize high-quality gestures and preserve semantics in limited domains, but require expert knowledge and produce inflexible, nond iverse gesture distributions.

5.1. Speech-Gesture Datasets

Speech-gesture datasets have grown substantially in size and speaker diversity, driven partly by pose estimation from video. This expansion comes with a persistent trade-off between scalable, varied data and accurate 3D motion, especially for fingers.

  • Dataset scale: Recent datasets exceed 100+ hours and include more speakers, with increased diversity enabled by 3D pose estimation from video.
  • Motion quality: Only a small fraction of datasets contain high-quality finger motion, limiting expressive and meaningful gesture generation.
  • Collection methods: Gesture motion is obtained mainly through optical motion capture or pose estimation from monocular video.
  • Collection methods: Motion-capture datasets are smaller and expensive to collect but provide high-quality 3D motion and the best finger capture.
  • Collection methods: Video-based datasets can be an order of magnitude larger and more naturalistic, but generally have lower motion quality and are limited to 2D motion.
  • Data requirements: The data required for adaptation depends on the application, and target-speaker data is generally more useful than equivalent data from non-target speakers.

5.2. Statistical and early machine learning approaches

Statistical and early machine-learning systems model speech-gesture relationships through distributions, probabilistic sequence models, and motion graphs. They increase flexibility and non-determinism over rule-based systems, while remaining constrained by simplified variables and data representations.

  • Statistical modeling: Statistical systems model speech-gesture relationships through gesture distributions rather than expert-encoded rules, using conditional probabilities or prior distributions.
  • Gesture profiles: Kipp modeled individual gesture profiles from annotated co-speech data, including properties such as handedness, timing, and communicative function.
  • Gesture profiles: Neff et al. extended gesture profiles with a character-specific animation lexicon and generated final animation through kinematic or dynamic simulation.
  • Bayesian models: Bayesian approaches modeled gesture morphology using object referents, discourse context, and the previously performed gesture.
  • Prosody-based models: Hidden Markov and hierarchical conditional models used prosodic features to select or generate gesture sub-units associated with speech rhythm and tone.
  • Motion graphs: Motion graphs generated synchronized conversational body motion by searching large databases under plausibility constraints while balancing efficiency and stylistic diversity.
  • Strengths and limitations: Statistical systems offer more flexibility and conversational non-determinism than rule-based systems, but typically model only a limited number of independent variables.

5.3. Deep learning approaches

Deep-learning gesture generation has expanded through diverse generative models, input modalities, and control strategies, improving perceived naturalness while leaving communicative appropriateness and style specification as open challenges.

  • Generative models: Deep generative models learn gesture distributions with few assumptions, using gradient-based optimization and architectures including VAEs, GANs, and normalizing flows.Deterministic regression can produce averaged motion, motivating probabilistic and adversarial components.
  • Open challenges: Deep learning substantially improves perceived naturalness, but communicative efficacy and appropriateness remain challenging compared with earlier intent-driven approaches.Later systems incorporate semantics, yet the review identifies appropriateness as an unresolved issue.
  • Input modalities: Researchers organize deep-learning systems by input modality, including audio, text, combined audio-text, non-linguistic modalities, and control parameters.The review uses this taxonomy to categorize systems that generate hand gestures.
  • Hybrid and retrieval approaches: Motion graphs combine audio, text, style, and rhythm signatures to retrieve gesture clips while minimizing transitions, performing on par or better than motion capture for GENEA 2022 Naturalness.The method matches signatures using Hamming distance and graph optimization.
  • Context and interaction: Interlocutor-aware models achieve statistically significant improvements in generated behaviors, showing the value of modeling conversational partners.This finding spans normalizing-flow and conditional-adversarial approaches for head, facial, and full-body behavior.
  • Control input: Style control remains difficult because animator-facing abstract intent does not map cleanly to biomechanical parameters and can require training data comparable to the dataset size.Recent work therefore explores more data-efficient style specification.

6. Key Challenges of Gesture Generation

Gesture generation remains difficult because human gestures are spontaneous, idiosyncratic, and non-periodic, while evaluation must separate human-likeness from contextual appropriateness. The review identifies evaluation, data, motion quality, grounding, and multimodal synthesis as central challenges.

  • Challenges: Rule-based systems produce well-formed gestures but lack flexibility and diversity, whereas data-driven systems improve both while retaining major challenges.Hand-designed rules are non-exhaustive and may not reflect naturally occurring gestures.
  • Evaluation: Evaluation is difficult because gesture generation is stochastic and human perception of gestures is highly subjective.Human-likeness is often assessed through perceptual studies, while appropriateness and specificity are harder to isolate.
  • Evaluation: Matched-versus-mismatched gesture studies help evaluate appropriateness while reducing interference from differences in motion human-likeness.The paradigm asks participants to choose between gesture motions paired with matching or mismatching contexts.
  • Evaluation: Objective motion metrics are attractive because subjective studies are expensive and time-consuming, but prediction errors are problematic for one-to-many gesture realizations.Different valid motions for the same speech can have low mutual agreement, making MSE an imperfect measure.
  • Evaluation: Motion-only metrics assess statistical similarity to captured motion but cannot determine synchronization, semantic relevance, or contextual appropriateness.The review notes limited objective techniques for diversity and appropriateness, despite newer semantic metrics such as SRGR.

6.2. Data

Co-speech gesture generation is data-limited, with finger motion especially difficult to capture reliably. More scalable extraction from in-the-wild video could substantially expand both dataset size and motion authenticity.

  • Data: Gesture generation has less data than machine-learning fields such as text, speech, and images, with finger motion a particular capture bottleneck.Motion capture is high quality but laborious, while unreliable finger data may require prediction from the rest of the body.
  • Data: Reliable 3D gesture extraction from monocular in-the-wild video could transform the field by expanding data and improving perceived authenticity.The review compares this potential shift with large datasets that powered advances in text and image generation.

6.3. Human-Like Gestures

Deep generative systems are improving in perceived human-likeness, but convincing gesture motion remains below completely human-like ratings and is affected by capture and avatar limitations.

  • Human-Like Gestures: Perceptual quality has been a primary research target because it is easier to estimate than gesture appropriateness for speech.The review relates this emphasis to the availability of standardized objective metrics in other generative fields.
  • Human-Like Gestures: Large evaluations provide strong evidence that the human-likeness of leading gesture-generation systems is improving.The review notes that head motion may be easier to make indistinguishable from human motion than arm and finger motion.
  • Human-Like Gestures: GestureMaster’s synthesized upper- and full-body gestures were rated higher than the original human speaker’s motion capture in GENEA Challenge 2022.The review cautions that motion-capture artifacts, especially for fingers, may have reduced the perceived human-likeness of the reference motion.
  • Human-Like Gestures: GENEA Challenge 2022 median human-likeness scores were 69 for the best upper-body system and 71 for the best full-body system, versus 63 and 70 for human motion capture.Neither the best system nor human motion capture approached the rating of 100 corresponding to completely human-like motion.
  • Human-Like Gestures: The review expects large-data deep-learning approaches to continue shaping gesture generation, following the trajectory of text-to-speech synthesis.It describes data-driven generative modeling as having become state of the art in co-speech gesture generation.

6.4. Multimodal Grounding

Human-like movement alone is insufficient: generated gestures must be grounded in speech, speaker state, interlocutor behavior, and spatial context. The review highlights weak correlations, ambiguous realizations, limited data coverage, and inadequate authoring control as persistent obstacles.

  • Multimodal Grounding: Gesture generation must produce movement that is both human-like and appropriately grounded in the interaction context.Relevant grounding dimensions include temporal information, semantic content, speaker characteristics, interlocutor behavior, and spatial information.
  • Temporal Grounding: Gestures are temporally related to acoustic prosody, motivating systems that explicitly use prosodic features for rhythmic gesture generation.Beat gestures have been studied as a central connection between gesture timing and speech acoustics.
  • Semantic Grounding: Speech acoustics alone are insufficient for semantically appropriate gestures because speech semantics and gesture form correlate weakly.Text provides a compact representation of semantic content, and including it has improved perceived gesture quality, although semantic content remains difficult to measure.
  • Identity, Style, Emotion, and Personality: Gesture variation depends on speaker identity, emotion, and context, while current systems remain far from authoring characters with imagined personalities.Large multi-speaker datasets enable personalization, and short motion clips support zero-shot style adaptation, but novel-character control remains limited.
  • Interaction and Spatial Grounding: Models must account for interlocutor behavior and spatial information, since listener reactions and deictic gestures depend on external conversational and environmental context.A phrase such as “You need to go that way” lacks the directional information needed to produce a communicatively meaningful deictic gesture.
  • Ambiguity: Because many factors combine ambiguously and exhaustive datasets are unlikely, probabilistic models are a compelling way to handle multiple possible gesture realizations.The review compares this ambiguity with variation in prosody for repeated speech and notes the combinatorial explosion of possible conditions.
  • Creative Intent: Automatic generation scales motion production but struggles with high-level creative control and can deviate from intended specifications.Existing controls may use low-level signals such as wrist height or velocity, while multi-speaker systems may not capture scenario-specific gesture variability.

6.5. Multimodal Synthesis

The review describes efforts to unify verbal and non-verbal synthesis, including full-body gesture modeling and single networks that generate speech audio and gesture motion.

  • Recent work explores full-body gesture motion, including the head and lower body.
  • Some systems train text-to-speech and speech-to-gesture models on shared data before merging them into one network.
  • Joint speech-and-gesture generation may improve modeling efficiency by providing prosodic prominence directly rather than requiring gesture systems to extract it from audio.
  • Generating gestures from text or audio is described as restricting systems to redundant gestures, motivating deeper representations of communicative intent.

7. Broader Impact

Gesture synthesis could support assistive, educational, healthcare, and entertainment applications, while raising concerns about bias, deepfakes, and effects on human behavior.

  • High-quality gesture synthesis could support tutors, rehabilitation trainers, health and eldercare agents, and personal assistants.
  • Human-data models raise concerns about whether training data represent diverse nonverbal behavior across language groups, ethnicities, and social strata.
  • Synthetic nonverbal motion could facilitate deepfakes and nefarious uses such as political propaganda or fake news.
  • Because people may entrain to synthetic models, researchers and developers should devise measures to mitigate possible effects on users’ behavior.

8. Conclusion

The review finds substantial progress in deep-learning gesture generation, but emphasizes that the field remains young and faces major challenges in data, motion quality, evaluation, control, stochasticity, and contextual grounding.

  • The field has progressed from early rule-based systems to extensive recent work using deep-learning approaches.
  • Most systems focus on monologue generation, while research is beginning to address dialogue and richer context.
  • Key challenges include better datasets, stronger subjective and objective evaluation, higher-quality and more meaningful motion, and adequate control.
  • Future systems must better address gesture stochasticity and grounding in multi-person interaction and spatial conversation context.
Loading 2301.05339v4…