Source-linked AI summary
Gesticulator: A framework for semantically-aware speech-driven gesture generation
Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexanderson, Iolanda Leite, Hedvig Kjellström
TL;DR
Existing systems generally use either acoustic or semantic speech representations, limiting the gesture types they can model together. This paper combines both representations to generate continuous joint-rotation gestures, and reports that using both modalities can improve models while its gestures are preferred over a CNN-GAN baseline. The approach requires labor-intensive annotated transcriptions and is constrained by the dataset’s sub-optimal vocabulary.
Problem
Existing co-speech gesture systems typically use a single speech modality, limiting joint modelling of acoustically linked beat gestures and semantically linked gestures.
Method
The model combines acoustic and semantic speech features to generate continuous gestures represented as joint angle rotations.
Results
Using both speech modalities can improve continuous gesture-generation models, and participants preferred the model’s gestures over the CNN-GAN baseline.
Takeaways & Limitations
Autoregressive connections can enforce gesture continuity, while PCA may remove perceptually important variation and reduce gesture range.
Takeaways & Limitations
The method requires labor-intensive annotated text transcriptions, and the dataset’s sub-optimal vocabulary makes semantic relations difficult to learn.
Abstract
from arXiv · showhide
During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current end-to-end co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying "high"): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code and video are available at the project page https://svito-zar.github.io/gesticulator .
1 INTRODUCTION
Co-speech gestures support engaging human-agent interaction, but existing data-driven systems typically model only acoustic or semantic speech information. The paper introduces a continuous gesture-generation model that combines both modalities.
- Co-speech gestures accompany speech content at levels ranging from partial word meanings to situation descriptions.
- Appropriate nonverbal behavior can make virtual-agent interactions more engaging and influence perceptions of an agent’s emotions.
- Rule-based systems are constrained by a discrete gesture inventory, motivating data-driven approaches that generate continuous motion.
- The proposed model maps speech acoustic and semantic features into continuous 3D gestures.
- The study compares architectures and modelling choices, then evaluates audio and semantic modalities using objective and subjective measures.
- The work extends the Trinity College dataset with manual text transcriptions and provides evaluation videos.
2.1 Background
Speech and gesture are strongly correlated, with gesture categories differing in their dependence on speech content or acoustics. The paper formulates generation as predicting a pose sequence from frame-level speech features using temporal context.
- Iconic, metaphoric, and deictic gestures depend on speech semantics, whereas beat gestures correlate with acoustic prosody.
- Gesture-speech alignment has been studied across languages, including French, German, and English.
- In English, gestures typically precede corresponding speech by an average of 0.22 s, with onset timing varying across gesture types.
- The model uses 1 s of future speech and 0.5 s of past speech to predict corresponding gestures.
- Given a sequence of speech features, the task is to generate a corresponding sequence of poses for an agent.
- Each pose is represented as joint rotations across body keypoints and three rotational axes.
2.2 Related Work
Data-driven gesture generation has progressed from audio-only and text-based systems toward multimodal inputs, but prior multimodal approaches generally predict discrete gesture classes. Continuous motion quality is a separate concern addressed through smoothing-related modelling choices.
- The review focuses on data-driven gesture-generation methods.
- Most prior systems use audio as the only speech-input modality, including models that generate arbitrary 3D motion.
- Other systems map text transcripts to gesture concepts, functions, or continuous gestures using probabilistic or recurrent models.
- Only a handful of prior works use multiple speech modalities, and the cited examples predict discrete gesture sets or classes.
- Continuous gesture generation avoids concatenation discontinuities found in playback-based motion graphs.
- Prior continuous-motion systems use postprocessing, velocity penalties, or recurrent connections to encourage smooth output.
3 TRAINING AND TEST DATA
The model is trained and tested on the Trinity Gesture Dataset, whose audio and motion-capture recordings were supplemented with manually reviewed transcriptions. Text is converted to frame-level semantic features and aligned with audio using word durations and special encodings.
- The Trinity Gesture Dataset contains 244 minutes of audio and motion-capture recordings from a male actor speaking freely on varied topics.
- The retained motion data cover 15 upper-body joints, excluding lower-body data and fingers because of poor data quality.
- Audio was transcribed with automatic speech recognition, then manually reviewed to correct errors and add punctuation.
- Testing used 50 ten-second segments: 30 random segments and 20 semantically linked segments identified through unanimous annotator agreement.
- 3.2 Audio-Text Alignment: BERT encodes non-filler words, whose vectors are repeated according to word duration to create frame-level text features.
- 3.2 Audio-Text Alignment: Filler words and silence receive fixed vectors, with silence encoded distinctly, and ASR timings nonuniformly upsample text features to audio length.
4 SPEECH-DRIVEN GESTURE GENERATION
The model generates upper-body motion from frame-synchronized acoustic and semantic speech features. It aligns text and audio, incorporates temporal context and previous poses, and trains with position and velocity losses.
- Input and motion representations: Acoustic and semantic speech features are combined to generate upper-body gesture motion at 20 frames per second.Audio uses log-power mel-spectrograms, while semantics use BERT representations supplemented by frame-wise scalar features.
- Input and motion representations: Motion-capture joint angles are converted to exponential-map features and reduced with PCA to 12 components retaining 92% of training variance.
- Model architecture: The feed-forward architecture concatenates encoded text and audio, uses a 0.5-second past and 1-second future speech window, and feeds output poses back autoregressively.
- Training procedure: Training uses aligned 70-frame sequences, with 40 central frames optimized by a loss combining pose-position MSE and velocity MSE.
- Training procedure: Seven epochs of pretraining without autoregression and progressively increasing teacher forcing prevent previous poses from overpowering speech information.
5 EVALUATION MEASURES
The evaluation combines objective motion statistics with subjective judgments of human-likeness and speech-relatedness. Objective measures examine positional error and motion dynamics, while user studies compare perceived gesture quality.
- Objective measures: Objective evaluation measures RMSE, acceleration, jerk, and acceleration histograms after converting generated gestures to 3D joint positions.
- Objective measures: Acceleration and jerk are averaged across frames and 14 joints, while velocity histograms compare generated and ground-truth motion in 1 cm/s bins.
- Subjective measures: Subjective studies assess whether virtual-character movements are human-like and reflect or help communicate the character’s speech.
- Study protocol: Participants evaluate shuffled side-by-side video pairs after training, with each study containing random segments, semantic segments, and attention checks.
6 ABLATION STUDY
The ablation study examines model components using objective motion statistics, wrist-velocity distributions, and perceptual comparisons. Results highlight autoregression, audio, text, and PCA as influential design choices, while FiLM and velocity loss have limited effects.
- Study design: The study trains seven system variants, including the full model, and compares them using objective and perceptual evaluations.
- Average motion statistics: Lower jerk indicates smoother motion, and the proposed model exhibits lower jerk than the original motion.
- Average motion statistics: Removing autoregression produces excessive jerk and loses continuity, identifying autoregression as important for smooth generated motion.
- Velocity histograms: Autoregression reduces fast wrist movements and makes velocity histograms more similar to ground truth, whereas removing PCA increases velocity and variability.
- Perceptual comparisons: Removing text drastically reduces perceived human-likeness and speech linkage, while removing audio is preferred less across all four perceptual questions.
- Perceptual comparisons: Removing autoregression significantly worsens perceived naturalness, without affecting whether gestures are semantically linked to speech.
- Perceptual comparisons: Removing FiLM improves Q3 and Q4 preferences, velocity loss has little perceptual effect, and No PCA is preferred on Q2, Q3, and Q4.
- Objective–subjective relation: Perceptual ratings change little for minor motion-statistic differences, while jerk closer to ground truth is preferred and excessive jerk is associated with unnatural motion.
7 ADDITIONAL EVALUATIONS AND COMPARISONS
The proposed model was benchmarked against CNN-GAN and ground-truth gestures using objective and perceptual evaluations. It was preferred over CNN-GAN, while ground-truth motion remained strongly preferred and semantic-linkage judgments were highly subjective.
- Comparison setup: The proposed model was compared with the CNN-GAN state-of-the-art baseline using objective and subjective evaluations.The comparison included motion statistics and pairwise preference tests.
- Objective evaluation: Acceleration and jerk for the proposed method were roughly half the ground-truth values, whereas CNN-GAN reached roughly twice the ground-truth values.These averages were computed over 50 test sequences.
- Perceptual comparison: 27 participants preferred the proposed model over CNN-GAN for human-likeness with p<.0001 and speech-reflectiveness with p<.02.The tests used binomial analyses excluding ties with Holm-Bonferroni correction.
- Ground-truth comparison: Ground-truth motion was preferred in 84–93% of comparisons across all questions, with every difference statistically significant.The ground-truth comparison retained 20 participants after exclusions.
- Semantic linkage: All three annotators marked more semantically linked gestures with text input than without text, but Cronbach’s alpha was below 0.5.The paired counts were 2 vs 0, 21 vs 9, and 9 vs 4; the low agreement indicates semantic classification was highly subjective.
8 CONCLUSIONS AND FUTURE WORK
The paper presents a data-driven model for continuous co-speech gestures linked to both audio and speech semantics. Its findings support multimodal input, autoregressive continuity with dedicated training, and caution against PCA-based motion reduction, while identifying annotation effort and vocabulary as limitations.
- Contributions: The model is presented as the first data-driven system capable of generating continuous gestures linked to both audio and speech semantics.The work targets co-speech gesture generation for a wider range of continuous motions.
- Findings: Using both audio and text can improve continuous gesture-generation models.The conclusion treats multimodal speech input as beneficial for the studied task.
- Findings: Autoregressive connections can enforce gesture continuity with few parameters, while pretraining prevents autoregressive information from overpowering speech inputs.The training scheme was introduced after observing convergence toward static poses.
- Findings: PCA in motion space can remove perceptually important variation and reduce the range of generated gestures.This conclusion concerns PCA use as in the compared prior work.
- Findings: Participants preferred the proposed gestures over the CNN-GAN baseline.This conclusion summarizes the perceptual comparison.
- Limitations and future work: The main limitation is reliance on labor-intensive annotated datasets containing text transcriptions.The authors suggest training directly on automatic speech-recognition transcriptions as a possible alternative.
- Limitations and future work: The dataset’s sub-optimal vocabulary makes learning semantic relations between gestures and text challenging.Of 50k total words, 4,230 are unique, and the first eight words account for 30% of spoken words.
- Limitations and future work: Future work includes stochastic modeling, larger datasets, and improved semantic coherence by treating gesture types separately.These directions are stated as extensions of the current model.