Source-linked AI summary
Learning Speech-driven 3D Conversational Gestures from Video
Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, Christian Theobalt
TL;DR
Speech-driven conversational animation must generate multiple plausible body gestures while coordinating them with facial and head motion. The paper addresses this with a unified CNN/GAN model and a large in-the-wild 3D annotation pipeline, achieving strong user-study ratings for naturalness and synchronization.
Problem
Existing speech-driven animation methods often separate body and face synthesis and rely on limited studio motion-capture data, while conversational body gestures remain multi-modal.
Method
A unified CNN uses shared facial, body, and hand processing with adversarial training, learned from monocular 3D annotations extracted from more than 33 hours of in-the-wild video.
Results
User studies rated the synthesized results 4.05 ± 0.85 for naturalness and 4.00 ± 0.91 for synchronization with speech, compared to other baselines.
Takeaways & Limitations
The framework supports simultaneous full 3D face, body, and hand gesture prediction for speech-driven virtual characters.
Takeaways & Limitations
The paper does not claim state-of-the-art face-only expression synthesis and leaves that comparison for future work.
Abstract
from arXiv · showhide
We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. Synthesis of conversational body gestures is a multi-modal problem since many similar gestures can plausibly accompany the same input speech. To synthesize plausible body gestures in this setting, we train a Generative Adversarial Network (GAN) based model that measures the plausibility of the generated sequences of 3D body motion when paired with the input audio features. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people. To this end, we apply state-of-the-art monocular approaches for 3D body and hand pose estimation as well as dense 3D face performance capture to the video corpus. In this way, we can train on orders of magnitude more data than previous algorithms that resort to complex in-studio motion capture solutions, and thereby train more expressive synthesis algorithms. Our experiments and user study show the state-of-the-art quality of our speech-synthesized full 3D character animations.
1. Introduction
The paper targets speech-driven synthesis of synchronized 3D body, hand, face, and head motion for conversational virtual characters. It combines a large in-the-wild training corpus with a unified model designed for expressive, multi-modal gesture generation.
- Motivation: Speech-driven animation aims to replace manual key-frame work for conversational gestures and facial expressions in virtual characters.The intended applications include easier authoring and real-time embodied conversational avatars.
- Motivation: Prior methods typically generated co-verbal body gestures or facial expressions separately and often relied on complex studio motion capture.This limited the ability to capture large corpora with subject variation and personal idiosyncrasies.
- Approach: The proposed system jointly generates synchronized 3D arm, torso, hand, face, and head motion from speech.The approach is presented as the first to combine these conversational animation modalities in one framework.
- Data: More than 33 hours of in-the-wild talking-person video are annotated for 3D face, body, and hand data to train the system.The annotations are obtained with monocular reconstruction and pose-estimation methods.
- Approach: A shared CNN encoder learns correlations among facial expression, body, and hand gestures, with separate decoder heads producing their motion sequences.The model also synthesizes beat gestures and motion reflecting mood and conversational style.
- Approach: The method uses adversarial training because multiple plausible gestures can accompany the same speech input.This is intended to avoid convergence toward mean poses and preserve expressive body motion.
2. Related Work
Prior work addressed speech-driven body gestures, head motion, and facial animation largely as separate problems using rules, learned models, or studio-captured data. More recent methods improve generation with deep learning, semantic information, and generative models, while this paper combines the relevant body modalities in one framework.
- Problem setting: Unlike speech-driven facial animation, conversational gesture synthesis is more multi-modal because body motion has less unique correspondence with audio.The paper therefore studies face and gesture generation together rather than separately.
- Rule-based methods: Earlier systems used hand-crafted linguistic or audio rules to synchronize body gestures and facial expressions with speech.Rule-based methods depend on language-specific expertise and may not handle non-phoneme sounds easily.
- Data-driven methods: Data-driven methods replaced expert rules with gesture scripts, probabilistic models, recurrent networks, and generative approaches.Examples include HMMs, CRBMs, GPLVMs, LSTMs, and GANs for speech-driven motion synthesis.
- Recent models: Recent work incorporates text semantics, normalizing flows, and mixture models to generate multiple gestures or transfer style between speakers.These approaches provide forms of variation or control, including gesture speed, symmetry, and speaker style.
- Training data: Deep speech-driven animation commonly depends on large audio-motion corpora collected with expensive in-studio motion-capture systems.In-the-wild video offers a route to much larger training corpora than studio capture.
- Positioning: The paper extends an in-the-wild 2D gesture dataset with 3D face, hand, and body annotations and adds head rotation and facial expression synthesis.This broadens prior 2D upper-body and hand generation toward full 3D conversational animation.
3. Dataset Creation
The dataset is created by automatically extracting 3D face, head, hand, and upper-body annotations from in-the-wild conversational video. This addresses the training-data bottleneck while retaining long temporal windows and subject diversity.
- Motivation: Complex in-studio capture is identified as a major bottleneck for obtaining sufficient speech-animation training data.The proposed alternative uses community video with accompanying audio.
- Training data: Figure 2 depicts in-the-wild conversational videos annotated with 3D dense-face, hand-pose, and body-pose parameters.These annotations form the training data for the speech-driven model.
- Source corpus: The source corpus contains 144 hours of video from 10 subjects recorded while talking to the camera in standing and sitting poses.The original annotations included 2D arm, hand, and sparse face landmarks.
- 3D annotation: The paper augments the corpus with 3D face performance capture and monocular 3D body and hand pose estimation.The resulting targets include face parameters, head pose, torso motion, and hand and upper-body gestures.
- Robustness: Body tracking is selected to handle self-occlusion, occlusion by people or desks, and incomplete camera framing.Hand tracking uses tightly cropped hand images and interpolation for missing poses between visible observations.
- Filtering: Low-confidence face or hand predictions are filtered, and four of the ten source subjects are removed because of low-resolution data.These steps improve robustness to monocular tracking errors.
- Audio features: Audio features combine 13 MFCC coefficients with log mean energy computed over overlapping temporal windows.Speech characteristics support facial-expression prediction, while temporal MFCC changes provide rhythm information for beat gestures.
4. Method
The method maps speech features to synchronized 3D face, body, and hand motion using a shared encoder with separate decoders. It combines supervised regression with adversarial training to produce expressive, audio-synchronized body and hand gestures.
- Network architecture: Speech features are mapped to temporal sequences of 3D facial expression, body pose, and hand pose parameters.The generator uses a 1D convolutional encoder-decoder architecture for this mapping.
- Network architecture: A shared encoder learns correlations across face, body, and hand motion, while separate decoders generate each sequence.The architecture is based on a single encoder and distinct face, body, and hand decoder heads.
- Adversarial training: The discriminator predicts whether paired audio and pose features are real or generated, encouraging gestures that are plausible and synchronized with speech.The adversarial input uses audio together with predicted body and hand pose; facial expression parameters are excluded because body gestures are treated as multimodal.
- Training procedure: Training uses 64-frame sub-sequences sampled with sliding-window overlap, and networks are trained on subject-specific datasets.Subject-specific training is used to capture individual gesture characteristics.
- Training objective: The regression loss combines facial-expression L2 error with body and hand L1 errors.The body term covers 3D body keypoint locations and head orientation, while the hand term covers 3D hand keypoint locations.
- Training objective: The loss weights are w1 = 0.37, w2 = 600, and w3 = 840 so the three regression terms are equally weighted.The overall objective additionally combines direct supervision with adversarial training, using w = 5 for the adversarial term.
5. Results
The paper evaluates its full face, body, and hand synthesis through user studies and facial keypoint comparisons. Participants rated the method highly for naturalness and speech synchronization, while facial expression quality was comparable to or slightly better than baselines but not claimed as state of the art.
- User studies: User studies assess naturalness and audio synchronization for synthesized full-character and gesture-only animations.The first study evaluates face, body, and hand gestures; the second isolates body and hand gesture quality while standardizing facial expressions.
- User studies: The method was rated more natural and more in-sync with audio than the MoGlow-based gesture model in the gesture-only study.Participants were instructed to ignore facial-expression quality, and all videos used facial expressions predicted by the proposed method.
- Facial expression evaluation: Facial lip-keypoint evaluation over 578 test sequences found similar or slightly better performance than other proposed baselines.The comparison used generated face vertices against image-based tracker keypoints in a neutral head pose.
- Facial expression evaluation: The unified architecture provides decent simultaneous face-expression synthesis, but the authors do not claim advances in face-only expression synthesis.The paper explicitly leaves face-only state-of-the-art improvement outside its scope and for future work.
6. Discussion
The discussion highlights the multimodal and noisy nature of conversational gesture synthesis, where plausible gestures are not uniquely determined by speech. It also motivates adversarial evaluation and notes subject-specific qualitative results.
- Body gestures lack a one-to-one mapping with speech, making stable expressive synthesis difficult alongside noisy monocular data.
- Lower validation L1 or L2 loss does not always correspond to qualitatively better gesture synthesis, supporting the importance of adversarial loss.
- Figure 4 presents subject-specific qualitative results using separately trained models for each represented subject.
- Different generator–discriminator update schedules can produce distinct gestures that are nevertheless equally plausible.
- The discriminator is proposed as an independent plausibility metric for rating speech-conditioned gesture synthesis.
7. Conclusion
The paper presents full 3D speech-driven prediction of facial, body, and hand gestures for virtual characters. It uses in-the-wild monocular tracking to obtain approximately 33 hours of training annotations and an adversarial audio–pose penalty to generate expressive, synchronized gestures.
- The approach predicts full 3D face, body, and hand gestures from speech to drive virtual characters or embodied conversational agents.
- Approximately 33 hours of in-the-wild footage provide 3D face, body, and hand pose annotations for training.
- An adversarial penalty on 3D pose and its combination with audio produces expressive body gestures synchronized with speech.
A. On 3D Annotations of In-The-Wild Video
The paper constructs training annotations from in-the-wild video using monocular 3D tracking, while acknowledging that these pseudo-ground-truth labels are less accurate than multi-view capture and cannot be quantitatively validated directly.
- In-the-wild videos are annotated with monocular 3D tracking approaches for body, hand, and face data.
- The pseudo-ground-truth annotations are less accurate than those from standard multi-view motion- or performance-capture systems.
- The quality of the 3D annotations cannot be quantitatively measured against human annotations in this dataset.
- Hand visibility is often limited, especially in standing-subject videos, so filtering allows some occluded-hand cases to remain in the dataset.
- Gaussian filtering with σ = 1.5 temporally smooths predicted body, hand, and head motion as well as ground-truth sequences.
B. Network Architecture
The supplied passages provide only a reference to the network architecture diagram and a table caption describing discriminator-based audio–body synchronization classification.
- Figure S3 is identified as a diagrammatic representation of the network architecture.
- Table S1 reports discriminator performance for classifying synchronized versus unsynchronized audio–body pairs on Oliver test sequences.
C. Further Analysis of the Discriminator
The discriminator can provide synchronization feedback during training, but its use as a general gesture-plausibility metric is not yet reliable. Longer training sequences improve model performance, while classifier generalization requires more varied off-sync data.
- Metric limitations: A classifier trained only on ground-truth motion sequences gives inconsistent plausibility rankings on baseline-generated motions.It rates the proposed model above ground truth and gives the highest score to the CNN-only baseline, contradicting the user-study result.
- Metric limitations: A more general gesture-plausibility metric requires an in-sync/off-sync dataset containing varied gesture-noise characteristics.Randomly mismatched audio–gesture pairs are insufficient for reliable evaluation across generated motion types.
- Training feedback: The discriminator can provide the generator with synchronization-quality feedback during training, unlike a vanilla gesture-only discriminator.The experiment identifies audio–gesture synchronization as a useful training signal.
- Context length: Longer training sequences lead to better model performance.The context-window experiment links longer training sequences with improved performance.