Source-linked AI summary
Learning Individual Styles of Conversational Gesture
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, Jitendra Malik
TL;DR
The paper asks whether person-specific conversational gestures can be predicted from speech in the wild despite asynchronous, multimodal motion and noisy pose supervision. It introduces an audio-to-pose model with temporal context and adversarial training, and reports that it outperforms other methods while releasing a large speaker-specific dataset.
Problem
The paper asks how much gesture motion can be predicted from raw speech audio despite asynchronous, multimodal gestures and limited human annotations.
Method
The model translates raw audio into speaker-specific arm and hand pose sequences using temporal context, L1 regression, and adversarial training on noisy pose detections.
Results
The model quantitatively outperforms several baselines and a previous adapted method.
Takeaways & Limitations
The released 144-hour dataset and speech-to-gesture task support computational analysis of conversational gesture and further cross-modal research.
Takeaways & Limitations
Audio lacks high-level semantics and clean separation from other sounds, while automatic pose estimates introduce substantial noise, especially for fingers.
Abstract
from arXiv · showhide
Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at http://people.eecs.berkeley.edu/~shiry/speech2gesture .
1. Introduction
The paper studies whether speech and gesture are related by predicting person-specific arm and hand motion from raw audio. It addresses asynchronous, multimodal gestures and noisy pose supervision with temporal context, regression, and adversarial learning.
- Speech and gesture are parallel communication channels that can complement spoken content with non-verbal information.
- The paper asks to what extent gesture motion can be predicted from raw speech audio, amid debate over whether the modalities are redundant or complementary.
- Gesture and speech are asynchronous and multimodal, while large-scale human annotations are infeasible, motivating training from automatic 2D pose detections on unlabeled video.
- The method uses large past-and-future temporal context to handle asynchronicity and support smooth gesture prediction.
- An adversarial discriminator supplements regression so predicted motion avoids mean-like outputs and remains realistic for the current speaker.
- The released dataset contains 144 hours of person-specific video from 10 speakers spanning varied backgrounds and topics.
2. Related Work
Prior work studied gesture recognition and generation largely in controlled settings or with text, rules, motion capture, or simplifying temporal assumptions. This paper instead targets person-specific co-speech gesture prediction from audio in the wild.
- Psychological studies often used few subjects, choreographed story retelling, and manual analysis, whereas this work studies gestures in the wild with data-driven methods.
- Gesture timing relative to speech remains disputed because gestures may cooccur with, precede, or follow the corresponding utterance.
- Computer vision has recognized sign language, emblematic, head, and co-speech gestures, but this paper predicts co-speech gestures from audio.
- Conversational-agent systems have included manually specified rules and learned mappings from speech or text to gesture.
- Earlier learned systems commonly relied on lab-recorded data, precise 3D pose, syllable-boundary timing, wrist-motion assumptions, or explicit language information.
- Related work also synthesized gestures or synchronized audio and visual motion, while this paper focuses on personalized in-the-wild speech-to-gesture translation.
3. A Speaker-Specific Gesture Dataset
The authors release a 144-hour in-the-wild video dataset designed for data-driven study of individual speakers’ speech and gesture. It provides skeletal pose sequences from automatic detection, whose quality is assessed against human labels.
- The dataset contains 144 hours of in-the-wild video from 10 speakers, including talk-show hosts, lecturers, and televangelists.
- Several hours are collected per speaker to support individually modeling diverse gesturing styles and topics.
- Gesture pose is represented over time with 49 OpenPose keypoints covering the neck, shoulders, elbows, wrists, and hands.
- The released skeletal keypoints are provided for every video frame at 15fps, but they are pseudo ground truth rather than human annotations.
- Automatic pose detection has more error than human annotation but enables training on substantially larger quantities of data.
- The authors compare automatic pose detections with human-observer labels to assess whether annotation accuracy supports quantitative conclusions.
4. Method
The method maps speech audio to temporal 2D pose sequences with a fully convolutional encoder–UNet architecture. L1 regression supplies supervision from noisy detections, while temporal context and adversarial training encourage smooth, speaker-plausible motion.
- The model learns speech-to-gesture mapping with L1 regression from corresponding audio and automatically detected pose sequences.
- A convolutional audio encoder converts a 2D spectrogram into a 1D signal, and the translation model predicts a temporal stack of 2D poses.
- The model predicts the whole pose sequence at once using full utterance context, rather than recurrently, to promote temporal coherence and smoothness.
- The UNet bottleneck supplies past and future context, while skip connections preserve high-frequency information needed for fast motion.
- Because L1 regression can produce overly smooth mean motions, an adversarial discriminator operates on pose differences to encourage realistic speaker motion.
- Training uses approximately 4 seconds of audio to predict 64 pose vectors corresponding to about 4 seconds at 15Hz.
5. Experiments
The method quantitatively outperforms several baselines, including an adapted previous method, for speech-to-gesture translation.
- The method quantitatively outperforms several baselines and an adapted previous method.
5.1. Setup
The experiments compare against pose, random-gesture, nearest-neighbor, and RNN baselines using L1 loss and PCK, while noting limitations of PCK for this task.
- Baselines: The study compares against median-pose, randomly chosen gesture, nearest-neighbor, and RNN-based baselines.The RNN baseline uses a 1-layer LSTM with MFCC features.
- Evaluation Metrics: L1 regression loss is the main metric, supplemented by percent of correct keypoints (PCK).PCK counts a keypoint as correct within α max(h, w) pixels of the ground truth.
- Evaluation Metrics: PCK is less suited than L1 for this cross-modal prediction task because it is thresholded and sensitive to gesture magnitude.
5.2. Quantitative Evaluation
Experiments evaluate numerical accuracy, perceptual realism, speaker specificity, and the added information provided by speech beyond the initial pose.
- Numerical Comparison: On 2,048 test intervals per speaker, the model outperforms most baselines for most speakers.The no-GAN condition is slightly better than the GAN condition, and the model generally outperforms the RNN baseline.
- Human Study: The human study compares real and generated motion in 4- and 12-second clips using the percentage of trials in which participants were fooled.
- Human Study: Realistic generated gestures fool humans at rates comparable to real motion for the dynamic speaker.For the stationary speaker, real motion is selected more often because generated motion has prediction noise.
- The Predicted Gestures are Person-Specific: Different-speaker models perform better than random motion but worse than the input speaker’s median pose and much worse than the matching speaker model.The diagonal structure of Figure 4 reflects this person-specific performance pattern.
- Speech is a Good Predictor for Gesture: Adding speech to the initial pose significantly improves average prediction accuracy across speakers.The improvement is statistically significant with p < 10^-3, and most gains come from speakers making large motions during speech.
5.3. Qualitative Results
The paper qualitatively compares predicted gestures with baselines and ground-truth sequences, with supplementary videos providing clearer temporal visualization.
- Figure 5 qualitatively compares speech-to-gesture outputs with baselines and ground-truth gesture sequences.Supplementary videos better convey the temporal behavior of the results.
6. Conclusion
The paper frames person-specific gesture prediction from in-the-wild speech as a computational approach to studying speech–gesture connections. It introduces a large dataset and reports that the model outperforms other methods, while identifying limitations and future research directions.
- The paper proposes predicting person-specific gestures from in-the-wild speech to computationally study connections between speech and gesture.
- The model outperforms other methods in an experimental evaluation.
- Audio provides prosody, intonation, rhythm, and tone, but does not directly encode high-level language semantics or separate speech from other sounds.
- Pose estimations are treated as ground truth despite introducing substantial noise, especially for speakers’ fingers.
- The work points toward video analysis, conversational agents, and further research on predicting motion from audio.
7. Appendix
The appendix describes dataset construction, annotation-quality assessment, temporal gesture-unit segmentation, and unsupervised gesture-dictionary learning. It also illustrates individual gesture styles through clustered motion examples.
- Data collection and processing: The dataset contains 60,000 video intervals averaging 8.7 seconds, totaling 144 hours.
- Quality of dataset annotations: Annotation quality is assessed by comparing automatic pseudo ground truth with labels from three human observers on a 200-frame subset.
- Gesture unit segmentation: Gesture units are segmented as atomic motion sequences that begin at rest and return to rest after the final gesture.
- Gesture unit segmentation: Temporal segmentation uses prediction error, with high error marking likely transitions from rest or unfamiliar motion.
- Dictionary learning: Gesture units are represented by the first five principal components, capturing 93% of variance, and compared using dynamic time warping.
- Individual styles of gesture: Clusters provide an unsupervised definition of each individual’s typical gestures, illustrated with central examples from Jon Stewart’s dictionary.