Source-linked AI summary
Analyzing Input and Output Representations for Speech-Driven Gesture Generation
Taras Kucherenko, Dai Hasegawa, Gustav Eje Henter, Naoshi Kaneko, Hedvig Kjellström
TL;DR
Speech-driven gesture generation needs a data-driven alternative to rigid rule-based methods for natural human-agent interaction. The paper learns a compact motion representation, predicts it from speech, and decodes it into gestures; representation learning improves objective and subjective performance, including perceived naturalness.
Problem
Rule-based gesture generation is rigid and labor-intensive, motivating data-driven speech-to-gesture generation for natural human-agent interaction.
Method
The method learns a lower-dimensional motion representation with an encoder-decoder, trains SpeechE to predict it from speech, and decodes the prediction into motion.
Results
Representation learning improved objective and subjective speech-to-gesture performance, with significantly more natural gestures than the baseline.
Takeaways & Limitations
Representation learning is a useful extension for speech-driven gesture generation, while speech acoustics alone produce mostly beat gestures.
Takeaways & Limitations
The method requires substantial high-quality parallel speech-and-motion training data for good prediction performance.
Abstract
from arXiv · showhide
This paper presents a novel framework for automatic speech-driven gesture generation, applicable to human-agent interaction including both virtual agents and robots. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates. Our approach consists of two steps. First, we learn a lower-dimensional representation of human motion using a denoising autoencoder neural network, consisting of a motion encoder MotionE and a motion decoder MotionD. The learned representation preserves the most important aspects of the human pose variation while removing less relevant variation. Second, we train a novel encoder network SpeechE to map from speech to a corresponding motion representation with reduced dimensionality. At test time, the speech encoder and the motion decoder networks are combined: SpeechE predicts motion representations based on a given speech signal and MotionD then decodes these representations to produce motion sequences. We evaluate different representation sizes in order to find the most effective dimensionality for the representation. We also evaluate the effects of using different speech features as input to the model. We find that mel-frequency cepstral coefficients (MFCCs), alone or combined with prosodic features, perform the best. The results of a subsequent user study confirm the benefits of the representation learning.
1 INTRODUCTION
The paper targets natural speech-driven gesture generation for human-agent interaction, replacing rigid rule-based systems with data-driven representation learning. It evaluates motion representations and speech features, with user-study evidence that representation learning improves perceived gesture naturalness.
- Conversational agents need non-verbal behaviors because people use them to communicate intent, emotions, and attitudes.
- Rule-based gesture systems are rigid, labor-intensive to author, and unable to capture the richness of human gesticulation.
- The paper applies representation learning to motion and compares alternative speech feature extractors for speech-driven gesture generation.
- The authors propose a speech-driven non-verbal behavior generation method applicable to any embodiment.
- Representation learning improved the perceived naturalness of generated gestures over the baseline in a subsequent user study.
2 REPRESENTATION LEARNING FOR SPEECH-MOTION MAPPING
The proposed system learns a compact motion representation, maps speech features into that representation, and decodes it into gesture coordinates. It extends a baseline speech-to-motion network by replacing direct raw-motion prediction with an encoder-decoder pipeline.
- Problem formulation: Speech-driven gesture generation maps frame-level speech features to corresponding sequences of 3D human-body keypoint coordinates.
- Baseline speech-to-motion mapping: The baseline processes overlapping speech windows with fully connected layers, a recurrent layer, and an output layer predicting pose and velocity.
- Proposed approach: The proposed approach learns a lower-dimensional motion representation, predicts it from speech, and chains the learned mappings to produce motion output.
- Motion representation learning: Reducing motion dimensionality simplifies the learning problem and reduces redundancy by concentrating important information into fewer values.
- Motion representation learning: A denoising autoencoder uses MotionE to encode motion m into representation z and MotionD to decode z back into motion m.
- Speech-to-representation mapping: SpeechE predicts compact motion representations using the baseline architecture, after which MotionD decodes them into joint coordinates at test time.
3 EXPERIMENTAL SETUP
The experiments evaluate speech and motion representations for speech-driven gesture generation using a Japanese gesture-speech dataset, objective and subjective measures, and repeated trials. They compare audio feature types, learned representation dimensionalities, and motion plausibility statistics.
- Dataset: The experiments used motion-capture data and headset-recorded MP3 speech from two Japanese individuals conducting an interview conversation.Motion data were stored in BioVision Hierarchy format.
- Dataset: The dataset contains 1,047 utterances, with 957 used for training, 45 for validation, and 45 for testing.Training used 171 minutes of data at 20 fps, yielding 206,000 training frames linked to 64 joint positions.
- Speech features: The study compared MFCC, spectrum-based, and prosodic speech features, reflecting a trade-off between simple learnable inputs and richer features requiring more data.Prosodic inputs represented pitch and energy information outside the specific words spoken.
- Speech features: MFCCs used 26 coefficients at 100 analysis frames per second, while spectrogram features were 64-dimensional at 200 frames per second before downsampling.Frequencies below 20 Hz and above 8000 Hz were removed from the spectrum-based features.
- Evaluation: The evaluation combined objective error measures with subjective measures and repeated every condition five times, reporting means and standard deviations.Repeated trials accounted for randomness from factors such as initial network weights.
- Evaluation: Average position error measures coordinate distance, while average speed and jerk distributions assess whether generated motion is plausible rather than an exact reproduction.The authors argue that gesture generation should produce natural candidate motions whose motion statistics follow similar distributions.
4 RESULTS AND DISCUSSION
The evaluation examines motion-representation dimensionality, speech features, motion statistics, and perceived gesture quality. Representation learning improves objective performance and perceived naturalness, while MFCCs provide the strongest overall speech representation among those tested.
- Motion representation dimensionality: A 325-dimensional motion representation minimizes average position error and yields the lowest motion jerkiness, while outperforming the baseline on both measures.The 325-dimensional embedding is smaller than the original 384-dimensional representation; its jerkiness advantage is slight relative to estimate uncertainty, whereas the average-jerk difference is highly significant.
- Motion representation dimensionality: Training results vary noticeably across runs, suggesting that random initial weights can lead optimization to different local optima.
- Input speech features: MFCCs achieve the lowest average position error but produce higher acceleration and jerkiness than spectrogram features.Spectrogram features match ground-truth acceleration and jerk better than the other studied features, despite having suboptimal average position error.
- Detailed motion analysis: Across acceleration distributions, the proposed model is closer to ground truth than the baseline, with MFCCs especially effective for hand motion.For shoulders, all speech-feature variants remain far from the actual acceleration distribution; for hands, MFCC-based motion is much closer to ground truth, while adding prosodic features performs similarly to MFCCs alone.
- User study: The user study finds significantly higher perceived naturalness for representation-learning gestures, but no significant differences in time consistency or semantic consistency.The proposed model received mean naturalness 4.79 versus 4.16 for the baseline, with p<0.002; the other scales were not significant.
5 RELATED WORK
Related work spans data-driven body-motion and gesture-generation methods, with particular relevance to approaches that learn compact motion representations and predict them from speech. The paper builds on prior speech-driven 3D motion generation by adding motion-representation learning.
- Data-driven body motion generation: Data-driven body-motion methods increasingly use deep learning for applications including animation, games, and simulation.
- Data-driven body motion generation: Autoencoder-based representation learning has produced accurate, parsimonious human-motion predictors, sometimes conditioned on external control signals.
- Speech-driven gesture generation: Gesture generation is shifting toward data-driven methods, although some systems combine learned models with rules or discrete gesture categories.
- Speech-driven gesture generation: Regression methods that predict arbitrary movements are presented as more flexible and scalable than discrete, predefined gestures.
- Speech-driven gesture generation: Prior speech-driven work generated 3D motion sequences, while this paper extends that direction with motion-representation learning.
6 CONCLUSIONS AND FUTURE WORK
The paper introduces representation learning for speech-driven gesture generation and reports improved objective and subjective performance, especially naturalness. It identifies data requirements and future extensions involving probabilistic motion, text, and humanoid robots.
- Conclusions: The model first learns a motion representation, then predicts that representation from speech instead of directly predicting raw joint coordinates.This extends prior deep-learning approaches with representation learning.
- Conclusions: Representation learning improves objective and subjective performance, with significantly more natural gestures than the baseline.The proposed and baseline models received similar ratings for time consistency and semantic consistency.
- Limitations: Good prediction performance requires substantial, high-quality parallel speech-and-motion training data.The authors suggest motion-estimation techniques applied to public video recordings as a possible way to obtain such datasets.
- Future Work: A probabilistic model could generate different plausible gestures for the same utterance.The proposed direction includes using a Variational Autoencoder to represent a distribution over plausible motions.
- Future Work: Adding text to speech acoustics could expand the model beyond mostly beat gestures to metaphoric and deictic gestures.The paper links this direction to the semantic content of utterances.
- Future Work: The learned model could be applied to humanoid robots so speech is accompanied by appropriate co-speech gestures.The paper gives the NAO robot as an example application.