Source-linked AI summary
ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech
Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, Marc-André Carbonneau
TL;DR
Speech-driven gesture generation must handle multiple appropriate motions for a speech line while supporting diverse styles. ZeroEGGS generates gestures from speech and a short reference motion, enabling zero-shot style control; its subjective evaluations report better motion naturalness, speech appropriateness, and style portrayal than the baseline.
Problem
Pre-recorded gesture animation takes substantial manual effort, scales poorly to unscripted settings, and may not synchronize with speech rhythm.
Source: §1 · p. 1Method
ZeroEGGS generates full-body gestures from speech while encoding a short example motion as a style embedding for zero-shot style control.
Source: §1 · p. 2Results
ZeroEGGS outperformed the baseline in subjective evaluations of naturalness, speech appropriateness, and style portrayal.
Source: §6 · p. 8Takeaways & Limitations
Experiments report generalization to new voices, languages, and unseen styles, allowing one model to serve different characters without retraining for each new style.
Source: §6 · p. 8Generated motion falls significantly behind ground truth, and subtle style-specific motions may be over-smoothed or absent.
Source: §6 · p. 8
Abstract
from arXiv · showhide
We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the same input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles.
1. Introduction
Gesture generation must scale beyond manually triggered animations while matching speech and preserving diverse speaker styles. ZeroEGGS uses short example motions for zero-shot style control and probabilistic variation.
1. Introduction
- Pre-recorded animations require substantial manual effort, scale poorly to unscripted settings, and may not synchronize with speech rhythm.These limitations motivate automatic, data-driven gesture generation.Source: §1 · p. 1
- Realistic gesture generation remains difficult, and controlling expressivity tied to speaker state and identity is especially challenging.Source: §1 · p. 2
- Speech-to-gesture generation is one-to-many: a speech line can fit many gestures, while characters also need distinct movement styles.The model must capture broad motion variation while preserving character idiosyncrasies.Source: §1 · p. 2
- Prior style methods depend on training examples for each target speaker and style, limiting generalization and requiring costly per-style data capture.Style can be difficult to name, making an example motion a practical way to specify it.Source: §1 · p. 2
- Short style examples let ZeroEGGS generalize to styles absent from training without requiring style labels.The model also supports latent-space style manipulation and repeated sampling of different motions for the same speech and style.Source: §1 · p. 2
2. Related Work
Gesture-generation research includes rule-based and data-driven approaches, with limitations in scalability, motion quality, and style control. ZeroEGGS builds on example-based style conditioning to address limited flexibility in prior controls.
2. Related Work
- Rule-based methods map speech markers to gestures using manually created rules, but diversity depends on the rules and database, and manual labor limits scale.Some use text analysis, while others use acoustic features [MXL∗13].Source: §2 · p. 2
- Data-driven methods learn speech–motion correlations, but generated-motion systems face one-to-many mapping and can produce mean-collapsed, lethargic motion.Generative adversarial approaches have had varying success depending on dataset size and gesture complexity [FNM19,GBK∗19,RGP21].Source: §2 · p. 2
- Generated gestures still lag motion-captured gestures in naturalness and can fail to beat mismatched real motion in speech appropriateness [KJY∗21,GBK∗19].Source: §2 · p. 2
- Existing style controls use handcrafted statistics or animation qualities, which may not describe the wide range of styles a model can learn.Examples include motion speed, height, spatial extent, symmetry, and Laban Effort and Shape qualities [AHKB20, YPJ∗21, SGD21].Source: §2 · p. 2
- Example-based style transfer has been applied to locomotion, where a discriminator is trained using labeled styles and unseen-style generalization is possible [AWL∗20], and dance, whose model was not shown to generalize beyond training styles [VPHB∗21].Source: §2 · p. 2
3. System Overview
ZeroEGGS combines speech and reference-style embeddings in an autoregressive gesture-generation pipeline. Its encoders summarize audio and motion clips, and the generator produces a gesture sequence conditioned on both.
3. System Overview
- The system encodes speech as a sequence S and reference motion style as a fixed-size vector e, then conditions an autoregressive Gesture Generator on both to produce animation Y.Training maximizes the likelihood of the gesture animation.Source: §3 · p. 3
3.1. Speech Encoder
- The Speech Encoder converts log-amplitude mel spectrograms and frame energy into a sequence of speech embedding vectors S.Resampling, 1D convolutions, nonlinear operators, and a frame-wise linear layer produce one embedding per frame.Source: §3.1 · p. 3
3.2. Style Encoder
- The Style Encoder maps normalized animation-frame features to a sampled Gaussian style embedding that summarizes the reference clip.Its sequence network produces Gaussian parameters by pooling over the encoded animation sequence.Source: §3.2 · pp. 3–4
3.3. Gesture Generator
- The Gesture Generator autoregressively produces pose frames from speech embeddings and the reference-style vector, using the previous pose state as input.A two-layer GRU decoder generates pose encodings; a separate initializer sets its hidden states from the initial pose, facing direction, and style embedding.Source: §3.3 · p. 4
4. Implementation and Training
The section documents the recorded speech-motion data, model components, and conditional-VAE training objective and procedure.
4.1. Dataset and Data Preparation
- The dataset contains 67 English monologue sequences by one female actor, covering 19 motion styles and 135 minutes of synchronized motion and audio.Motion was recorded at 60 fps using a 75-joint skeleton that includes hands and fingers; mirrored animation data was added for training.Source: §4.1 · p. 4
4.2. Model Implementation
- The speech encoder has an approximately one-second receptive field, the style embedding is 64-dimensional, and generation uses a Transformer block and recurrent decoder.The decoder uses two GRU cells, each with a hidden-state size of 1024.Source: §4.2 · pp. 4–5
4.3. Training & Losses
- Training uses 256-frame sequences and style examples of 256–512 frames, omits teacher forcing, and randomly varies animation and speech speed by 10%.Training on the model’s own predictions slows convergence but is described as improving its ability to recover from its own errors.Source: §4.3 · p. 5
- Training uses a conditional-VAE objective that minimizes the negative ELBO, combining gesture reconstruction loss with KL regularization of the style posterior against a Gaussian prior.The regularization term is weighted using the cost-annealing strategy from [BVV∗16].Source: §4.3 · p. 5
- The reconstruction loss combines errors for joint positions, rotations, translational and rotational velocities, and facing direction in local and world spaces.It also penalizes velocity errors computed on-the-fly by finite differences; the facing-direction term is intended to prevent rotational drift.Source: §4.3 · p. 5
5. Experiments and Results
Experiments assess generalization, style control, and subjective quality. ZeroEGGS generated gestures for unseen styles and speakers, supported latent-space style manipulation, and outperformed MoGlow on several user-study measures.
5.1. Unseen Styles and Speakers
- Tests included unseen Oration, retained examples from other styles, a new speaker paired with unseen Oration, and French, German, and gibberish speech.Figures 5 and 6 show generated gestures for unseen styles and for an unseen style with a new speaker.Source: §5.1 · p. 5, Figure 5 · p. 6, Figure 6 · p. 6
5.2.1. Blending Styles
- Interpolating between Old and Oration embeddings gradually changed the character’s posture and hand position.The variational style embedding provides a continuous space for mixing multiple samples.Source: §5.2.1 · p. 5, Figure 7 · p. 6
5.2.2. Control via PCA Components
- PCA dimensions roughly corresponded to body sway and to hand height and radius; changing them affected root velocity and hand-height trajectories.The PCA manipulation examples use a Neutral-style gesture, with the first component changed by one standard deviation of the Neutral style.Source: §5.2.2 · p. 6, Figure 8 · p. 7
5.3.1. Study Design
- The three user studies assessed naturalness, speech appropriateness, and style portrayal with 131 participants using randomized MUSHRA-like ratings.Participants compared ground-truth motion, mismatched motion, ZeroEGGS, and MoGlow; style portrayal was rated without sound.Source: §5.3.1 · pp. 6–7, §5.3 · p. 6
5.3.2. Results
- ZeroEGGS outperformed MoGlow on all three measures for its dataset and on appropriateness and style recognizability for TSG; ground-truth motion scored best across measures.The three measures were naturalness, appropriateness for speech, and style portrayal.Source: §5.3.2 · p. 7
- On the authors’ dataset, mismatched motion was not significantly different from ZeroEGGS for naturalness, while ZeroEGGS scored higher for appropriateness (p < .01); on TSG, mismatched motion was on par.The naturalness results also report mismatched motion below ground truth and above MoGlow.Source: §5.3.2 · pp. 7–8
- For style portrayal, mismatched motion from another style scored below ZeroEGGS, while MoGlow was not significantly different from either.The authors attribute ZeroEGGS’s lower-than-average result for Happy to the system-by-style interaction.Source: §5.3.2 · p. 8
- The authors caution that MUSHRA-like scores indicate relative model performance rather than absolute performance.They also note that MoGlow was trained on the single-style TSG dataset after retraining it on their dataset yielded jittery, low-quality output.Source: §5.3.2 · p. 8
6. Discussion
ZeroEGGS outperformed its baseline in naturalness, appropriateness for speech, and style portrayal, while its example-based style representation supports latent-space control. The discussion also identifies limits in naturalness, subtle-motion generation, moving-speaker performance, semantic gesture coverage, and foot contact.
6. Discussion
- The variational style embedding supports manipulation and interpolation, weighted mixing of reference styles, and resampling varied gestures for the same input.These controls can support transitions between styles within an animated sequence.Source: §6 · p. 8
- ZeroEGGS outperformed the baseline in naturalness, appropriateness for speech, and style portrayal in subjective evaluations.The authors attribute gains mainly to example-based conditioning and also note that smooth, stable motion contributes to perceived naturalness.Source: §6 · p. 8
- Generated motion still falls significantly behind ground truth in naturalness, and subtle style-specific motions can be over-smoothed or omitted.The authors cite infrequent head nods in the Agreement style despite their abundance in training data.Source: §6 · pp. 8–9
- Performance decreased slightly on the unseen Trinity Speech-Gesture dataset, where frequent speaker displacement is not represented in the training data.The authors identify support for character displacement as future work and say the method performs well on more static styles.Source: §6 · p. 9
- ZeroEGGS generates small amounts of foot sliding, which the authors describe as a common issue in non-physically-based animation that IK-based post-processing can address.Source: §6 · p. 9
- Raw spectral audio features support robustness to language and speaker voice, but limit generation to beat gestures without speech semantics.The authors suggest semantic speech markers as a possible future input; ground truth scored higher for speech appropriateness in the user study.Source: §6 · p. 9
7. Conclusion
ZeroEGGS generates stylized gestures from speech using a short example clip to specify style, with generalization to unseen styles, voices, and languages. The conclusion reports state-of-the-art gesture animations and identifies several directions for future work.
7. Conclusion
- ZeroEGGS generates stylized gesture animations from speech, with style specified by a short example clip.Source: §7 · p. 9
- The model generalizes to unseen styles, voices, and languages, while its latent style representation provides control over generation.Source: §7 · p. 9
- Future work includes user-specified disentanglement of the style latent space, semantic gestures, and architectures based on affine transform coupling layers.Source: §7 · p. 9