Source-linked AI summary
Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders
Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, Linchao Bao
TL;DR
Audio-to-gesture generation must model a one-to-many relationship because the same speech can correspond to different gestures. This paper uses a conditional VAE with shared and motion-specific latent codes, plus training strategies, to generate more realistic, diverse, and controllable motions.
Problem
Audio-to-gesture generation is challenging because the same speech can correspond to different gestures, while existing one-to-one models tend to produce averaged, nond diverse motions.
Method
The method splits latent representations into shared and motion-specific codes and uses mapping, relaxed motion, bicycle, and diversity losses to train multimodal gesture generation.
Results
The final model improves realism, diversity, and multimodality, and supports inserting specified motions through motion-specific code editing with smooth transitions.
Takeaways & Limitations
The approach enables diverse gesture synthesis from audio while allowing users to control generated motion by specifying motions on the timeline.
Takeaways & Limitations
Invalid motions can produce abnormally high multimodality scores, so multimodality must be interpreted together with motion validity.
Abstract
from arXiv · showhide
Generating conversational gestures from speech audio is challenging due to the inherent one-to-many mapping between audio and body motions. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, resulting in plain/boring motions during inference. In order to overcome this problem, we propose a novel conditional variational autoencoder (VAE) that explicitly models one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code mainly models the strong correlation between audio and motion (such as the synchronized audio and motion beats), while the motion-specific code captures diverse motion information independent of the audio. However, splitting the latent code into two parts poses training difficulties for the VAE model. A mapping network facilitating random sampling along with other techniques including relaxed motion loss, bicycle constraint, and diversity loss are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than state-of-the-art methods, quantitatively and qualitatively. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline. Code and more results are at https://jingli513.github.io/audio2gestures.
1. Introduction
Audio-to-gesture generation is difficult because the same speech can correspond to different gestures, while one-to-one models tend to produce averaged, low-diversity motion. Audio2Gestures addresses this by separating shared audio-motion information from motion-specific variation and using training mechanisms that support random sampling.
- The same spoken words can elicit different gestures because speakers’ mental and physical states vary.
- One-to-one audio-to-motion models average multiple valid motions, producing gestures that lack diversity.
- Random noise and relaxed motion loss help prevent decoder degeneration and allow multimodal motion generation without requiring the exact target motion.Relaxed motion loss penalizes only joints whose deviations exceed a threshold, letting motion-specific codes adjust the final motion while shared codes retain control.
- The model splits its latent representation into shared and motion-specific codes to regress training data while generating diverse motions.
- The method is evaluated on both 3D and 2D gesture-generation tasks against several state-of-the-art methods.
2. Related Work
Earlier audio-to-motion methods select or predict gestures but remain limited by database coverage, restricted motion patterns, or averaged outputs. Audio2Gestures instead targets diverse co-speech gestures with a customized probabilistic model for both 2D and 3D data.
- Motion-database methods select and blend existing clips but cannot readily generate motions outside the database and scale poorly to large databases.
- Earlier deep models addressed simpler cyclic instrument-motion tasks, whereas co-speech gestures involve more complicated motion and speech relationships.
- Speech2Gesture’s fully convolutional model avoids recurrent error accumulation but still predicts averaged motion because training data contain one-to-many mappings.
- The Trinity Speech-Gesture Dataset provides 244 minutes of paired audio and 3D motion-capture data for studying audio–3D-motion correlation.
- Prior deep-learning methods mainly tested 2D keypoint locations because of limited 3D human-pose data.
- Audio2Gestures differs from related disentanglement methods by separating audio-motion shared information from motion-specific information in a customized VAE.
3. Preliminaries
The paper introduces VAE-based latent sampling and motion losses for supervising generated gestures. Its representation supports both 2D and 3D motion, using direct positions for 2D and rotations with forward kinematics for 3D.
- A VAE constrains latent-code distributions toward a target distribution, enabling output sampling from the latent space.
- The latent objective is optimized through the Evidence Lower Bound, with KL divergence comparing latent distributions.
- Motion reconstruction loss combines rotation, position, and speed losses with respective weights.It is defined as Lmot = λrot × Lrot + λpos × Lpos + λspeed × Lspeed.
- Rotation loss measures angular distance between predicted and ground-truth rotations, while position loss uses L1 distance between joint positions.
- Speed loss supervises joint-speed dynamics to help the model learn complicated motion changes.
- The model handles 2D data by predicting joint positions and 3D data by predicting rotations before calculating positions with forward kinematics.For 3D motion, forward kinematics takes each joint’s rotation about its parent and relative translation, such as bone length, to output joint positions.
4. Audio2Gestures
Audio2Gestures models one-to-many audio-motion generation by separating shared audio-motion information from motion-specific variation. A mapping network and specialized training losses make sampled motion-specific codes usable while preserving audio-related motion generation.
- Network structure: The conditional encoder-decoder splits latent representations into shared and motion-specific codes to represent multiple motions for similar audio.Audio and motion encoders extract the respective codes, while a common decoder recombines them to generate motion.
- Network structure: During inference, the model combines an audio-derived shared code with a randomly sampled motion-specific code to generate motion.The mapping network transforms random input into the motion-specific feature used by the decoder.
- Latent code learning: Training adds audio-conditioned reconstruction because otherwise the decoder can ignore shared codes and generate motion solely from motion-specific features.This extra data flow uses the audio-derived shared code and a mapping-network-generated motion-specific code.
- Latent code learning: A mapping network improves realism by aligning randomly sampled features with the motion-specific feature distribution.The distribution parameters are computed from the target motion-specific code per channel.
- Latent code learning: Relaxed motion loss penalizes only joints exceeding a distance threshold, allowing sampled motion-specific codes to vary final motion while retaining shared-code control.The loss addresses degeneration and avoids requiring random samples to reconstruct the exact original target.
- Latent code learning: Bicycle constraints and diversity loss address mode collapse by encouraging motion-code reconstruction and exploration of meaningful motion-specific codes.The bicycle constraint promotes a bijection between motion and motion-specific code, while diversity loss maximizes distance between sampled motions.
5. Experiments
The experiments evaluate Audio2Gestures on motion datasets using established metrics and comparisons with three state-of-the-art methods. The study also includes ablation analyses of individual model modules.
- Experimental design: The experimental section covers datasets, evaluation metrics, implementation details, comparisons with three state-of-the-art methods, and ablation studies.Additional results are provided on the project page.
- Datasets: The Trinity Gesture Dataset contains 23 paired audio-motion sequences totaling 244 minutes, with 56-joint motion recorded at 60 or 120 FPS.The dataset uses speech from a male native English speaker discussing topics including movies and daily activities.
5.2. Evaluation metrics
The evaluation uses realism, diversity, and multimodality metrics, supplemented by user studies rating realism, diversity, and audio-motion matching. Each metric captures a different aspect of generated motion quality, with validity caveats for diversity-based measures.
- Realism: Realism is evaluated using joint-position L1 distance and percentage of correct 3D keypoints (PCK).PCK counts keypoints whose prediction distance from the target is below threshold δ, set to 0.2 in the experiments.
- Diversity: Diversity measures differences among equal-length, non-overlapping clips within the same generated motion, using average L1 distance.Generated motions are split into 50-frame clips in the experiments.
- Multimodality: Multimodality measures differences between motions sampled across multiple runs for the same audio clip.Unlike diversity, it compares separate sampled motions rather than short clips within one long motion.
- Metric caveats: Higher diversity or multimodality scores are not necessarily better because jitter and invalid poses can also produce large motion differences.The metrics therefore favor higher variation only when the generated motion remains natural.
- User study: The user study compares methods through 20-second videos rated for realism, diversity, and matching degree, using scores from 5 for best to 1 for worst.Figure 4 summarizes ranking counts and average scores from 160 comparisons by 40 participants.
5.3. Implementation details
The experiments process Trinity and S2G-Ellen audio-motion data into frame-aligned inputs and use articulated body representations. Quantitative and ablation results are reported through separate tables, with sampling methods evaluated over repeated tests.
- Data processing: Trinity audio is resampled to 16kHz and converted into 64-dimensional log-mel features aligned with 30-FPS motion.The hop size is 533, computed from the 16,000Hz sample rate and 30-FPS motion rate.
- Data processing: Trinity motion is downsampled to 30 FPS, retargeted to SMPL-X, and represented with 6D joint rotations; finger motion is removed because of noise.SMPL-X includes 54 joints spanning the body, hands, and face.
- Evaluation protocol: Table 1 reports quantitative results on Trinity and S2G-Ellen, while Table 2 reports Trinity ablations with each row adding a component to the previous configuration.Methods that support sampling are evaluated over 20 tests, with average and best scores reported.
- Data processing: S2G-Ellen uses 64-frame clips, audio features processed as in Trinity, and body joints represented in coordinates relative to the root.The clips are approximately 4.2 seconds long.
5.4. Comparison with state-of-the-art methods
The method is compared with StyleGestures and Speech2Gesture on Trinity, alongside broader quantitative and user-study comparisons. Results favor the proposed approach on realism and diversity, while baselines exhibit limitations in sampling, long-term stability, or pose validity.
- Overall comparison: The proposed method outperforms previous state-of-the-art algorithms on realism and diversity in both quantitative and user-study results.The authors attribute this outcome to explicitly modeling the one-to-many audio-motion relationship.
- StyleGestures: StyleGestures supports sampling different motions for the same audio, but has less appealing motion quality and the lowest diversity score.Its LSTM output can become static, and error accumulation worsens long-term generation, especially at 5000 frames compared with 400-frame tests.
- Speech2Gesture: Speech2Gesture has similar realism but lower diversity than the proposed method and does not support generating multimodal motions.The comparison identifies a diversity and sampling limitation despite comparable realism.
- Speech2Gesture: Speech2Gesture with GAN generates many invalid poses and obtains the worst performance despite repeated training with learning rates from 0.0001 to 0.01.The authors suggest unstable GAN training may contribute to this result.
5.5. Ablation study
The ablation study shows that splitting the latent code alone harms realism and diversity, while mapping and additional losses progressively improve multimodal motion generation. The final model outperforms the baseline across all quantitative indicators.
- Evaluation: The ablation variants are evaluated over 20 runs, reporting averaged and best performance to account for randomness from the VAE and motion-specific feature sampling.This evaluation separates two sources of randomness in the model.
- Baseline: The baseline’s averaged and best scores are nearly identical, indicating that VAE randomness alone does not produce multimodal motions.The reported avg L1, avg PCK, and avg Diversity equal the corresponding best scores.
- +split: The +split variant explicitly models one-to-many mapping but worsens realism and diversity because sampled signals misalign with motion-specific features.The sampled signals differ from motion-specific features in statistical characteristics such as derivative means and variances.
- +mapping net: The mapping network aligns sampled signals with motion-specific features and improves most motion metrics, although multimodality and realism remain imperfect.It outperforms the baseline on L1, achieves similar PCK, and generates multimodal motions, but has lower diversity and realism than the baseline.
- Additional losses: Bicycle constraints increase multimodality from 3.44 to 3.68 and average diversity from 5.86 to 6.31.The diversity loss further improves diversity and multimodality while having little influence on realism.
5.6. Application
The application uses motion-specific codes to insert user-provided motion clips into generated sequences. Replacing sampled codes over a selected timeline produces smooth motion transitions.
- Timeline-controlled synthesis: Motion-specific codes strongly control the generated output, allowing an extracted code from a predefined clip to replace sampled codes between frames t and t + n.The model generates a smooth motion from the edited motion-specific code.
6. Conclusion
The paper models audio-to-motion one-to-many mappings by splitting latent codes into shared and motion-specific components, with customized training improving generated motion. It also enables smooth insertion of specified motions, although generated gestures are not very related to spoken content.
- Core conclusion: Splitting latent codes into shared and motion-specific components improves the realism, diversity, and multimodality of generated motion.The approach explicitly models the one-to-many audio-to-motion relationship.
- Application: Editing motion-specific codes lets users insert specific motions into generated sequences with smooth and realistic transitions.This provides direct control over the output motion.
- Limitation: Generated motions are not very related to what the person says, motivating future conditioning with word embeddings.This is identified as a limitation of the current model.