Source-linked AI summary
Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, Lequan Yu
TL;DR
Audio-driven co-speech gesture generation needs models that capture audio-gesture associations while maintaining temporal coherence. DiffGesture uses conditional diffusion with a multimodal Transformer, gesture stabilization, and classifier-free guidance, and experiments report coherent gestures with stronger audio correlations and state-of-the-art performance.
Problem
Co-speech gesture generation requires temporally coherent gesture sequences aligned with continual audio, while prior methods include GAN-based approaches with mode collapse and unstable training.
Method
DiffGesture applies conditional diffusion to skeleton and audio clips, using a Diffusion Audio-Gesture Transformer, Gesture Stabilizer, and implicit classifier-free guidance.
Results
Experiments on two benchmark datasets show coherent gestures with stronger audio correlations and performance surpassing state-of-the-art methods.
Takeaways & Limitations
DiffGesture provides a diffusion-based approach for efficient non-autoregressive generation that captures sequential cross-modal dependencies and reduces error accumulation.
Abstract
from arXiv · showhide
Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gesture generation. Specifically, we first establish the diffusion-conditional generation process on clips of skeleton sequences and audio to enable the whole framework. Then, a novel Diffusion Audio-Gesture Transformer is devised to better attend to the information from multiple modalities and model the long-term temporal dependency. Moreover, to eliminate temporal inconsistency, we propose an effective Diffusion Gesture Stabilizer with an annealed noise sampling strategy. Benefiting from the architectural advantages of diffusion models, we further incorporate implicit classifier-free guidance to trade off between diversity and gesture quality. Extensive experiments demonstrate that DiffGesture achieves state-of-theart performance, which renders coherent gestures with better mode coverage and stronger audio correlations. Code is available at https://github.com/Advocate99/DiffGesture.
1. Introduction
Co-speech gestures support communication and motivate audio-driven gesture generation for virtual avatars. DiffGesture adapts diffusion modeling to capture audio-gesture associations and temporal coherence, addressing limitations of prior approaches.
- Co-speech gestures help speakers express thoughts, listeners comprehend meanings, and conversations develop mutual intimacy.
- Audio-driven co-speech gesture generation synthesizes upper-body gesture sequences aligned with speech audio for embodied AI avatars.
- Earlier methods searched predefined speech-gesture pairs or learned speech-to-skeleton mappings from large-scale corpora.
- Existing conditional diffusion models mainly address static data, whereas co-speech generation requires temporally coherent gestures conditioned on continual audio clips.
- DiffGesture formulates conditional diffusion on skeleton and audio clips to capture cross-modal associations and maintain temporal coherence.
- Experiments on two benchmark datasets report coherent gestures, stronger audio correlations, and performance surpassing state-of-the-art methods.
- The framework combines a Diffusion Audio-Gesture Transformer, implicit classifier-free guidance, and a Diffusion Gesture Stabilizer with annealed noise sampling.
2. Related Work
Related work progresses from rule-based gesture stitching to neural mappings learned from large speech-gesture corpora, while also examining how input modalities affect generation.
- Rule-based methods predefine speech-gesture pairs and refine transitions between motions.
- Recent neural methods learn speech-to-gesture mappings from large training corpora annotated with pseudo labels from an off-the-shelf pose estimator.
- Related studies also investigate how input modality influences co-speech gesture generation.
3. Our Approach
DiffGesture formulates co-speech gesture synthesis as conditional diffusion over skeleton sequences and audio, then combines multimodal temporal modeling, stabilization, and implicit guidance. The framework addresses temporal coherence and ambiguous audio-to-gesture mappings during generation.
- Gesture-space diffusion: The framework defines forward diffusion and conditional reverse generation in gesture space for skeleton sequences conditioned on speech audio and initial poses.The forward process gradually adds Gaussian noise, while the reverse process denoises gesture representations using contextual audio and pose information.
- Problem formulation: The model represents each frame with unit direction vectors between adjacent joints and optimizes a denoising model to synthesize the skeleton sequence.The objective minimizes the discrepancy between the target sequence and the generated sequence conditioned on audio and the first M poses.
- Training objective: The conditional diffusion process uses Gaussian transitions and an MSE noise-prediction objective, with context masking enabling joint conditional and unconditional training.The model samples noisy states through the forward process and learns to predict noise for denoising.
- Audio-gesture Transformer: The Diffusion Audio-Gesture Transformer concatenates noisy gestures with temporally aligned context so self-attention can capture long-term dependencies without autoregressive error accumulation.Each frame’s skeleton and contextual condition form a token, enabling sequential modeling across the clip.
- Implicit classifier-free guidance: Implicit classifier-free guidance jointly trains conditional and unconditional models because speech-to-gesture mappings are one-to-many and explicit classifier guidance is difficult to apply.A scale parameter adjusts the trade-off between diversity and quality during sampling.
4. Experiments
Experiments evaluate DiffGesture on TED Gesture and TED Expressive using distributional, audio-motion correlation, diversity, qualitative, user-study, and ablation analyses. Across these evaluations, DiffGesture is reported to produce high-quality, coherent, diverse, and audio-aligned gestures.
- Evaluation Metrics: FGD measures the distance between synthesized and real gesture distributions, while Beat Consistency measures motion-audio beat correlation.FGD uses latent feature distributions; Beat Consistency compares detected motion beats with audio beats.
- Quantitative Results: DiffGesture achieves state-of-the-art performance on TED Gesture and TED Expressive, with especially large gains on TED Expressive.The comparison uses FGD, Beat Consistency, and Diversity; FGD is better when lower, while the other metrics are better when higher.
- Qualitative Results: DiffGesture produces diverse, human-like poses, whereas comparison methods often generate slow, invariant, unreliable, or stiff motions.The qualitative analysis highlights dull baseline outputs as evidence of mode collapse.
- Ablation Studies: Ablations show that implicit classifier-free guidance and the Diffusion Gesture Stabilizer consistently improve benchmark performance.The study removes classifier-free guidance during training and inference or removes the stabilizer during inference to assess their effects.
- User Study: In a user study, 18 participants rated naturalness, smoothness, and speech-gesture synchrony for unlabeled generated clips on a 1-to-5 scale.The study included 30 cases across TED Expressive and TED Gesture, and participants widely accepted DiffGesture's high-fidelity results.
- Ablation Studies: The Transformer-based diffusion backbone outperforms the GRU alternative on TED Gesture and TED Expressive.The comparison evaluates GRU and Transformer backbones under the same diffusion-based setup.
5. Conclusion
DiffGesture is a diffusion-based framework for co-speech gesture generation that combines a Diffusion Audio-Gesture Transformer with a Diffusion Gesture Stabilizer to produce coherent gestures with strong audio correlations.
- DiffGesture uses a diffusion-based framework for co-speech gesture generation.
- The Diffusion Audio-Gesture Transformer and Diffusion Gesture Stabilizer attend to conditioning information and support coherent generation.
- Its non-autoregressive pipeline helps generate results efficiently and reduce error accumulation.