PaperlayerView exact revision on arXiv

Source-linked AI summary

Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models

Simon Alexanderson, Rajmund Nagy, Jonas Beskow, Gustav Eje Henter

arXiv:2211.09707v2cs.LGcs.CVcs.GRcs.HCcs.SDeess.AS

TL;DR

Audio-driven human motion is ambiguous and difficult to predict from audio, motivating probabilistic generation for gestures and dance. The paper adapts diffusion models with Conformers for 3D pose sequences, adds classifier-free style control and product-of-experts ensembles, and reports strong motion quality with adjustable styles. The models remain limited by slow generation and incomplete capture of speech semantics and global musical structure.

  • Problem

    Audio-driven gestures and dance are highly ambiguous and variable given audio, creating a need for probabilistic motion-generation models.

  • Method

    The paper adapts DiffWave with Conformers to model conditional distributions of 3D pose sequences from acoustic features and uses classifier-free guidance for style control.

  • Results

    The proposed models outperform leading baselines across gesture and dance datasets and produce distinctive styles with adjustable expression strength.

  • Takeaways & Limitations

    Diffusion models support high-quality audio-driven motion generation with controllable stylistic expression and product-of-experts combinations such as style interpolation.

  • Takeaways & Limitations

    Generation takes around a second per motion example, and the models do not capture all speech semantics or global musical structure.

Abstract

from arXiv · show

Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthesising human motion that co-occurs with audio, e.g., dancing and co-speech gesticulation, since motion is complex and highly ambiguous given audio, calling for a probabilistic description. Specifically, we adapt the DiffWave architecture to model 3D pose sequences, putting Conformers in place of dilated convolutions for improved modelling power. We also demonstrate control over motion style, using classifier-free guidance to adjust the strength of the stylistic expression. Experiments on gesture and dance generation confirm that the proposed method achieves top-of-the-line motion quality, with distinctive styles whose expression can be made more or less pronounced. We also synthesise path-driven locomotion using the same model architecture. Finally, we generalise the guidance procedure to obtain product-of-expert ensembles of diffusion models and demonstrate how these may be used for, e.g., style interpolation, a contribution we believe is of independent interest. See https://www.speech.kth.se/research/listen-denoise-action/ for video examples, data, and code.

1 INTRODUCTION

Audio-driven human motion is valuable for applications including embodied agents, but gestures and dance are ambiguous and difficult to generate convincingly. The paper introduces diffusion-based audio-driven motion generation with style control and reports strong results across gesture and dance datasets.

  • Motivation: Audio-driven gestures and dance are difficult because the same audio can correspond to highly variable, individual motion.The paper argues that this ambiguity requires a strong probabilistic model.
  • Approach: Diffusion models provide a principled framework for learning expressive probability distributions for audio-driven 3D human motion.The approach uses deep-learning architectures without the restrictions of invertible flows or minimax GAN optimization.
  • Contributions: The paper pioneers diffusion models for gesture and dance generation, using Conformers and classifier-free guidance for adjustable stylistic expression.It also introduces a dance dataset and product-of-experts diffusion ensembles for tasks such as interpolation.
  • Results: Experiments outperform leading baseline systems on multiple gesture and dance datasets while producing distinctive, adjustable motion styles.The paper provides code, data, and pretrained systems alongside the reported results.

2 BACKGROUND AND PRIOR WORK

Prior work spans rule-based, neural, and probabilistic approaches to audio-driven gesture and dance synthesis, while diffusion models had not previously been used to generate motion from acoustic features. This paper adds diffusion-based motion generation with independently controllable style intensity and style identity.

  • Gesture generation: Earlier gesture systems used rules and prerecorded clips, while later deep-learning systems used recurrent or convolutional models for more scalable generation.These approaches commonly represented motion using 3D joint positions in Cartesian coordinates.
  • Probabilistic modeling: Probabilistic gesture models can represent a range of possible motions and sample distinct realizations beyond deterministic predictions.Prior examples included hidden semi-Markov models and combinations of adversarial learning with regression losses.
  • Gesture style control: Style control is important for communication, with prior methods often learning motion-style representations through encoders and latent-variable models.Other work defined style using continuous kinematic properties such as average hand height.
  • Gesture style control: This paper controls style intensity independently of style identity and uses that capability to combine and interpolate between styles probabilistically.This differs from prior style-control approaches described in the paper.
  • Dance synthesis: Dance synthesis has explored discrete motion units, genre labels, style transfer, and large sets of audio features, reflecting the complexity of dance distributions.The paper notes that extensive audio features may affect generalization across music pieces and genres.
  • Diffusion models: Related generative approaches include Conformer-based adversarial models and Transformer-based normalizing flows, but neither prior line used diffusion models for audio-to-motion generation.Prior diffusion work addressed audio waveform generation or text-conditioned motion rather than motion generated from acoustic features.

3 METHOD

The paper formulates audio-driven pose synthesis as conditional diffusion and adapts DiffWave to generate vector-valued motion sequences. Its architecture replaces dilated convolutions with Transformer/Conformer stacks and supports translation-invariant positional information.

  • The task is to generate human pose sequences from aligned audio features, optionally conditioned on a style vector.
  • Diffusion progressively adds Gaussian noise, then trains a neural network to reverse the process and generate observations from noise.
  • The denoising network predicts the noise added to training poses, thereby defining the denoising process and learned probability density.
  • Sampling starts from Gaussian noise and reverses N diffusion steps; faster synthesis is left for future work.
  • The model adapts DiffWave by removing upsampling and changing scalar waveform output to vector-valued pose output at the audio frame rate.
  • Stacks of Transformers or Conformers replace DiffWave’s dilated convolutions, combining long-range self-attention with convolutional processing for motion.
  • Translation-invariant self-attention supplies sequence-position information because the authors expect motion to be invariant to temporal translation.

4 EXPERIMENTS

Experiments evaluate full-body motion generation across five motion-capture datasets using subjective studies as the core assessment. The evaluation uses pairwise video comparisons, merit scores, and statistical testing, with additional objective metrics reported.

  • 4.1 Data and modelling: The experiments use five high-quality 3D motion-capture datasets and model full-body motion, which increases dimensionality and makes artefacts such as foot-sliding more visible.
  • 4.1 Data and modelling: Motion is represented at 30 fps with skeletal joint rotations in exponential-map coordinates relative to a T-pose.
  • 4.1 Data and modelling: Most style-controlled systems have matched style-unconditional models, enabling classifier-free guidance to adjust stylistic expression.
  • 4.1 Data and modelling: Hyperparameter tuning found noise scheduling especially important, with a larger schedule end-value producing improved motion.
  • 4.2 Evaluation methodology: Subjective evaluation is central because motion naturalness and style lack widely validated objective metrics, although objective metrics are also reported.
  • 4.2 Evaluation methodology: Four user studies compare motion through consecutive side-by-side 10-second clips, covering three gesture studies and one dance study.
  • 4.2 Evaluation methodology: Responses are converted into merit scores, analysed with one-way ANOVA and Tukey multiple-comparisons tests, and accompanied by 95% confidence intervals and win rates.

4.3 Gesture-generation experiments

The gesture experiments compare the proposed diffusion model with strong autoregressive baselines on TSG and ZeroEGGS, evaluating motion preference and stylistic distinctiveness. The proposed model improves motion preference over StyleGestures and ZeroEGGS, while guided diffusion increases style distinctiveness at a naturalness trade-off.

  • TSG dataset: On TSG, LDA ranked below ground-truth motion capture but above LDA-DW and StyleGestures in user-rated merit.All differences were significant; LDA versus LDA-DW had p < 0.01, while other differences had p < 0.001.
  • TSG dataset: The Conformer-based LDA architecture outperformed the original DiffWave-based ablation and StyleGestures on TSG motion quality.The authors also report that Conformers appeared less sensitive to hyperparameter settings.
  • ZeroEGGS dataset: The ZeroEGGS study tested four styles—happy, angry, old, and public speaking—using neutral-style speech to reduce interference from speech style.Nine 10-second neutral-style segments generated 36 segment-style combinations.
  • ZeroEGGS dataset: On ZeroEGGS, LDA achieved the best motion-preference score, outperforming ZeroEGGS, while LDA-G and ZeroEGGS were statistically tied.The preference advantage over ZeroEGGS was significant at p < 0.001; exaggerated style expression was less natural and less preferred.
  • ZeroEGGS dataset: LDA-G received the highest style-matching score, ahead of ZeroEGGS and unguided LDA.The comparisons were significant at p < 0.01 for LDA-G versus ZeroEGGS and p < 0.001 for ZeroEGGS versus LDA.

4.4 Music-driven dance synthesis

The paper evaluates a style-conditional diffusion model for music-driven dance using a new 373-minute motion-capture dataset and compares it with Bailando. The proposed model ranks below ground truth but above the baseline in subjective dance quality, while producing visually distinctive styles.

  • Dataset and setup: The new dance dataset contains 373 minutes of parallel music audio and high-quality dance motion capture across multiple genres and styles.It combines selected PSMD recordings with approximately 3.5 hours of additional recordings from three accomplished dancers.
  • Generated motion: The style-conditional dance outputs were visually distinctive even in still images, and demonstrations included sequences longer than a thousand frames.The proposed models were trained on joint rotations and converted to joint positions for user-study visualisation.
  • Dataset and setup: LDA, LDA-U, and Bailando were compared using held-out motion and songs across Jazz, Charleston, Hip-Hop, Locking, Krumping, and Casual styles.The study generated 36 10-second audio segments, with six segments per dance style.
  • Subjective evaluation: Ground-truth motion capture achieved the highest preference scores, followed by LDA, LDA-U, and Bailando, with all differences significant at p < 0.001.LDA exhibited an almost 40% win rate against ground truth.
  • Subjective evaluation: The authors conclude that LDA surpasses Bailando in subjective dance quality but does not reach human motion-capture quality.They suggest LDA-U’s lower rating may reflect its smaller set of music features and lack of supplied style control.

4.5 Objective metrics

The paper supplements human evaluations with distributional, diversity, and beat-alignment metrics for gesture and dance, and demonstrates generalisation to path-driven locomotion. These metrics broadly align with perception but cannot fully assess control fidelity and may be statistically limited by small datasets.

  • Metric scope: FID-based metrics compare real and synthetic feature distributions, but they cannot determine whether motion follows the associated style or audio control.This limitation applies to rhythm appropriateness and other control-input relationships.
  • Gesture metrics: Gesture quality was evaluated with Fréchet Gesture Distance using a pretrained gesture autoencoder as the feature extractor.FGD has a moderate but nonzero correlation with human-likeness ratings.
  • Dance metrics: Dance evaluation used FID_k, FID_g, diversity scores Div_k and Div_g, and beat-alignment score BAS.Lower Fréchet distances are better, while higher BAS indicates closer beat matching.
  • Metric interpretation: Objective scores mostly aligned with human perception, but LDA-DW matched the SG baseline on FGD despite significantly outperforming it in the user study.The authors attribute such outliers partly to limited statistical accuracy from small motion-capture test sets.
  • Dance metrics: The proposed dance model was more diverse than Bailando, with kinematic diversity comparable to natural dance and beat alignment close to professional recorded motion.The authors caution that good dancing is not reducible to beat matching.
  • Path-driven locomotion: Path-driven locomotion conditioned root motion on a ground-plane path and style, producing circular-path motion consistent with both controls.The RaisedLeftArm example showed no noticeable foot-skating and transitions between root-node speeds while turning.

5 PRODUCTS OF EXPERT DIFFUSION MODELS

The paper generalises classifier-free guidance into product-of-expert diffusion ensembles, enabling style interpolation, extrapolation, dynamic transitions, and combinations of diverse motion models.

  • Guided interpolation: Guided interpolation blends two conditional diffusion models during denoising and supports extrapolation when the interpolation weight lies outside [0, 1].The method differs from averaging discrete style inputs or latent embeddings, which may produce poorly covered or undefined regions.
  • General ensembles: The framework generalises to M models with weights that need not lie in [0, 1], provided their output spaces match.Experts may differ in architecture, conditioning information, and training data.
  • Theory: Product-of-expert ensembles combine diffusion-model predictions to restrict generated motion toward regions considered probable by every expert.Diffusion sampling makes these probabilistic products feasible for synthesis despite their historically difficult sampling problem.
  • Experiments: Interpolations produce a posture progression from hunched old to upright angry, a steady increase in movement range from still to public speaking, and contrasting locomotion styles.Wrist-speed statistics change monotonically but nonlinearly across style weights, while time-dependent weights enable dynamic transitions.
  • Experiments: 50-50 dance-fighting combinations preserve natural beat-synchronised motion while suppressing distinct dance and fighting moves that are unlikely under one expert.The resulting arm and hand heights generally remain appropriate for MMA, consistent with the intersection of the two distributions.

6 LIMITATIONS

The models generate visually strong motion but remain constrained by slow synthesis and limited access to semantic, musical, and long-range choreographic structure.

  • Generation speed: Diffusion sampling requires many denoising steps, and the paper’s architectures take around a second to generate each motion example.The work targets offline generation and does not optimise synthesis speed.
  • Generation speed: The parallel architecture is unsuitable for real-time interaction or game-engine integration, where autoregressive models might be a better fit.The authors expect recent diffusion innovations could considerably accelerate synthesis.
  • Motion understanding: Audio-driven models do not capture all aspects of purposeful gesturing and dancing, including speech semantics and global musical structure.The authors suggest adding text-derived semantic features such as BERT embeddings for communicative gestures.
  • Motion understanding: Limited rapidly changing musical features and short training windows hinder learning higher-level dance structure and longer choreographic correlations.A hierarchical model with explicit structure information is proposed as a possible direction.
  • Product-of-expert ensembles: Product-of-expert interpolation can yield less distinctive or less motion-filled outputs than either endpoint because it focuses on their distributional intersection.The authors identify this mathematically consistent behaviour as both a strength and a weakness.

7 CONCLUSIONS AND FUTURE WORK

The paper establishes diffusion models as a strong approach for audio-driven 3D motion generation and extends them with controllable product-of-expert composition. Future work targets faster synthesis, richer conditioning, and multimodal generation.

  • Conclusions: The proposed diffusion approach outperforms previous state-of-the-art models in motion quality across audio-driven motion applications and supports controllable stylistic expression.The paper also applies the architecture to path-driven locomotion.
  • Conclusions: Product-of-expert diffusion ensembles enable novel interpolation and combinations of diverse diffusion models with different control parameters.The formulation is presented as applicable to ensembling, interpolation, and beyond.
  • Future work: Future work includes accelerated generation, conditioning motion on audio and text, and synthesising speech and gesture together.Text conditioning is intended to support semantic and communicative gestures synchronised with speech.
  • Future work: Using pretrained diffusion models as strong priors could avoid modifying large existing models and reduce the risk of catastrophic forgetting associated with prolonged fine-tuning.This possibility is discussed for supervised tasks and reinforcement learning.

A.1 Data processing and model training

The pipeline standardizes and augments motion data, adapts conditioning for different datasets, and tunes Conformer depth allocation to motion characteristics. Training configurations vary across gesture, dance, and other datasets, while style conditioning differs for the ZE baseline.

  • Data processing: Inputs and outputs were standardized, with lateral mirroring and time stretching augmenting selected dancing and MMA training updates.Binary features were left on {0, 1}; time stretching used factors uniformly sampled from [0.9, 1.1].
  • Data processing: Finger motion was excluded and replaced with a fixed hand pose in all experiments.The paper attributes this choice to the difficulty and quality issues of finger motion capture.
  • Model tuning: With similar parameter counts, 10 blocks with 4 Conformers produced more deliberate motion, whereas 20 blocks with 2 Conformers increased overall motion.The former setting was used for most scenarios, while dance used the latter; differences were subtle and motion quality was similar.
  • Training configuration: Final models used 100 diffusion steps for gesture datasets and 150 steps for other datasets, with dataset-specific noise ranges and architecture settings.Most models used 8 attention heads, 256 attention channels, 512-dimensional embeddings, and 1024 feedforward and Conformer channels.
  • Style conditioning: Unlike the proposed systems, the ZE baseline conditioned style on an encoded motion clip sampled from a training video rather than one-hot style input.The sampled clips were 256–512 frames long.

A.2 User studies

The user studies recruited fluent English speakers from several English-speaking countries and evaluated paired 20-second videos in web browsers. Randomization, headphones, compensation, and attention checks were used to structure participation and data inclusion.

  • Participants and procedure: Participants were recruited through Prolific from the US, Canada, UK, Ireland, Australia, and New Zealand, and were required to be fluent in English.Studies ran in a web browser and generally presented 36 20-second comparison videos per participant.
  • Participants and procedure: Participants generally used headphones, while clip order and video order were randomized for each participant.Each participant typically saw motion from each audio clip once.
  • Participants and procedure: The median experiment completion time was 15 minutes, with approximately 12 GBP median hourly compensation.
  • Quality control: Two random attention checks were included, and participants failing both were disqualified from analysis.The check used spoken instructions, or displayed text when stimuli omitted audio.
Loading 2211.09707v2…