PaperlayerView exact revision on arXiv

Source-linked AI summary

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

Tenglong Ao, Zeyi Zhang, Libin Liu

arXiv:2303.14613v4cs.CVcs.GR

TL;DR

Stylized co-speech gesture systems have limited flexibility when controlled by predefined labels or example clips. GestureDiffuCLIP uses CLIP-guided multimodal style conditioning with latent diffusion and contrastive gesture–transcript alignment, and reports flexible generation plus superior user-study ratings against state-of-the-art approaches. Its scope is bounded by training data, architecture, and limitations of the vanilla CLIP text encoder and pose imitation.

  • Problem

    Predefined labels and example motion clips do not flexibly convey user intent, while speech-to-gesture ambiguity can weaken semantic correspondence.

  • Method

    GestureDiffuCLIP extracts style representations from text, video, or motion with CLIP, injects them into a latent diffusion generator via AdaIN, and uses contrastive gesture–transcript alignment.

  • Results

    The system generates realistic stylized gestures from varied style descriptions and outperforms state-of-the-art approaches in human likeness, appropriateness, and style correctness.

  • Takeaways & Limitations

    The framework supports zero-shot style control with unseen prompts and multimodal inputs, including text, motion sequences, and video clips.

  • Takeaways & Limitations

    Performance and robustness remain constrained by training data and architecture; vanilla CLIP text encoders cannot accept excessively long prompts reliably.

Abstract

from arXiv · show

The automatic generation of stylized co-speech gestures has recently received increasing attention. Previous systems typically allow style control via predefined text labels or example motion clips, which are often not flexible enough to convey user intent accurately. In this work, we present GestureDiffuCLIP, a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. We leverage the power of the large-scale Contrastive-Language-Image-Pre-training (CLIP) model and present a novel CLIP-guided mechanism that extracts efficient style representations from multiple input modalities, such as a piece of text, an example motion clip, or a video. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator via an adaptive instance normalization (AdaIN) layer. We further devise a gesture-transcript alignment mechanism that ensures a semantically correct gesture generation based on contrastive learning. Our system can also be extended to allow fine-grained style control of individual body parts. We demonstrate an extensive set of examples showing the flexibility and generalizability of our model to a variety of style descriptions. In a user study, we show that our system outperforms the state-of-the-art approaches regarding human likeness, appropriateness, and style correctness.

1 INTRODUCTION

GestureDiffuCLIP addresses limited controllability in stylized co-speech gesture synthesis by using CLIP to support flexible multimodal style prompts and contrastive learning to preserve speech–gesture semantics.

  • Motivation: Existing gesture systems struggle to synthesize arbitrary stylized gestures under user control.Label-based approaches are constrained by the number and granularity of available style labels.
  • Motivation: CLIP provides a shared latent space that can represent style prompts from text, motion, and video in a flexible interface.The resulting representations can guide generative models such as diffusion models.
  • Motivation: Speech-to-gesture generation is a many-to-many mapping, and imperfect temporal alignment can produce mean motions with weak semantic correspondence.The paper addresses this ambiguity with a gesture–transcript joint embedding space and semantic loss.
  • Contributions: The proposed system distills pretrained CLIP knowledge through self-supervised training, enabling zero-shot style generation from arbitrary text or video prompts.Training treats each gesture motion as its own style prompt and reconstructs it from extracted CLIP latents.
  • Contributions: GestureDiffuCLIP combines multimodal prompt-conditioned synthesis, latent diffusion with transformer and AdaIN layers, semantic alignment, and self-supervised learning without detailed labels.These elements are presented as the paper’s principal contributions.

2 RELATED WORK

Prior work spans rule-based, label-based, example-based, and neural motion-style methods, while GestureDiffuCLIP uses CLIP and AdaIN to support multimodal, fine-grained style control without labeled data.

  • Co-speech gesture synthesis: Rule-based gesture methods are interpretable and controllable but require substantial effort to create datasets and linguistic rules.Data-driven approaches were developed to reduce this manual burden.
  • Co-speech gesture synthesis: End-to-end gesture models generate natural-looking motion but face challenges in matching generated content to speech because speech and gesture have an inherent many-to-many relationship.Other systems explicitly model rhythm and semantics through architecture or training supervision.
  • Style control for human motion: Motion style-transfer methods transfer the style of a reference motion clip to source motion, with later work improving quality, efficiency, supervision, and body-part control.The surveyed methods include AdaIN-, VAE-, flow-, and few-shot-based approaches.
  • Style control for co-speech gestures: Earlier co-speech style-control systems depend on labeled data and may learn only a single style per generator or require low-resource adaptation.These limitations motivate more flexible style representations.
  • GestureDiffuCLIP: GestureDiffuCLIP extracts style embeddings from arbitrary text prompts with CLIP and injects them through AdaIN, supporting fine-grained multimodal prompts without style labels.Its autoregressive design is described as potentially suitable for real-time applications.

3 SYSTEM OVERVIEW

The system combines a speech-conditioned latent diffusion generator with a CLIP-based style encoder, while contrastive gesture–transcript embeddings provide semantic guidance during synthesis.

  • System overview: The system accepts speech audio, a transcript, and a text, video, or motion style prompt to generate full-body gestures aligned with speech rhythm and content.The style prompt describes the desired gesture style.
  • System overview: A latent diffusion generator produces speech-matched gesture sequences, while a CLIP-based encoder extracts style embeddings for conditioning.The generator operates in a compact latent motion space learned with VQ-VAE.
  • Style conditioning: Style embeddings are incorporated into the diffusion generator through an adaptive instance normalization layer.Classifier-free diffusion guidance is used alongside self-supervised learning.
  • Semantic alignment: The gesture–transcript module learns a shared embedding space by encoding paired transcript and gesture sequences and maximizing their similarity with contrastive loss.Transcript features are aggregated by max pooling before comparison with gesture embeddings.

4 MOTION REPRESENTATION

The motion representation compresses pose sequences into discrete latent codes with VQ-VAE, enabling diffusion-based generation in a compact space while preserving motion quality and diversity.

  • Pose representation: Each gesture motion is represented as a sequence of poses containing character displacement and 6D joint rotations.The sequence length is denoted by K and the character has J joints.
  • VQ-VAE representation: VQ-VAE provides a compact motion representation intended to preserve quality and diversity while enabling fast inference.The model uses an encoder–decoder architecture.
  • VQ-VAE representation: The VQ-VAE encoder converts the motion sequence into a downsampled latent-code sequence with latent dimension C.The downsampling rate is d = K/L, where L is the latent sequence length.
  • VQ-VAE representation: During reconstruction, the decoder quantizes latent codes to their nearest vectors in a learned codebook before decoding the sequence back into motion.The codebook contains N_VQ latent vectors.
  • Training: The encoder and decoder are trained with standard VQ-VAE training and then frozen for the remaining system training.Both continuous and quantized latent sequences are used by other system components.

5 GESTURE-TRANSCRIPT JOINT EMBEDDINGS

The paper learns a shared gesture-transcript embedding space with contrastive learning to connect speech semantics and gesture motion despite many-to-many and temporally imperfect correspondence. These embeddings support transcript retrieval and semantic saliency estimation for guiding semantically correct gesture generation.

  • Motivation: A gesture-transcript joint embedding space addresses the many-to-many mapping between speech and gesture that can otherwise produce mean motions and weak semantic correspondence.The method combines contrastive learning with temporal aggregation to handle ambiguous gesture-speech relationships.
  • Joint embedding construction: Gesture and transcript encoders map sentence-level inputs into a shared space, with transcript tokens temporally associated with audio and gestures.The resulting feature sequences can have different lengths while sharing the same feature dimension.
  • Joint embedding construction: Max pooling aggregates semantics-relevant information from each feature sequence, reducing sensitivity to imperfect temporal alignment between gestures and corresponding words or phrases.The pooled vectors serve as the transcript and gesture embeddings used for contrastive learning.
  • Contrastive learning: CLIP-style contrastive learning increases similarity for matched gesture-transcript pairs and decreases similarity for mismatched pairs using gesture-to-text and text-to-gesture cross-entropy.The loss is computed across a training batch, with softmax temperature τ controlling the similarity distributions.
  • Contrastive learning: Momentum distillation uses pseudo-targets from exponentially moving-average encoders to reduce instability caused by missing or multiple gesture correspondences.The technique addresses noisy correspondence in real data.
  • Applications: The joint embeddings support motion-based transcript retrieval and semantic saliency identification, with high-saliency words providing cues for semantically correct gesture generation.Retrieved sentences may differ in meaning while remaining naturally pairable with the query gesture.

6 STYLIZED CO-SPEECH GESTURE DIFFUSION MODEL

GestureDiffuCLIP generates gestures in a conditional latent diffusion model using speech audio, transcripts, and flexible style prompts. A transformer denoising network integrates rhythm, semantics, and CLIP-derived style through hierarchical conditioning, while separate encoders support text, motion, and video prompts.

  • Generator: The generator synthesizes latent gesture codes conditioned on audio, transcript, and style prompt, then decodes them into gesture motion with a pretrained VQ-VAE decoder.Audio captures rhythm and stress, transcripts provide semantic features, and the prompt determines overall style.
  • Latent diffusion: Inference uses DDPM sampling autoregressively, generating and denoising each frame’s latent code using previous motion, speech context, style prompt, and future audio frames.The strategy extends naturally to long sequences conditioned on previously generated gestures.
  • Latent diffusion: Latent diffusion progressively noises gesture representations toward N(0, I), then reverses the process to recover motion from random latent codes.The reverse process is performed with a denoising network that predicts added noise.
  • Denoising network: The transformer denoising network hierarchically fuses audio with noisy latents, transcript semantics through attention, and style through a CLIP-guided AdaIN layer.Causal attention restricts interactions to current and preceding data, while audio conditioning includes δ_a future frames.
  • Style prompting: Separate CLIP-based encoders represent text, motion, and video prompts, with a video encoder temporally aggregating CLIP image features into a shared embedding.The CLIP encoders are pretrained separately and frozen while the denoising network is trained.
  • Style prompting: Training uses gesture clips as style prompts rather than requiring detailed style labels, enabling zero-shot generation for arbitrary text or video prompts.The self-supervised scheme reconstructs motions from extracted CLIP latents despite training on motion prompts only.
  • Body-part control: The system extends fine-grained body-part style control by partitioning the body and learning separate motion VQ-VAEs for individual parts.Noise combination is used to combine body-part-specific controls.

7 EVALUATION

Evaluation shows that GestureDiffuCLIP supports flexible style control across prompt modalities, body parts, sentences, and guidance strengths while preserving realistic and semantically appropriate motion. User studies, quantitative metrics, and ablations support the effectiveness of its semantic and style components.

  • 7.2 Results: Text, video, and motion prompts guide distinct styles for the same speech, including anger, Hip-hop, environmental sway, and arm-up gestures.The demonstrations use a ZeroEGGS test speech and show cross-modal style conditioning.
  • 7.2 Results: Different prompts control different body parts while maintaining natural coordination among them.This demonstrates fine-grained body part-aware style control rather than only whole-body conditioning.
  • 7.2 Results: Changing the classifier-free guidance scale s strengthens or exaggerates the prompted style, such as raising the hands higher for “raise both hands.”The effect is visualized in Figure 9.
  • 7.2 Results: Style prompts can change at every sentence while generated gestures match each style and transition smoothly between them.The demonstrated sequence changes from “raise both hands” to “out of breath” and then “Usain Bolt.”
  • 7.3.2 User Study: The user study finds Ours tied with ground truth and Ours (w/o transcript) for human likeness while outperforming CaMN, and Ours scores highest for appropriateness.Both comparisons are reported with p<0.001, while removing transcript information significantly lowers appropriateness.
  • 7.4 Quantitative Evaluation: Ablations show that transcript embeddings improve semantic consistency, style loss improves style recognizability, and AdaIN avoids jittering and unnatural movements compared with concatenation fusion.Removing transcript-related components lowers appropriateness and SC; removing Lstyle lowers SRA; concatenation causes visibly unnatural motion.

8 CONCLUSION

GestureDiffuCLIP supports flexible stylized gesture synthesis from text, motion, and video prompts while aligning generated gestures with speech. The authors report broad style flexibility, but note limitations in prompt interpretation, physical motion quality, and real-time inference.

  • 8 CONCLUSION: The system extracts CLIP style embeddings from text, motion, and video prompts and injects them into a diffusion generator through AdaIN.This unified design also supports style control for individual body parts through noise combination.
  • 8 CONCLUSION: Gesture-transcript contrastive learning promotes semantic consistency between speech and generated gestures.The learned joint embedding space is intended to preserve semantic and rhythmic harmony with speech.
  • 8 CONCLUSION: Video prompts can produce gestures conditioned on diverse visual styles, including yoga, flying birds, dinosaurs, fire, and lightning.These examples illustrate conditioning on non-motion video content as style prompts.
  • 8 CONCLUSION: Long text prompts may be interpreted inaccurately, and poses far outside the training distribution may not be reproduced reliably.The authors specifically identify vanilla CLIP text encoders and out-of-distribution poses as constraints.
  • 8 CONCLUSION: Scaling the CLIP-style gesture-transcript models with larger datasets remains an open direction.The authors identify increased model power from larger training datasets as a future research opportunity.
  • 8 CONCLUSION: Generated motion exhibits slight foot sliding, while latent diffusion requires many inference steps and is challenging to use in real time.The paper suggests physically based generation and diffusion acceleration as future directions.

A DETAILS OF USER STUDY

The user study uses paired 10-second videos and evaluates multiple gesture-generation approaches across speech samples. Participants assess randomized side-by-side comparisons using questionnaires administered through the Credamo HBO tool.

  • A DETAILS OF USER STUDY: Each comparison pair contains two 10-second videos played sequentially from left to right using the same speech and character model.The questionnaires are administered through Credamo’s Human Behavior Online tool.
  • A DETAILS OF USER STUDY: The BEAT-based study creates 288 video pairs from 24 speech samples and 12 pairwise combinations involving GT, Ours, Ours without transcript, and CaMN.Each participant evaluates 24 pairs covering all 24 speech samples.

A.1.2 ZeroEGGS Dataset.

The ZeroEGGS study evaluates style-conditioned gesture synthesis using six neutral audio clips and four target styles. It compares ZE with the proposed system in randomized paired assessments.

  • A.1.2 ZeroEGGS Dataset: Six neutral ZeroEGGS test recordings are combined with happy, sad, angry, and old styles, producing 24 video clips for each system.The comparison is between ZE and Ours, with all 24 clips shown once per participant in randomized order.

A.2 Style-Control Study on the ZeroEGGS Dataset

The style-control study compares systems under dataset-label and random-prompt conditions using matched audio and style configurations. ChatGPT prompts are designed to generate detailed gesture and emotional-expression guidance from speech transcripts.

  • A.2 Style-Control Study on the ZeroEGGS Dataset: The dataset-label and random-prompt tests use the same six audio clips and four text prompts, differing only in the input prompt.Each test generates 24 clips per system for MD-ZE and Ours.
  • A.2 Style-Control Study on the ZeroEGGS Dataset: The prompt-generation setup asks ChatGPT to act as a public-speaking coach and provide body-language, gesture-style, and emotional-expression suggestions.Suggestions are requested after each transcript sentence based on its semantics and emotion.
  • A.2 Style-Control Study on the ZeroEGGS Dataset: Figure 15 presents the prompt inputs supplied to ChatGPT.The figure documents the prompt-generation inputs rather than reporting a performance comparison.

B IMPLEMENTATION DETAILS OF BASELINES

The baseline implementation details describe reproducing CaMN, selecting a better-performing model as baseline, and constructing MD-ZE from official pretrained components with motion retargeting. Figure 15 documents ChatGPT prompt inputs used in Section 7.5.

  • CaMN is reproduced using provided training codes on a larger original-paper dataset because its pretrained generation model was unavailable.The unreleased emotion label was discarded from the model conditions.
  • The reproduced CaMN model achieves an FGD value of 122.5, while the selected baseline achieves better FGD of 110.23.
  • MD-ZE is constructed from official pretrained MotionDiffuse and ZeroEGGS models, despite their different skeletons.
  • Motion prompts are converted from SMPL joint positions into rotations, saved as BVH, and retargeted to the ZeroEGGS skeleton using Blender's BVH Retargeter.
  • Figure 15 demonstrates the prompt inputs supplied to ChatGPT in Section 7.5.
Loading 2303.14613v4…