PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation

Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, Bolei Zhou

arXiv:2203.13161v1cs.CV

TL;DR

Co-speech gesture generation seeks speech-consistent body and gesture movements, while holistic pose synthesis struggles with fine-grained gestures. HA2G hierarchically extracts audio features and renders pose, achieving realistic gestures that outperform prior methods in objective and human evaluations.

  • Problem

    Holistic generation of all joints simultaneously struggles to produce fine-grained co-speech gestures.

  • Method

    HA2G uses a Hierarchical Audio Learner, a Hierarchical Pose Inferer, and audio-text contrastive learning to generate gestures hierarchically.

  • Results

    HA2G generates realistic, high-fidelity co-speech gestures and outperforms previous methods in objective evaluations and human studies.

  • Takeaways & Limitations

    Hierarchical cross-modal associations support fine-grained gesture generation across different human body parts and audio semantic granularities.

  • Takeaways & Limitations

    The model is trained on an English-based corpus, introducing language-related inductive bias; generating gestures across diverse languages remains future work.

Abstract

from arXiv · show

Generating speech-consistent body and gesture movements is a long-standing problem in virtual avatar creation. Previous studies often synthesize pose movement in a holistic manner, where poses of all joints are generated simultaneously. Such a straightforward pipeline fails to generate fine-grained co-speech gestures. One observation is that the hierarchical semantics in speech and the hierarchical structures of human gestures can be naturally described into multiple granularities and associated together. To fully utilize the rich connections between speech audio and human gestures, we propose a novel framework named Hierarchical Audio-to-Gesture (HA2G) for co-speech gesture generation. In HA2G, a Hierarchical Audio Learner extracts audio representations across semantic granularities. A Hierarchical Pose Inferer subsequently renders the entire human pose gradually in a hierarchical manner. To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations. Extensive experiments and human evaluation demonstrate that the proposed method renders realistic co-speech gestures and outperforms previous methods in a clear margin. Project page: https://alvinliu0.github.io/projects/HA2G

1. Introduction

Co-speech gestures clarify spoken content and support more expressive human–machine interaction, but holistic pose generation misses fine-grained cross-modal motion. HA2G addresses this by associating hierarchical audio semantics with hierarchical body movements.

  • Co-speech gestures supplement speech information, making communicated content clearer and more understandable.
  • Generating expressive body movements is important for facilitating human–machine interaction with embodied agents.
  • Deep models mapping compact audio or text representations to holistic pose sequences fail to capture micro-scale motions and cross-modal information.
  • Different gesture types correspond to different audio levels, with high-level semantics supporting metaphorical gestures and low-level beat or volume supporting rhythmic gestures.
  • HA2G builds hierarchical cross-modal associations across multiple granularities and generates gestures from coarse to fine using Hierarchical Audio Learner and Hierarchical Pose Inferer modules.
  • HA2G extracts hierarchical audio features, uses audio-text contrastive learning, and associates multi-level features with body parts for cascaded pose generation.

2. Related Work

Prior work studies audio-visual correspondence in related motion-generation tasks, while speech-driven gesture synthesis remains difficult because speech–gesture correspondence is more implicit and complicated.

  • Human-centered audio-visual learning has addressed music-to-dance and talking-face generation, but speech-to-gesture modeling remains largely unexplored.
  • Speech-driven gesture generation requires constructing speech–gesture correspondences that are more complicated and implicit than those in music-to-dance or talking-face generation.
  • Existing motion-generation methods include CNN-, RNN-, and GAN-based visual approaches that use historical motions as input.
  • HA2G instead focuses on identifying correlations between speech and gestures for speech-driven motion synthesis in conversational settings.

3. Our Approach

HA2G combines hierarchical audio representations, transcript-based contrastive learning, identity-conditioned feature blending, and cascaded pose generation to synthesize speech-driven gestures progressively from coarse to fine.

  • HA2G generates a target person’s co-speech gestures from speech audio, speaker identity, initial poses, and optionally text information.
  • The Hierarchical Audio Learner extracts low-, middle-, and high-level audio features from different encoder depths to represent distinct semantic granularities.
  • Transcript features provide high-level linguistic supervision: aligned high-level audio features are positives, while mismatched high-level and low/mid-level features serve as negatives.
  • The style coordinator transforms reference-frame identity features into weights that linearly blend low-, mid-, and high-level audio features for each motion hierarchy.
  • A six-level body hierarchy generates poses progressively, with each level using the previous inferred pose and its style-coordinated audio feature.
  • Cascaded bi-GRUs generate the final pose from initial frames through hierarchical intermediate poses, with Huber reconstruction, adversarial, and physical constraints supporting training.

4. Experiments

HA2G is evaluated on TED Gesture and TED-Expressive against established baselines using distributional, beat-consistency, diversity, qualitative, user-study, and ablation analyses. Results support stronger fine-grained and synchronous gesture generation, while revealing dataset-dependent gains and the contributions of hierarchical audio and pose modeling.

  • Experimental Settings: HA2G is compared with Attention Seq2Seq, Speech2Gesture, Joint Embedding, and Trimodal on TED Gesture and TED-Expressive under matched training settings.The evaluation also includes pseudo ground truth; methods without open-source code are excluded from comparison.
  • Quantitative Evaluation: FGD, Beat Consistency, and diversity evaluate gesture distribution quality, audio-motion synchronization, and variation across generated gestures.FGD uses latent feature distributions, Beat Consistency uses audio and kinematic beats, and diversity averages feature distances across generated pairs.
  • Quantitative Evaluation: HA2G outperforms existing methods on both datasets, with its strongest FGD advantage on TED-Expressive because TED Gesture lacks finger information.The authors attribute the dataset difference to the lower motion hierarchy available without finger annotations.
  • Qualitative Evaluation: Qualitative comparisons show that HA2G produces synchronous and more diverse motions, whereas baselines exhibit slow, rigid, asynchronous, or unnatural poses.Speech2Gesture also produces synchronous motions but shares twisted-hand artifacts; HA2G is reported as more diverse than that single-level design.
  • Ablation Study: Ablations show that multi-level audio features, audio-text contrastive learning, pose hierarchy, distinct level-specific features, and physical regularization improve performance or pose realism.Removing text still outperforms Trimodal with text, while removing hierarchy degrades fine-grained associations and removing physical regularization reduces realistic pose quality.

5. Discussion

HA2G combines hierarchical audio representations, contrastive audio-text learning, and coarse-to-fine pose generation for co-speech gestures. The authors report superior performance, while identifying English-language training data as a limitation.

  • HA2G extracts discriminative audio representations across semantic granularities through a contrastive learning strategy.
  • Its Hierarchical Pose Inferer uses physical regularization to render the entire human pose gradually in a hierarchical manner.
  • Extensive experiments demonstrate superior performance and high-fidelity co-speech gesture generation.
  • The model is trained on an English-based corpus, introducing language-related inductive bias and motivating work on diverse languages.

Supplemental Document: Learning Hierarchical Cross-Modal Association

The supplemental document concerns co-speech gesture generation.

  • The document addresses co-speech gesture generation.
  • Its title identifies the topic as co-speech gesture generation.
  • The available passage provides only the topic phrase “for Co-Speech Gesture Generation.”

A. More Details about Dataset

The dataset section describes data collection, filtering, annotation considerations, and dataset statistics for TED Gesture and TED-Expressive.

  • The data section discusses drawbacks of expensive MoCap collection for building large corpora across speaking contents and styles.
  • Selected clips require visible speaker keypoints, nontrivial motion, and duration longer than 5 seconds.
  • Pose annotations serve as pseudo ground truth, with ExPose performance and filtering used to support their suitability for training.
  • The section records dataset statistics in Table 5 and describes speech-text preprocessing using transcript word onset timestamps.

B. Architecture Details

The architecture extracts multi-level audio features, blends them with speaker and motion hierarchy information, and generates poses progressively with auxiliary modeling and evaluation components.

  • Hierarchical audio features are aligned in shape and time, then blended and passed to cascaded bi-GRUs for coarse-to-fine pose generation.
  • Audio-text contrastive learning is used to obtain more discriminative cross-modal feature representations.
  • The motion hierarchy follows the body’s tree structure, progressively detaching joints from the fingers toward the main body structure.
  • ResNetSE34 stage-2, stage-3, and stage-4 outputs are transformed into low-, mid-, and high-level audio features.
  • Later pose hierarchies use outputs from preceding levels, allowing successive layers to generate poses with information from earlier pose levels.
  • A GAN discriminator, pose auto-encoder, and physical constraints support training or evaluation of generated poses.

C. Training Stage and Inference Stage

Training uses speech audio, transcripts, reference frames, and initial poses, while inference removes the transcript requirement and can synthesize styles and poses when references are unavailable.

  • Training Stage: Training encodes speech audio into multi-level features, transcripts into text features for contrastive learning, and reference frames into identity and initial-pose representations.The reference frames also provide style coordination and initial poses.
  • Inference Stage: At inference, the transcript is omitted because it serves only as an auxiliary input during training.
  • Inference Stage: Without reference frames or initial poses, the method samples a speaker-identity style vector and initial poses from the dataset.
  • Physical Constraint: The physics constraint represents poses as bone-direction vectors and regularizes included angles with Gaussian log-likelihoods.The angle statistics are estimated from pseudo-ground-truth hand annotations because no benchmark provides accurate finger keypoints for co-speech settings.

E. Analysis on Beat Consistency Score Metric

The paper adapts Beat Consistency to co-speech motion by tracking included-angle changes rather than summed keypoint velocity, then analyzes threshold sensitivity across methods.

  • Metric Design: Beat Consistency measures motion–audio beat correlation using changes in included angles between bones.This design accommodates speech gestures, where arms may remain relatively still while fingers move at different scales.
  • Metric Design: The method normalizes each frame’s summed angle change by the mean absolute angle change before beat extraction.
  • Threshold Analysis: Beat candidates are local optima whose differences from neighboring time steps exceed a threshold, filtering trivial changes while retaining sudden motion changes.
  • Threshold Analysis: Our method achieves superior Beat Consistency performance with high robustness to threshold variation, while Speech2Gesture performs better only at low thresholds by matching trivial beats.

G.1. Ablation Study on TED Gesture Dataset

The TED Gesture ablation evaluates module contributions and reference-frame sensitivity, with limited hierarchy because the dataset lacks finger annotations.

  • Ablation Study: TED Gesture lacks finger annotations, producing a lower motion hierarchy and less significant performance improvement from individual modules.
  • Metric Analysis: Figure 3 evaluates Beat Consistency across thresholds from 0.01 to 0.3 in increments of 0.01.
  • Reference-Frame Sensitivity: Using additional initial poses yields only marginal performance gains, indicating robustness to the reference-frame hyperparameter M.The compared settings include M = 1, 4, and 7 frames, with M = 4 inherited from Trimodal.

G.3. Randomness of Diversity Metric

The Diversity evaluation reduces sampling randomness by using 500 pairs and repeating the evaluation ten times with different random seeds.

  • Evaluation Procedure: The Diversity metric is evaluated on 500 randomly sampled pairs, exceeding the 200-pair procedure used in prior work.
  • Evaluation Procedure: Repeating the evaluation ten times yields comparatively small differences between groups, supporting reproducibility of the Diversity results.
  • Visualization: Figure 4 shows mosaicked examples from a setting with a very strong prior on hand pose to remove personally identifiable information.
  • Visualization: Table 12 reports the randomness analysis for the Diversity metric.

H. Limitations and Future Work

The work is scoped to upper-body co-speech gestures and remains limited by dataset biases and difficulty modeling very subtle movements. Future work targets full-body generation, minor pose movements, and improved filtering of problematic samples.

  • Limitations: The method generates upper-body gestures only, so it does not model full-body actions such as walking.The authors identify full-body co-speech gesture generation as future work.
  • Limitations: Strong hand-pose priors in some TED Talk samples, including speakers holding objects, can affect training.The authors specifically mention speakers or chalk in their hands as examples of such priors.
  • Limitations: The model has difficulty capturing subtle movements such as shrugs because the dataset contains few examples of these actions.The authors attribute this difficulty to sparse training examples and plan to improve modeling of minor pose movements.
  • Future Work: Future work will improve full-body gesture generation, capture minor pose movements, and filter samples with strong priors through the automatic dataset pipeline.These directions are stated as remedies for the current scope, motion-modeling, and data-quality limitations.
Loading 2203.13161v1…