PaperlayerView exact revision on arXiv

Source-linked AI summary

When Semantics Matter: Reliability-Aware Semantic-Rhythm Control for Co-Speech Gesture Generation

Zhirui Xing, Long Ye, Kaige Li, Ziyi Xu, Ming Meng

arXiv:2609.36685v1cs.CV

TL;DR

Co-speech gesture systems can follow speech rhythm while underusing textual semantics, especially when cues are incomplete or noisy. This paper estimates text's predictive contribution and adapts semantic-versus-rhythm control, while conditioning generation on acoustic reliability. It reports best results across reported metrics on BEAT and TED-Expressive.

Abstract

from arXiv · show

Co-speech gesture generation aims to synthesize natural gestures that are both temporally synchronized with speech and semantically consistent with the spoken content. Although recent methods can generate rhythmically plausible motions, they often rely heavily on acoustic prosody while underutilizing textual semantics, especially when semantic annotations are incomplete, noisy, or unavailable. Consequently, the generated gestures may follow speech rhythm while failing to express the intended semantics. To address this problem, we propose a reliability-aware semantic-rhythm control framework for co-speech gesture generation. We first learn a discrete motion prior that represents continuous gestures in a compact and structured motion-code space. We then introduce a dual-branch semantic contribution estimation mechanism consisting of a full multimodal branch and an audio-only branch. Their distributional discrepancy is formulated as conditional information gain to quantify how much textual semantics changes the predicted motion. Based on this estimate, a controllable semantic-rhythm objective selectively strengthens semantic guidance in content-relevant segments while limiting unnecessary semantic intervention in rhythm-dominant segments. Furthermore, we treat background noise as an acoustic reliability condition and introduce noise-conditioned feature modulation together with beneficial latent perturbation to improve generation robustness under realistic acoustic environments. Experiments on benchmark datasets demonstrate that the proposed framework achieves a favorable balance among semantic expressiveness, rhythmic synchronization, motion diversity, and robustness, enabling reliable and controllable co-speech gesture generation.

Introduction

Existing systems can capture speech rhythm yet underuse textual semantics, while uniform text–audio fusion may inject irrelevant cues. The proposed framework measures semantic contribution and acoustic reliability to balance semantic expression with rhythmic synchronization.

  • Acoustic prosody supports rhythmic gestures, but it is less effective for semantically grounded iconic, metaphoric, and deictic gestures.Uniformly fusing text and audio can introduce irrelevant semantic information and weaken useful rhythmic cues.Source: Introduction · p. 2
  • The framework combines a discrete motion prior, multimodal and audio-only prediction branches, conditional information gain, semantic–rhythm control, and acoustic reliability conditioning.The full branch uses audio, text, and speaker identity; the audio-only branch excludes text.Source: Introduction · p. 2
  • Conditional information gain estimates the predictive contribution of textual semantics and guides stronger semantic expression in content-relevant segments while limiting intervention in rhythm-dominant segments.Source: Introduction · p. 2
  • Acoustic reliability modulation and beneficial latent perturbation are introduced to improve robustness under noisy speech.Source: Introduction · p. 2

Related Work

Speech-driven gesture methods have advanced in motion realism, diversity, and rhythmic alignment, while semantic-aware methods add language and other high-level controls. The reviewed approaches generally do not measure text's predictive utility beyond acoustic prosody or distinguish semantic-dominant from rhythm-dominant segments.

Speech-Driven Gesture Generation

  • Speech-driven methods progressed from recurrent, adversarial, flow-based, and variational models to hierarchical, diffusion, discrete-motion, flow-matching, and causal-token approaches.Despite gains in realism, diversity, and rhythmic alignment, these methods generally do not quantify textual semantics' contribution to each prediction.Source: Speech-Driven Gesture Generation · p. 2

Semantic-Aware Gesture Generation

  • Semantic-aware methods use retrieval, coherence constraints, diffusion, language models, motion examples, and gesture captions to improve semantic grounding or control.Examples include LivelySpeaker, SemGes, GestureHYDRA, MECo, and CoordSpeaker.Source: Semantic-Aware Gesture Generation · pp. 2–3
  • Most semantic-aware methods do not measure whether text adds predictive utility beyond acoustic prosody or distinguish semantic-dominant from rhythm-dominant segments.Source: Semantic-Aware Gesture Generation · p. 3

Method

The framework predicts structured motion codes and selectively balances semantic guidance with acoustic rhythm. It also conditions generation on acoustic reliability and uses beneficial latent perturbations for noisy speech.

Problem Formulation and Framework Overview

  • The framework determines when textual semantics should influence gestures and when acoustic rhythm should remain dominant.Its components are a discrete motion prior, dual-branch generator, conditional semantic information gain with controllable optimization, and acoustic reliability conditioning with latent perturbation.Source: Problem Formulation and Framework… · p. 3

Discrete Motion Prior Learning

  • A discrete motion prior maps continuous gestures into reusable motion codes, letting the generator predict structured tokens instead of unconstrained frame-level joint coordinates.The prior is trained with pose reconstruction, velocity consistency, codebook updating, and commitment regularization.Source: Discrete Motion Prior Learning · p. 3

Dual-Branch Multimodal Motion-Code Generation

  • The full branch uses audio, text, and speaker identity, while the audio-only branch omits text; both predict distributions over the same frozen motion codebook.The branches share the rhythm encoder and prediction space, making their discrepancy primarily reflect textual semantics.Source: Dual-Branch Multimodal Motion-Code… · pp. 3–4
  • The framework quantifies textual contribution through discrepancy between full and audio-only predictive distributions, then estimates segment-level semantic relevance.The relevance estimator can use segment annotations or weak targets from text saliency and cross-modal consistency.Source: Conditional Semantic Information Gain… · p. 4
  • A margin-based objective enforces minimum semantic contribution in relevant segments while limiting unnecessary semantic intervention in rhythm-dominant segments.The margins provide training-time control over semantic expressiveness and acoustic-rhythm preservation.Source: Conditional Semantic Information Gain… · p. 5
  • Noise-conditioned modulation adapts generator representations to estimated acoustic reliability, while a condition-dependent perturbation module adds stochastic latent variation.A benefit-ranking constraint retains perturbations that improve decoded reconstruction, with a magnitude regularizer limiting excessively large deviations.Source: Acoustic Reliability Conditioning and… · p. 5

Datasets and Implementation Details

Experiments use BEAT and TED-Expressive, with dataset-specific splits and aligned 34-frame clips. The section also specifies training settings and evaluation metrics.

Datasets and Implementation Details

  • BEAT and TED-Expressive are split into training, validation, and test sets at ratios of 19:2:2 and 8:1:1, respectively.Motion is segmented into 34-frame clips, root-normalized, and aligned with audio and text.Source: Datasets and Implementation Details · p. 5
  • The model is trained for 300 epochs on a single NVIDIA GPU using AdamW with batch size 64, learning rate 2 × 10−4, and weight decay 1 × 10−4.Training uses linear warm-up, cosine annealing, and gradient clipping at a maximum norm of 1.0.Source: Datasets and Implementation Details · pp. 5–6
  • Generation quality is evaluated using FGD, Beat Consistency, Diversity, and Semantic-Relevant Gesture Recall.Lower FGD is better; higher BC, Diversity, and SRGR indicate stronger rhythmic alignment, motion variation, and semantic relevance.Source: Datasets and Implementation Details · p. 6

Comparison with State-of-the-Art Methods

The proposed method ranks first across reported metrics on BEAT and TED-Expressive. Its results span motion realism, synchronization, semantic relevance, and diversity.

Comparison with State-of-the-Art Methods

  • The method ranks first on all reported metrics across BEAT and TED-Expressive, balancing realism, synchronization, semantic relevance, and diversity.The comparison covers BEAT and TED-Expressive; TED-Expressive results are reported as generalizing across in-the-wild conditions.Source: Comparison with State-of-the-Art Methods · p. 6
  • On BEAT, FGD is 4.362, BC is 0.805, Diversity is 322.455, and SRGR is 0.261; versus MIBURI, FGD falls 0.085 and the other metrics rise 0.002, 0.721, and 0.001.The reported gains correspond to FGD, BC, Diversity, and SRGR, respectively.Source: Comparison with State-of-the-Art Methods · p. 6
  • On TED-Expressive, FGD is 4.318, BC is 0.775, and Diversity is 153.427; versus MIBURI, FGD falls 0.088 while BC and Diversity rise 0.006 and 1.111.The method ranks first on all three reported TED-Expressive metrics.Source: Comparison with State-of-the-Art Methods · p. 6

Qualitative Comparison

Qualitative examples compare generated gestures across four speaking dynamics. The method varies motion amplitude, emphasis, posture, and trajectory with the depicted dynamics.

Qualitative Comparison

  • For gradually calming speech, the method reduces motion amplitude and guides the hands toward a stable pose, unlike GlobalDiff’s persistently pronounced motion.The comparison is illustrated with three temporally ordered poses and dashed arrows marking hand trajectories.Source: Qualitative Comparison · p. 6
  • For increasingly impassioned speech, the method shows clearer upper-body expansion and stronger hand emphasis than GlobalDiff.The examples present three ordered poses to show gesture evolution.Source: Qualitative Comparison · p. 6
  • For consistently calm speech, the method maintains restrained gestures and stable posture; under intense speech, it sustains intensity with a more coherent directional trajectory than MIBURI.MIBURI shows noticeable variations in calm speech and more dispersed pose transitions under intense speech.Source: Qualitative Comparison · p. 6

Ablation Studies

The ablations evaluate semantic contribution modeling, acoustic reliability modeling, and qualitative behavior across acoustic conditions. Results support selective semantic guidance and show that combining reliability components improves robustness under noise.

Semantic Contribution Modeling

  • Across four speaking dynamics, the method better adapts gesture amplitude, direction, and temporal evolution to changing vocal intensity than GlobalDiff and MIBURI.The comparison overlays three temporally ordered poses per example and uses dashed arrows for dominant hand trajectories.Source: Figure 4 · p. 7
  • With all components enabled, the full model achieves the best reported FGD, BC, Diversity, and SRGR scores: 4.362, 0.805, 322.455, and 0.261, respectively.Source: Semantic Contribution Modeling · p. 7

Acoustic Reliability Modeling

  • At 0 dB SNR, the full model reduces FGD from 6.736 to 5.742 and improves BC from 0.672 to 0.731 and SRGR from 0.194 to 0.224.AdaLN and latent perturbation independently improve robustness, their combination performs best, and AdaLN yields slightly stronger gains under both tested noise levels.Source: Acoustic Reliability Modeling · p. 7

Clean 10 dB 0 dB

  • Under corrupted acoustic inputs, reliability-aware modeling prevents abrupt gesture-amplitude changes and maintains stable upper-body motion, supporting coherent transitions and speech-rhythm synchronization.Figure 5 presents qualitative comparisons under different acoustic conditions.Source: Acoustic Reliability Modeling · p. 8, Figure 5 · p. 8

Conclusion

The framework estimates textual semantics’ contribution to gesture generation and balances semantic guidance with acoustic rhythm. It also models acoustic reliability, and experiments on BEAT and TED-Expressive show improvements in motion quality, semantic relevance, and diversity.

Conclusion

  • The framework estimates textual information’s contribution with a dual-branch conditional information gain mechanism, balances semantic expressiveness with rhythmic synchronization, and uses acoustic reliability modulation and stochastic latent perturbation to stabilize generation under corrupted audio.Experiments on BEAT and TED-Expressive demonstrate consistent improvements in motion quality, semantic relevance, and generation diversity.Source: Conclusion · p. 8

Related papers

Loading 2609.36685v1…