PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures

Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman, M. Hamza Mughal, Christian Theobalt, Ashwin Ram, Jürgen Steimle, Vera Demberg

arXiv:2605.30608v3cs.CL

TL;DR

The paper addresses the difficulty of aligning spoken language with semantic gestures when direct transcript-motion alignment overemphasizes low-level kinematics. It introduces transcript-grounded semantic motion anchors that describe physical form and communicative intent as auxiliary contrastive supervision, improving retrieval and user preference in downstream generation.

  • Problem

    Direct transcript-motion alignment can overemphasize low-level kinematics and miss the communicative intent of sparse, semantically meaningful gestures.

  • Method

    The method discretizes 3D gestures into motion primitives, verbalizes their physical form, and grounds those descriptions in transcripts to create semantic motion anchors for auxiliary contrastive supervision.

  • Results

    The method outperforms prior approaches across BEAT2 retrieval directions and improves text-to-gesture R@1 from 39.1 to 42.3, an 8.2% relative gain over the direct baseline.

  • Takeaways & Limitations

    Users significantly preferred gestures retrieved by the method in downstream retrieval-augmented generation, indicating better matching of communicative intent in practice.

  • Takeaways & Limitations

    The method models only a subset of gesture attributes, adds offline anchor-generation overhead, relies on a closed-source LLM, and may generalize unevenly across cultures, languages, and demographic groups.

Abstract

from arXiv · show

Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.

1 Introduction

The paper addresses the difficulty of aligning spoken language with semantically meaningful gestures and introduces semantic motion anchors to represent gesture form and communicative intent. Anchor-based supervision improves retrieval performance and supports semantically grounded downstream gesture generation.

  • Semantic gestures are important for communication and co-speech gesture synthesis and understanding, but shared language-motion representations remain difficult to learn.
  • Direct motion-text alignment can emphasize low-level kinematics and frequent beat gestures, underrepresenting sparse semantic gestures that convey communicative intent.
  • Semantic motion anchors describe gesture physical form and communicative intent in structured natural language, reducing sensitivity to irrelevant kinematic variation.
  • The framework discretizes continuous 3D gestures, verbalizes motion primitives into physical-form descriptions, and grounds them in transcripts with an LLM for auxiliary supervision.
  • On BEAT2, text-to-gesture R@1 rises from 39.1 to 42.3, a +3.2 point absolute gain and an 8.2% relative improvement over the direct text-gesture motion baseline.
  • The study reports gains across retrieval directions and significant user preference for retrieved gestures in downstream generation, supporting the method’s semantic retrieval objective.

2 Related Work

Related work shows that existing co-speech gesture systems improve naturalness and alignment but often produce generic, beat-dominated motion. Retrieval and semantics-aware methods address this problem, although transcript-motion alignment remains implicit and can confuse kinematic with communicative similarity.

  • Co-speech gesture research distinguishes semantic representational gestures from frequent beat gestures that primarily mark rhythm.
  • Speech-conditioned generation methods improve naturalness and temporal alignment but often produce generic or beat-dominated gestures.
  • Standard text-to-motion retrieval benchmarks use captions that directly describe performed actions, making language-motion relations relatively literal.
  • Co-speech retrieval is more implicit because transcripts provide discourse context rather than directly describing gestures, so direct alignment can conflate kinematic and communicative similarity.

3 Semantic Motion Anchor Supervision for Text-to-Gesture Retrieval

The method constructs semantic motion anchors by converting gesture motion into structured physical descriptions and grounding them in speech context. These anchors provide modality-matched auxiliary contrastive supervision alongside the primary transcript-motion retrieval objective.

  • Problem formulation: Text-to-gesture retrieval pairs spoken transcripts with 3D gesture sequences and adds a semantic motion anchor during training as auxiliary contrastive supervision.
  • Anchor generation: Anchor generation converts continuous gesture motion into compact natural language through motion tokenization, token verbalization, and transcript-grounded reasoning.
  • Motion tokenization: A two-stream RVQ-VAE compresses upper-body motion into discrete body-hand tokens, with each token representing an 8-frame segment.
  • Token verbalization: Deterministic verbalization maps token geometry to descriptions of observable hand, body, spatial, and kinematic properties, producing a temporal physical-motion narrative.
  • Transcript grounding: An LLM grounds the physical narrative in the transcript through handedness, motion, intent, and verification checks to produce a compact description of form and function.
  • Anchor-supervised contrastive learning: Training uses transcript-motion contrastive learning plus physical-form, communicative-intent, and anchor-bridge losses in a shared retrieval space.
  • Anchor-supervised contrastive learning: Physical-form descriptions supervise motion while intent descriptions supervise transcripts, preserving modality matching without replacing the main retrieval task.

4 Evaluation of Semantic Motion Anchor Generation

The paper evaluates generated semantic motion anchors against human annotations and automated judgments. SEMANTIX supplies human-annotated gesture clips with gold descriptions covering physical form and communicative intent.

  • Evaluation design: The evaluation compares generated anchor quality with human expert annotations and automated assessment.
  • SEMANTIX dataset: SEMANTIX contains 878 human-annotated TED Expressive and BEAT2 semantic gesture clips paired with gold physical-form and communicative-intent descriptions.
  • Annotation schema: Gold descriptions specify handedness, hand shape, hand orientation, spatial position, motion trajectory, and communicative intent.
  • Automated assessment: LLM-based evaluation assigns separate 1–5 Likert scores for physical gesture similarity and communicative intent similarity against gold references.
  • Validation: Agreement between automated evaluation and human judgments is measured with Spearman rank correlation on 100 generated anchors from TED and BEAT2.

5 Text-Gesture Retrieval

The evaluation tests semantic motion anchor supervision through BEAT2 retrieval, ablations, semantic alignment, and cross-dataset transfer. Results show stronger top-rank retrieval, meaningful anchor-content effects, improved semantic matching, and partial transfer through anchor proxies.

  • Main retrieval results: Our method outperforms all baselines on both BEAT2 gesture-to-text and text-to-gesture retrieval directions.Across seven random-seed runs, gains are concentrated at the highest-ranking results.
  • Main retrieval results: 14.2%: gesture-to-text R@1 improves relative to JEGAL, while text-to-gesture R@1 improves by 7.6%.Relative to JEGAL, gesture-to-text MRR also improves by 9.4%, and text-to-gesture MRR by 6.1%.
  • Anchor ablations: Semantic anchor content improves retrieval beyond the auxiliary contrastive structure’s in-domain regularization benefit.Meaningful anchors outperform fixed Gaussian unit-vector targets across R@5 and MRR with p < 0.05.
  • Sensitivity analysis: Peak mean MRR occurs at λp = 0.01–0.05 and moderate λs = 0.10–0.15.Performance degrades as λp increases regardless of λs, whereas the intent branch remains stable across a broad λs range.
  • Semantic alignment: Qualitative examples show that our model retrieves gestures matching the intended semantic label, unlike text-only and random-anchor baselines.The baselines retrieve gestures with mismatched communicative functions in both examples.
  • Cross-dataset generalization: Anchor proxies partially recover TED-to-TED retrieval over the text-contrastive baseline and outperform random anchors in TED-to-BEAT2 transfer.Direct BEAT2 motion embeddings perform near chance on TED-to-TED because of the kinematic domain gap, while meaningful anchor content supports transfer.
  • Downstream application: Users preferred gestures retrieved by the proposed approach over RAG-Gesture in the downstream generation study.The study compared retrieval components within a retrieval-augmented co-speech gesture generation setting.

6 Conclusion

The paper introduces semantic motion anchors for text-gesture retrieval and reports improved retrieval and downstream user preference over a retrieval-augmented baseline.

  • Semantic motion anchors represent gesture motion through physical form and communicative intent.
  • The framework converts 3D motion into upper-body tokens, verbalizes structured physical descriptions, and grounds them in transcripts for auxiliary supervision.
  • Semantic motion anchor supervision improves retrieval over direct text-motion alignment on BEAT2.
  • Random-anchor controls attribute in-domain gains to auxiliary supervision, while their cross-dataset collapse indicates the importance of meaningful anchor content.
  • Users significantly preferred gestures retrieved by this approach over those retrieved by RAG-Gesture.

7 Limitations

The semantic motion anchors model only part of gesture-relevant variation, and the pipeline has computational, licensing, and generalization constraints.

  • Semantic motion anchors do not fully model gesture phases or subtle finger articulation.
  • The pipeline uses a simple contrastive setup, leaving alternative ways to incorporate anchors for future work.
  • Anchor generation adds computational overhead and depends on a closed-source LLM, although generation occurs once offline for training anchors.
  • Because the method is trained primarily on BEAT2 and TED, performance may not generalize equally across cultures, languages, or demographic groups.

A.1 RVQ-VAE Architecture and Training

The RVQ-VAE compresses upper-body motion through separate body and hand streams, reconstructs the skeleton, and balances temporal detail against token compactness and verbalization reliability.

  • Training data: The training corpus combines TED Expressive and BEAT2, covering over 120 hours of co-speech motion tracking data.
  • Architecture: The model separately encodes body motion and hand articulation through independent 1D convolutional streams before shared decoding.
  • Architecture: Each stream uses three-stage residual quantization, with codebook sizes (128, 128, 128) for body and (128, 64, 32) for hands.
  • Architecture: The RVQ-VAE processes 8-frame snippets of 38 upper-body joints and splits them into body and hand streams for independent encoding.
  • Training: Codebook entries with EMA counts below 1.0 are replaced with current-batch encoder outputs to prevent inactive codes from remaining unused.
  • Temporal compression: A downsampling factor of 8 trades reconstruction fidelity and compact token sequences against verbalization reliability.

A.2 Rule-based Motion Primitive Verbalization

Motion tokens are reconstructed and converted into interpretable physical attributes through deterministic rules, producing lookup dictionaries for token combinations.

  • Each motion token is defined by residual codebook indices from the body and hand streams.
  • The method reconstructs each token’s 8-frame skeleton sequence before extracting interpretable physical attributes.
  • Deterministic geometric rules derive physical descriptions across body-hand token combinations and organize them into lookup dictionaries.

A.2.1 Body-stream Attributes

The body stream verbalizes each primitive through body-normalized wrist and arm attributes, including spatial placement, elbow bend, reach, and motion direction.

  • A.2.1 Body-stream Attributes: Body-stream attributes classify each wrist by vertical level, depth, horizontal placement, elbow bend, reach, and motion direction.These attributes are extracted independently for the body stream and converted into structured descriptions.
  • A.2.1 Body-stream Attributes: Wrist placement is described relative to the body using vertical levels, body-centered horizontal categories, and front-versus-behind torso depth.Horizontal labels include crossed-inward, extended-outward, torso-side, and body-centre; depth distinguishes in-front-of-torso from behind-torso.
  • A.2.1 Body-stream Attributes: Elbow bend is categorized from the computed elbow angle as sharply-bent, bent, slightly-bent, or straight.The categories use thresholds below 45°, 90°, and 135°, with angles at least 135° labeled straight.
  • A.2.1 Body-stream Attributes: Arm reach is the wrist-to-shoulder distance divided by total arm length, with labels near-body, mid-reach, or extended.The thresholds are ρ < 0.4, 0.4 ≤ ρ ≤ 0.7, and ρ > 0.7, respectively.
  • A.2.1 Body-stream Attributes: Arm motion is detected over an 8-frame window and labeled held or decomposed into vertical, horizontal, and depth directions.When multiple axes are active, their direction labels are concatenated.

A.2.2 Hand-stream Attributes

The hand stream extracts palm orientation and hand shape, verbalizes body-hand primitives over time, and uses transcript context to infer communicative intent in semantic motion anchors.

  • A.2.2 Hand-stream Attributes: Hand primitives encode palm orientation and hand shape alongside the body-stream descriptions.Palm orientation is computed from palm-plane geometry and mapped to speaker-relative labels; hand shape is classified from finger curl.
  • A.2.2 Hand-stream Attributes: Hand shape classification first detects index-pointing, then assigns open-flat, open-relaxed, curled, or fist labels from mean finger curl.Index-pointing requires low index curl and higher curl in the other three fingers.
  • A.2.2 Hand-stream Attributes: Deterministic templates map extracted attributes into body or hand descriptions, which are concatenated chronologically across 8-frame primitives.A 24-frame gesture yields three verbalizations for each stream, and matching bilateral descriptions may be collapsed into one bimanual description.
  • A.2.2 Hand-stream Attributes: The semantic description combines physical motion with communicative intent inferred from the gesture’s transcript context.Speech triggers include quantity words, deixis, contrastive language, emphatic short phrases, and references to a singled-out entity.
  • A.2.2 Hand-stream Attributes: The prompt’s structured reasoning variant gives the best overall pose score while intent scores remain high across prompt variants.The authors therefore use the structured reasoning prompt for downstream experiments.
  • A.2.2 Hand-stream Attributes: The retrieval evaluation compares semantic-anchor supervision against transcript-motion baselines and controls under matched training and evaluation settings.Figure 6 reports a consistently higher cumulative top-k retrieval fraction for the semantic-anchored model, especially at low ranks.
Loading 2605.30608v3…