PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak

arXiv:2610.11669v1cs.CV

TL;DR

Holistic co-speech gesture evaluation still lacks reliable alignment between objective metrics and human perception, especially for semantic appropriateness. The paper benchmarks models under a shared protocol, validates metrics against human judgments, and finds that composites improve alignment across five dimensions while SGP is selectively aligned with speech-aware judgments.

  • Problem

    Objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify in holistic co-speech gesture generation.

    Source: §1 · pp. 1–2

  • Method

    The paper standardizes model comparison, analyzes objective metrics against human judgments, and derives target-specific composites alongside SGP from LLM-enriched semantic gesture annotations.

    Source: §1 · p. 2, §4.6.1 · p. 9, §5 · p. 10

  • Results

    Target-specific composites improve alignment with human judgments across all five dimensions, with the largest gains for absence of animation errors and content match; SGP aligns selectively with speech-aware judgments, unlike SC.

    Source: §5 · p. 10

  • Takeaways & Limitations

    No single objective metric consistently reflects every perceptual dimension, while complementary signals add perceptual information beyond the strongest individual metric with target-dependent benefits.

    Source: §5 · p. 10

    SGP detects whether annotated semantic intervals avoid collapsing into generic beat gestures, but does not verify exact intended meaning or gesture category.

    Source: §6 · p. 11

Abstract

from arXiv · show

Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations. Unlike Semantic Score (SC), which shows no significant association with the evaluated perceptual dimensions, SGP is selectively aligned with speech-aware human judgments. We construct five target-specific composite metrics aligned with the subjective dimensions. These composites improve perceptual alignment across all five dimensions, with the largest gains for absence of animation errors and content match, indicating that complementary objective signals can better approximate human judgments than individual metrics alone. Overall, our results show that objective metrics require validation against subjective evaluations.

1. Introduction

Evaluation of holistic co-speech gestures has not kept pace with generation, and existing metrics may not reflect human perception or semantic appropriateness. The paper proposes a unified framework for validating metrics, evaluating semantic gestures, and comparing models under standardized conditions.

  • Current evaluation is fragmented, and objective metrics may miss perceptual quality and semantic mismatches even when gestures are natural and synchronized.Different study settings also make reported performance difficult to compare directly.Source: §1 · pp. 1–2
  • The framework analyzes objective–subjective correlations across five perceptual dimensions and constructs target-specific composite metrics from complementary signals.The dimensions include human-likeness, motion diversity, absence of animation errors, speech timing, and content match.Source: §1 · p. 2
  • The benchmark compares four representative holistic gesture-generation models under a unified dataset, rendering pipeline, and evaluation protocol.This shared setup is intended to support direct and fair performance analysis.Source: §1 · p. 2
  • The authors introduce fine-grained semantic gesture annotations and Semantic Gesture Preservation (SGP), a new objective measure assessed against human perception.The annotation framework uses LLMs and contributes an annotated corpus of semantic gesture descriptions.Source: §1 · p. 2

2. Related Work

Co-speech gesture generation has shifted toward holistic, semantically conditioned motion, while evaluation still faces uncertain metric validity and limited fine-grained semantic assessment. Prior work motivates human-centered validation and multimodal semantic annotation.

2.1. Co-speech gesture generation

  • Generation has shifted from prosody-driven rhythmic motion toward holistic body, hand, and face synthesis conditioned on richer linguistic and semantic information.Recent approaches include separate modeling of semantic gestures, retrieval of relevant motion, and semantic-consistency objectives.Source: §2.1 · p. 2

2.2. Human-Centered Validation of Objective Metrics

  • Human judgments remain sensitive to study design, while objective metrics offer reproducibility but have uncertain perceptual correspondence; combined metrics are proposed as an alternative.For co-speech gestures, combined metrics for holistic and semantics-aware methods had not yet been explored in the cited discussion.Source: §2.2 · p. 3

2.3. Assessing Semantic Grounding in Co-Speech Gestures

  • Existing semantic measures [LZI∗22, AZL23, LGOY25, PLÖ∗26, LYS∗23] only partly capture gesture presence, conveyed meaning, and consistency with speech.The paper addresses limited scalable annotation by using multimodal models for fine-grained descriptions and deriving a metric from micro-level annotations.Source: §2.3 · p. 3

3. Methodology

The methodology curates objective motion metrics, introduces a metric based on fine-grained semantic annotations, and evaluates holistic gesture models against human judgments. It also analyzes metric–judgment correlations to derive composite metrics.

  • The study curates objective metrics covering multiple aspects of motion and adds a semantic metric based on MLLM-derived fine-grained motion annotations.Source: §3 · p. 3
  • A perceptual study extends existing subjective criteria with motion diversity, absence of animation errors, and content match.Source: §3 · p. 3
  • The authors evaluate recent holistic and semantics-aware models, analyze objective–subjective correlations, and derive potential composite metrics.Source: §3 · p. 3

3.1. Benchmark Objective Metrics

The benchmark organizes objective metrics across latent, distributional, and raw-signal levels to measure physical plausibility and communicative quality in co-speech gesture generation.

3.1. Benchmark Objective Metrics

  • The taxonomy spans distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness.The metrics operate at latent, distributional, and raw-signal levels.Source: §3.1 · p. 3

3.1.1. Latent-space Distribution-Based Metrics

  • Fréchet Gesture Distance measures real–generated motion-distribution discrepancy in learned latent space, while Density and Coverage assess local reference-manifold alignment and support coverage.FGD depends on its learned feature encoder and does not directly capture speech–gesture semantics.Source: §3.1.1 · p. 3, §3.1.1.2 · pp. 3–4

3.1.2. Geometric Fidelity Metrics

  • Chamfer Distance measures average surface-geometry similarity, Hausdorff Distance captures the largest surface deviation, and Mean Joint Distance measures aligned body-and-hand pose error.Dice Coefficient measures frontal-plane trajectory overlap but discards depth and temporal ordering.Source: §3.1.2.1 · p. 4, §3.1.2.2 · p. 4, §3.1.2.3 · p. 4

3.1.3. Motion Quality and Physical Plausibility Metrics

  • Foot Contact compares stable ground-contact rates, while relative Log Dimensionless Jerk compares generated and reference smoothness, with values nearer zero indicating greater similarity.Foot Contact reflects lower-body stability and artifacts such as foot sliding rather than speech-related gesture quality.Source: §3.1.3 · p. 4, §3.1.3.2 · p. 4

3.1.4.1. Beat Consistency

  • Beat Consistency measures gesture–speech beat synchronization, indicating when gestures occur but not whether they match spoken content.Higher values indicate stronger temporal correspondence between gesture and acoustic beats.Source: §3.1.4.1 · pp. 4–5

3.1.5. Semantic Appropriateness Metrics

  • SRGR weights pose agreement by semantic relevance but may penalize valid alternative gestures, whereas Semantic Score uses learned embedding similarity and depends on embedding quality.SGP instead assesses preservation of semantic gesturing at annotated intervals without requiring exact reference-pose reproduction; it specifically tracks collapse into beat gestures, not exact meaning or gesture type.Source: §3.1.5.1 · p. 5, §3.1.5.2 · p. 5, §3.1.5.3 · p. 5

3.2. LLM-Based Semantic Gesture Annotation

The authors enrich BEAT2’s semantically relevant gesture intervals with structured multimodal annotations to support semantic evaluation. The annotations distinguish observable gesture form from contextual interpretation and are checked for consistency and interpretability.

3.2. LLM-Based Semantic Gesture Annotation

  • Gemini 2.5 Pro annotates each predefined semantic interval using its rendered motion segment and target word or expression, while analyzing only that interval and grounding output in visible motion.The existing BEAT2 annotations provide intervals and coarse gesture categories but limited information about physical realization or communicative meaning.Source: §3.2 · p. 5
  • Each annotation captures gesture form, contextual meaning, gesture type, and confidence, separating visible performance from interpretation.Gesture types are iconic, metaphoric, deictic, or beat; for “had,” an example describes a downward right-hand motion interpreted as possession and classified as iconic.Source: §3.2 · p. 6
  • A subset is manually assessed for gesture-type agreement, contextual-meaning correctness, and gesture-form description quality, treating Gemini outputs as annotations to validate rather than semantic ground truth.The enriched annotations are used in the Semantic Gesture Preservation metric.Source: §3.2 · p. 6

3.3. Benchmark Models

The benchmark surveys holistic co-speech gesture models spanning different methodological paradigms in a unified full-body motion space; architectural details are deferred to the original papers.

3.3.1. EMAGE

  • EMAGE jointly synthesizes facial, upper- and lower-body, hand, and global translation motion using masked reconstruction, region-specific VQ-VAEs, and audio-conditioned generation.Content Rhythm Attention balances rhythmic speech cues with transcript semantics, and the model supports partially predefined spatial-temporal gesture constraints.Source: §3.3.1 · p. 6

3.3.2. GestureLSM

  • GestureLSM uses flow matching and spatial-temporal attention to model interactions across body regions, conditioning on acoustic and transcript features.A latent shortcut model and beta-distributed timestep sampling reduce inference steps while maintaining high-quality, speech-coordinated holistic motion generation.Source: §3.3.2 · p. 6

3.3.3. SemTalk

  • SemTalk separates rhythm-related base motion from sparse semantic motion, then uses frame-level semantic scores to selectively activate and fuse semantic motion.Its hierarchical coarse-to-fine attention propagates speech-driven cues across body regions, while separate region representations preserve their dynamics.Source: §3.3.3 · p. 6

3.3.4. SemConFlow

  • SemConFlow combines region-specific motion priors with multimodal semantic alignment and contrastive flow matching to favor semantically congruent speech–motion trajectories.Its experiments on BEAT2 and SHOW demonstrate strong performance in motion realism, diversity, synchronization, and semantic consistency.Source: §3.3.4 · p. 6

3.4. Perception Study Design

The perceptual study separates visual motion quality from speech-dependent appropriateness using muted and audio conditions. It evaluates four models and ground truth on semantically rich clips, with 101 valid participants.

3.4. Perception Study Design

  • The muted condition rated human-likeness, motion diversity, and absence of animation errors; the audio condition rated speech timing and content match on seven-point Likert scales.The conditions distinguish visual motion quality from speech-dependent appropriateness.Source: §3.4 · p. 7
  • 101 valid participants rated the study after quality control: 49 in the muted condition and 52 in the audio condition.The study recruited 103 participants, excluding two muted-condition participants after quality control.Source: §3.4 · p. 7
  • The study compared EMAGE, SemConFlow, SemTalk, and GestureLSM with BEAT2 ground truth using matched 10-second clips and a shared rendering setup.Stimuli were selected for high occurrence of iconic, metaphoric, and deictic gestures, comprising 80 semantic events.Source: §3.4 · p. 7
  • The study produced 75 source-sequence pairs and 60 generated-motion observations for objective–subjective correlation analysis, using mean ratings as subjective targets.The 75 observations include five motion sources; ground-truth observations are excluded from the 60 generated-motion observations.Source: §3.4 · p. 7

4. Experiments

The experiments compare holistic gesture-generation methods under shared evaluation conditions, then test how objective metrics and target-specific composites align with human judgments. Results show that model rankings and metric–human associations vary across perceptual dimensions.

4.1. Datasets / 4.2. Implementation Detail

  • BEAT2-Standard contains 27 hours; benchmark methods share a training-data regime and identical test sequences, representation conversion, post-processing, rendering, and metric computation.BEAT2-Standard uses the official 85%/7.5%/7.5% train/validation/test split across 25 speakers.Source: §4.1 · p. 7, §4.2 · p. 7

4.3. Objective Evaluation

  • SemConFlow leads most distributional, geometric, and speech-alignment metrics, but rankings vary across metric families; EMAGE leads Density, Coverage, relative LDLJ, and SGP.SemTalk leads Foot Contact, and no single method dominates every metric family.Source: §4.3 · pp. 7–8

4.4.1. Main Analysis.

  • BEAT2 has the highest mean ratings across all five perceptual dimensions, while SemConFlow leads generated systems on all five; visual and speech-conditioned rankings differ.Ratings came from 49 muted-condition and 52 audio-condition participants, aggregated over 15 unique sequences.Source: Figure 2 · p. 8, §4.4.1 · p. 8

4.4.2. Relationship among perceptual dimensions.

  • Across generated motions, all five ratings correlate positively (ρ = 0.595–0.974), with speech timing and content match most tightly linked (ρ = 0.974), yet the dimensions remain non-interchangeable.Human-likeness and absence of animation errors are also strongly associated (ρ = 0.928).Source: §4.4.2 · p. 8, Table 2 · p. 9

4.5. Objective–Subjective Correlation Analysis

  • Metric–human alignment is target-specific: Coverage associates with speech timing and content match, whereas SC is nonsignificant and SGP selectively tracks speech-aware judgments.LDLJREL best correlates with the muted human-likeness and animation-error dimensions; BC is negatively correlated with all five perceptual dimensions.Source: §4.5 · pp. 8–9, Figure 3 · p. 9

4.6.1. Learning Target-Specific Perceptual Composites

  • Five target-specific OLS predictors use standardized objective metrics, exclude Chamfer for redundancy with Hausdorff, retain SC, and use LOOCV over 60 generated observations.Each predictor targets one of the five perceptual dimensions.Source: §4.6.1 · pp. 9–10

4.6.2. Cross-Validated Perceptual Alignment

  • All five composites outperform their strongest individual metric; correlations span 0.5375 for motion diversity to 0.7824 for content match, with largest gains for animation errors (+0.0819) and content match (+0.0805).Gains are target-dependent: motion diversity and speech timing improve by +0.0117 and +0.0246, respectively.Source: Table 3 · p. 10, §4.6.2 · p. 10

5. Conclusions

The benchmark combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. Its results show that objective metrics vary in perceptual relevance, while target-specific composites and SGP provide more selective alignment with human judgments.

5. Conclusions

  • Retraining recent methods under a shared BEAT2 protocol and unified rendering and evaluation pipeline enables more controlled, directly comparable assessment across systems.The benchmark combines standardized comparison with human-centered metric validation and fine-grained semantic evaluation.Source: §5 · p. 10
  • Target-specific composites improve alignment with human judgments across all five dimensions, with the largest gains for absence of animation errors and content match.The benefit of combining metrics remains target-dependent.Source: §5 · p. 10
  • Unlike Semantic Score (SC), which has no significant association with evaluated perceptual dimensions, Semantic Gesture Preservation (SGP) aligns selectively with speech-aware judgments.SGP measures whether annotated semantic intervals avoid collapsing into generic beat-like motion, not whether exact intended meaning is preserved.Source: §5 · p. 10

6. Limitations and Discussion

The evaluation is limited by its data structure and selected study windows, and its semantic annotations and metric do not establish exact meaning preservation. The reported objective–subjective relationships are specific to the evaluated models, sequences, and metric implementations.

6. Limitations and Discussion

  • The perceptual analysis covers 15 unique speech sequences and four systems, producing 60 generated model-sequence observations rather than 60 fully independent samples.Study windows were enriched for semantic gestures, improving sensitivity to semantic evaluation but potentially overrepresenting semantically dense speech relative to broader BEAT2.Source: §6 · p. 11
  • Gemini 2.5 Pro annotations are scalable semantic supervision, not semantic ground truth, and SGP does not verify exact intended meaning or gesture category.Objective–subjective relationships are specific to the evaluated models, sequences, and metric implementations; broader datasets, models, and studies are needed.Source: §6 · p. 11

7. Appendix

The two perceptual studies share an interaction flow but assess different dimensions: the muted condition focuses on visual motion quality, while the audio condition assesses speech-dependent properties.

7.1. User Study Interface and Instructions

  • In both studies, participants watch each clip in full before ratings become available, and attention checks are distributed throughout.Muted checks use instructed responses; audio checks ask about the preceding clip’s content.Source: §7.1 · p. 13, Figure 4 · p. 13, Figure 6 · p. 13
  • In the muted condition, participants rate human-likeness, motion diversity, and absence of animation errors based only on visual motion.Each criterion uses a seven-point scale, from very poor to excellent.Source: §7.1 · p. 13
  • In the audio condition, participants view animation with speech and separately rate speech timing and content match.The instructions distinguish temporal coordination from semantic correspondence; ratings use a seven-point scale.Source: §7.1 · p. 13, Figure 5 · p. 13

8. LLM-Based Semantic Gesture Annotation

The annotation procedure turns predefined semantic intervals into structured descriptions of visible gesture form and inferred communicative meaning, then checks the generated annotations. A manual validation study judged 75% of sampled annotations correct.

8.1. Annotation Procedure

  • The procedure segments semantic-word intervals, pairs each clip with its target word, prompts Gemini to describe gesture form and meaning, assigns a gesture class, and checks the output.The requested form description covers handedness, hand shape, orientation, spatial location, and movement trajectory; classes are iconic, metaphoric, deictic, or beat.Source: §8.1 · p. 14
  • Each interval’s annotation records its target word, fine-grained gesture description, inferred contextual meaning, and gesture class, separating observable form from inferred function.The tuple is represented as ai = (wi,di,mi,ci).Source: §8.1 · p. 14

8.2. Annotation Example

  • A representative event-level example uses a rendered motion clip of approximately 2–3 seconds centered on the target semantic interval, paired with its target word and structured annotation prompt, to produce structured semantic annotation.Source: §8.2 · p. 14, Figure 8 · p. 14

8.3. Human Validation

  • Manual validation compared sampled clips with annotations, checking five visible-motion components and whether class and contextual meaning fit the observed gesture and expression.An annotation counted as correct when it captured the principal visible motion and its semantic interpretation was compatible with the assessed communicative function.Source: §8.3 · p. 14
  • 75% of evaluated LLM-generated annotations were judged correct, supporting their use as scalable semantic supervision while motivating human verification for ambiguous instances.The validation assessed both visual description and semantic interpretation.Source: §8.3 · p. 14

Related papers

Loading 2610.11669v1…