PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark for Spatially Grounded Gesture Generation

Anna Deichler, Rishabh Dabral, Fethiye Irmak Dogan, Anindita Ghosh, Jonas Beskow

arXiv:2610.03105v1cs.CVcs.GRcs.HC

TL;DR

Generated gestures can look natural while indicating the wrong object, and existing evaluation lacks a common framework for measuring referential accuracy. The paper introduces a benchmark and protocol that separate temporal alignment, spatial grounding, and perceived naturalness; its results show that stronger geometric grounding need not bring a naturalness advantage.

  • Problem

    Co-speech gesture evaluation lacks a common data, task, and metric framework for measuring whether generated gestures indicate their intended 3D referents.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces pointing-annotated naturalistic dialogue data, a target-directed generation task, a flow-matching baseline, and a protocol that separately evaluates temporal alignment, spatial grounding, and perceived naturalness.

    Source: §1 · p. 2, §4.2 · p. 3, §9 · p. 9

  • Results

    RePointer exceeds captured human motion on spatial grounding without being rated more natural, and no single objective score captures referential gesture quality.

    Source: §1 · p. 2, §9 · p. 9, §8 · p. 9

  • Takeaways & Limitations

    Geometric precision, timing, and perceived motion quality are distinct dimensions of referential gesture quality and should be reported separately.

    Source: §9 · p. 9, §8 · p. 9

    The benchmark’s naturalistic data come from five rooms of one VR corpus, with a single held-out room supplying all 100 test clips.

    Source: §7 · p. 8

Abstract

from arXiv · show

Communication in shared space interweaves verbal and non-verbal signals, and pointing gestures anchor language to the environment: "put the cup on that one" is uninterpretable without the gesture that fixes the referent. Yet no common framework exists for evaluating whether generated gestures indicate their intended referent; distributional metrics reward a gesture aimed at the wrong object as long as it looks natural. We introduce a benchmark for spatially grounded gesture generation, comprising ~2K pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents, a task in which systems must decide when, how and where to point within conversational speech, and a protocol that separates temporal alignment, spatial grounding and perceived naturalness. We also provide a flow-matching baseline, MM-Conv-Flow. Evaluating it alongside an independent retrieval-based system and captured human motion, we find that geometric grounding can exceed that of human pointing without any gain in perceived naturalness, showing that referential gesture quality must be measured along separate dimensions.

1. Introduction

Referential gesture generation needs to indicate the intended object, a capability that distributional motion metrics do not measure. The paper introduces a benchmark, task, and evaluation protocol, then shows that geometric grounding and perceived naturalness can diverge.

1. Introduction

  • A gesture can look natural and synchronize with speech yet point at the wrong object, so distributional metrics alone miss referential accuracy.The missing framework comprises data, a task formulation, and metrics for spatial grounding.Source: §1 · p. 2
  • The benchmark pairs approximately 2K naturalistic dialogue clips with 3D referents and asks systems to decide when and where to point during continuous speech.The task withholds the reference motion and apex frame at inference.Source: §1 · p. 2
  • Its protocol separates temporal alignment and spatial accuracy, combines them as STG, and rates rendered motion naturalness in a crowdsourced study.This makes timing, pointing accuracy, and perceived naturalness distinct evaluation dimensions.Source: §1 · p. 2
  • MM-Conv-Flow is a flow-matching baseline adapted to the benchmark with target-conditioned residual branches, relational pointing losses, and directional refinement.It is trained on the released training split and evaluated under the released protocol.Source: §1 · p. 2
  • RePointer exceeds captured motion on spatial grounding without a naturalness advantage, while combined grounding comparisons with ground truth depend on temporal-scoring parameters.These results motivate reporting spatial, temporal, and naturalness measures separately.Source: §1 · p. 2

2. Related Work

Prior work addresses motion generation and referential gestures, but datasets and evaluation protocols have made direct comparison difficult. This work builds on naturalistic VR dialogue data and adds pointing annotations and a held-out evaluation split.

2.1. Spatially Grounded Gesture Generation

  • Earlier co-speech models focus on motion distribution and audio alignment, while spatially grounded approaches use differing datasets and protocols that prevent direct comparison.The benchmark supplies a common corpus, task, and evaluation protocol.Source: §2.1 · p. 2

2.2. Referential Gesture Datasets

  • Existing datasets include captured scene-and-motion signals and synthetic pointing, whereas MM-Conv links natural dialogue references to objects in a 3D scene.Nearly half of MM-Conv’s referring expressions are pronominal, making context relevant to reference resolution.Source: §2.2 · pp. 2–3

3. Corpus and Annotations

  • MM-Conv contributes 1,842 pointing clips from five-room VR dialogue, with manually reviewed pointing annotations and dominant-hand labels.A random-forest classifier guided review, while human-assigned gesture tiers determined inclusion.Source: §3 · p. 3
  • SGS-HSI adds 1,138 isolated single-target clips, but its synthesised speech and constructed phrase–gesture alignment do not extend the naturalistic referential domain.Its clips provide clean target geometry and captured motion.Source: §3 · p. 3
  • The fixed 100-item evaluation set is drawn from held-out MM-Conv clips, with speaker-stratified sampling and a room-level split.SGS-HSI is split by source gesture so test clips do not share a source with training.Source: §3 · p. 3, Table 1 · p. 3

4. Benchmark

The benchmark asks systems to generate target-directed pointing motion from speech and scene-related inputs, without supplying the reference apex. Its evaluation separates timing, spatial grounding, and motion quality, alongside a baseline evaluated on the held-out test split.

4.1. Task Formulation

  • Systems receive speech, transcript, target coordinates, dominant hand, body shape, and scene graph, then generate target-directed SMPL-X motion without a supplied apex frame.The task isolates gesture generation from referent resolution because the target coordinate is provided.Source: §4.1 · p. 3

4.2. Evaluation Protocol

  • The protocol scores apex timing and target-directed spatial accuracy separately, multiplies them as STG, and reports motion-quality metrics separately from grounding.The reported motion measures include FGD, average jerk, L1 velocity diversity, and beat alignment; they do not measure whether the intended target is indicated.Source: §4.2 · pp. 3–4
  • The five-room evaluation holds out Room 211, and the 100 test targets vary in distance and apparent angular extent.Each point in Figure 3 represents one test clip.Source: Figure 2 · p. 4, Figure 3 · p. 4
  • Temporal scoring uses a fixed 10-frame tolerance and 15-frame decay, while spatial scoring evaluates a forearm-direction ray against the target’s extent.The spatial score uses the detected apex, and its 20° decay scale is a benchmark design parameter rather than a perceptual threshold.Source: §4.2 · p. 4

4.3. MM-Conv-Flow

  • MM-Conv-Flow is a benchmark reference system trained on the training split and evaluated on the held-out test split for direct protocol-based comparison.Its full hyperparameters are reported in Appendix C.Source: §4.3 · p. 4

Co-Speech Gesture Backbone (Phase 1)

Phase 1 trains a co-speech motion backbone with conditional flow matching; Phase 2 injects target geometry and adds relational pointing supervision.

Co-Speech Gesture Backbone (Phase 1)

  • The 170.5 M-parameter, 12-layer MotionDiT learns 300-dimensional SMPL-X motion from BEAT2 using conditional flow matching and speech-conditioned audio–text features.It uses windowed cross-attention and adaptive temporal attention for audio–gesture synchrony, plus smoothness and bone-length losses.Source: Co-Speech Gesture Backbone (Phase 1) · pp. 4–5

Spatial Conditioning (Phase 2)

  • Phase 2 encodes pelvis-relative target direction and distance with a hand embedding, then adds identity-initialised spatial residual branches to the Phase 1 backbone.Mode A freezes the backbone and adds gated residual bias; Mode B unfreezes it and adds spatial cross-attention, with higher capacity but risk to the co-speech prior if spatial gradients dominate.Source: Spatial Conditioning (Phase 2) · p. 5
  • Spatial supervision combines forearm–target alignment and arm-extension penalties only on target-bearing samples, using a soft-min so the best-pointing frames dominate.The arm need not aim at the target throughout the clip, and no apex annotation is used.Source: Spatial Conditioning (Phase 2) · p. 5
  • Phase 2 trains for 200 epochs with a 70/30 mix of pointing clips and BEAT2, audio dropout, and explicit jerk and velocity smoothness losses.BEAT2 rows have zero spatial weight and bypass spatial conditioning.Source: Spatial Conditioning (Phase 2) · p. 5
  • Table 2 reports development-stage variant comparisons on 455 held-out clips with an earlier temporal scorer, so its temporal and STG values are not comparable to the final Table 3 evaluation.The starred configuration is the one used in the final evaluation.Source: Table 2 · p. 6

Inference-Time Directional Refinement

Inference-time directional refinement applies a partial kinematic correction toward the target; it reduces angular error from 19.5° to 11.0°.

Inference-Time Directional Refinement

  • The correction target preserves the current arm length, while gating uses the model’s own arm extension rather than frame indices.The baseline reconstructs the world-frame target from the estimated pelvis at each step.Source: Inference-Time Directional Refinement · p. 6
  • At each Euler step, refinement predicts a clean endpoint and applies an extension-weighted partial correction along the wrist–elbow–shoulder chain toward the shoulder–target ray.The wrist, elbow, and shoulder receive fixed correction weights of 1.0, 0.5, and 0.25, respectively; the weighted propagation preserves arm length only approximately.Source: Inference-Time Directional Refinement · p. 6
  • Directional refinement reduces angular error from 19.5° to 11.0° beyond the Mode B spatial-conditioning result.The parameters were selected on a validation split drawn from training rooms and fixed before held-out-room evaluation.Source: Inference-Time Directional Refinement · p. 6

5. Evaluation

The evaluation compares MM-Conv-Flow, RePointer, and captured motion on spatial grounding, temporal alignment, motion quality, and perceived naturalness. RePointer leads on geometric grounding, but its advantage does not establish better perceived naturalness, and its combined comparison with captured motion depends on temporal scoring parameters.

5.1. RePointer

  • RePointer retrieves a full-body SMPL-X anchor, adapts it to query timing and target geometry, then selects a candidate using crossfitted scorers.Adaptations include apex-preserving temporal warping, target-ray inverse kinematics, and a short tapered hold; root translation, lower body, and inactive arm remain unchanged.Source: §5.1 · p. 6

5.2.1 Grounding Analysis

  • RePointer scores 0.905 on spatial grounding, above MM-Conv-Flow (0.691) and captured motion (0.767), but this geometric advantage does not establish communicative accuracy.Its advantage over MM-Conv-Flow is +0.214 [95% CI: +0.163, +0.272], and its score above captured motion is +0.140 [95% CI: +0.088, +0.196].Source: §5.2.1 · pp. 6–7
  • RePointer achieves Score𝑎=0.784 versus 0.499 for MM-Conv-Flow; median absolute apex errors are 0.40 s and 0.92 s, respectively.RePointer scores higher across all 806 tested combinations of temporal tolerance and decay parameters; the GT value is treated as a detector/reference diagnostic, not a human timing baseline.Source: §5.2.1 · p. 7, Figure 5 · p. 7
  • Under primary temporal parameters, mean STG is 0.719 for RePointer, 0.720 for GT, and 0.361 for MM-Conv-Flow; RePointer–GT ordering changes with parameter settings.STG is the product of temporal and spatial scores, averaged across clips; the authors therefore draw no conclusion about RePointer relative to captured motion from STG.Source: §5.2.1 · p. 7

5.2.2 Motion Quality.

  • RePointer has the lowest EMAGE FGD (2.364), versus 4.551 for MM-Conv-Flow and 4.738 for held-out GT, but retrieval-based initialization overlaps the FGD reference distribution.RePointer also has higher L1 velocity diversity and beat-alignment scores than both alternatives, but these metrics do not establish greater perceived naturalness.Source: §5.2.2 · p. 7

5.3. Perceptual Evaluation

  • The perceptual study rates naturalness because objective grounding scores do not capture whether generated motion looks human-like.The 23 clips were rendered under GT, MM-Conv-Flow, and RePointer with the same audio, avatar, room, and camera; only body motion varied.Source: §5.3 · p. 7, §5.3.1 · p. 7

5.3.1 Protocol.

  • Participants rated each clip on a five-point naturalness scale after full playback, with ratings gated on at least 85% of the clip being played.Sessions included 23 stimulus trials and two attention checks; the median session lasted 5.5 minutes.Source: §5.3.1 · p. 7

5.3.2 Results.

  • GT motion is rated significantly more natural than both generated systems, while MM-Conv-Flow’s higher MOS than RePointer is not statistically significant.The study reports 62 participants and 1,410 ratings; Figure 6 shows per-clip and overall MOS with 95% confidence intervals.Source: §5.3.2 · p. 7, §5.2.2 · p. 7, Figure 6 · p. 8

6. Validity Analysis

The validity analysis shows that objective grounding measures capture distinct properties and have important interpretive boundaries. Geometric precision is not evidence of communicative success, and aggregate grounding comparisons can depend on temporal metric design.

6. Validity Analysis

  • Score𝑏 can reward evaluator-aligned forearm geometry without establishing communicative usefulness or perceived naturalness.The retrieval-based system can retain semantically unrelated motion while adapting its arm toward the target, so good spatial grounding may coexist with inappropriate residual gesture semantics.Source: §6 · p. 8
  • RePointer’s ordering against GT on combined grounding changes with temporal tolerance and decay parameters, so the aggregate comparison is not robust.RePointer remains above MM-Conv-Flow across tested parameterisations; reporting temporal and spatial scores separately, alongside raw timing error, exposes the source of an STG result.Source: §6 · p. 8

7. Limitations

The benchmark’s scope is bounded by its corpus, task formulation, and evaluation protocol. The authors identify further limits in naturalistic data coverage and in what the perceptual study tests.

7. Limitations

  • The benchmark draws on five rooms from one VR corpus, with all 100 test clips coming from one held-out room.SGS-HSI adds clean single-target pointing, but its synthetic speech and constructed phrase–gesture alignment do not extend the naturalistic referential domain.Source: §7 · p. 8
  • Supplying the target isolates gesture generation from reference resolution, leaving joint language, speech, and motion generation outside the current task.The authors identify joint generation as a possible future benchmark variant.Source: §7 · p. 8
  • Score𝑏 measures target-directed forearm geometry, while sensitivity to elbow and wrist pose-estimation or calibration errors remains unquantified.Score𝑎 also depends on a frozen apex detector and fixed tolerance, which are benchmark design choices rather than perceptual thresholds.Source: §7 · p. 8
  • The MOS study measures naturalness, not whether observers can identify the intended referent from generated motion.The current test set lacks controlled distractor structure, so a forced-choice task could be trivial or ill-posed; a matched-distractor item set is needed.Source: §7 · pp. 8–9

8. Discussion

The retrieval-based system exceeded captured human motion on spatial grounding and distributional metrics without being rated more natural. The discussion argues that grounding, timing, and perceived motion quality require separate measures, while current systems use target coordinates rather than learning end-to-end spatial grounding.

8. Discussion

  • Both evaluated systems use supplied target coordinates, so their grounding scores partly reflect geometric operators rather than solely learned scene-aware gesture generation.A harder benchmark would withhold the target coordinate and require referent inference from the scene and utterance.Source: §8 · p. 9
  • RePointer substantially exceeded MM-Conv-Flow on spatial grounding, but their naturalness ratings did not differ significantly; RePointer's better FGD, velocity diversity, and beat alignment brought no MOS advantage.Source: §8 · p. 9

9. Conclusion

  • The benchmark reports temporal alignment, spatial grounding, and perceived naturalness separately; a distractor-controlled referent-identification study is a next step toward measuring communicative success directly.Source: §9 · p. 9
  • The retrieval-based system exceeded captured human motion on spatial grounding and distributional motion metrics without being rated more natural.Source: §9 · p. 9

A. Temporal-Score Sensitivity

The sensitivity analysis varies temporal-score parameters while holding detector predictions and spatial scores fixed. RePointer's advantage over MM-Conv-Flow persists across the tested grid, whereas absolute scores and the RePointer–GT combined-grounding comparison depend on parameter choice.

A. Temporal-Score Sensitivity

  • The primary temporal-score setting is τ=10 and σ𝑡=15 frames, defining benchmark credit rather than an empirically established perceptual timing threshold.Source: A. Temporal-Score Sensitivity · p. 11

B. Apex Detector

  • The sensitivity sweep tests τ∈{0,…,30} and σ𝑡∈{5,…,30} in one-frame steps, with detected and reference apexes and Score𝑏 fixed.The frozen logistic-regression apex detector uses kinematic features and is trained on recordings disjoint from evaluation; the sweep does not independently validate detection.Source: A. Temporal-Score Sensitivity · p. 11, B. Apex Detector · p. 11

A. Temporal-Score Sensitivity

  • Mean Score𝑎 varied from 0.30 to 0.97 for RePointer and 0.11 to 0.84 for MM-Conv-Flow across the grid, while median apex errors remained 0.40 s and 0.92 s, respectively.The raw median errors are independent of the scoring parameters.Source: A. Temporal-Score Sensitivity · p. 11
  • RePointer scored higher than MM-Conv-Flow under all 806 parameter combinations, with mean Score𝑎 differences of +0.13 to +0.34 and combined STG differences of at least +0.21.The smallest lower 95% confidence bounds were +0.06 for Score𝑎 and +0.14 for STG.Source: A. Temporal-Score Sensitivity · p. 11
  • GT scored higher than RePointer on Score𝑎 throughout the grid, a diagnostic of detector agreement with the kinematically defined reference apex rather than human timing variability or a human-performance ceiling.Source: A. Temporal-Score Sensitivity · p. 11
  • The mean RePointer–GT STG difference changed sign across settings, and its 95% confidence interval included zero for 60% of the grid and every setting near the primary parameters.At σ𝑡=15 frames, the sign changed between τ=10 and τ=11; the primary-setting scores were 0.719 versus 0.720.Source: A. Temporal-Score Sensitivity · p. 11, Figure 8 · p. 11

C. MM-Conv-Flow Reproducibility Details

MM-Conv-Flow separates training into a co-speech motion prior and spatial adaptation, with spatial conditioning injected into selected transformer layers. The appendix documents its architecture, parameter counts, training mixture, and phase-specific losses.

C. MM-Conv-Flow Reproducibility Details

C.1. Architecture Configuration

  • MM-Conv-Flow uses a 12-layer MotionDiT backbone and injects target-conditioned spatial information through gated residual bias in layers 4–11 and, in Mode B, cross-attention in layers 8–11.The spatial projections are zero-initialized; Mode B adds approximately 0.8M parameters, yielding 478.1M total parameters, of which 171.3M are trainable.Source: §C.1 · p. 12, Figure 9 · p. 12

C.2. Training Configuration

  • P1 trains a general co-speech motion prior on BEAT2, while P2 starts from the final P1 checkpoint and adapts on a 70/30 mixture of pointing and BEAT2 samples.The retained BEAT2 component preserves general co-speech motion during spatial adaptation.Source: §C.2 · p. 12
  • P2 uses a relational loss combining alignment and extension terms, while P1 uses no relational or spatial loss; several regularizers are disabled in P2.In P2, Lrel = 1.5Lalign + 0.5Lextend has overall weight 0.5.Source: §C.2 · p. 12

D. Auxiliary Motion Metrics

The appendix defines auxiliary motion metrics for distributional similarity, smoothness, velocity variation, and rhythmic alignment, distinguishing these from referential-apex timing. A supplementary video presents three motion conditions on held-out clips under shared scene and audio conditions.

D. Auxiliary Motion Metrics

  • FGD compares each system's motion-feature distribution with the training-reference distribution, with lower values indicating closer distributional agreement.Source: D. Auxiliary Motion Metrics · p. 12
  • Average jerk characterizes motion smoothness, while L1 velocity diversity measures velocity variation without a preferred direction because variation may reflect richer or excessive motion.Lower average jerk indicates smoother motion, but smoothness alone does not imply naturalness.Source: D. Auxiliary Motion Metrics · p. 12
  • Beat alignment measures motion-beat correspondence with audio beats, whereas Score𝑎 evaluates referential-apex timing.Source: D. Auxiliary Motion Metrics · p. 13

E. Supplementary Video

  • The supplementary video shows GT MoCap, MM-Conv-Flow, and RePointer side by side on five held-out clips, with shared audio, avatar, room, and camera.Only body motion differs across conditions; clip 298 illustrates a retrieval failure mode.Source: E. Supplementary Video · p. 13
Loading 2610.03105v1…