PaperlayerView exact revision on arXiv

Source-linked AI summary

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

Peizhen Li, Longbing Cao, Yang Zhang

arXiv:2609.33095v1cs.RO

TL;DR

Listener motion generation must coordinate speaker-cue timing with ongoing listener motion and represent locally variable facial events alongside overall trajectories. REALM combines delay-aware gated fusion with stochastic expression refinement; it improves over evaluated baselines on ViCo and L2L and is also deployed on Ameca with perceptual evaluation.

  • Problem

    Listener generation must account for lagged speaker cues and ongoing listener motion while capturing local facial variation that deterministic reconstruction can attenuate.

    Source: §1 · pp. 1–2

  • Method

    REALM uses delay-centered attention and adaptive gating to fuse speaker audio with listener history, then adds audio-conditioned stochastic expression residuals while retaining the coarse pose.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    REALM improves over evaluated baselines on ViCo and L2L, achieves the lowest expression and pose L1 errors among evaluated methods, and is most preferred in the robot user study.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Ameca deployment and a perceptual user study demonstrate the applicability of the generated motion to physical embodiment.

    Source: §5 · p. 9

    REALM operates on offline conversational windows; streaming human–robot interaction requires explicit control of temporal information access and end-to-end latency.

    Source: §5 · p. 9

Abstract

from arXiv · show

Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal alignment with stochastic expression refinement. We propose REALM (Reactive Embodied Audio-driven Listening Model), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code: https://github.com/lipzh5/REALM Demo: https://youtu.be/Tf5mpd5S8VQ

1 INTRODUCTION

Listener generation must coordinate lagged speaker cues with the listener’s ongoing motion and represent local facial variation beyond a coarse trajectory. REALM addresses these challenges with delay-aware adaptive fusion and stochastic expression refinement, and is evaluated on conversational benchmarks and an Ameca robot.

1 INTRODUCTION

  • Generation must account for speaker-cue timing and listener motion history because responses can lag cues and speaker activity need not prompt a matching motion change.Recent listener motion provides context for response development, while preceding speaker cues inform timing and form.Source: §1 · p. 1
  • Brief expressions and blinks vary locally alongside overall trajectories, and deterministic reconstruction can attenuate this variation.The proposed refinement models expression variation without directly perturbing coarse pose parameters.Source: §1 · p. 2
  • A delay-centered attention prior and learned gate combine temporally aligned speaker context with listener history for coarse motion prediction.The shifted ALiBi bias favors speaker representations near a nominal response lag while allowing content-dependent alignment.Source: §1 · p. 2
  • The coarse decoder predicts expression and pose trajectories, then audio-conditioned stochastic residuals refine expression while leaving coarse pose unchanged.This separates trajectory reconstruction from local expression variation.Source: §1 · p. 2
  • 8.71 to 3.91: expression FID∆fm on ViCo Dtest; REALM also had the lowest expression and pose L1 errors on ViCo and L2L and was most preferred in the robot user study.The evaluation also deploys generated motion on Ameca and includes a perceptual study of physical behavior.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work conditions listener motion on speaker cues, conversational context, or listener history, while physical embodiment adds a gap between facial-motion representations and robot action constraints. REALM combines delay-aware conditioning and expression refinement, and its Ameca deployment uses robot-specific retargeting.

Listening Head Generation

  • L2L (Ng et al. (2022)) and ARIG (Guo et al. (2025)) use autoregressive prediction, while CustomListener (Liu et al. (2024b)) introduces past-guided generation.Related approaches also include Transformer-based generation, dyadic representation learning, and diffusion-based motion modeling.Source: §2 · p. 3
  • REALM complements prior listener-history approaches with delay-centered speaker alignment, adaptive weighting of speaker evidence and history, and stochastic expression residuals that retain coarse pose parameters.The framework targets response alignment and local facial variation.Source: §2 · p. 3

Embodied Avatars and Human-Robot Interaction

  • Mapping generated facial coefficients to robot actuators must account for the domain gap and mechanical constraints; the Ameca pipeline adds smoothing and calibration.The mapping is calibrated relative to a mechanical neutral configuration.Source: §2 · p. 3

3 METHOD

REALM conditions listener-motion generation on preceding motion and speaker audio, then uses delay-aware fusion and stochastic expression refinement. Its pipeline also retargets the generated motion to a physical robot.

3.1 HISTORY-CONDITIONED GENERATION WITH A DELAY PRIOR

  • Each target listener frame is generated from a speaker-audio window and preceding listener history, using history for continuity and audio for responsive changes.The coarse-to-fine framework pairs this history-aware alignment with a separate stochastic refinement stage for local facial dynamics.Source: §3.1 · p. 3

3.2 REACTIVE GATED SPEAKER–LISTENER FUSION

  • A shifted ALiBi prior favors speaker representations near nominal delay τ, while a learned gate balances aligned speaker evidence against listener history.The prior is soft rather than a minimum latency, and the fusion mask alone does not guarantee end-to-end streaming; gate peaks align closely with reaction onsets empirically, without a rigid causal constraint.Source: §3.2 · pp. 4–5

3.3 COARSE-TO-FINE STOCHASTIC REFINEMENT

  • The refinement pathway uses audio-conditioned stochastic residuals based on coarse expression; residuals alter only expression, leaving the rigid pose unchanged.This parameterization permits multiple expression realizations for the same conditioning input.Source: §3.3 · p. 5

3.4 PHYSICAL GROUNDING AND ROBOTIC EMBODIMENT

  • Robot execution maps generated facial motion into robot controls under hardware constraints, then calibrates relative displacement from mechanical neutral rather than transferring absolute human-derived positions.The mapping applies actuator-range clipping and resolves overlapping control semantics.Source: §3.4 · p. 6

4 EXPERIMENTS

REALM achieves the lowest point-wise L1 errors for expression and pose across ViCo and L2L, and its Ameca deployment is rated highly by evaluators. The experiments also examine delay sensitivity and component ablations, while qualitative comparisons highlight expression dynamics in physical embodiment.

4 EXPERIMENTS

  • The evaluation covers ViCo and L2L, measuring point-wise accuracy, distributional realism, speaker–listener correlation, and motion variability against five baselines.ViCo includes in-domain and out-of-domain test splits.Source: §4 · p. 6

4.1 QUANTITATIVE RESULTS

  • REALM achieves the lowest point-wise L1 errors for expression and pose across both ViCo and L2L.Tables 1 and 2 report quantitative comparisons across the datasets.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 7
  • The ViCo ablation evaluates Gated Fusion, the Refinement Module, and Shifted Attention; pose metrics depend on the coarse stage because refinement changes only expressions.Source: Table 3 · p. 7
  • An 8-frame delay shift (about 267 ms) performs best among tested settings, while a 12-frame shift worsens performance, especially in rPCC.The nominal delay centers the attention prior; it is not a universal physiological reaction time.Source: §4.1 · p. 8, Table 4 · p. 8

4.2 ROBOTIC EMBODIMENT AND USER STUDY

  • In two representative Ameca sequences, REALM recovers contextually appropriate smiles and blinks that ListenFormer often smooths away, resembling human ground truth.The visual comparison uses frames from sequences driven by speaker context.Source: Figure 3 · p. 8, §4.2 · p. 9
  • On Ameca, REALM achieves the highest overall preference, with pairwise score differences from the strongest baseline statistically significant at p < 0.01.The user study involved 25 evaluators; REALM also scored higher in synchrony, while ListenFormer often lacked high-frequency micro-dynamics.Source: §4.2 · p. 9

5 CONCLUSION AND FUTURE WORK

REALM combines delay-centered gated fusion with stochastic expression refinement and improves over evaluated baselines on ViCo and L2L. Ameca deployment and a perceptual study support its applicability to physical embodiment, while streaming use and transfer across robot embodiments remain future work.

5 CONCLUSION AND FUTURE WORK

  • REALM improves over evaluated baselines on ViCo and L2L, and Ameca deployment with a perceptual user study demonstrates applicability to physical embodiment.Source: §5 · p. 9
  • REALM currently uses offline conversational windows; streaming interaction requires explicit temporal-access control and end-to-end latency handling.Source: §5 · p. 9
  • The physical deployment uses robot-specific retargeting, so transfer across embodiments requires morphology-aware mappings and actuator-limit validation on each platform.Source: §5 · p. 9

AI USE STATEMENT

The authors describe using generative AI to polish prose and refine software code, while retaining responsibility for reviewing the resulting work.

AI USE STATEMENT

  • Generative AI tools were used to polish parts of the paper for readability and help refine code by identifying and fixing bugs.Source: AI USE STATEMENT · p. 10
  • The authors state that AI tools were not used to generate synthetic datasets, develop theoretical models, formulate mathematical claims, design experiments, or interpret results.Source: AI USE STATEMENT · p. 10
  • The authors reviewed polished text and manually verified and tested AI-suggested code fixes, taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study uses licensed public datasets and bounded inverse-kinematics control for physical safety, and reports an anonymized, double-blinded evaluation with no personally identifiable information collected.

ETHICS STATEMENT

  • ViCo and L2L are publicly available datasets used under their respective academic licenses.Source: ETHICS STATEMENT · p. 10
  • Physical robot control uses deterministic inverse kinematics with hard-coded mechanical safety bounds rather than unconstrained neural end-to-end control.Source: ETHICS STATEMENT · p. 10
  • The MOS study used double-blinded, anonymized video reviews with 25 participants and collected no personally identifiable information.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement says the source code and training scripts are supplied in the supplementary materials, with implementation details documented in the appendix.

REPRODUCIBILITY STATEMENT

  • The supplementary materials include the REALM source code and training scripts.Appendix B documents the two-stage curriculum loss, core-module hyperparameters, dataset processing, and evaluation-metric definitions; Appendices D and A detail inference-time gating and blink analyses.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix D and Appendix A document inference-time gating data and structural blink analyses, respectively.The statement also says Appendix B contains implementation details.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors state that implementation details are documented in the appendix to keep the conceptual narrative focused.Source: REPRODUCIBILITY STATEMENT · p. 10

A EMPIRICAL ANALYSIS OF THE GATING MECHANISM

The gating analyses relate predicted reactivity to listener motion on the listener timeline, while the appendix specifies the network realization of fusion and stochastic refinement. The reported associations do not establish speaker-to-listener delay or causal effects of the gate.

A EMPIRICAL ANALYSIS OF THE GATING MECHANISM

  • The gating analysis compares gate peaks with velocity-thresholded listener-motion onsets and compares ground-truth velocity in high- versus low-gate frames within each clip.The stated criteria are temporal alignment with physical reaction onsets and statistical correlation with physical motion velocity.Source: A EMPIRICAL ANALYSIS OF THE GATING… · p. 14

A.1 TEMPORAL ALIGNMENT WITH TRUE REACTION ONSETS

  • Across 515 velocity-defined onsets, the nearest predicted gate peak had a median offset of 0 frames and a mean offset of −0.50 frames.These listener-timeline offsets do not estimate speaker-cue-to-listener-response delay or isolate the shifted-attention contribution; peak density and a chance-alignment baseline also matter.Source: §A.1 · p. 14

A.2 INTRA-CLIP CORRELATION WITH MOTION VELOCITY

  • 0.0051 higher mean ground-truth velocity in the top versus bottom gate-value deciles (p = 3.2×10−5) is associated with higher predicted reactivity within clips.The authors caution that this association does not establish that gate activations cause motion changes or prevent long-horizon drift.Source: §A.2 · p. 14

B.1 REACTIVE GATED FUSION PARAMETERIZATION

  • The fusion parameterization encodes a shifted-ALiBi delay prior with a causal mask, then obtains fused context from multi-head attention outputs.Speaker-audio and listener-history windows are separately encoded; the attention outputs are concatenated and projected.Source: §B.1 · pp. 14–15

B.2 STOCHASTIC REFINEMENT PARAMETERIZATION

  • The refinement parameterization encodes the coarse expression trajectory, conditions stochastic residuals on speaker representation, and projects the resulting expression residual into the complete motion.The text specifies Gaussian latent noise and a temporal self-attention encoder followed by an output projection.Source: §B.2 · p. 15

B.3 DATASET STATISTICS AND REPRESENTATIONS

This material describes the datasets and metrics, training and model configuration, robot mapping, and stochastic micro-dynamics design. The pipeline uses a staged coarse-to-fine objective and constrains its physical outputs through calibration and safety checks.

B.3 DATASET STATISTICS AND REPRESENTATIONS; B.4 EVALUATION METRIC DEFINITIONS

  • ViCo and L2L provide distinct evaluation settings, and motion is assessed for accuracy, realism, speaker–listener correlation, and variability relative to ground-truth targets.ViCo includes in-domain and unseen-identity test splits; L2L is an in-the-wild corpus with motion coefficients and pre-extracted audio features.Source: §B.3 · pp. 15–16, §B.4 · p. 16

B.5 ROBOTIC EMBODIMENT RETRAINING

  • For robot experiments, the models are retrained on 52 semantic facial blendshapes extracted with FaceVerse V2 to support anatomical mapping to Ameca controls.Source: §B.5 · p. 16

B.6 TWO-STAGE TRAINING OBJECTIVES

  • Training first learns coarse trajectories, then freezes the coarse generator while refinement learns expression residuals; the second stage combines coordinate preservation with adversarial supervision.The stated objectives include reconstruction, velocity regularization, and a boundary constraint in the coarse stage, and spatial consistency plus adversarial loss in refinement.Source: §B.6 · pp. 16–17

B.7 MODEL CAPACITY AND COMPUTE RESOURCES; B.8 MODULE CONFIGURATIONS

  • REALM has 1.50M trainable parameters and 0.02G FLOPs per forward pass at W = 16, and its core modules use a shared 128-dimensional latent space.The module configurations include a four-layer reaction decoder and a refinement module combining dilated Conv1D with a Transformer Encoder.Source: §B.7 · p. 18, §B.8 · p. 18

C PHYSICAL GROUNDING: INVERSE KINEMATIC MAPPING AND CONTROL SAFETY

  • The physical grounding pipeline maps facial coefficients to robot joints, smooths them over five frames, calibrates relative to a neutral posture, and clamps commands to hardware limits.The pipeline also applies motor rate-of-change constraints and firmware current-overload cutoffs.Source: C PHYSICAL GROUNDING: INVERSE… · p. 18

C.1 KINEMATIC SUBSPACE FORMULATIONS

  • The kinematic mapping separates ocular, head, and lower-face subsystems, using synergistic and antagonistic blendshape couplings for lip actuation.Angular head and gaze targets are parameterized in radians, and nonlinear max-pooling is used for overlapping bilateral muscle groups such as nasal sneering.Source: §C.1 · p. 19

D STRUCTURAL ANALYSIS OF MICRO-DYNAMICS: BLINK EXTRACTION

  • REALM injects stochastic noise in latent feature space rather than directly into output coordinates, with audio-derived modulation and channel attention shaping its dynamics.A self-attention encoder imposes temporal coherence; the stated design aims to map latent stochasticity into biomechanically structured movement rather than anatomically decoupled jitter.Source: D STRUCTURAL ANALYSIS OF… · p. 20

D.1 METHODOLOGY AND EVALUATION METRICS

The blink evaluation measures rate and duration using rendered facial landmarks, and the reported results contrast REALM with omitted-refinement and unmodulated-noise variants. The accompanying physical-robot study used randomized, double-blinded side-by-side comparisons with human evaluators.

D.1 METHODOLOGY AND EVALUATION METRICS

  • Blink evaluation reconstructs videos, detects eye-aspect scores from rendered facial landmarks, and measures absolute blink-rate and average-duration errors.The method avoids relying solely on raw blendshape coefficients because eyelid closures can be entangled with adjacent facial activations.Source: §D.1 · p. 20

D.2 ANALYSIS OF RESULTS

  • 273.56 ms is REALM’s reported blink duration, compared with 292.01 ms for ground truth; unmodulated noise produced durations averaging over 630 ms.Omitting refinement produced almost zero blinks, while REALM recovered high-frequency dynamics with the reported duration close to the ground-truth duration.Source: §D.2 · p. 20, Table 8 · p. 20

E SUBJECTIVE USER STUDY DETAILS; E.1 PROCEDURE AND PARTICIPANTS

  • The physical Ameca user study recruited 25 evaluators, each rating a randomized subset of five ViCo test sequences in side-by-side comparisons.The comparisons included REALM and five baselines; method ordering was randomized and double-blinded for every sequence.Source: E SUBJECTIVE USER STUDY DETAILS · p. 21, §E.1 · p. 21

E.2 EVALUATION METRICS

REALM scores highest across four perceptual criteria in the reported robot evaluation and combines temporal alignment, adaptive context weighting, and stochastic expression refinement. Its empirical comparisons remain bounded by the methods and configurations evaluated.

E.2 EVALUATION METRICS

  • The 5-point evaluation covers naturalness, audio-visual synchrony, contextual appropriateness, and overall preference.The criteria assess lifelike movement, temporal alignment with speaker cues, context-fitting reactions, and holistic interaction preference.Source: §E.2 · p. 21

E.3 ANALYSIS

  • REALM achieved the highest scores across all four robot-study metrics; deterministic baselines scored lowest, while ListenFormer lacked high-frequency micro-dynamics.The report attributes REALM’s natural, synchronized, engaging physical behavior to stochastic refinement preserved by relative motion calibration.Source: §E.3 · p. 21

F.1 MODELING CHOICES.

  • REALM combines a delay-centered attention bias, a learned gate for speaker context and listener history, and stochastic expression residuals while retaining coarse pose coordinates.The approach builds task-specific inductive biases for balancing continuity, responsiveness, and local expression variation.Source: §F.1 · p. 22

F.2 SCOPE AND BASELINE SELECTION.

  • Quantitative comparisons focus on listener expression and pose coefficients conditioned on speaker audio and listener history, requiring compatible representations, inputs, and evaluation procedures.Source: §F.2 · p. 22
  • The paper reports no reproduced ARIG or Wang et al. results because implementations were unavailable, leaving compatible-protocol evaluation as an outstanding comparison.Their modeling choices are discussed qualitatively.Source: §F.2 · p. 22
  • UniLS is included, and the empirical conclusions are restricted to the methods and configurations evaluated.Source: §F.2 · p. 22
Loading 2609.33095v1…