PaperlayerView exact revision on arXiv

Source-linked AI summary

Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE

Yuhang Hu, Junlin Lyu, Yuan Fang, Xiaoying Shen, Lechen Zhang, Jiong Lin, Judah Goldfeder, Steven Tian, Hod Lipson

arXiv:2610.19635v1cs.ROcs.AI

TL;DR

Fixed tracking rules do not capture the temporal structure of human gaze, and robot gaze must coordinate target choice with eye, head, and eyelid motion. The paper combines asynchronous target selection and reactive fixation with a target-conditioned Transformer-VAE trained on retargeted human motion. RGT-Base has the lowest reported overall seven-joint MSE among tested controllers on the shared ground-truth-target benchmark.

  • Problem

    Fixed scripted controllers have limited flexibility across targets and motion styles, while the paper asks how to reproduce human gaze dynamics under target-conditioned robot control.

    Source: §1 · p. 1

  • Method

    A VLM selects a referent, reactive perception maintains fixation, and a Transformer-VAE uses recent full-head motion and a planned eye trajectory to generate robot commands.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

  • Results

    2.19×10−4 rad2, RGT-Base has the lowest overall seven-joint MSE among tested controllers on the ground-truth-target benchmark.

    Source: §4.1 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The physical demonstration shows the targeting and control components working together, while the benchmark supports a comparison of robot-space motion generation under shared targets.

    Source: §1 · p. 2, §5 · p. 9

    The ground-truth-target benchmark does not measure referent-selection or fixation errors, and robot-space MSE does not establish perceived naturalness.

    Source: §6 · p. 9

Abstract

from arXiv · show

Gaze coordinates attention in human interaction through coupled eye, head, and eyelid motion. A humanoid robot must choose where to look while generating a suitable movement toward that target; fixed tracking rules alone do not model the temporal structure of human gaze. We separate these tasks with an asynchronous vision-language model that selects a scene referent, a reactive perception module that maintains fixation, and a Realistic Gaze Transformer (RGT) that generates target-conditioned motion. RGT is a Transformer-VAE trained on markerless human recordings retargeted to the robot's 11-DoF joint space: self-supervised pretraining encodes recent full-head history, and autoregressive fine-tuning exposes the model to its own predictions during rollout. At inference, a four-dimensional binocular-eye trajectory conditions coordinated neck and eye motion; an auxiliary blink predictor triggers eyelid templates during playback. On a two-second benchmark in which all controllers receive the same ground-truth target, RGT-Base obtains the lowest reported overall seven-joint MSE, and physical demonstrations show the integrated targeting and control pipeline. The evidence establishes robot-space kinematic fidelity and real-time RGT inference; perceived naturalness and open-vocabulary target-selection accuracy remain questions for controlled evaluation.

1 Introduction

Gaze conveys social cues, but fixed scripted controllers offer limited flexibility across targets and motion styles. The paper separates referent selection from reactive fixation and target-conditioned robot motion, then evaluates motion generation with shared ground-truth targets while demonstrating the integrated pipeline physically.

1 Introduction

  • Gaze supports attention, turn-taking, intimacy, and emotional signaling, and appropriate eye contact has been linked to perceived trustworthiness.Physical embodiment gives robots a way to reproduce these social cues through motion.Source: §1 · p. 1
  • The framework separates slow VLM referent selection, real-time fixation tracking, and RGT generation of coordinated neck, binocular-eye, and eyelid commands.The benchmark gives every controller the same ground-truth eye target, while physical demonstrations show the integrated targeting pipeline.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Fixed scripted controllers are useful starting points but have limited flexibility across targets and motion styles, motivating learned human-motion priors for joint-space reproduction.The paper asks whether such a prior improves reproduction under target-conditioned, closed-loop robot control.Source: §1 · p. 1
  • The contributions are a VLM-guided reactive targeting interface, a target-conditioned Transformer-VAE with autoregressive fine-tuning, and a markerless data and evaluation pipeline.The pipeline tests closed-loop fidelity, stochastic diversity, participant adaptation, and runtime.Source: §1 · p. 2

2 Related Work

Prior robot-face work has expanded from scripted facial presets toward learned control, while gaze has received less attention within that paradigm. Robot gaze controllers and generative motion representations provide relevant foundations, but learning coordinated gaze directly from human motion remains the stated motivation.

Learning on Humanoid Facial Robots

  • Learning-based facial-robot work spans motor-babbling mimicry, recognition- or preference-guided expression, anticipatory smile coexpression, and multilingual lip synchronization.The paper positions synchronized robot-coordinate neck, binocular-eye, and eyelid trajectories as its gaze-specific setting.Source: §2 · p. 2

Robot Gaze Control

  • Robot gaze control progressed from scripted eye-head movement to bio-inspired and model-based strategies, including pursuit, saccades, and joint attention.Saran et al. used monocular deep learning for real-time gaze following; the authors still describe realistic gaze as an open challenge.Source: §2 · pp. 2–3

Representation Learning for Motion

  • Motion-representation research includes contrastive and temporal-convolutional embeddings, VAE and Transformer-VAE generation, and newer diffusion models for controllable motion synthesis.The cited application range includes gaze-conditioned prediction and text-conditioned human motion generation.Source: §2 · p. 3

3 Methods

The method separates target selection from the generation of robot-space gaze motion. It conditions a Transformer-VAE on recent full-head history and a planned binocular-eye trajectory, then trains the model to handle autoregressive rollout.

3 Methods

  • The formulation uses an 11-DoF humanoid face and maps a full-head motion history and future eye trajectory to predicted full-head motion and blink probabilities.The decoder is trained to track the eye target while preserving temporal structure from the human-motion prior.Source: §3 · p. 3

3.1 VLM-Guided Gaze Targeting

  • A VLM maps the scene and language instruction to a referent, and a reactive perception module refines it into frame-rate fixation used to form the RGT condition.The two stages separate deliberative target choice from continuous fixation maintenance.Source: §3.1 · p. 4

3.2 Data Preprocessing

  • The markerless webcam data contains 11 motion channels; preprocessing repairs abrupt eye-gaze spikes, filters noise, extracts blink labels, and resamples signals to 60 Hz.The retargeted trajectories are a shared kinematic reference rather than optical-motion-capture ground truth.Source: §3.2 · p. 4

3.3 Eyeball-Trajectory Generator

  • The four-dimensional eye condition contains right- and left-eye yaw and pitch, using recorded binocular gaze during training and synthesized commands on the real robot.The auxiliary template path realizes blink playback; optional eye-command perturbations apply only to deployment conditions.Source: §3.3 · p. 4

3.4 Realistic Gaze Transformer (RGT)

  • RGT compresses recent full-head motion into a latent code and combines it with the eye condition to predict future facial motion and blink probabilities.With no encoder-to-decoder skip connection, the latent summarizes recent state and style; sampling it can produce different realizations for the same target.Source: §3.4 · pp. 4–5

3.5 Model Training Procedure

  • Training first learns a motion prior from 60-frame histories, then autoregressive fine-tuning conditions later prediction blocks on the model’s own earlier outputs.The fine-tuning rollout exposes the model to accumulating prediction errors while tracking eye commands.Source: §3.5 · p. 5
  • In the illustrated leftward shift, the ground-truth sequence has eyes move before the head, and RGT follows that broad temporal ordering.Eyes-Only and Head-Only instead show distinct eye-fixed and head-fixed movement patterns.Source: Figure 3 · p. 6

4 Experimental Results

The benchmark compares learned and heuristic controllers under the same ground-truth eye target, and RGT-Base has the lowest reported overall seven-joint MSE among the learned models. Further tests examine qualitative motion, subject adaptation, and training ablations, with results bounded by single-run and small-cohort evaluations.

4.1 Quantitative and Qualitative Results

  • The representative leftward gaze shift shows RGT following the recorded eyes-before-head ordering, unlike Eyes-Only and Head-Only, which move only one component.The frames are one selected example, and no controlled observer ratings establish which motion people perceive as more natural.Source: §4.1 · pp. 6–7
  • On the closed-loop seven-joint benchmark over 1,848 S1 clips, RGT-Base had the lowest learned-model MSE: 2.19×10−4 rad2 versus RNN’s 3.04 × 10−4 rad2, a descriptive 27.9% reduction.Source: Table 1 · p. 7, §4.1 · p. 7
  • Single-step evaluation favors Flow Matching, but its closed-loop error is higher, consistent with errors accumulating when generated actions enter subsequent histories.Autoregressive fine-tuning trains RGT on self-generated histories; rollout evaluation therefore provides information not captured by teacher-forced reconstruction alone.Source: §4.1 · p. 7

4.2 Adaptation Across Different Human Subjects

  • 1.8 × 10−3 to 3.6 × 10−4 rad2 is the reported overall error change after subject-specific adaptation, with improvement in all five sampled subjects.Each adaptation uses that subject’s S2 training prefix and evaluates on its held-out tail; the small non-clinical cohort does not establish broader generalization.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Ablation Studies

  • The no-skip VAE has APD 4.4 × 10−6 rad2 versus 5.5 × 10−7 rad2 for the skip model, while their reconstruction MSEs remain of the same order.APD measures diversity across K=10 rollouts, not fidelity or observer preference; the deterministic variant has slightly lower reconstruction error.Source: Table 3 · p. 8, §4.3 · p. 8
  • Autoregressively fine-tuned models have lower prediction errors across behavior groups, and Base has lower overall MSE than Small or Large in this run.The larger model’s difference is not explained by the table; the authors adopt Base as the default RGT.Source: §4.3 · p. 8, Figure 4 · p. 8

5 Conclusion

The framework separates semantic target selection from generation of coordinated eye, head, and eyelid motion. In the ground-truth-target benchmark, RGT-Base has the lowest overall seven-joint MSE among tested controllers, while physical demonstrations show the integrated pipeline.

5 Conclusion

  • An asynchronous VLM and reactive perception module provide targets, while RGT generates coordinated 11-DoF commands from a one-second motion history and planned binocular-eye trajectory.The Transformer-VAE learns from retargeted human recordings, uses autoregressive fine-tuning, and predicts blink timing for template playback.Source: §5 · p. 8
  • RGT-Base has the lowest overall seven-joint MSE among tested controllers in the ground-truth-target benchmark.Physical demonstrations show the targeting and control components working together.Source: §5 · p. 9
  • The next HRI evaluation should test whether people infer the robot’s attention during handover or collaborative search and whether they prefer its gaze to alternatives.The conclusion presents these as questions for future evaluation, not established outcomes.Source: §5 · p. 9

6 Limitations and Future Work

The benchmark isolates robot motion by giving every controller a ground-truth target, so it does not establish target-selection accuracy or fixation under occlusion. Kinematic results also have limits in statistical certainty, perceptual validation, generalization, and rollout duration.

6 Limitations and Future Work

  • Ground-truth targets isolate movement quality but leave VLM referent selection and reactive fixation under occlusion unevaluated.The authors call for target-perturbation sweeps and labeled open-vocabulary grounding cases.Source: §6 · p. 9
  • The VLM runs below the control rate and supplies no explicit policy for gaze shifts during turn-taking or mutual-gaze maintenance.The present system also does not adapt to a partner’s affect in closed loop.Source: §6 · p. 9
  • The MSE table is one stochastic run without fixed evaluation seeds or confidence intervals, and MSE and APD are not observer ratings.Repeated runs would quantify uncertainty, while controlled studies are needed for perceived naturalness, legibility, preference, and trust.Source: §6 · p. 9
  • The six-participant non-clinical sample and markerless webcam retargeting constrain generalization, while scored rollouts cover only two seconds.The authors also note that blink templates simplify playback and do not offer directly interpretable control of blink style or its relation to attention.Source: §6 · p. 9

A Appendix and Supplementary Materials

The accompanying repository provides the RGT implementation, markerless collection and preprocessing pipeline, and MuJoCo simulation documentation; the complete participant trajectory dataset is not currently public.

A Appendix and Supplementary Materials

A.1 Single-step Sliding-Window Evaluation

The single-step sliding-window protocol evaluates per-step prediction using ground-truth history, avoiding error compounding. Flow Matching leads under this protocol, but its ranking reverses in closed-loop evaluation.

A.1 Single-step Sliding-Window Evaluation

  • Each frame uses real history [j −60, j) and the ground-truth target for the next chunk, then advances the window by one frame.Because the history remains ground truth, the protocol measures per-step prediction quality without error compounding.Source: §A.1 · p. 13
  • The reversal between single-step and closed-loop rankings shows that single-step accuracy does not transfer to autoregressive deployment.The authors therefore use long-horizon closed-loop rollout as the primary evaluation in the main paper.Source: §A.1 · p. 13
  • Flow Matching has the lowest overall joint-angle MSE under the single-step protocol: 1.2 × 10−5, roughly 7× better than the next-best learned model.It is strongest on five of six behaviors and overall; the closed-loop ranking reverses, with Flow Matching the weakest learned baseline.Source: Table 4 · p. 13, §A.1 · p. 13

A.2 Hardware and Control

The robot head has 11 actuated degrees of freedom across the neck, eyes, and eyelids. A PD controller converts action inputs into joint torques, with high proportional gains used for rapid head and eye tracking.

A.2 Hardware and Control

  • The humanoid head has 11 DOFs: three neck, four eye, and four eyelid joints.Motion limits are set to resemble natural human anatomical constraints.Source: §A.2 · p. 13
  • A proportional-derivative controller converts action inputs into joint torques.The proportional and derivative gains are predefined based on actuator dynamics.Source: §A.2 · p. 13
  • Relatively high proportional gains support fast and accurate tracking of rapid head and eye movements.Source: §A.2 · p. 13

A.3 VLM and Visual Perception Module

An off-the-shelf VLM selects a scene referent from camera images and, when available, speech transcripts; reactive perception turns that selection into frame-rate fixation. Structured outputs and asynchronous VLM calls connect deliberative selection to continuous gaze control.

A.3 VLM and Visual Perception Module

  • The VLM jointly selects an attended referent and generates an optional spoken response from visual input and a possibly empty transcript.Its constrained output specifies a normalized bounding box, using the face box when the target is a person.Source: §A.3 · pp. 13–14
  • The off-the-shelf VLM uses a fixed prompt and structured JSON output, with hosted and fully local deployment options.The same prompt and output contract are used in both deployments.Source: §A.3 · p. 14
  • The VLM runs asynchronously below the 60 Hz control rate, while reactive perception maintains fixation and the controller uses the latest result.Stale VLM frames are discarded to bound latency.Source: §A.3 · p. 14
  • FastSAM segments the VLM-designated region, and the mask centroid is projected into the eyeball frame as the fixation point.The class-agnostic segmenter preserves targets beyond a fixed category list.Source: §A.3 · p. 14

A.4 Dataset Collection, Description and Representation

The dataset uses markerless webcam recordings processed into 11-dimensional, timestamped facial-motion sequences. It includes structured behavior sequences for pretraining and a primary benchmark, plus participant-specific adaptation sequences; the complete dataset is not currently public.

A.4 Dataset Collection, Description and Representation

  • A commodity webcam and pretrained MediaPipe models produce timestamped 11-DoF motion records without specialized motion-capture hardware.The representation includes head orientation, binocular gaze, and upper and lower eyelid motion.Source: §A.4 · p. 15
  • S1 contains six structured behavior sequences for pretraining and the primary benchmark; S2 contains five participant-specific adaptation sequences from Subjects 2–6.Source: §A.4 · p. 15
  • The six structured behaviors vary in head amplitude, eye–head contribution, and overall dynamics, and sequences are represented as time series of 11 joint angles.Table 5 summarizes S1 using processed 60-Hz command-grid rows.Source: §A.4 · p. 15, Table 5 · p. 15
  • The complete dataset is not currently public.Source: §A.4 · p. 15

A.5 Data Preprocessing Details

The appendix describes preprocessing, blink synthesis, recorded eye–head associations, and generation of the online eye-condition stream. The lagged correlations show earlier eye motion in these recordings, while the condition generator produces smooth transitions under speed and joint limits.

A.5 Data Preprocessing Details

  • Timestamped facial-motion sequences are resampled to 60 FPS; eye-gaze spikes are interpolated across, while blink-related eyelid dips are flattened and later reintroduced at synthesis.Blink labels are extracted from upper-eyelid motion, and soft labels use a Gaussian kernel.Source: §A.5 · p. 16

A.6 Blink Template Extraction and Synthesis

  • Blink templates are extracted from markerless recordings and played back online when the model’s blink-probability head triggers, while head and gaze follow the generated trajectory.The source describes 20-frame templates and a 100-frame refractory period between triggers.Source: §A.6 · pp. 16–17

A.7 Data Analysis

  • ρ=0.86 for the reported eye–head yaw/pitch comparison, and lagged correlation peaks at L20 (333 ms), with ρ20=0.88.The peak is consistent with earlier eye motion; adjacent frames are temporally correlated, making naive independent-frame significance testing inappropriate.Source: §A.7 · pp. 17, 19, Figure 5 · p. 18

A.8 Online Generation of the Eyeball-Condition Signal

  • The eye-condition generator interpolates between anchors to produce an approximately 0.33-second transition, with velocity and mechanical limits applied to the resulting angles.The resulting 4-D ocular command stream has less than 1 ms latency.Source: §A.8 · p. 19

Related papers

Loading 2610.19635v1…