Source-linked AI summary
SARAH: Spatially Aware Real-time Agentic Humans
Evonne Ng, Siwei Zhang, Zhang Chen, Michael Zollhoefer, Alexander Richard
TL;DR
Conversational agents need spatially aware motion because speech-aligned gestures alone do not make them respond naturally to a moving user. SARAH generates causal full-body motion from user trajectory and dyadic audio, with inference-time gaze control, achieving state-of-the-art quality at over 300 FPS and real-time avatar deployment.
Problem
Existing conversational motion methods lack situated spatial reasoning, often ignoring user movement or requiring non-causal access that prevents streaming inference.
Method
SARAH combines a causal transformer-based VAE with interleaved latent tokens and flow matching conditioned on user trajectory and dyadic audio, plus classifier-free gaze guidance.
Results
Over 300 FPS, SARAH achieves state-of-the-art motion quality on Embody 3D and outperforms non-causal baselines that are 3× slower.
Takeaways & Limitations
The causal, real-time system supports spatially aware conversational agents in streaming headset and real-time avatar environments.
Takeaways & Limitations
The method may generalize poorly to underrepresented spatial configurations or gaze behaviors, and gesture style and locomotion remain uncontrollable.
Abstract
from arXiv · showhide
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.
1 Introduction
SARAH addresses the lack of spatial awareness in conversational agents by generating causal, streaming full-body motion that responds to both speech and user movement. It combines natural gesture alignment, spatial orientation, controllable gaze, and real-time deployment.
- Contribution: SARAH generates full-body motion that aligns gestures with dyadic audio while orienting and reacting to the user’s spatial movement in real time.The method is designed to satisfy conversational, spatial-awareness, controllability, and causal streaming requirements simultaneously.
- Motivation: Current conversational motion methods often ignore interlocutor movement, rely on stationary or forward-facing interactions, or require non-causal computation unsuitable for streaming.Existing dyadic datasets also lack the spatial dynamics needed for reactive behavior.
- Method: The architecture combines a causal transformer-based VAE with interleaved latent tokens and a flow matching model conditioned on user trajectory and both speakers’ audio.The latent design supports streaming inference while preserving temporal coherence.
- Method: A classifier-free gaze guidance mechanism lets users adjust eye-contact intensity at inference while the model learns natural spatial alignment from data.This decouples learning the motion distribution from controlling gaze preference.
2 Related work
Related work has largely modeled gesture or gaze in monadic, stationary, forward-facing, or non-causal settings. SARAH instead targets fine-grained proxemics in dynamic dyadic conversations with supervised causal generation.
- Monadic gesture generation: Most prior gesture-generation methods synthesize single-speaker motion from audio, text, or semantics without awareness of an interlocutor.These approaches include recurrent, feed-forward, autoregressive-transformer, and semantics-conditioned systems.
- Proxemics: Oculesic and proxemic signals are widely used as priors for social prediction tasks, whereas SARAH directly predicts these signals for conversational motion.Its supervised formulation learns fine-grained proxemic information rather than using it only as an auxiliary prior.
- Dyadic interaction: Dyadic gaze and head-motion methods commonly assume forward-facing video calls or stationary pairs, losing global proxemic information.SARAH identifies dynamic spatial interaction as an underrepresented setting.
- Positioning: SARAH claims the first explicit modeling of fine-grained proxemics in dynamic, interactive dyadic conversations.It combines supervised spatial modeling with causal generation for this setting.
- Real-time generation: Recent generative motion methods often use acausal diffusion or causal diffusion variants that remain too slow for real-time streaming.Prior work has explored causal transformers, diffusion forcing, and distillation to reduce inference cost.
3 Real-time, Auto-regressive Motion Synthesis
The method generates conversational full-body motion from user head position and both audio streams using a causal, autoregressive architecture designed for streaming. Its Euclidean motion representation, causal VAE, flow matching generator, and gaze guidance support stable motion, real-time inference, and adjustable eye contact.
- Motion synthesis: The model generates agent motion conditioned on the user’s floor-projected head position and both participants’ audio streams.The user’s head position is used because full-body pose is often unavailable in headset-based systems.
- Motion representation: Each joint is represented as a 3D icosahedron whose centroid provides position and SVD recovers orientation, with mesh geometry added for surface detail.The representation avoids error propagation from local rotations and includes normalization relative to the first frame.
- Motion representation: The Euclidean representation improves performance over traditional joint-angle parameterizations.The paper attributes the comparison to Table 1 and contrasts the representation with joint-angle approaches requiring inverse kinematics.
- Causal VAE: A causal transformer VAE interleaves latent tokens at a fixed stride, with encoder and decoder attention restricted to preceding frames and earlier latents.This design supports streaming inference while preserving causal access patterns.
- Motion generator: A causal transformer-based flow matching model predicts motion latents from user position and both audio streams using modality-specific conditioning.Flow matching transports Gaussian noise toward data while strict causal masking supports autoregressive generation.
- Controllable gaze guidance: The gaze score measures alignment between the agent’s facing direction and the direction toward the user, approaching 1 for direct facing and −1 for facing away.At inference, classifier-free guidance uses a target gaze score to modulate eye-contact intensity while preserving natural variation.
- Dataset: The dyadic Embody 3D subset provides audio and 3D motion from conversations with dynamic spatial interactions, including freely walking and shifting participants.It is described as the first dataset to capture 3D spatial proxemics in conversation.
4 Experiments
Experiments evaluate realism, diversity, gaze alignment, physical plausibility, and expressiveness against retrieval, generative, and ablation baselines. The method provides competitive motion quality while remaining causal, real-time, and controllable.
- Evaluation setup: The evaluation measures five axes: FGD, FGDacc, Foot Slide, Wrist Var, and Head Ang.These metrics assess distributional similarity, acceleration dynamics, skating artifacts, gesture expressiveness, and gaze alignment.
- Retrieval baselines: Ours achieves 0.83 gaze alignment, compared with Random’s 0.28 and NN’s 0.59, while balancing wrist variance at 105.0 against ground truth’s 137.6.Both retrieval methods achieve near-zero foot sliding at 0.01, but their context-agnostic or common-clip sampling produces less suitable gesture variance.
- Generative baselines: A2P records higher FGD (2.01 vs. 1.28), lower wrist variance (69.4 vs. 105.0), and weaker gaze alignment (0.71 vs. 0.83) than Ours.Both diffusion methods run at 90 FPS, 3× slower than Ours, and rely on future context.
- Generative baselines: SHOW runs at 230 FPS but has worse foot sliding (0.27 vs. 0.01), lower wrist variance (65.0 vs. 105.0), and weaker gaze alignment (0.61 vs. 0.83) than Ours.Its audio-only conditioning lacks user-position information needed for spatially aware orientation.
- Ablations: Removing the VAE raises FGD from 1.28 to 1.95 and halves inference speed from 300 to 150 FPS, while foot sliding remains 0.01.The ablation indicates that latent compression primarily benefits distributional matching and efficient generation.
- Gaze controllability: Increasing gaze guidance from 0.0 to 1.0 raises gaze alignment from 0.56 to 0.96, while guidance 0.8 matches ground truth at 0.81 and lowers FGD to 0.92.The no-guidance case has FGD 1.28, so moderate guidance improves both alignment and motion quality in this evaluation.
5 Conclusion
The paper presents a real-time method for spatially-aware conversational motion that orients toward moving users while producing speech-aligned gestures. It also identifies training-data bias, limited controllability beyond gaze, and the need for architectural changes for multi-party interaction.
- Its architecture combines a causal transformer-based VAE with flow matching conditioned on user trajectory and dyadic audio.
- A gaze alignment score with classifier-free guidance decouples learning from control, allowing eye-contact preferences to be adjusted.
- Over 300 FPS and state-of-the-art quality outperform non-causal baselines that are 3× slower.
- Underrepresented spatial configurations or gaze behaviors may generalize poorly, while gesture style and locomotion remain uncontrolled and multi-party conversations require architectural modifications.
A.1 Video Results
The video results show the model generating streaming conversational motion that turns toward users, adapts gestures to emotional contexts, and supports adjustable gaze. Comparisons with existing methods and an interactive VR setup illustrate its intended deployment.
- The model produces conversationally appropriate gestures while naturally turning toward the user and transitioning between speaking and listening modes.
- It generates contextually appropriate body language across stressed, excited, angry, and celebratory emotional contexts.
- Increasing the gaze score produces more direct facing, while dropping it out yields in-distribution gaze behavior from the training dataset.
- The method produces livelier gestures than MDM, more realistic motion than Audio2Photoreal, and fewer motion artifacts than TalkSHOW.
- Its real-time operation enables interactive VR agents using LLM-generated dialogue and text-to-speech models.
A.2 Training Details
The training setup uses AdamW optimization, fixed-length motion subsequences, a train/validation/test split, and causal streaming logic for audio features. The VAE produces a compact latent sequence from each 400-frame input.
- Optimization: The model uses AdamW with β1 = 0.9, β2 = 0.999, weight decay 1 × 10−4, and a learning rate peaking at 1 × 10−4 after warmup.
- Data Processing: Training uses an 80/10/10 train/validation/test split and randomly sampled subsequences of T=400 frames.
- Data Processing: Although HuBERT-Large is not fully causal, test-time streaming avoids future-frame access by using the current context and previous T−s frames.
- Latent Dimension: The VAE latent dimension is D_z=256; with stride s=4 and sequence length T=400, each sequence produces K=100 latent tokens.
A.3 Inference Details
Inference generates motion in short chunks for real-time streaming and renders the resulting joint motions as photorealistic avatars. The streaming protocol preserves temporal consistency while using a midpoint solver.
- Streaming Protocol: Inference generates motion in chunks of s=4 frames while retaining the last two tokens and inpainting history frames for temporal consistency.
- Streaming Protocol: Each chunk uses a midpoint solver with 4 iterations, totaling 8 function-evaluation steps.
- Streaming Protocol: The streaming configuration achieves 60 fps at test time for real-time deployment.
- Photorealistic Rendering: Photorealistic rendering uses joint parameter motions, facial expression and body-pose frames, and viewpoint direction to synthesize registered geometry and view-dependent texture.