PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality

Richard Xu, Dakarai Crowder, Alexis E. Block, Wenzhen Yuan

arXiv:2610.34534v1cs.RO

TL;DR

Robots must coordinate speech, movement, and contact while adapting to unpredictable human behavior, yet touch and embodied multi-turn evaluation remain underexplored. This paper presents a speech-, vision-, and touch-integrating framework and evaluates it in VR against human teleoperation. Autonomous ratings were positive but significantly lower than teleoperation, with timing, clarification, and spatial coordination identified as relevant factors in the gap.

  • Problem

    Robots struggle to integrate multimodal input, adapt to unpredictable behavior, and coordinate responses, while social touch and first-person multi-turn evaluation remain underexplored.

    Source: §I · p. 1

  • Method

    The framework integrates speech, vision, and social touch, represents inputs as text or images, and maintains interaction history for multi-turn robot speech and movement.

    Source: §I · p. 1

  • Results

    Participants rated the autonomous system above the scale midpoint for naturalness and social appropriateness, but rated teleoperation significantly higher on both measures.

    Source: §I · p. 2, §VII · p. 7

  • Takeaways & Limitations

    The observed breakdowns and feedback point to conversational timing, clarification, and spatial coordination as relevant challenges for narrowing the gap with teleoperation.

    Source: §VIII · p. 8

    The study was conducted in VR, so the findings require validation with physical robots in real-world environments.

    Source: A. Limitations · p. 8

Abstract

from arXiv · show

Social robots must coordinate speech and movement while interpreting multimodal human input. We present a multimodal framework that integrates speech, vision, and touch to generate contextually appropriate robot speech and movement. We evaluated the framework in virtual reality with 18 participants across five social interaction scenarios, comparing its performance against a human-teleoperated baseline. Although participants rated the autonomous system above the neutral midpoint for both naturalness and social appropriateness, the teleoperated baseline received significantly higher ratings on both measures. Behavioral coding and qualitative feedback revealed that conversational timing, clarification of ambiguous requests, and spatial coordination were important factors contributing to this performance gap. These findings highlight key challenges in developing autonomous robots capable of natural, multimodal social interaction.

I. INTRODUCTION

Robots must coordinate speech, movement, and contact while adapting to changing social interactions, but integrating multimodal input and evaluating embodied, multi-turn interaction remain challenges. This work presents and evaluates a speech-, vision-, and touch-integrating framework in VR, where autonomous ratings were positive but below teleoperation.

I. INTRODUCTION

  • Robots must coordinate speech, body movement, and physical contact while adapting to interruptions, misunderstandings, and shifting social contexts.Prior work often used single-turn interactions or videos rather than first-person interaction with an embodied robot.Source: §I · p. 1
  • The framework integrates speech, vision, and social touch to generate coordinated robot speech and movement, using interaction history to support multi-turn interaction.It was deployed on a virtual humanoid robot for participant interaction through a VR headset.Source: §I · p. 1
  • The evaluation compared the autonomous system with a human-teleoperated baseline across five scenarios using participant ratings, open-ended feedback, and coded interaction failures.VR provided consistent scenarios, dense contact signals, and close-proximity interaction without injury risk.Source: §I · p. 1
  • The autonomous system was rated above the neutral midpoint for naturalness and social appropriateness, but teleoperation scored significantly higher on both measures.The study also found no statistically significant condition-by-scenario interaction.Source: §I · p. 2

II. RELATED WORK

Prior social-touch systems often rely on predefined reactions or scripted control, while embodied-agent work largely leaves touch unexamined. VR offers a controlled setting for HRI evaluation.

A. Social-Physical Human-Robot Interaction

  • Social-touch robots commonly use predefined behaviors or structured activities, and speech-and-touch systems have often relied on Wizard-of-Oz control or scripts.A humanoid form may raise expectations of greater social intelligence, while touch interpretation in context and autonomous responses beyond fixed behaviors remain an open aim.Source: A. Social-Physical Human-Robot… · p. 2

B. Physically Embodied Agents for Social Interaction

  • LLM- and VLM-based social robots coordinate verbal and nonverbal behavior, but social touch remains largely unexamined in this work.Subjective ratings or isolated behaviors provide little insight into interaction breakdowns and user responses.Source: B. Physically Embodied Agents for… · p. 2

C. Virtual Reality for Human-Robot Interaction

  • VR offers a controlled platform for HRI studies, though prior work has observed differences such as increased interpersonal distances compared with other presentation settings.Prior studies report that perceptions, trust, and engagement can often be preserved across live, VR, and screen-based interactions.Source: C. Virtual Reality for Human-Robot… · p. 2

III. SYSTEM

The system combines speech, touch, and visual context with interaction history to produce action–speech pairs. Its input pipeline converts speech and touch into triggers and representations for the VLM.

III. SYSTEM

  • Speech and tactile inputs trigger a VLM response, while visual input supplies context without initiating a response; the VLM outputs action–speech pairs conditioned on recent interaction history.The history stores up to 200 recent entries and is cleared at the start of each scenario.Source: §III · p. 2, Figure 2 · p. 2

A. Multi-modal Perception & Input Processing

  • Speech is transcribed in real time, and a finalized transcript after a brief end-of-speech pause triggers prompt generation and VLM inference.The speech input is collected through the VR headset microphone.Source: A. Multi-modal Perception & Input… · p. 3
  • Touch events are represented as spatiotemporal point clouds from detected hand–robot collisions, with points recording location, force, and time.Each hand is represented by 24 spherical colliders, and contact collection ends after 0.5 seconds without contact.Source: A. Multi-modal Perception & Input… · p. 3, Figure 3 · p. 3
  • A classifier maps touch events to gesture, contact location, and repetition information using features such as duration, force, contact count, centroid movement, and location.The training dataset comprised 1,756 samples across eight social-touch gestures from seven pilot participants; leave-one-subject-out accuracy was 78.2 ± 8.2%.Source: A. Multi-modal Perception & Input… · p. 3

B. Social Reasoning & Behavior Generation

The VLM receives a prompt combining interaction context, execution state, and history, alongside an image from the robot’s eye camera. Separate interruption mechanisms handle new user input and ongoing robot behavior.

B. Social Reasoning & Behavior Generation

  • Each prompt combines a fixed pre-prompt, the robot’s current execution state, interaction history, and an image from the robot’s eye camera.The pre-prompt includes motion primitives and an instruction to return a should-interrupt flag; the execution state includes the current action and queued motions.Source: B. Social Reasoning & Behavior… · pp. 3–4
  • Interaction history is a chronological text list, with current speech and touch appended as a transcript or a touch-event summary.Touch summaries include gesture class, classifier confidence, repetition count, and the body part touched.Source: B. Social Reasoning & Behavior… · p. 4
  • New inputs can invalidate pending VLM requests, prompt motion interruption decisions, or stop robot speech when the user begins speaking.Simultaneous touch and speech are coalesced into one prompt; when a busy robot does not interrupt, the input remains as context.Source: B. Social Reasoning & Behavior… · p. 4

IV. EVALUATION

The system was evaluated in VR using a simulated Reachy and a controlled comparison with a human-teleoperated baseline. The setup supported speech and direct physical interaction while keeping the operator and participant in separate rooms.

IV. EVALUATION

  • The VR study compared the autonomous system with a human-teleoperated baseline across five social scenarios.The simulated Reachy setup enabled collision detection, safe physical interaction, and controlled environmental conditions.Source: §IV · p. 4

A. Setup

  • The experiment used haptic gloves, a mirrored VR monitor, and a camera recording the participant.The gloves had six linear resonant actuators per glove, positioned at the fingers and palm.Source: A. Setup · p. 4
  • Participants could interact with the articulated virtual robot through speech or direct contact that moved its arms and head.The base remained rigid unless a base-movement command was being executed.Source: A. Setup · p. 4
  • A single trained operator used the robot’s camera feed, participant audio, and the autonomous system’s motion primitives to control the baseline remotely.The autonomous system used voice-cloning text-to-speech to match the operator’s voice; LLM generation accounted for most response latency (M = 2.3 s, SD = 0.5 s).Source: Human-Teleoperated Baseline · p. 4

B. Interaction Scenarios

Five scenarios elicited functional and emotional interactions, including situations that encouraged physical contact without requiring touch. Participants familiarized themselves with the robot, then completed both conditions across all five scenarios in a counterbalanced, within-subject design.

B. Interaction Scenarios

  • The five scenarios covered functional and emotional interactions, with two designed to encourage physical interaction without explicitly requiring touch.The make-way and vacuum scenarios used a blocked narrow passage and loud vacuum noise, respectively.Source: B. Interaction Scenarios · p. 4
  • Scenarios prompted participants to ask for a joke, get the robot to move, cheer it up, gain its attention, or request a drink.The prompts corresponded to five distinct situations shown in Fig. 5.Source: Figure 5 · p. 5

C. User Study Procedure

  • Participants calibrated their virtual avatar and freely interacted with the robot during familiarization until they felt comfortable.They received an example prompt to introduce themselves to the robot.Source: C. User Study Procedure · p. 5
  • Eighteen participants completed all five scenarios in both conditions, with condition order counterbalanced within each scenario and participants blind to condition.One of 19 recruited participants was excluded after reporting random questionnaire responses.Source: C. User Study Procedure · p. 5

D. Measures and Analyses

The study collected slider ratings, open-ended feedback, and video-coded interaction behaviors. Statistical models compared ratings and breakdowns across conditions and scenarios while accounting for repeated participant measurements.

  • Participants rated interaction naturalness and appropriateness on continuous slider scales after each interaction.The scales ranged from unnatural to natural and inappropriate to appropriate.Source: D. Measures and Analyses · p. 5
  • One author inductively coded open-ended responses and video recordings to identify interaction breakdowns and observable positive or negative behaviors.Breakdowns were operationalized as observable communication or task-execution failures, or behaviors suggesting failure, such as repetition.Source: D. Measures and Analyses · p. 5
  • Negative binomial regression modeled breakdown counts by condition, scenario, and their interaction, with standard errors clustered by participant.A Wilcoxon signed-rank test compared participants’ mean total-error counts between conditions.Source: D. Measures and Analyses · p. 5
  • Linear mixed-effects models tested ratings by condition, scenario, and their interaction, with participant random intercepts and prior robot attitude as a covariate.A separate mixed-effects model compared autonomous ratings with the neutral midpoint.Source: D. Measures and Analyses · p. 5
  • Figure 6 reports per-scene ratings, coded interaction events, and whether participants touched the robot in each condition.Its first 12 coded behaviors are breakdowns and its final three are positive interaction behaviors.Source: Figure 6 · p. 6

V. RESULTS

The autonomous system rated above the neutral midpoint on naturalness and appropriateness, but the teleoperated baseline scored significantly higher on both. Autonomous interactions also had more breakdowns, especially in the vacuum scenario, while the baseline commonly clarified ambiguous input.

A. Perceived Interaction Quality

  • Teleoperated interactions received significantly higher naturalness and appropriateness ratings, although autonomous ratings exceeded the neutral midpoint on both measures.Autonomous ratings were farther above the midpoint for appropriateness; condition-by-scenario interactions were not significant (all p values > 0.38).Source: Table II · p. 6, Table III · p. 6, A. Perceived Interaction Quality · p. 6

B. Where Interactions Break Down

  • Autonomous interactions had more breakdowns (M = 1.54, SD = 1.42) than teleoperated interactions (M = 0.17, SD = 0.21; p < 0.001).The difference was largest in the vacuum scenario; timing and responsiveness failures were most frequent, while the teleoperator commonly asked for clarification when input was ambiguous.Source: B. Where Interactions Break Down · p. 6
  • Incorrect behavior selection and failed tasks were uncommon breakdown categories, with navigation errors as an exception.Source: B. Where Interactions Break Down · p. 6

C. Touch Incidents

Touch occurred more often in autonomous interactions, while participants’ feedback emphasized conversational timing and clarification as important to verbal interaction.

C. Touch Incidents

  • Participants touched the robot in 47.8% of autonomous interactions and 34.4% of teleoperated interactions, using touch for attention, affection, or incidentally.In 12 cases, participants touched the robot only in the autonomous condition; the reverse never occurred.Source: C. Touch Incidents · p. 6

VI. QUALITATIVE ANALYSIS

  • Qualitative analysis of open-ended responses identified timing and clarification as two verbal behaviors participants discussed.Timing was mentioned by 10 of 18 participants and clarification by 8 of 18.Source: §VI · pp. 6–7, A. Verbal Interaction · p. 7

A. Verbal Interaction

  • Ten participants described timing concerns, including delayed responses and pacing that disrupted the joke scenario’s conversational flow.Participants valued the teleoperated robot’s pause before the punchline; the autonomous robot delivered it immediately.Source: A. Verbal Interaction · p. 7
  • Eight participants preferred clarification for ambiguous requests, such as asking which drink the user wanted instead of choosing automatically.Participants described collaborative resolution of ambiguity as preferable.Source: A. Verbal Interaction · p. 7

B. Embodied Coordination

Participants rated the autonomous system above the scale midpoint, but rated the teleoperated baseline significantly higher on naturalness and social appropriateness. Feedback and behavioral analysis pointed to coordination, timing, clarification, and differing expectations about robot emotion.

B. Embodied Coordination

  • Fourteen of 18 participants evaluated movement and orientation alongside speech, and described failures in movement, orientation, or touch responsiveness as disrupting interaction flow.Expectations included contextually appropriate movement, facing the participant while speaking, and responding appropriately to touch.Source: B. Embodied Coordination · p. 7

VII. DISCUSSION

  • The teleoperated baseline received significantly higher ratings than the autonomous system for both naturalness and social appropriateness.Autonomous-system mean ratings were above the scale midpoint, and no condition-by-scenario interaction was statistically significant.Source: §VII · p. 7
  • Autonomous interactions showed user repetitions, unacknowledged input, and interruptions; the 3.7 s mean response latency exceeded the cited 250 ms typical conversational turn gap.The discussion identifies pacing, clarification of ambiguous requests, and spatial awareness as areas for improvement; navigation errors occurred in the autonomous system but not the baseline.Source: §VII · p. 7
  • Touch was more frequent in autonomous interactions, and participants may have used it as an alternative channel when speech was unsuccessful.The paper notes that further evaluation is needed to determine whether touch improves recovery from communication breakdowns.Source: §VII · p. 7

C. Emotional Expectations

  • Participants differed in their responses to robot sadness; whether it seemed natural or unsettling appeared to depend on viewing the robot as a tool or social agent.Some participants preferred emotion when it had sufficient context and the robot responded reciprocally.Source: §VII · p. 7, C. Emotional Expectations · p. 7

A. Limitations

The study’s results are bounded by its virtual-reality setting and possible advantages in teleoperator familiarity and speech expressiveness. The conclusion identifies physical-robot deployment as future work.

A. Limitations

  • The VR study’s findings require validation with physical robots in real-world environments.The teleoperator’s familiarity with the virtual environment and the voice-cloned TTS system’s less expressive prosody may also have influenced comparisons.Source: A. Limitations · p. 8

VIII. CONCLUSION

  • Future work will address timing, clarification, and spatial coordination and deploy the framework on a physical robot.The conclusion reports that participants rated the system generally appropriate and natural and used touch alongside speech.Source: §VIII · p. 8
Loading 2610.34534v1…