PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied Referring Expression Comprehension in Human-Robot Interaction

Md Mofijul Islam, Alexi Gladstone, Sujan Sarker, Ganesh Nanduru, Md Fahim, Keyan Du, Aman Chadha, Tariq Iqbal

arXiv:2512.06558v1cs.RO

TL;DR

Robots need to interpret human instructions that combine speech with embodied cues, but existing datasets lack broad coverage of natural, diverse interactions. The paper introduces Refer360 and MuRes, an information-bottleneck module that reinforces salient modality-specific representations. MuRes improves performance across referring-expression and VQA evaluations, and the authors present Refer360 as a benchmark for embodied interaction research.

  • Problem

    Existing datasets have perspective and viewpoint biases, limited nonverbal-gesture coverage, and a focus on indoor settings, restricting evidence for embodied instruction comprehension across diverse HRI contexts.

    Source: §1 · p. 2

  • Method

    The authors collect Refer360 across diverse settings and views, and introduce MuRes, a guided residual module that selectively reinforces salient modality-specific signals in pretrained representations.

    Source: §1 · p. 2

  • Results

    Across referring-expression and VQA evaluations, MuRes improved model performance, including 4.58% higher VisualBERT VQA accuracy on ScienceQA.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Refer360 provides a multimodal benchmark for studying embodied referring expressions, while MuRes offers a lightweight adapter approach that consistently improved evaluated multimodal models.

    Source: §9 · p. 8

Abstract

from arXiv · show

As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.

1 Introduction

The introduction frames embodied instruction comprehension as a dataset and representation challenge: existing resources have viewpoint, gesture, and setting gaps, while multimodal fusion can miss modality-specific cues. It presents Refer360 and MuRes as responses, reporting improvements across referring-expression and VQA evaluations.

1 Introduction

  • HRI instructions combine speech with cues such as gaze, gesture, and posture, which help resolve references such as “this one” or “that side.”The introduction links unambiguous instruction understanding with smoother integration into human teams and more natural interaction.Source: §1 · p. 1
  • Existing datasets are limited by perspective-tied language, single-view recordings, partial gesture coverage, and predominantly constrained indoor collection.The introduction argues these gaps limit support for comprehension in diverse, unconstrained HRI contexts.Source: §1 · p. 2
  • Refer360 contains 13,990 multimodal interactions from 66 participants across 392 sessions, spanning 3.2 million synchronized frames and 17.62 hours of recording.Its collection includes indoor and outdoor settings and multiple sensor modalities.Source: §1 · p. 2
  • MuRes uses an information bottleneck and guided residual connections to reinforce salient modality-specific cues while forming complementary multimodal representations.The authors describe it as a lightweight adapter for existing multimodal models.Source: §1 · p. 2
  • MuRes improved multimodal-model performance across referring-expression and VQA evaluations, including 4.58% higher VisualBERT VQA accuracy on ScienceQA.The introduction also reports CLIP IOU-25 improvements of 3.4% on Refer360 and 4.99% on CAESAR-PRO.Source: §1 · pp. 2–3

2 Related Work

Prior embodied referring-expression datasets vary in modality, viewpoint, and realism, while multimodal models often emphasize alignment over complementary modality-specific cues. These limitations motivate broader data and fusion approaches for embodied HRI.

2.1 Embodied Referring Expression Datasets

  • Prior E-RFE datasets commonly focus on indoor, static-view, scripted, limited-modality, or synthetic settings, leaving limited samples for realistic HRI.The related-work discussion describes the second embodied-interaction line as interpreting human instructions through verbal and nonverbal cues.Source: §2.1 · p. 3
  • Existing embodied datasets include verbal-plus-pointing, gaze and body-motion, and synthetic verbal-plus-nonverbal examples, but each has scope constraints.The cited examples include,, and, respectively.Source: §2.1 · p. 3

2.2 Multimodal Representation Learning

  • Alignment-focused self- and cross-attention can miss complementary signals such as gaze and gesture that matter for interpreting embodied instructions.Prior multimodal methods have improved VQA through alignment-based training objectives.Source: §2.2 · p. 3

3 Refer360: E-RFE Dataset

Refer360 is a multimodal dataset of natural referring interactions across varied real-world contexts, combining first- and third-person views with gaze, gesture, and other sensor streams. Its collection system uses a robot-mounted Kinect and an eye tracker to capture synchronized perspectives.

3 Refer360: E-RFE Dataset

  • Refer360 combines egocentric video and gaze with exocentric RGB, depth, infrared, audio, skeletal data, and robot camera motion across varied contexts.The recordings include both laboratory and unconstrained outdoor environments.Source: §3 · pp. 3–4

3.1 Data Collection System

  • The collection system pairs a participant-worn Pupil eye tracker with an Azure Kinect mounted on a telepresence robot to capture first- and third-person data.The robot could be repositioned during collection, supporting naturalistic viewpoint changes in HRI scenarios.Source: §3.1 · p. 4

3.2 Participants

  • The study recruited 66 participants, with one participant’s data excluded because they did not consent to release it.The reported average participant age was 26.66 years, with a standard deviation of 3.36 years.Source: §3.2 · p. 4

4 Data Collection Procedure

Participants gave object-referencing instructions under constrained and unconstrained conditions, while varied objects, environments, and viewpoints captured situated communication. Post-processing segmented and synchronized interactions, extracted canonical frames, and validated transcriptions and alignment.

4 Data Collection Procedure

  • Participants completed object-referencing sessions under constrained guidance to use verbal and nonverbal cues or unconstrained instructions with no prescribed format.They could describe targets from different perspectives, such as “the box to my right” versus “the box to your left.”Source: §4 · p. 4

4.1 Dataset Post-processing

  • Pointing and gaze-fixation events were timestamped to align data streams, and five experts validated interaction segmentation, synchronization, and audio transcriptions.The collection also used interaction start and end times from the Pupil Cloud event log to segment recordings.Source: §4.1 · p. 5, §4 · p. 5
  • Post-processing split Kinect recordings into modality streams and interaction segments, matched eye-tracker recordings by timestamps, and transcribed Kinect audio.Canonical frames were logged when participants pointed precisely to the target object.Source: §4.1 · p. 5

5 Dataset Analysis

Refer360 captures many brief interactions across diverse sessions, while participant surveys show a strong preference for multimodal object referencing.

5 Dataset Analysis

  • 392 sessions yielded 13,990 interactions over 17.62 hours, including approximately 3.2 million multimodal frames and 28,736 canonical frames.Sessions averaged 2.69 minutes; interactions averaged roughly 4.53 seconds.Source: §5 · p. 5
  • The dataset’s short interaction cycles and sustained session diversity support studying embodied communication at fine-grained and contextual levels.Source: §5 · p. 5
  • 96.97% of surveyed participants favored combining verbal instructions with gestures such as pointing and gaze; none chose gestures alone.Only 3.03% reported relying on verbal instructions alone.Source: §5 · p. 5

6 MuRes: Multimodal Guided Residual Model

MuRes adds a guided information bottleneck to residual fusion, selecting salient visual and language signals while retaining prior representations for downstream tasks.

6 MuRes: Multimodal Guided Residual Model

  • MuRes addresses frozen visual-language representations whose fusion can lose salient information, while ordinary residual connections reinforce entire representations without a bottleneck.Source: §6 · p. 5
  • Projected visual and language representations query the original extracted representations through cross-attention to obtain guided residual representations.The cross-attention keys and values are the original visual and language representations.Source: §6 · p. 6, Figure 3 · p. 6
  • Adding guided residuals to projected representations retains residual connections’ prior-representation reinforcement and yields fused features for downstream learning.The paper also notes residual connections can improve gradient flow.Source: §6 · p. 6
  • The model is trained for object bounding-box prediction and multiple-choice visual question answering, using regression and classification losses respectively.Source: §6 · p. 6

7 Experimental Setup

Experiments compare MuRes variants and vanilla fusion across embodied referring-expression and visual question-answering tasks.

7 Experimental Setup

  • The evaluation integrates MuRes into CLIP, DualEncoder, ViLT, BLIP-2, and VisualBERT and tests embodied referring-expression comprehension on Refer360 and CAESAR-PRO.ScienceQA and A-OKVQA are used for visual question answering.Source: §7 · p. 6
  • The tested variants use visual-only, language-only, or joint visual-and-language queries; vanilla residual and no-residual models provide comparison baselines.Source: §7 · p. 6
  • Table 3 compares vision-language models on bounding-box E-REF comprehension across Refer360 and CAESAR-PRO, with visual-only and language-only variants denoted V and L.Source: Table 3 · p. 7

8 Experimental Results

MuRes improves reported performance on embodied referring-expression comprehension and visual question answering. Visual reinforcement raises CLIP’s Refer360 and CAESAR-PRO IOU-25 scores, while joint visual-language reinforcement performs best across the reported VQA variations.

8.1 Embodied Referring Expression

  • MuRes enhances embodied referring-expression performance on both Refer360 and CAESAR-PRO; CLIP IOU-25 rises from 25.80% to 29.20% on Refer360 and from 37.92% to 42.91% on CAESAR-PRO.Source: §8.1 · p. 7
  • Vanilla residual connections offer modest gains over models without residual fusion, whereas MuRes selectively reinforces salient representation content.The authors describe the information bottleneck as extracting relevant information rather than indiscriminately reinforcing entire representations.Source: §8.1 · p. 7
  • Visual reinforcement raises CLIP’s Refer360 IOU-25 from 25.80% to 29.20%, while language reinforcement raises it to 28.30%.The authors interpret this difference as suggesting object grounding relies predominantly on visual information.Source: §8.1 · p. 7

8.2 Visual Question-Answering Tasks

  • MuRes improves performance across evaluated VQA variations compared with models without residual connections, and joint visual-language reinforcement consistently yields the highest improvements.Source: §8.2 · pp. 7–8
  • CLIP’s ScienceQA accuracy increases from 21.31% to 51.85% with MuRes, while VisualBERT’s ScienceQA performance rises from 34.95% to 37.13% with visual reinforcement and 37.63% with language reinforcement.Source: §8.2 · p. 8

9 Overal Discussion

Refer360 broadens embodied interaction research with synchronized, multi-view verbal and nonverbal data across varied settings, while MuRes provides a lightweight fusion mechanism. The discussion presents both as tools for grounding models in embodied contexts and points to extending the dataset toward embodied question answering.

9 Overal Discussion

  • Refer360 pairs synchronized egocentric, exocentric, and top-down recordings with verbal instructions and nonverbal gestures across varied indoor and outdoor settings.The discussion highlights perspective-invariant reference resolution and gesture–gaze fusion as uses for this data.Source: §9 · p. 8
  • MuRes uses an information bottleneck in residual connections to amplify salient modality-specific signals before fusion, and improves performance across the evaluation setups.The discussion characterizes it as a lightweight adapter between generic pretrained encoders and fine-grained E-RFE signals.Source: §9 · p. 8
  • Refer360 could be extended to broader visual question answering and an embodied question-answering benchmark integrating situated reasoning with embodied cues.The paper presents this as a future possibility, not an existing dataset capability.Source: §9 · p. 8

10 Conclusion

The conclusion presents Refer360 as a diverse multimodal benchmark and MuRes as a bottleneck module for extracting salient modality-specific representations. The resources section reports which dataset and reproduction materials are available or still pending.

10 Conclusion

  • Refer360 combines multiple visual viewpoints, depth, infrared, 3D skeletal data, audio, and robot-camera motion to capture real-world embodied interactions.The paper describes it as the first embodied referring-expression comprehension dataset curated with this range of sensor data.Source: §10 · p. 8
  • Across four datasets spanning embodied referring-expression understanding and visual question answering, adding MuRes to existing multimodal models improves downstream performance.The conclusion also notes that current multimodal models struggle to bridge frozen visual-language representations and salient modality-specific cues.Source: §10 · p. 8

A Resources

  • The processed dataset (49.71 GB) is linked, while raw data, source code, and model checkpoints are pending acceptance or publication; a Singularity container is shared for reproduction.The Docker Hub link is withheld to maintain anonymity.Source: A Resources · p. 11

B Additional Experimental Results: Quantitative Analysis

The additional qualitative results illustrate VisualBERT responses with and without MuRes on ScienceQA diagram questions and ambiguous A-OKVQA questions. The examples indicate that the useful modality reinforcement varies with the question.

B Additional Experimental Results: Quantitative Analysis

  • On the ScienceQA map question, visual MuRes yields the correct answer “Europe,” while language-only MuRes yields “Asia.”The passage attributes the difference to the need to understand the highlighted region’s spatial location.Source: B Additional Experimental Results:… · p. 11, Figure 4 · p. 11
  • For the ScienceQA question about a property shared by three objects, only joint visual-and-language MuRes answers “Transparent.”Either modality reinforced alone gives a response that is not the most accurate.Source: B Additional Experimental Results:… · p. 11
  • The figure also presents VisualBERT reasoning on ambiguous visual questions from A-OKVQA, alongside diagram questions from ScienceQA.Its caption reports improved visual question answering with MuRes on both datasets.Source: Figure 4 · p. 11

C.1 Data Collection System

The collection system combines external and egocentric sensing with synchronized event logging to capture embodied instructions across varied settings. Post-processing segments and aligns the streams, transcribes speech, and organizes session data for the dataset.

C.1 Data Collection System

  • The collection setup combines an Azure Kinect mounted on a telepresence robot with Pupil Smart Glass and a Python application coordinating sensor capture.The system records visual, depth, infrared, IMU, skeletal, gaze, and audio data from complementary viewpoints.Source: §C.1 · p. 11, §C.1.1 · p. 11, §C.1.2 · p. 12

C.1.2 Data Collection Application.

  • The Python application records UNIX timestamps for interaction boundaries and canonical events, using Space for start/end and G for the moment of pointing or gaze.Q terminates a session; matching timestamps support post-processing synchronization between the Kinect and Pupil recordings.Source: §C.1.3 · p. 12

C.1.3 Time-based Synchronization.

  • Refer360 collection spans controlled lab and outside-lab settings, with 75 objects selected from those environments.Participants could use subject, robot, or neutral perspectives; constrained sessions suggested verbal and nonverbal cues, while unconstrained sessions did not prescribe them.Source: §C.1.4 · pp. 12–13
  • Participants completed demographic and post-task surveys, and the study received university IRB approval.Sessions lasted about an hour and recorded ego-view, exo-view, IMU, skeleton, and audio streams.Source: §C.1.4 · p. 13

C.2 Dataset Processing

  • Processing separates Kinect RGB, depth, and infrared video, while storing IMU and skeleton time series, session metadata, audio, and Pupil video in corresponding files.The Kinect video streams and audio are extracted into separate files, with accelerometer and gyroscope readings split apart.Source: §C.2 · p. 13
  • Post-processing segments interactions and canonical frames by recorded timestamps, matches the corresponding Pupil recording, and transcribes Kinect audio with Whisper.Five human experts manually verified synchronization and segmentation before human annotation.Source: §C.2 · p. 14
  • Each processed session has its own folder structure, with transcription text and separate ego- and exo-view video files.Figure 7 depicts the resulting structure, including audio, video, skeletal, and sensor data across RGB, depth, and infrared modalities.Source: §C.2 · p. 14, Figure 7 · p. 14

Related papers

Loading 2512.06558v1…