PaperlayerView exact revision on arXiv

Source-linked AI summary

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel

arXiv:1904.10151v2cs.CVcs.CL

TL;DR

Robots still struggle to understand natural-language instructions and identify objects in unfamiliar environments, limiting their practical interaction with the visual world. The paper introduces REVERIE, a dataset and task requiring navigation to remote objects and explicit object identification, together with an Interactive Navigator-Pointer baseline. Existing methods perform poorly, while the proposed approach provides a stronger baseline but remains substantially behind human performance.

  • Problem

    Robots lack the learned environmental knowledge and language-to-vision abilities that let children perform object-oriented tasks in unfamiliar environments.

  • Method

    The paper introduces REVERIE, a task and benchmark dataset requiring agents to navigate unseen indoor environments and identify remote objects from natural-language instructions.

  • Results

    Existing navigation and referring-expression methods achieve only 7.07% REVERIE success on the test split versus 77.84% human performance, while the Interactive Navigator-Pointer provides a strong baseline.

  • Takeaways & Limitations

    REVERIE exposes the significant challenge of jointly understanding high-level instructions, navigating, and grounding referred objects, with a large remaining gap to human performance.

Abstract

from arXiv · show

One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language. Overcoming this challenge requires the ability to perform a wide variety of complex tasks in response to multifarious instructions from humans. In the hope that it might drive progress towards more flexible and powerful human interactions with robots, we propose a dataset of varied and complex robot tasks, described in natural language, in terms of objects visible in a large set of real images. Given an instruction, success requires navigating through a previously-unseen environment to identify an object. This represents a practical challenge, but one that closely reflects one of the core visual problems in robotics. Several state-of-the-art vision-and-language navigation, and referring-expression models are tested to verify the difficulty of this new task, but none of them show promising results because there are many fundamental differences between our task and previous ones. A novel Interactive Navigator-Pointer model is also proposed that provides a strong baseline on the task. The proposed model especially achieves the best performance on the unseen test split, but still leaves substantial room for improvement compared to the human performance.

1. Introduction

REVERIE introduces a practical embodied vision-and-language task in which agents navigate unfamiliar indoor environments to identify remote objects from concise natural-language instructions. The paper contributes a benchmark dataset and an interactive Navigator-Pointer baseline, while showing that existing methods perform poorly on this task.

  • Task: REVERIE requires an agent to navigate toward and identify a remote target object specified by a practical natural-language instruction in a real indoor environment.The target is not visible at the starting location, and success requires returning a bounding box around the specified object.
  • Task: The task emphasizes explicit object grounding, unlike VLN’s point navigation and EQA’s question answering, making its concise instructions closer to practical human commands.The instructions describe tasks such as locating objects rather than requiring manipulation.
  • Dataset: The benchmark extends the Matterport3D Simulator with object annotations, including labels and bounding boxes projected across viewpoints and camera angles.This supports evaluation at different navigable locations and viewing directions.
  • Dataset: The dataset contains 10,567 panoramas across 90 buildings, 4,140 target objects, and 21,702 crowd-sourced instructions averaging 18 words.The collected instructions include phenomena such as spatial relations, imperatives, dangling modifiers, and co-references.
  • Evaluation: Combining state-of-the-art navigation and referring-expression models yields no promising results, motivating an interactive Navigator-Pointer model as a stronger baseline.The study also reports human performance to quantify the remaining machine–human gap.

2. Related Work

REVERIE differs from prior embodied vision-and-language tasks by requiring navigation to a remote object and explicit object grounding from concise, high-level instructions. Its dataset and evaluation support viewpoint-aware object localization in realistic indoor environments.

  • Referring Expression Comprehension: Referring expression tasks localize an object in a single image, whereas REVERIE requires finding an initially unseen object after navigation.The target may only be accessed after reaching a closed location.
  • Viewpoint-Aware Grounding: REVERIE introduces viewpoint variation because candidate objects span panoramas and their visual appearance, bounding-box size, and aspect ratio can change across views.The simulator projects object annotations onto images from different viewpoints and camera angles.
  • Vision-and-Language Navigation: Unlike conventional VLN, REVERIE uses semantic-level commands for practical object-finding tasks instead of step-by-step navigation instructions.The instructions resemble references such as “the cold tap in the first bedroom on level two.”
  • Comparison with Embodied Tasks: REVERIE evaluates explicit target-object grounding rather than only point navigation or question answering.Success requires selecting a target bounding box, with IoU ≥0.5 when candidate boxes are unavailable.

4. The Interactive Navigator-Pointer Model

The Interactive Navigator-Pointer model jointly navigates toward a remotely referenced object and grounds the target through interaction between navigation and referring-expression modules.

  • 4. The Interactive Navigator-Pointer Model: The model contains navigator, pointer, and interaction modules that use referring-expression information to guide action prediction.The interaction module sends pointer-derived information to the navigator, enabling more accurate action prediction.
  • 4.1. The Navigator Module: The navigator uses FAST-short with an LSTM, attention, and backtracking to select viewpoints and determine the stopping location.It maintains candidate and ending queues, then selects the viewpoint with the largest accumulated logit as the actual stop location.
  • 4.2. The Pointer Module: The pointer uses MAttNet to decompose instructions into subject, location, and relationship components for object matching.Object representations include in-box appearance, relative location, and surrounding-object relationships before producing a weighted matching score.
  • 4.3. The Interaction Module: The interaction module selects the top-3 objects in each candidate view and encodes their labels and visual regions into candidate viewpoint features.These features are concatenated with viewpoint features and sent to the navigator; the pointer acts as hard attention over target-related objects.
  • 4.4. Training: Training combines navigation and referring-expression objectives, including action-selection cross-entropy, progress-monitor mean squared error, and ranking loss.The navigation loss balances action selection and progress monitoring, while the referring-expression loss ranks positive pairs above negative pairs.

5. Experiments

Experiments evaluate baseline, state-of-the-art, human, and referring-expression-only performance on REVERIE. Results show substantial task difficulty, while interactive navigation improves over non-interactive combinations.

  • 5. Experiments: Experiments compare baseline and state-of-the-art navigation methods combined with MAttNet, and also measure human and referring-expression-only performance.The referring-expression-only setting places the agent at the ground-truth target location before object comprehension.
  • 5.1. Baselines: Random achieves around 1% REVERIE success, while sequence-to-sequence baselines degrade substantially on unseen splits.The Shortest model achieves perfect performance because it receives the ground-truth path to the goal.
  • 5.2. REVERIE Experimental Results: 7.07% is the best REVERIE success rate for combining state-of-the-art navigation and MAttNet on the test split, versus 77.84% human performance.FAST-Short’s navigation SPL drops from 43% on R2R Val-UnSeen to 6.17% on REVERIE Val-UnSeen.
  • 5.2. REVERIE Experimental Results: Language-only FAST-Lan-Only shows a large drop on seen and unseen splits, indicating that both language and visual information are needed.This tests whether the dataset can be solved effectively from instruction language alone.
  • 5.2. REVERIE Experimental Results: The interactive model consistently outperforms non-interactive models and gains 4.2% on the test split.FAST-Short serves as the ablated model without the proposed interaction module.
  • 5.3. Referring Expression-Only: Referring-expression-only models achieve around 50% test accuracy, still leaving a 40% gap to human performance.This isolates object comprehension at the ground-truth goal viewpoint rather than evaluating navigation and grounding jointly.

6. Conclusion

REVERIE advances human-robot collaboration by testing high-level natural-language navigation and target-object identification in previously unseen real-image-rendered buildings. Results show that this combined task remains challenging and far from human performance.

  • REVERIE evaluates an agent’s ability to follow high-level natural-language instructions, navigate, and identify a target object in previously unseen rendered real buildings.
  • 80% accuracy on ReferCOCO contrasts with the difficulty of applying state-of-the-art referring-expression models to REVERIE.
  • Existing vision-and-language methods can be plugged into REVERIE, but understanding and executing high-level instructions remains significantly challenging.
  • The combination of instruction navigation and referring-expression comprehension shows a large gap to human performance.

Supplements

The supplementary material provides detailed explanations of evaluation metrics, collected data, data-collection tools, human testing, and visualizations of REVERIE results.

  • The supplementary material explains evaluation metrics, collected-data examples, data-collection tools, human testing, and visualizations of REVERIE results.

1. Evaluation Metrics

REVERIE evaluates navigation and object-grounding performance using success, oracle success, path efficiency, trajectory length, and bounding-box overlap.

  • Navigation Success requires the target object to be observable at the agent’s stopping viewpoint.Objects are considered visible within 3 meters of the current location.
  • Navigation Oracle Success counts navigation as successful when the target object is observable from any passed viewpoint.
  • Navigation SPL measures navigation success weighted by the length of the navigation path.The metric uses task success, shortest start-to-goal distance, and the agent’s path length.
  • Navigation Length is the trajectory length measured in meters.
  • REVERIE Success requires the predicted bounding box to achieve IoU (intersection over union) ≥0.5 with the ground-truth box.

2. Typical Samples of The REVERIE Task

Typical REVERIE samples illustrate variation in object categories, goal regions, path instructions, and target-object referring expressions.

  • Figure 1 presents typical REVERIE samples spanning diverse object categories, goal regions, path instructions, and target-object referring expressions.

3. Data Collecting Tools

The REVERIE data-collection tool is WebGL-based and supports workers with real-time reference information and instructional materials during annotation.

  • The REVERIE dataset is collected using a WebGL-based tool shown in Figures 2 and 3.
  • The interface provides location-dependent information about the agent’s level, region, and surrounding building structure.
  • At the goal location, the tool highlights the target object with a red 3D rectangle and displays its category label and same-category object count.
  • Text and video instructions help workers produce high-quality annotations.

4. Human Performance Test

The human-performance test measures whether workers can navigate to a remote object’s goal location and identify the target from nearby candidates.

  • Workers receive an instruction about a remote object, navigate to its goal location, and select the target object from candidate objects.
  • The testing interface lets workers look around and move forward or backward by dragging or clicking.

5. Visualisation of REVERIE Results

REVERIE examples combine object categories, goal regions, path instructions, and referring expressions, while result visualizations show trajectories and object grounding from representative methods.

  • The visualized REVERIE results include the state-of-the-art FAST-short method and the baseline R2R-SF method.
  • Dataset samples vary in object category, goal region, path instruction, and object referring expression.
  • The collection interfaces include worker instructions, assistant information, and a user input field.
  • Human workers navigate to a goal location before identifying the target object, with only objects within 3 meters highlighted.
  • A sample instruction asks the worker to navigate to a closet and identify a pile of clothes using its shelf position.
  • Figure 5 visualizes trajectories and referring-expression grounding for three typical methods.
Loading 1904.10151v2…