PaperlayerView exact revision on arXiv

Source-linked AI summary

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel

arXiv:1904.10151v2cs.CVcs.CL

TL;DR

REVERIE targets the challenge of following natural-language instructions to navigate to and identify remote objects in real indoor environments. The paper introduces a task and dataset, evaluates existing methods, and proposes an Interactive Navigator-Pointer baseline; the best reported REVERIE success rate was 7.07% on the test split, versus 77.84% human performance.

  • Problem

    The paper addresses how to evaluate agents’ ability to follow high-level natural-language instructions to navigate to and identify remote objects in real indoor environments.

    Source: §1 · p. 2, §6 · p. 8

  • Method

    The authors introduce the REVERIE task and benchmark dataset, evaluate existing navigation and referring-expression methods, and propose an Interactive Navigator-Pointer baseline.

    Source: §1 · p. 2

  • Results

    7.07% REVERIE success rate on the test split was achieved by combining state-of-the-art navigation and referring-expression models, versus 77.84% human performance.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    Following high-level instructions through navigation and target-object identification remains a challenging task with a large gap to human performance.

    Source: §6 · p. 9

Abstract

from arXiv · show

One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language. Overcoming this challenge requires the ability to perform a wide variety of complex tasks in response to multifarious instructions from humans. In the hope that it might drive progress towards more flexible and powerful human interactions with robots, we propose a dataset of varied and complex robot tasks, described in natural language, in terms of objects visible in a large set of real images. Given an instruction, success requires navigating through a previously-unseen environment to identify an object. This represents a practical challenge, but one that closely reflects one of the core visual problems in robotics. Several state-of-the-art vision-and-language navigation, and referring-expression models are tested to verify the difficulty of this new task, but none of them show promising results because there are many fundamental differences between our task and previous ones. A novel Interactive Navigator-Pointer model is also proposed that provides a strong baseline on the task. The proposed model especially achieves the best performance on the unseen test split, but still leaves substantial room for improvement compared to the human performance.

1. Introduction

REVERIE asks an agent to follow a natural-language instruction through an unfamiliar indoor environment and identify a remote object, combining language understanding, navigation, and visual grounding. The paper introduces a large annotated dataset and an interactive navigator-pointer baseline, while finding existing methods inadequate and measuring the human-machine gap.

Introduction

  • Children can apply knowledge of familiar environments and interpret language in unfamiliar settings, capabilities that robots currently lack and that limit their applications.The paper motivates REVERIE as a way to advance real-world vision-and-language research.Source: §1 · pp. 1–2
  • REVERIE requires an agent to navigate toward a remote target object from a natural-language instruction and return a bounding box identifying it.The task tests inference about the object's likely location as well as language-conditioned visual identification.Source: §1 · p. 2
  • Unlike VLN [1] and EQA [7], REVERIE evaluates explicit object grounding, while RefExp [22] selects an object in one image rather than requiring navigation to discover it.The target is absent from the initial view, leaving at least an order of magnitude more object candidates than RefExp.Source: §1 · p. 2
  • The dataset contains 10,567 panoramas across 90 buildings, 4,140 target objects, and 21,702 crowd-sourced instructions averaging 18 words.It extends the Matterport3D Simulator [1] with object labels and bounding boxes; instructions include spatial relations, long or dangling modifiers, and coreferences.Source: §1 · p. 2, Table 1 · p. 2
  • State-of-the-art navigation and referring-expression methods combined for REVERIE show no promising results; the paper proposes an interactive navigator-pointer baseline and reports human performance on the test set.The human evaluation is intended to quantify the machine-human gap.Source: §1 · p. 2

2. Related Work · 3. The REVERIE Dataset

REVERIE asks an agent to follow a concise natural-language instruction through an unseen indoor environment and precisely localize a remote object. The sections distinguish this task from related work and describe its evaluation, simulator, data collection, and dataset characteristics.

2. Related Work

  • Unlike image-based referring-expression tasks, REVERIE requires navigation to a remote object, selection across viewpoint panoramas, and recognition despite changing observation angles.Prior referring-expression work matches images and expressions or decomposes expressions into appearance, location, and object relationships [29]; REVERIE adds remote localization and viewpoint variation.Source: §2 · p. 3
  • REVERIE targets objects rather than locations, uses semantic-level commands, and requires bounding-box localization, unlike navigation-only, oracle-assisted, and answer-or-action tasks.Its instructions resemble “the cold tap in the first bedroom on level two” rather than step-by-step directions. Unlike related oracle-assisted object-finding tasks [23] [24] [25] and TOUCHDOWN’s outdoor location finding [5], REVERIE tests whether an agent follows an instruction alone; unlike EQA [7] [10], it outputs a precise object location.Source: §2 · p. 3

3. The REVERIE Dataset

  • The dataset section covers the task definition, evaluation metrics, simulator, data-collection policy, and analysis of collected instructions.These components describe the REVERIE task and dataset as a whole.Source: §3 · p. 3

3.1. The REVERIE Task

  • The target is absent from the starting scene; the agent navigates using panoramic observations and may output a bounding box once per episode at any step.The agent can stop at a navigable viewpoint or remain at its current viewpoint, and localization is successful if the target is identified within 3 meters, including from different viewpoints or camera views.Source: §3.1 · p. 4

3.2. Evaluation Metrics

  • REVERIE success requires selecting the correct candidate box or achieving IoU ≥0.5; navigation succeeds only when the agent stops within 3 meters of the target.Navigation is also measured by success rate, oracle success rate, success rate weighted by path length (SPL), and path length in meters [1].Source: §3.2 · p. 4

3.3. The REVERIE Simulator

  • The Matterport3D-based simulator lets an embodied agent move between panoramic viewpoints and adjust its camera pose, rendering a view at each viewpoint.Object annotations account for visibility and bounding-box changes as the camera moves or rotates; the dataset includes 20k object annotations.Source: §3.3 · pp. 4–5

3.4. Data Collection

  • More than 1,000 workers contributed around 2,648 hours to collect three referring expressions per target object using an interactive WebGL simulator.Workers saw a path animation, were shown an object at the goal, and could look around; room and object information helped them produce unambiguous commands.Source: §3.4 · p. 5

3.5. Dataset Analysis

  • REVERIE contains 21,702 instructions averaging 18 words, and 4,140 target objects across 489 categories; its splits include unseen validation and test scenes.The instructions average 18 words, compared with 29 for R2R [1] and 8 for RefCOCOg [30]. The training set has 59 scenes and 10,466 instructions; validation has 63 scenes and 4,944 instructions, including an unseen split of 10 scenes and 3,573 instructions; the test set has 6,292 instructions in 16 unseen scenes.Source: §3.5 · p. 5

4. The Interactive Navigator-Pointer Model

The model couples navigation and object pointing so that visual grounding can guide action prediction, while navigation can help the pointer reach the target location.

4.1. The Navigator Module

  • FAST-short [15] uses an attention-based sequence-to-sequence LSTM with backtracking; it selects the next viewpoint by accumulated logits and stops at the highest-logit location.Its visual and textual co-grounding signal is informed by, while the interaction module updates the visual features used for panoramic candidates.Source: §4.1 · p. 6

4.2. The Pointer Module

  • MAttNet [29] attends to subject, location, and relationship phrases, scores objects using in-box appearance and relative location, and combines the matching scores with a weighted sum.The location representation uses relative offsets and area ratios to up to five surrounding objects of the same category.Source: §4.2 · p. 6

4.3. The Interaction Module

  • The interaction module selects the pointer’s top three objects per candidate view, encodes their labels with a bidirectional LSTM, and sends concatenated text and visual features to the navigator as hard attention.Visual features are averaged ResNet FC7 outputs for the selected object regions; the pointer highlights target-related objects for each viewpoint.Source: §4.3 · p. 7

4.4. Loss Functions

  • The joint objective combines cross-entropy action selection and mean-squared-error progress monitoring with a referring-expression ranking loss.The action and progress terms form the navigation loss; the ranking loss trains object-expression matching.Source: §4.4 · p. 7

5. Experiments

Experiments show that interactive navigation and object pointing improve REVERIE results, while the task remains far more challenging for models than for humans. Results also show that navigation and referring-expression comprehension each contribute to the challenge.

5.1. Implementation Details

  • For training, 640×480 simulator images and goal-viewpoint images with target-object boxes train the pointer, whose object information then assists navigator training.Goal-viewpoint images are used where the target object is visible.Source: §5.1 · p. 7

5.2. REVERIE Experimental Results

  • Random achieves around 1% success; R2R-TF and R2R-SF perform better on Val-Seen but drop on unseen splits, while Student-Forcing generally outperforms Teacher-Forcing.Shortest performs perfectly because it is given the ground-truth path to the goal.Source: §5.2 · p. 8
  • FAST navigation with MAttNet achieves 7.07% REVERIE success on the test split, far below human performance of 77.84%.The paper also reports that FAST-Short navigation SPL on Val-UnSeen falls from 43% on R2R to 6.17% on REVERIE.Source: §5.2 · p. 8, Table 3 · p. 8
  • FAST-Lan-Only drops substantially on both seen and unseen splits, indicating that the task requires jointly considering language and visual information.The model receives only the instruction as input.Source: §5.2 · p. 8
  • The interactive navigator–pointer model consistently outperforms non-interactive models, gaining 4.2% on the test split over FAST-Short.FAST-Short serves as the ablated model without the interaction module.Source: §5.2 · p. 8, Table 3 · p. 8
  • At the ground-truth goal viewpoint, referring-expression models achieve around 50% test accuracy, yet remain about 40% below humans.These results are substantially better than the joint navigation-and-referring-expression results.Source: §5.2 · p. 8, Table 4 · p. 8

6. Conclusion

REVERIE introduces a task and dataset for evaluating agents that navigate to targets in previously unseen real-image buildings from high-level natural-language instructions. Its conclusions emphasize that executing such instructions and combining navigation with referring-expression comprehension remain challenging.

6. Conclusion

  • REVERIE introduces a task and dataset evaluating agents’ ability to navigate unseen real-image buildings and identify target objects from high-level natural-language instructions.The paper describes it as the first task to evaluate this capability.Source: §6 · pp. 8–9
  • Combining instruction navigation with referring-expression comprehension remains challenging, and baseline and Navigator-Pointer results indicate a need for further research.The paper concludes that this combination has a large gap to human performance.Source: §6 · p. 9
  • Existing vision-and-language methods can be readily applied to REVERIE, but understanding and executing high-level instructions remains a significant challenge.The paper also identifies a large gap between task performance and human performance.Source: §6 · p. 9

Supplements

The supplement documents evaluation metrics, collected data and collection tools, human testing, and visualizations of REVERIE results.

Supplements

  • The supplement explains evaluation metrics, illustrates collected data and collection tools, describes human testing, and visualizes several REVERIE results.Source: Supplements · p. 11

1. Evaluation Metrics

The section defines navigation success by visibility of the target at the stopping viewpoint, with a 3-meter visibility threshold, and specifies oracle, path-efficiency, trajectory-length, and task-level success metrics.

1. Evaluation Metrics

  • Navigation Success requires observing the target at the stop viewpoint, with objects treated as visible within 3 meters of the current location.The 3-meter threshold is intended to encourage agents to approach the target more closely.Source: §1 · p. 11
  • Navigation Oracle Success requires that the target be observable at any viewpoint the agent passed.Source: §1 · p. 11
  • Navigation SPL weights navigation success by path length, using the shortest start-to-goal distance and the agent's trajectory length.Source: §1 · p. 11
  • Navigation Length is the trajectory length in meters.Source: §1 · p. 11
  • REVERIE Success requires the output bounding box to have IoU of at least 0.5 with the ground-truth box.Source: §1 · p. 11

2. Typical Samples of The REVERIE Task

Figure 1 presents typical REVERIE task samples that illustrate diversity in object category, goal region, path instruction, and target-object referring expression.

2. Typical Samples of The REVERIE Task

  • Figure 1 illustrates diverse object categories, goal regions, path instructions, and target-object referring expressions in typical REVERIE samples.Source: §2 · p. 11

3. Data Collecting Tools

The authors built a WebGL-based data-collection tool that gives workers location-specific reference information and guidance for producing high-quality annotations.

3. Data Collecting Tools

  • The WebGL tool updates workers with the agent’s level and region, highlights the target object at the goal, and shows its label and same-category object count.It also reports how many regions in the build have the same function as the current region.Source: §3 · p. 11
  • Text and video instructions help workers understand how to produce high-quality annotations.Source: §3 · p. 11

4. Human Performance Test

The human performance test uses a WebGL tool to measure the machine–human performance gap. Workers navigate toward an instructed remote object and select it from object candidates.

4. Human Performance Test

  • The WebGL-based tool tests human performance to assess the machine–human performance gap.The paper shows the tool in Figure 4.Source: §4 · p. 11
  • Workers receive an instruction about a remote object and select one target from a range of object candidates.Source: §4 · p. 11
  • Workers navigate to the goal location by looking around and moving forward or backward via dragging or clicking.Source: §4 · p. 11

5. Visualisation of REVERIE Results

The section presents visual examples of REVERIE results from FAST-short and R2R-SF, showing navigation trajectories and object-referring-expression grounding.

5. Visualisation of REVERIE Results

  • Figure 5 visualizes several REVERIE results from FAST-short and R2R-SF.The figure caption describes trajectories and referring-expression grounding for three typical methods.Source: §5 · p. 11, Figure 5 · p. 16
  • In Figure 5, colored dots denote reachable locations, while different colors distinguish locations in different Matterport regions.Source: Figure 5 · p. 16

Related papers

Loading 1904.10151v2…