PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating to Objects Specified by Images

Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jitendra Malik, Stefan Lee, Devendra Singh Chaplot

arXiv:2304.01192v1cs.CVcs.RO

TL;DR

Navigating to a particular object instance from an image requires both visual disambiguation and exploration of an unknown environment. This paper uses a modular, zero-fine-tuning system for the task, achieving 56% success versus 25% for the best baseline on HM3D InstanceImageNav.

  • Problem

    Image Goal Navigation must identify a particular object instance from an image and explore an unknown environment to find it.

    Source: §1 · p. 1

  • Method

    A modular system decomposes InstanceImageNav into exploration, instance re-identification, goal localization, and local navigation using off-the-shelf components with zero fine-tuning.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    56% success versus 25% for the best baseline on HM3D InstanceImageNav, and 2.3x better in Val than OVRL-v2-IIN.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The framework can serve as a strong baseline for evaluating trained navigation policies and as a catalyst for modular semantic navigators.

    Source: §6 · p. 8

    Benchmark success requires stopping within 1.0 m of the depicted instance, which must be oracle-viewable by turning and looking up or down.

    Source: §3 · p. 3

Abstract

from arXiv · show

Images are a convenient way to specify which particular object instance an embodied agent should navigate to. Solving this task requires semantic visual reasoning and exploration of unknown environments. We present a system that can perform this task in both simulation and the real world. Our modular method solves sub-tasks of exploration, goal instance re-identification, goal localization, and local navigation. We re-identify the goal instance in egocentric vision using feature-matching and localize the goal instance by projecting matched features to a map. Each sub-task is solved using off-the-shelf components requiring zero fine-tuning. On the HM3D InstanceImageNav benchmark, this system outperforms a baseline end-to-end RL policy 7x and a state-of-the-art ImageNav model 2.3x (56% vs 25% success). We deploy this system to a mobile robot platform and demonstrate effective real-world performance, achieving an 88% success rate across a home and an office environment.

1. Introduction

The introduction motivates image-specified navigation to particular object instances and presents a modular system evaluated in simulation and on a mobile robot.

1. Introduction

  • Image-specified goals let users target a particular object instance, but require disambiguating similar-looking objects and exploring to find the goal.The example is locating a particular bed while exploring bedrooms.Source: §1 · p. 1
  • 56% vs. 25% success on HM3D InstanceImageNav; the system also outperforms an end-to-end RL baseline 7-fold and a state-of-the-art ImageNav model 2.3x.The 56% vs. 25% comparison is against the best baseline; the RL baseline achieves 8.3% success.Source: §1 · p. 2
  • The system decomposes the task into exploration, instance re-identification, goal localization, and local navigation, using off-the-shelf components without fine-tuning.The overview instantiates these stages with frontier exploration, feature matching, masked feature projection, and analytical planning.Source: §1 · p. 2, Figure 2 · p. 2
  • The system is deployed on a mobile robot in two real-world environments, where it achieves 88% success and is shown finding a chair 10 m away.The figure pairs the image-goal task with a real-world deployment example.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors connect modular navigation with sample efficiency and real-world execution, contrasting it with end-to-end methods’ sample complexity, overfitting, and sim-to-real limitations.Their system uses zero fine-tuning and reports effective real-world performance.Source: §1 · p. 2

2. Related Work

Related work spans learned ImageNav policies, modular semantic navigation, and object instance re-identification; this paper applies a modular approach to image-specified instances.

2. Related Work

  • ImageNav uses an image as a visual goal, commonly sampled randomly in previously unseen environments; many approaches learn end-to-end vision-to-action policies with deep reinforcement learning.The paper studies the instance-based setting, where goal images depict object instances [24].Source: §2 · pp. 2–3
  • Modular semantic navigation decomposes tasks such as ObjectNav and open-vocabulary object localization; prior work also reports modular systems avoiding visual sim-to-real degradation [18].This paper extends modular navigation to image-specified targets.Source: §2 · p. 3
  • Object instance re-identification asks whether an image depicts the same object as an anchor image, with prior methods ranging from local features to deep descriptors and transformer re-ranking.The paper addresses embodied re-identification, acquiring novel instance views through egomotion.Source: §2 · p. 3
  • The paper uses SuperGlue-based keypoint matching for embodied re-identification and shows that the matches also inform instance localization.Its task differs from standard image retrieval because novel views are acquired through egomotion.Source: §2 · p. 3

3. InstanceImageNav Task Setup

The task setup specifies an agent starting at a random pose in an unexplored indoor environment and navigating to the object instance shown in an RGB goal image.

3. InstanceImageNav Task Setup

  • InstanceImageNav places an agent at a random pose in an unexplored indoor environment and directs it to the object instance depicted in an RGB image [24].The setup follows the ImageNav track of the 2023 Habitat Navigation Challenge to support comparison and transfer between simulation and reality.Source: §3 · p. 3
  • At each step, observations contain the goal RGB image, egocentric RGB and depth images, and pose relative to the starting pose.The goal-image camera is independent of the agent camera; the goal image is 512 × 512 RGB, while egocentric RGB and depth are 640 × 360.Source: §3 · p. 3
  • The discrete action set is MOVE-FORWARD, TURN-LEFT, TURN-RIGHT, and STOP; forward motion is 0.25 m and turns are 30°.Source: §3 · p. 3
  • Success requires issuing STOP within 1.0 m Euclidean distance of the depicted instance, which must be oracle-viewable by turning or looking up or down.Source: §3 · p. 3
  • Real-world execution uses a Hello Robot Stretch, modeled in simulation as a 1.41 m-high, 0.17 m-radius cylinder with a forward-facing RGBD camera at 1.31 m.Source: §3 · p. 3

4. Method

The method factors InstanceImageNav into exploration, goal-instance re-identification, goal localization, and local navigation, then instantiates the stages in a modular system.

4.1. Proposed Modules

  • Mod-IIN operationalizes this factorization as a purely modular system without retrained or fine-tuned components.It is evaluated in simulation and reality.Source: §4.1 · p. 4, §4 · p. 4
  • Frontier-based exploration greedily visits the nearest frontier on a 2D occupancy map updated from egocentric depth and pose.Frontiers mark the boundary between explored and unexplored regions.Source: §4.1 · p. 4
  • Re-identification extracts SuperPoint keypoints and descriptors, matches goal and egocentric features with SuperGlue, and thresholds summed match confidence.The binary classifier is applied at each time step.Source: §4.1 · p. 4
  • τ = 24.5 maximizes F-measure on a pairwise goal-image dataset used to balance false positives and false negatives.The dataset contains 121 objects and 2,270 goal images.Source: §4.1 · pp. 4–5, Figure 3 · p. 5
  • To avoid targeting background or adjacent objects, the system masks the goal instance, projects matched features through depth into the world, and adds voxelized goal points to the map.Detic supplies the goal-image mask; the resulting target points are combined with the exploration map.Source: §4.1 · p. 5
  • Local navigation uses an incremental fast-marching planner to reach the nearest reachable frontier or goal point from a partial occupancy map.Source: §4.1 · p. 5

5. Experiments

Experiments compare Mod-IIN with prior methods, examine module variations and failure causes, and demonstrate the system in simulation and real-world environments.

5.1. Experimental Setup

  • The simulation uses HM3D-SEM scenes and a validation split of 795 unique object instances, reporting SR, SPL, NE, and Max-ST.The IIN RL baseline is trained from scratch, while OVRL-v2 is evaluated zero-shot and after fine-tuning for InstanceImageNav.Source: §5.1 · pp. 5–6
  • Re-identification ablations compare confidence-sum and match-count keypoint classifiers with ResNet- and CLIP-based image encoders.The confidence-sum classifier thresholds summed correspondence confidence; the match-count variant thresholds the number of matched pairs.Source: §5.1 · p. 6
  • Localization ablations compare projecting matched keypoints within a Detic instance mask against a central crop, class-based localization, and oracle masks.The oracle localization variant projects egocentric pixels observing the goal instance.Source: §5.1 · pp. 6–7

5.2. Simulation Results

  • Mod-IIN achieves 0.561 SR versus 0.083 SR for the IIN RL baseline and outperforms OVRL-v2-IIN by 2.3x in validation.Mod-IIN requires zero fine-tuning, whereas the IIN RL baseline was trained for 3.5 billion steps across 64 GPUs.Source: Table 1 · p. 6, §5.2 · p. 7

5.3. Failure Analysis.

  • Among 100 failed episodes, 41% were re-identification false negatives and 23% false positives, which often matched background features.The authors suggest improved re-identification, views closer to the goal image, or foreground-background reasoning as possible mitigations.Source: Figure 4 · p. 7, §5.3 · pp. 7–8
  • Exploration, localization, and local-navigation errors accounted for 13%, 12%, and 11% of failures, respectively.Exploration errors occurred when the goal was not viewed within the time budget; localization errors involved projecting points outside the goal instance.Source: §5.3 · p. 8

5.4. Qualitative Example

  • In a simulation episode, the agent navigated to the correct dining chair among identical objects after projecting goal keypoints at step 229.The agent reached the goal and stopped at step 237.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5. Real-World Deployment

  • The Hello Robot Stretch succeeded in 7 of 8 episodes across an office and an apartment, using five image goals captured by mobile phone.The goals depicted a couch, chair, and plant in the office, and a bed and plant in the apartment.Source: §5.5 · p. 8, Figure 6 · p. 8

6. Discussion

The authors frame InstanceImageNav as four modular sub-tasks and report that their zero-fine-tuning system outperforms existing learned policies and transfers to real-world execution. They identify memory-less detection and localization as a limitation for persistent environments.

6. Discussion

  • The system decomposes InstanceImageNav into exploration, goal re-identification, goal localization, and local navigation, using off-the-shelf components without fine-tuning.The authors present it as a baseline for trained navigation policies and a framework for modular semantic navigators.Source: §6 · p. 8
  • The authors report that the system outperforms existing state-of-the-art end-to-end learned policies and transfers to real-world execution.Source: §6 · p. 8
  • Memory-less detection and localization limit the system’s fit for sequential tasks in persistent environments, which would benefit from compressed goal-image memory.Source: §6 · p. 8
  • The Oregon State effort was partly supported by DARPA’s Machine Common Sense program; the authors state that the views are not official sponsor or US Government policy.Source: §6 · p. 8

Supplementary Material

A threshold search on 400 training episodes found that the threshold selected by maximal F-measure also optimized downstream system success.

A. Verifying Threshold Optimality

B. Re-ID Thresholds of Alternate Methods

The alternate re-identification methods show that keypoint-based approaches score higher than ResNet and CLIP, while a static central crop is also described as a goal-localization alternative.

B. Re-ID Thresholds of Alternate Methods

  • Keypoint-based methods achieve higher PR AUC and maximal F-measure than ResNet and CLIP; Keypoint-Conf’s 0.972 versus Keypoint-Match’s 0.962 maximal F-measure aligns with a 0.022 SR downstream performance difference.Source: B. Re-ID Thresholds of Alternate Methods · p. 11

C. Crop-Projected Goal Localization Details

  • Crop-Projected replaces an instance-specific segmentation mask with a static central crop shared across episodes, projecting matched keypoints inside it to the goal map channel.The reported effective crop covers the central 1/3 of the image and extends down to a height of 1/8.Source: C. Crop-Projected Goal Localization… · p. 11

D. Implementation Details

The implementation uses pretrained ResNet, CLIP, SuperGlue, and Detic components, with a Stretch V1 robot and RGBD camera for real-world experiments.

D. Implementation Details

  • Model variations use ImageNet-1K-pretrained ResNet50 or ViT-B/32 CLIP weights, and the implementation uses pretrained SuperGlue with indoor-trained weights from ScanNet.Source: D. Implementation Details · p. 11
  • Figure 7 identifies threshold performance on a Train-split subset, while Figure 8 shows static central crops on Val-split goal images.Source: Figure 7 · p. 11, Figure 8 · p. 11
  • Detic uses its default 21K-trained model with a custom vocabulary of chair, sofa, bed, toilet, potted plant, and TV monitor.Source: D. Implementation Details · p. 11
  • Real-world experiments use a Stretch V1 robot with its default Intel RealSense D435i RGBD camera.Source: D. Implementation Details · p. 11

E. Real-World Qualitative Example

A real-world episode illustrates the method navigating from a hallway to a pictured chair, then re-identifying, localizing, and reaching it despite real-world observation and pose-estimation noise.

E. Real-World Qualitative Example

  • The agent traverses a furnished office, passes a couch, enters the chair’s room, re-identifies and localizes the chair, then navigates to it.The example uses RGBD observations from Stretch and poses estimated with Hector SLAM on ROS.Source: E. Real-World Qualitative Example · p. 11
  • Figure 9 plots precision-recall and F-measure curves for each goal instance re-identification method.Source: Figure 9 · p. 12
  • The goal image comes from a mobile-phone camera at a view chosen by the task-issuing user, without reliance on robot hardware.The passage presents this as demonstrating ease of goal specification and method robustness.Source: E. Real-World Qualitative Example · p. 12

F. Complete Real-World Results

Table 3 presents the complete real-world episode descriptions and results for Mod-INN.

F. Complete Real-World Results

Related papers

Loading 2304.01192v1…