Sim-to-Real Transfer for Vision-and-Language Navigation

Peter Anderson; Ayush Shrivastava; Joanne Truong; Arjun Majumdar; Devi Parikh; Dhruv Batra; Stefan Lee

Source-linked AI summary

Sim-to-Real Transfer for Vision-and-Language Navigation

Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, Stefan Lee

arXiv:2011.03807v1Nov 7, 2020cs.CVcs.CLcs.RO

TL;DR

The paper asks whether a robot can follow unconstrained natural-language navigation instructions in an unseen environment, despite the gap between simulated discrete actions and continuous robot control. It transfers a simulation-trained VLN agent using a subgoal model and evaluates it in a paired office simulator and physical setting, finding stronger transfer with prior mapping than without it.

Problem
The paper studies whether a robot can follow unconstrained natural-language navigation instructions in an unseen environment, where discrete simulated actions must connect to continuous physical control.
Source: §1 · p. 1
Method
The authors transfer an R2R-trained VLN agent to a robot, using a subgoal model to propose nearby waypoints from RGB images and laser scans, plus domain randomization to mitigate visual differences.
Source: §1 · p. 1, §4 · p. 5, §6 · p. 8
Results
With an occupancy map and navigation graph, success was 46.8% on the robot versus 55.9% in simulation; without prior mapping, success was 22.5%.
Source: §1 · p. 2, §6 · p. 8, §5 · p. 8
Takeaways & Limitations
The results suggest that language groundings learned in simulation can transfer to a physical environment not seen in training when an occupancy map and navigation graph are prepared in advance.
Source: §6 · p. 8
In the no-prior-map cold-start setting, transfer is much less reliable, with subgoal prediction errors accounting for most of the success-rate drop.
Source: §5 · p. 8, §6 · p. 8

Abstract

from arXiv · show

We study the challenging problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions. Recent work on the task of Vision-and-Language Navigation (VLN) has achieved significant progress in simulation. To assess the implications of this work for robotics, we transfer a VLN agent trained in simulation to a physical robot. To bridge the gap between the high-level discrete action space learned by the VLN agent, and the robot's low-level continuous action space, we propose a subgoal model to identify nearby waypoints, and use domain randomization to mitigate visual domain differences. For accurate sim and real comparisons in parallel environments, we annotate a 325m2 office space with 1.3km of navigation instructions, and create a digitized replica in simulation. We find that sim-to-real transfer to an environment not seen in training is successful if an occupancy map and navigation graph can be collected and annotated in advance (success rate of 46.8% vs. 55.9% in sim), but much more challenging in the hardest setting with no prior mapping at all (success rate of 22.5%).

1 Introduction

The paper transfers a simulation-trained Vision-and-Language Navigation agent to a physical robot, addressing the gap between discrete simulated actions and continuous robot control. In a paired office evaluation, transfer is more successful with prior mapping than in the no-map setting.

1 Introduction

The study addresses instruction following in unseen environments, where simulation-only or component-level evaluations may miss real sensing, actuation, localization, and integration challenges.The authors transfer an R2R-trained VLN agent to a physical robot for a full-system evaluation.Source: §1 · p. 1
The subgoal model uses 360° RGB images and laser scans to identify nearby navigable waypoints that the VLN agent can consider as high-level actions.The framework also uses domain randomization to mitigate visual differences between simulation and reality.Source: §1 · pp. 1–2
The authors digitize and annotate a 325m2 office environment, collecting 111 natural-language instructions over 1,334m of trajectories for parallel simulator and robot evaluation.The simulator replica supports comparison in the same previously unseen test environment.Source: §1 · p. 2, Figure 1 · p. 2
With a map, success was 46.8% on the robot versus 55.9% in simulation; without prior mapping, success was 22.5%.The authors attribute the mapped-setting reductions to visual-domain differences (3.9%) and viewpoint differences (5.2%), and the no-map decline mainly to subgoal prediction errors.Source: §1 · p. 2
The contributions include an annotated simulator environment, a ROS-based transfer framework, a subgoal model, and an empirical analysis of sim-to-real differences.The framework interfaces a trained VLN agent with standard ROS components.Source: §1 · p. 2

2 Related Work

Prior language-guided robot evaluations often differ from this paper’s setting: they use constrained environments, training environments, limited commands, or tasks other than navigation. Meanwhile, VLN research has advanced in simulation, motivating investigation of transfer to physical platforms.

Instruction-Following Robots

Prior robot evaluations rarely test unconstrained navigation instructions in a building unseen during training, instead using artificial landmarks, training environments, outdoor settings, few commands, or manipulation tasks.Examples include the wheelchair of Hemachandra et al., quadcopters of Huang et al. and Blukis et al., and the robot teammate of Oh et al..Source: §2 · p. 3

Vision-and-Language Navigation (VLN)

In standard VLN, agents ground natural-language directions in visual observations and actions in photorealistic indoor simulations, with test environments unseen during training and no prior exploration.Recent work improves performance through speaker models, data augmentation, progress estimation, and backtracking.Source: §2 · p. 3

Pano-Simulators and Datasets

Panoramic simulation datasets now cover indoor and outdoor navigation tasks, making transfer of agents trained in these environments to physical platforms an open need.Examples include R2R/Matterport3D, StreetLearn, and language-navigation datasets such as TouchDown and Talk2Nav [44].Source: §2 · p. 3

3 Sim-to-Real Experimental Setting

The experimental setting pairs a physical office test site with a digitized simulator replica and a newly collected instruction dataset. Robot hardware, pose tracking, and standardized navigation metrics support comparison across real and simulated runs.

Coda Test Environment

Coda is a 325m2 shared office space with varied rooms, glass walls, reflections, and changing light, while its lack of personal items helps limit simulator-to-real drift over time.The space includes an elevator lobby, corridors, a lounge, and breakout areas.Source: §3 · p. 3

Simulator Construction

The simulator replica uses a Matterport scan, textured mesh, panoramic images, poses, and a visibility graph to construct a navigation graph for reachable viewpoints.Six unreachable viewpoints were excluded, and graph edges longer than 5m were removed.Source: §3 · p. 3

Navigation Instructions

The Coda dataset contains 111 instructions describing 37 sampled shortest-path trajectories, totaling 1,334m; each instruction was followed by a separate annotator for validation.After discarding the highest-error instruction per trajectory, average human follower success was 93%.Source: §3 · pp. 3–4

Robot Platform

The TurtleBot2 carries a 360° RGB camera and a 270° laser scanner, with standard ROS packages supporting mapping, localization, obstacle avoidance, and path planning.The VLN agent and subgoal model run remotely during evaluation.Source: §3 · p. 4

Evaluation Metrics

Success requires a final position within 3m of the goal, and evaluation reports trajectory length, navigation error, success, SPL, and path-fidelity metrics.NDTW and SDTW trajectories use 100 equally spaced points to remove sampling differences between simulation and robot runs.Source: §3 · p. 4

Robot Pose Tracking

Robot pose is tracked with AMCL after constructing a 0.05m-resolution occupancy map and registering it to the Matterport3D coordinate frame.The estimated pose error is typically an order of magnitude below the 3m success radius.Source: §3 · p. 4

4 Adapting a VLN Agent to a Physical Robot

Transferring a simulator-trained VLN agent to a robot requires addressing visual differences and the gap between graph-based discrete actions and physical navigation. The approach combines domain randomization, a subgoal model, and ROS navigation components.

VLN Agent

The transferred agent uses panoramic 360° RGB observations and selects neighboring Matterport3D viewpoints about 2.1m away, whereas the robot requires physical motion.The simulator teleports the agent to the selected viewpoint, so transfer must address visual, action-space, navigation, and localization differences.Source: §4 · p. 4

Visual Domain Adaptation

6.4% of success rate is lost due to visual domain differences; training with random color jitter is used to mitigate them.Brightness, contrast, and saturation varied by a factor of 0.3, and hue by 0.01, during simulated training.Source: §4 · pp. 4–5

Action Space Adaptation

The subgoal model supplies nearby waypoint candidates for the VLN agent, while standard ROS navigation components execute the selected action without relying on the simulator navigation graph.The framework combines learned subgoals with classical navigation modules.Source: §4 · p. 5, Figure 2 · p. 5
The subgoal model fuses a radial laser-scan occupancy map with panoramic CNN features in a U-Net to predict waypoint probabilities across range and heading bins.When it misses a valid waypoint between narrow bookshelves, robot navigation fails.Source: §4 · p. 5, Figure 3 · p. 6
The model is trained on Matterport3D viewpoints using simulated 270° laser scans and Sinkhorn divergence against neighboring graph viewpoints, excluding routes requiring stairs.The paper reports Sinkhorn divergence as more effective than cross-entropy because it respects the underlying metric space.Source: §4 · p. 5
29% of predicted waypoints are within 0.5m of a ground-truth neighboring viewpoint, 60% within 1m, and 74% within 1.5m.These measurements use the Theta V camera at viewpoints in the Coda navigation graph.Source: §4 · p. 6

Obstacle Avoidance and Path Planning

A depth camera and 270° laser scanner provide complementary obstacle observations, which ROS move_base uses for cost maps, planning, and waypoint execution.The depth camera can detect overhanging obstacles that may be missed by the planar laser scan.Source: §4 · p. 6

Verification

Mock ROS nodes using simulator images and graph data verified that the robot implementation matched the original R2R-EnvDrop model within 1%.The ROS-based harness wraps the standard VLN agent API.Source: §4 · p. 6

5 Results and Analysis

Transfer to Coda was substantially less reliable without prior maps and navigation graphs than with them. The Coda simulator was slightly easier than R2R test, while visual adaptation reduced—but did not eliminate—the camera-related performance drop.

5 Results and Analysis

55.9% success on Coda simulation exceeded the 49.2% R2R test result, supporting Coda as an in-domain transfer baseline.Table 1 characterizes Coda alongside R2R validation-unseen and test environments.Source: §5 · p. 7, Table 1 · p. 7
6.4% and 6.2% drops in success rate and SPL with the robot camera were reduced to 3.9% and 4.3% after visual domain adaptation.The comparisons use Coda simulator images captured with the robot camera versus Matterport panoramas.Source: §5 · p. 7
22.5% instruction-following success in the no-map setting was a further 24.3% drop from the with-map setting, where physical Coda reached 46.8%.Table 2 covers 1,334m of language-guided trajectories and reports success rate and path fidelity.Source: Table 2 · p. 7, §5 · pp. 7–8
Around 9.1% lower success than Coda simulation accompanied feasible map-enabled transfer; out-of-position viewpoints were attributed to degraded visual inputs.The with-map physical evaluation covered 111 instructions and 1.3km without collisions or navigation failures.Source: §5 · pp. 7–8
Without a map or navigation graph, each episode required fresh SLAM and subgoal waypoint predictions, with false-positive and false-negative waypoints explaining most of the performance drop.The authors consider transfer in this cold-start setting an open research problem.Source: §5 · p. 8

6 Conclusion and Future Directions

The paper reports largely successful transfer with advance mapping, but much less reliable cold-start transfer. It identifies subgoal waypoint prediction as a target for future improvement or removal.

6 Conclusion and Future Directions

With advance occupancy mapping and a navigation graph, transfer was largely successful with an approximately 9% reduction in instruction-following success, while unmapped cold-start transfer was much less reliable.The stated sources of the mapped-setting reduction are visual differences (approximately 4%) and viewpoint differences (approximately 5%).Source: §6 · p. 8
In the cold-start setting, subgoal-model failures to predict matching neighboring graph waypoints were the main stated reason transfer was less reliable.Source: §6 · p. 8
Future work could improve or eliminate the subgoal model by training the VLN agent with low-level actions; continuous-motion simulators are one proposed route, though their visual artifacts may exacerbate domain differences.Source: §6 · p. 8
Even in the no-map setting, pose tracking used a separate, isolated ROS navigation stack with map access.Source: §6 · p. 8