PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-to-Real Transfer of Vision-Language Navigation in Continuous Environments Using an Ackermann-Steered Mobile Robot

Chalindu Abeywansa, Sahan Gunasekara, Devindi De Silva, Seniru Dissanayake, Ranga Rodrigo, Peshala Jayasekara

arXiv:2610.07192v1cs.AIcs.RO

TL;DR

Traditional VLN models often depend on navigation graphs, panoramic views, and precise localization, while simulation-to-real transfer also faces visual and control differences. This paper adapts a continuous-environment VLN model using real-world robot data and fine-tuning. The authors report real-world navigation with simulation-to-real performance drops of 19% for SPL and 12% for nDTW.

  • Problem

    Traditional VLN approaches often rely on graphs, panoramic imagery, and precise localization, while simulation-to-real transfer faces visual-domain and action-space differences.

    Source: §I · p. 1

  • Method

    The system trains a cross-modal-attention VLN model on simulated VLN-CE data, then applies photometric adjustments and fine-tunes it with real-world robot data.

    Source: §I · p. 1, E. Fine-Tuning the System · p. 4

  • Results

    0.26 SPL and 0.46 nDTW are reported for the fine-tuned real-world model, with simulation-to-real drops of 19% for SPL and 12% for nDTW.

    Source: 2) Performance in Real-World… · p. 6, 3) Analysis of Results · p. 6

  • Takeaways & Limitations

    The approach demonstrates real-world instruction following on an Ackermann-steered robot without navigation graphs or 360-degree views.

    Source: §I · p. 2

Abstract

from arXiv · show

Vision-Language Navigation (VLN) enables robots to navigate through environments using natural language instructions, making human-robot interaction intuitive. Traditional VLN models often rely on navigation graphs, 360-degree views, and perfect localization which pose significant challenges when adapting these models to real-world settings. This work addresses these limitations by performing a simulation-to-real domain shift of a VLN approach that operates in continuous environments without requiring navigation graphs or panoramic views. The proposed system integrates vision-language models that align visual inputs and linguistic instructions within a shared embedding space, facilitating natural language-driven navigation. We employ a Cross-Modal Attention (CMA) based architecture trained on an existing dataset in a simulated environment and fine-tune it using real-world data collected from a custom-built Ackermann-steered robot equipped with a camera and a LiDAR sensor. By utilising linear photometric adjustments and fine-tuning on a limited number of episodes, our model successfully adapts to real-world environments, achieving effective navigation while running offline on dedicated hardware. Experimental results, evaluated using Success weighted by Path Length (SPL) and Normalized Dynamic Time Warping (nDTW) metrics, demonstrate the robustness and adaptability of our approach. Keywords: Vision-Language Navigation, Cross-Modal Attention, Natural Language Instructions, Sim-to-Real Transfer, Autonomous Navigation, Ackermann-steering.

I. INTRODUCTION

Traditional VLN systems rely on graph-based navigation, panoramic views, and precise localization, while simulation-to-real transfer also faces visual and action-space gaps. This work adapts continuous-environment VLN for an Ackermann-steered robot and reports successful real-world navigation with minimal fine-tuning.

I. INTRODUCTION

  • Traditional VLN systems often assume oracle navigation, known graphs, panoramic RGB and depth imagery, and perfect localization, complicating transfer to continuous real-world environments.Visual differences and the shift from discrete simulated actions to continuous robot control add further sim-to-real challenges.Source: §I · p. 1
  • The approach trains on simulated VLN-CE data, then uses linear photometric adjustments and fine-tuning to adapt the model to real-world settings.It encodes image and depth data with two ResNet50 networks, instructions with a Bi-LSTM, and predicts actions through cross-modal attention.Source: §I · p. 1
  • The system is designed to navigate without navigation graphs or 360-degree views, and experiments demonstrate instruction following across diverse real-world settings.Source: §I · p. 2
  • The framework transfers a VLN-CE model to real indoor environments on an Ackermann-steered robot using minimal fine-tuning.The authors describe this as the first attempt at VLN with an Ackermann-steered robot, to the best of their knowledge.Source: §I · p. 2
  • The work provides a ROS implementation for offline VLN deployment, which the authors describe as enabling practical applications.Source: §I · p. 2

II. RELATED WORK

The related work traces VLN from discrete graph-based navigation toward continuous environments, while noting that real-world transfer remains constrained by assumptions such as panoramic views and exact localization.

II. RELATED WORK

  • The section frames related work as an evolution from discrete action spaces toward continuous environments, while identifying remaining challenges and research gaps.Source: §II · p. 2

A. Vision-Language Navigation

  • VLN began with RGB-based navigation from high-level language commands in the R2R benchmark, followed by work on visual-language interpretation and attention,,,,.Early models were generally limited to discrete environments and action spaces with fixed viewpoints and graph-based path planning.Source: A. Vision-Language Navigation · p. 2

B. Navigation Graphs

  • Graph-based methods structure movement through nodes and edges,, but 360-degree views and exact localization may be unavailable in real-world settings.Source: B. Navigation Graphs · p. 2

C. Sim-to-Real Transfer

Prior VLN-CE work moves beyond discrete graph assumptions but remains largely simulation-based, while sim-to-real approaches face visual and sensing differences. The paper proposes an embodied, graph-free approach using egocentric vision and language with a cross-modal attention model.

C. Sim-to-Real Transfer

  • Prior sim-to-real work used domain randomization and subgoal prediction, but its performance degraded without premapped environments and annotated navigation graphs; LM-Nav relies on preexploration and large pretrained models.The paper contrasts these approaches with operation in unknown environments without precomputed maps and with limited processing power.Source: C. Sim-to-Real Transfer · p. 2

D. Vision-Language Navigation in Continuous Environments

  • VLN-CE, supports longer-horizon navigation in continuous environments, but related systems still use precomputed navigation graphs,, and panoramic visuals and remain simulation-only.Source: D. Vision-Language Navigation in… · p. 2

III. METHODOLOGY

  • The proposed embodied system uses the robot’s egocentric view and language without a predefined navigation graph, predicting discrete actions for execution through a ROS handler with fail-safe obstacle avoidance.The system is tested on a custom Ackermann-steered robot.Source: §III · pp. 2–3, Figure 1 · p. 3

A. Generalized System Architecture

The architecture combines visual, depth, and language inputs to predict navigation actions. Cross-modal attention feeds recurrent action prediction, and ROS connects the model to robot sensing, control, and obstacle avoidance.

A. Generalized System Architecture

  • RGB and depth images are photometrically adjusted and encoded with dual pretrained ResNet50 networks, while GLoVE instruction embeddings are encoded with a Bi-LSTM for resource-constrained deployment.Source: A. Generalized System Architecture · p. 3

1) Cross-Modal Attention (CMA) Model:

  • The first GRU tracks visual observations, and cross-modal attention generates instruction, visual, and depth attention used by a second GRU to predict the next action embedding.The visual embeddings are concatenated with the previous action embedding before entering the first GRU.Source: 1) Cross-Modal Attention (CMA) Model · p. 3

2) Integration with ROS:

  • ROS manages sensory-data acquisition, motor control, and real-time obstacle avoidance while mediating communication between the VLN model and robot hardware.The authors intend the ROS integration to support adaptation to other robots already modeled with ROS.Source: 2) Integration with ROS · p. 3

B. Dataset and Simulation Environment

The system uses VLN-CE data and staged training to develop navigation in simulation, then deploys it on an Ackermann-steered robot with LiDAR-based obstacle handling.

B. Dataset and Simulation Environment

  • VLN-CE provides continuous trajectories derived from R2R, and its 10,837 training, 1,839 validation, and 3,408 test episodes support initial training.The Matterport3D environments provide variation in spatial layouts, textures, and lighting.Source: B. Dataset and Simulation Environment · p. 3

C. Training in Simulated Environment

  • Simulation training combines DAgger imitation learning from oracle actions with supervised learning on ground-truth paths and instructions.The supervised phase follows precomputed paths using low-level actions until convergence.Source: C. Training in Simulated Environment · p. 4

D. Robot Platform

  • The Ackermann platform uses an elevated RGB-D camera and LiDAR; its chassis and suspension support a stable, near-human camera viewpoint.The platform includes an NVIDIA Jetson AGX Xavier for onboard processing.Source: D. Robot Platform · p. 4, Figure 2 · p. 4
  • LiDAR obstacle avoidance adjusts speed and steering from front, left, and right sector distances, while command interpolation smooths motion.The authors report that the steering mechanism had less impact on overall VLN performance.Source: D. Robot Platform · p. 4

E. Fine-Tuning the System

Real-world fine-tuning uses joystick-collected RGB-D, instruction, and action data, with photometric adjustments before the CMA model is updated.

E. Fine-Tuning the System

  • Joystick-driven episodes provide RGB frames, depth images, instructions, and the human navigator’s corresponding actions.Photometric adjustments are applied to RGB and depth inputs to align them with simulated training conditions.Source: E. Fine-Tuning the System · p. 4
  • Fine-tuning extracts 512-dimensional embeddings and updates only the final linear layer mapping them to four navigation actions.The actions are stop, move forward, turn right, and turn left; cross-entropy loss is backpropagated to update that layer.Source: E. Fine-Tuning the System · pp. 4–5
  • The authors report that this targeted update let limited real-world data bring performance close to simulation.They describe the result as improved real-world adaptability.Source: E. Fine-Tuning the System · p. 5

IV. EXPERIMENTS AND RESULTS

The experiments assess simulated and real-world VLN using SPL and nDTW, and examine real-world data collection and photometric correction before evaluation.

IV. EXPERIMENTS AND RESULTS

  • The evaluation uses SPL and nDTW to assess navigation and the effects of training and fine-tuning strategies.The approach follows evaluation practices cited in the paper.Source: §IV · p. 5

1) Simulation Environment:

  • Simulation experiments cover 5,247 VLN-CE test and validation episodes in Habitat environments containing obstacles, doors, and furniture.The simulated settings were designed to approximate real-world scenarios as far as possible.Source: 1) Simulation Environment · p. 5

2) Data Collection and Fine Tuning:

  • Real-world data span six environments and 20 trajectories with 2,500+ RGB-D frames and over 140 navigation configurations; RGB scaling uses α = 1.5 and β = 1.2, and depth uses gamma correction of 0.35.The caption reports that corrected depth, being closer to simulation scenes, enabled successful navigation; the selected corrections worked across all six indoor environments.Source: 2) Data Collection and Fine Tuning · p. 5, Figure 3 · p. 5

3) Evaluation in Real-World Environment:

The real-world evaluation tests instruction-following across varied indoor settings, while SPL measures success in relation to path efficiency.

3) Evaluation in Real-World Environment:

  • The robot was evaluated for 18 episodes across six indoor environments, with paths compared against human navigation paths recorded using the same hardware.The environments include offices, hallways, and open spaces with varied obstacles, lighting, and surface types.Source: 3) Evaluation in Real-World Environment · p. 5

B. Evaluation Metrics

1) Success weighted by Path Length (SPL) [28]:

  • SPL adjusts episode success by path efficiency, relating robot path length to ideal path length across episodes.The passage defines N as the number of episodes, Si as episode success, pi as robot path length, and li as ideal path length.Source: 1) Success weighted by Path Length… · p. 5

2) Normalized Dynamic Time Warping (nDTW) [28]:

nDTW compares the robot’s actual trajectory with a reference path derived from natural-language instructions to assess path alignment. The figures show instruction-specific routes and path tracking.

2) Normalized Dynamic Time Warping (nDTW) [28]:

  • nDTW compares the robot’s actual trajectory with an instruction-derived reference trajectory to quantify path alignment and navigation accuracy.The metric offers insight into navigation precision.Source: 2) Normalized Dynamic Time Warping… · p. 5
  • Figure 4 shows the robot’s vision, instruction, and path tracking for two different routes through the same environment.Panels (a) and (b) correspond to different instructions.Source: Figure 4 · p. 6
  • Figure 5 distinguishes each instruction’s ideal path in green from the robot’s tracked path in blue across several locations.The routes shown are for different instructions.Source: Figure 5 · p. 6

C. Results

The model scored 0.32 SPL and 0.52 nDTW in simulation; after fine-tuning, it achieved 0.26 SPL and 0.46 nDTW in real-world environments, improving over simulation-only baselines.

1) Performance in Simulation:

  • In simulation, the VLN-CE model achieved SPL 0.32 and nDTW 0.52 across validation episodes.These scores provide the simulation performance baseline.Source: 1) Performance in Simulation · p. 6

2) Performance in Real-World Environments:

3) Analysis of Results:

The analysis reports a limited simulation-to-real performance drop and better map-free performance than the cited map-based sim-to-real implementation. The conclusion presents this as a real-world transfer of VLN-CE to an Ackermann-steered robot, while identifying further work to narrow the gap.

3) Analysis of Results:

  • Sensor noise, visual discrepancies such as lighting, and environmental dynamics are cited as factors behind the real-world performance drop.The paper connects these factors to lower success and longer or inferior paths, which can also reduce nDTW.Source: 3) Analysis of Results · p. 6
  • Approximately 19% for SPL and 12% for nDTW were the reported simulation-to-real performance drops; the model surpassed the cited map-based sim-to-real implementation without a map.The paper also reports that the feature-fields implementation performs slightly better in simulation but drops significantly in real environments compared with this model.Source: 3) Analysis of Results · p. 6, Table II · p. 7, §V · p. 7

V. CONCLUSION

  • The work presents the first transfer of a VLN-CE model to a real-world Ackermann-steered robot, navigating unseen indoor environments without navigation graphs or 360-degree views.The approach uses targeted fine-tuning with real-world data and minimal photometric adjustments.Source: §V · p. 7
  • Further narrowing the sim-to-real gap calls for improved transfer techniques and training on more diverse simulated environments.The conclusion also identifies outdoor deployment as future work, with outdoor lighting, trajectories, and vocabulary among the stated challenges.Source: §V · p. 7

Related papers

Loading 2610.07192v1…