PaperlayerView exact revision on arXiv

Source-linked AI summary

Robostral Navigate

Abdelaziz Bounhar, Abhijeet Somani, Aditi Kabra, Adrian Valente, Adrien Petralia, Adrien Sade, Alan Jeffares, Albert Jiang, Aleksandr Timashov, Alexandre Cahill, Alexandre Gavaudan, Alexandre Laval, Alexandre Sablayrolles, Amelie Heliou, Amos You, Andre Jonasson, Andrew Bai, Andrew Ehrenberg, Andrew Zhao, Angele Lenglemetz, Anmol Agarwal, Antonia Calvi, Arata Suzuki, Arjun Majumdar, Arthur Fournier, Artjom Joosen, Avinash Sooriyarachchi, Aylin Guliz Akkus, Aysenur Karaduman, Baptiste Bout, Baptiste Roziere, Baudouin De Monicault, Benjamin Holzschuh, Benjamin Lefaudeux, Benjamin Tibi, Bernhard Stadlbauer, Blazej Osinski, Camille Le Scao, Chaoran Yu, Charlotte Cronjager, Chen-Yo Sun, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clemence Lanfranchi, Corentin Barreau, Corentin Sautier, Cristiana-Diana Diaconu, Cyprien Courtot, Daniel Marczak, Darius Dabert, Diego de Las Casas, Dominik Nuss, Dylan Rubini, Dzmitry Soupel, Elizaveta Demyanenko, Elliot Chane-Sane, Emilien Fugier, Emmanuel Gottlob, Erik Aas, Etienne Goffinet, Etienne Millon, Eujeong Choi, Fabian Paischer, Fabian Schlager, Faruk Ahmed, Federico Baldassarre, Filip Szatkowski, Florian Wiesner, Gabrielle Berrada, Gaetan Ecrepont, Gaetan Lepage, Gaspard Blanchet, Gaspard Donada-Vidal, Gauthier Delerce, Gauthier Guinet, Genevieve Hayes, Georgii Novikov, Giada Pistilli, Gianluca Galletti, Guillaume Breton, Guillaume Kunsch, Guillaume Lample, Guillaume Martin, Guillaume Raille, Gunjan Dhanuka, Gunshi Gupta, Han Zhou, Harshil Shah, Hasan Furkan Vural, Hedi Hadiji, Hope McGovern, Hugo Cisneros, Hugo Thimonier, Indraneel Mukherjee, Ivan Cuevas Salazar, Jacques Sun, Jan Ludziejewski, Jason Rute, Jean Quentin, Jean-Hadrien Chabran, Jean-Malo Delignon, Jie Zhang, Joachim Studnia, Joep Barmentlo, Johannes Brandstetter, John Harvill, Jonas Amar, Jonas Schweizer, Josephine Delas, Josselin Somerville, Julien Denize, Julien Tauran, Kartik Khandelwal, Khyathi Raghavi Chandu, Kilian Tep, Kush Jain, Larissa Laich, Laura Calem, Laurence Aitchison, Laurent Callot, Laurent Fainsin, Leo Cotteleer, Leonard Blier, Lingxiao Zhao, Louis Martin, Louis Serrano, Lucile Saulnier, Ludovic Ho Fuh, Luis Montero, Maarten Buyl, Manon Chossegros, Marcin Mozejko, Margaret Jennings, Markus Hennerbichler, Martin Alexandre, Mathieu Poiree, Mathieu Schmitt, Mathilde Guillaumin, Matthieu Andre, Matthieu Dinot, Matthieu Futeral, Maurits Bleeker, Mauro Comi, Max Mynter, Maxim Berman, Maxime Darrin, Maxime Louis, Maximilian Augustin, Maximilian Muller, Melina Jingting Laimon, Mert Unsal, Mia Chiquier, Michael Pilcer, Michal Pietruszka, Michal Zajac, Mikhail Biriuchinskii, Minh-Quang Pham, Minwoo Kang, Morgane Riviere, Namit Katariya, Nathan Grinsztajn, Nathan Simpson, Neeraj Aggarwal, Neha Gupta, Ola Mysiak, Oliver Leicht, Olivier Bousquet, Olivier Duchenne, Parag Jain, Patricia Wang, Patrick Blies, Patrick von Platen, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavan Kumar Reddy, Pavel Kuksa, Philippe Pinel, Philomene Chagniot, Pierre Stock, Pierre-Andre Savalle, Piotr Milos, Prateek Gupta, Pravesh Agrawal, Quentin Desreumaux, Quentin Torroba, Quercus Hernandez, Ram Ramrakhya, Randall Isenhour, Ranjit Parva, Raul Perez Pelaez, Reinhard Sonnleitner, Remi Delacourt, Richard Kurle, Rishi Shah, Rob Romijnders, Rohin Arora, Romain Sauvestre, Roman Soletskyi, Rosalie Millner, Rupert Menneer, Sagar Vaze, Samuel Barry, Samuel Belkadi, Samuel Humeau, Sanchit Gandhi, Sandeep Subramanian, Sarthak Mittal, Saskia Adaime, Sean Cha, Sebastian Kaltenbach, Shashwat Dalal, Shashwat Verma, Sherif Waly, Shrimai Prabhumoye, Siddhant Waghjale, Siddharth Gandhi, Simon Lepage, Simon Sorg, Soham Ghosh, Sophie Marbach, Srijan Mishra, Stanislas Lange, Steve Hong, Sumukh Aithal, Szymon Antoniak, Tarun Kumar Vangani, Teven Le Scao, Theo Cachet, Thibaut Lavril, Thomas Chabal, Thomas Coste, Thomas Defard, Thomas Foubert, Thomas Robert, Thomas Wang, Tianyu Zhang, Tim Lawson, Timothee Lacroix, Tobias Kronlachner, Tom Bewley, Tom Edwards, Tomas Hodan, Tuhin Das, Tyler Wang, Ulrick BLE, Umar Jamil, Umberto Tomasini, Valentin Mace, Van Phung, Vedant Nanda, Victor Jouault, Victor Letzelter, Victor Paltz, Victor Poucheret, Vincent Maladiere, Vincent Pfister, Virgile Richard, Vladislav Bataev, Wassim Bouaziz, Wen Ding Li, William Havard, William Marshall, Xinghui Li, Xingran Guo, Xinyu Yang, Yann Dreze, Yassine El Ouahidi, Yassir Bendou, Yihan Wang, Yimu Pan, Yves Martin des Taillades, Zaccharie Ramzi, Zhenlin Xu, Zsofia Csakany

arXiv:2607.20785v3cs.ROcs.AI

TL;DR

Navigation systems often rely on depth, multiple cameras, or pre-built maps, restricting compatible robots and requiring environment-specific calibration. Robostral Navigate predicts image-space waypoints from monocular RGB and achieves state-of-the-art results on R2R-CE and RxR-CE.

  • Problem

    Leading navigation systems depend on depth sensors, LiDAR, multiple cameras, or pre-built maps, limiting compatible robots and requiring environment-specific calibration.

    Source: §1 · p. 2

  • Method

    Robostral Navigate uses a single monocular RGB stream and predicts navigation waypoints by pointing to target locations in the current camera view.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Across R2R-CE and RxR-CE, Robostral Navigate sets state of the art; on R2R-CE, it achieves 77.4% success rate, 10.5 points above Qwen-RobotNav-4B.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The work is a first step toward a general-purpose embodied agent, demonstrating that state-of-the-art navigation can have minimal sensing requirements.

    Source: §5 · p. 9

Abstract

from arXiv · show

Deploying navigation systems at scale requires a recipe that minimizes sensor assumptions, generalizes across robot embodiments, and trains efficiently. Yet, today's best systems depend on depth sensors, multi-camera rigs, or pre-built maps, limiting the hardware they support and increasing deployment cost. We introduce Robostral Navigate, an 8B vision-language model built around this scalability objective. The model consumes only a stream of monocular RGB images - the most ubiquitous sensor across robotic platforms and predicts waypoints by pointing to the next target location in the current camera view. Operating purely in image space, rather than robot-specific coordinates, makes the policy naturally robust to changes in camera intrinsics and scene scale, enabling deployment across wheeled, legged, and aerial robots without recalibration. We generate 2.4 million trajectories across 350k simulated scenes to reduce the reliance on real-world data collection and scale easily. We further introduce a prefix-caching training recipe that packs entire episodes into single training sequences, reducing training tokens by 22x and cutting training time from months to days. A tree-based attention mask prevents conditioning on previous ground-truth actions, encouraging visually grounded action prediction, and reinforcement learning is used to further improve exploration and recovery capabilities. On the Room-to-Room and Room-Across-Room in Continuous Environments (R2R-CE and RxR-CE) benchmarks, Robostral Navigate sets a new state of the art. On R2R-CE, it achieves a 77.4% success rate, surpassing the best monocular method by 10.5 points and the strongest depth- or multi-camera system by 5.3 points despite using only a single RGB camera. On RxR-CE, it reaches 75.1% success rate, outperforming all monocular baselines.

1 Introduction

Robostral Navigate is an 8B vision-language navigation model designed to scale across robot embodiments using only monocular RGB images. It predicts image-space waypoints and trains efficiently in simulation, with reinforcement learning to improve exploration and recovery.

1 Introduction

  • The 8B model predicts the next target as an image-space waypoint and desired arrival orientation, avoiding robot-specific metric coordinates to transfer across camera configurations and embodiments.It consumes a monocular RGB stream, a sensor broadly available across robotic platforms.Source: §1 · p. 2
  • Training entirely in simulation uses approximately 2.4 million trajectories across 350k scenes; prefix caching packs episodes into single sequences, cuts training tokens by 22×, and reduces month-long runs to days.The attention-masking technique preserves all learning signals while compressing episodes into single training sequences.Source: §1 · pp. 2–3
  • Online CISPO reinforcement learning on a curated hard subset of trajectories yields an additional 4% improvement in success rate while targeting exploration and recovery from failures.It is applied after supervised training.Source: §1 · p. 3
  • On R2R-CE validation unseen, Robostral Navigate achieves 77.4% success rate, beating Qwen-RobotNav-4B (66.9%) by 10.5 points and Qwen-RobotNav-8B (72.1%) by 5.3; on English-only RxR-CE validation unseen, it reaches 75.1% success rate and 68.7% SPL.The RxR-CE result uses only a single monocular RGB camera.Source: §1 · p. 3

2 Modeling

Robostral Navigate separates high-level visual navigation planning from low-level action generation, using image-grounded waypoints to connect the two models.

2 Modeling

  • The system combines an 8B vision-language model for instruction reasoning and coarse path planning with a 121M diffusion policy for translating paths into robot motion.The models operate in series at different inference frequencies.Source: §2 · p. 3

2.1 Architecture Overview

  • The VLM takes a language instruction and a history of monocular RGB frames, then predicts waypoints as image coordinates or displacement commands.It is initialized from a dense 8B VLM designed for spatial grounding; visual tokens are appended to the instruction, and frame history supports tracking progress and revisited locations.Source: §2.1 · pp. 3–4

2.2 Navigation via Pointing with Displacement Fallback

  • When the destination is visible, the model jointly predicts image coordinates and metric displacements; when it is outside the view, it omits coordinates and predicts displacements only.Because only approximately 10% of training data has invisible destinations, displacement prediction also serves as a co-training task; the model infers STOP when the task is complete.Source: §2.2 · p. 4

2.3 Pointing to Actions

  • A 121M diffusion transformer [Peebles and Xie, 2023] converts camera-frame waypoints into a one-second chunk of 30 base-relative displacement and rotation actions for a motion controller.Its inputs include the VLM prediction, robot height and radius, the VLM context frame, and the latest frame, accounting for latency between those images.Source: §2.3 · p. 4, §2.1 · p. 4

2.4 Cross-Robot Generalization

  • Training randomizes robot dimensions and camera placement, and the same VLM and diffusion-policy weights transfer to the Galaxea R1 [Galaxea AI, 2026] and Hiwonder JetAuto without retraining.Randomized ranges include robot height 0.4–1.8 m, radius 0.15–0.45 m, camera height 70–100% of robot height, and pitch 0–25°; only the low-level platform controller differs between deployments.Source: §2.4 · p. 4, Figure 2 · p. 5

3 Training

Robostral Navigate is trained on large-scale simulated RGB navigation trajectories, using prefix-tree training to reduce sequence redundancy and online reinforcement learning to improve recovery and exploration.

3.1 Data Generation

  • 2.4 million trajectories across 350k scenes pair instructions with RGB observations and navigation actions, using varied environments, layouts, objects, lighting, and architecture.Farthest-point sampling diversifies start and goal positions, producing trajectories of varied lengths, including cross-floor routes.Source: §3.1 · pp. 5–6

3.2 Supervised Training with Prefix Trees

  • Prefix-tree attention prevents conditioning on previous ground-truth actions while preserving the training signal of per-time-step samples and avoiding redundant prefix encoding.Tokens can attend to earlier tokens in the shared trunk or their own branch, but not another branch.Source: §3.2 · p. 6
  • 22× fewer training tokens and training time reduced from months to days result from encoding full episodes once without dropping action-prediction targets.The formulation reduces token complexity from O(T^2) to O(T).Source: §3.2 · p. 6

3.3 Online Reinforcement Learning via CISPO

  • Online CISPO training lets the agent learn from states it visits, addressing behavioral-cloning exposure bias and missing information-seeking and recovery behaviors.CISPO [Chen et al., 2025] uses group-relative advantage estimation; the reward is −max(2, dist_to_goal), with distance measured geodesically in meters.Source: §3.3 · p. 7
  • A 35k-task hard subset focuses reinforcement-learning rollouts on cases the supervised policy fails to solve reliably, including complex layouts, ambiguous instructions, and long-horizon tasks.Tasks are packed by scene; scene-contiguous ordering creates batches with shared visual context and empirically outperforms random shuffling.Source: §3.3 · pp. 7–8

4 Results

Robostral Navigate achieves strong results on the R2R-CE and RxR-CE instruction-following benchmarks using a single RGB camera. Online reinforcement learning improves success rates over supervised fine-tuning, including on unseen environments.

4 Results

  • Navigation evaluation reports NE, SR, OS, and SPL, measuring final distance to goal, stopping within 3 m, coming within 3 m at any point, and path efficiency, respectively.SPL weights success by the shortest-path-to-actual-path ratio, penalizing detours.Source: §4 · p. 8

4.1 Main Results

  • 77.4% SR and 74.2% SPL on R2R-CE validation unseen establish a new state of the art across all metrics; SR exceeds Qwen-RobotNav-4B by 10.5 points and depth- or multi-camera Qwen-RobotNav-8B by 5.3 points.The model uses only a single RGB camera with no auxiliary sensors.Source: §4.1 · p. 8
  • 79.43% seen and 77.43% unseen R2R-CE SR after online RL improve over the SFT baseline by 3.47 and 4.03 points, respectively.Peak validation SR reached 80.46% on seen and 77.43% on unseen environments.Source: §4.1 · p. 9
  • 75.1% SR, 68.7% SPL, and 3.47 m NE on RxR-CE outperform Qwen-RobotNav-8B with depth on SPL and NE while remaining competitive on SR.Against the single-camera Qwen-RobotNav-8B, SR improves by 1.7 points and SPL by 5.2 points.Source: §4.1 · p. 9
  • 75.1% RxR-CE SR after online RL improves on the 71.2% SFT baseline by 3.9 points.The passage reports this result as state of the art and notes that it generalizes to long-horizon instruction-following tasks.Source: §4.1 · p. 9

5 Conclusion

Robostral Navigate achieves state-of-the-art performance on R2R-CE and RxR-CE with a single-RGB-camera policy, efficient simulation-based training, and reinforcement learning. The authors describe it as a first step toward a general-purpose embodied agent.

5 Conclusion

  • Robostral Navigate achieves state-of-the-art performance on the R2R-CE and RxR-CE benchmarks.The 8B vision-language model is designed to minimize sensor assumptions, generalize across embodiments, and train efficiently from simulation alone.Source: §5 · p. 9
  • The policy predicts navigation actions by pointing and requires only a single RGB camera for deployment.The authors view the work as a first step toward a general-purpose embodied agent, demonstrating the combination of large-scale simulation, efficient training, and reinforcement learning with minimal sensing requirements.Source: §5 · p. 9
  • 22× fewer training tokens are used by prefix caching while preserving all learning signals; reinforcement learning with CISPO further finetunes the policy.The authors report that this combination yields an agent that is efficient at solving navigation tasks, as measured on R2R-CE and RxR-CE.Source: §5 · p. 9

Related papers

Loading 2607.20785v3…