PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-6-Astra Lights Up Embodied Navigation: Evaluation in Zero-Shot Vision-and-Language Navigation in Continuous Environments

Guangzhao Dai, Qianru Sun, Qi Wu, Bin Zhu

arXiv:2609.29861v1cs.RO

TL;DR

This paper asks whether a general-purpose foundation model can navigate unfamiliar environments through its own perception, reasoning, and decisions. It evaluates GPT-6-Astra with monocular RGB and a minimal interface, finding strong zero-shot benchmark performance alongside persistent route-execution and goal-verification challenges.

  • Problem

    The paper asks whether a general-purpose foundation model can navigate unfamiliar environments by directing its own interaction with them.

    Source: §1 · p. 1

  • Method

    GPT-6-Astra is evaluated on zero-shot continuous VLN through a minimal Codex-harness interface using monocular RGB and primitive actions, without navigation-specific fine-tuning.

    Source: §1 · p. 2

  • Results

    79.0% success rate with ultra reasoning exceeds reported zero-shot SpatialAnt and Qwen-RobotNav success rates by 13.0 and 6.9 percentage points, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors encourage VLN research to build on foundation-model capabilities while identifying limitations and where navigation-specific methods or training could improve them.

    Source: §1 · p. 2

    The evaluation is limited to R2R-CE-100 with one run per episode, and different Codex versions prevent isolating the effect of reasoning effort.

    Source: §7 · p. 8, §6.3 · p. 8

Abstract

from arXiv · show

We investigate whether GPT-6-Astra, a general-purpose foundation model, can navigate unfamiliar environments using its own perception, reasoning, and decision-making capabilities. Our evaluation focues on zero-shot vision-and-language navigation in continuous environments (VLN-CE) through a minimal interface in the Codex harness, aiming to unleash GPT-6-Astra's full potential for navigation. Using monocular RGB, GPT-6-Astra decides when to observe, how to move, and when to stop, without navigation-specific fine-tuning, a trained waypoint predictor, or a pre-built scene map. Our evaluation yields four main findings. First, \textbf{\textit{GPT-6-Astra achieves strong zero-shot navigation performance using only monocular RGB observations}}. On the common-adopted zero-shot R2R-CE benchmark, ultra reasoning achieves a success rate of \textbf{\textit{79.0\%}}, exceeding the strongest reported zero-shot and supervised success rates by \textbf{\textit{13.0}} and \textbf{\textit{6.9}} percentage points, respectively. Second, \textbf{\textit{GPT-6-Astra advances multi-stage language instructions into coherent, adaptive navigation}} by grounding spatial relations, tracking task progress, and revising its actions. Third, \textbf{\textit{reliable route execution and goal verification remain challenging, even with ultra reasoning}}. Plausible local landmark matches do not consistently lead to correct task completion. Fourth, \textbf{\textit{these capabilities motivate rethinking the role of embodied learning}}. Future VLN research should build on foundation models to advance generalizable and reliable embodied intelligence.

1. Introduction

The introduction asks whether a general-purpose foundation model can navigate unfamiliar environments by directing its own interaction, and evaluates GPT-6-Astra through a minimal interface. It reports strong zero-shot results alongside persistent route-following and goal-verification difficulties, motivating further work on reliable navigation.

Main findings

  • VLN tests whether a foundation model can integrate perception, reasoning, memory, and action over extended interaction to reach an instructed destination.The task requires recognizing landmarks, grounding spatial relations, tracking progress, coordinating movement, recovering from mistakes, and stopping appropriately.Source: §1 · pp. 1–2
  • The study uses a minimal Codex-harness interface to examine GPT-6-Astra's navigation capabilities without introducing a new navigation method.The harness sustains the session while GPT-6-Astra makes navigation decisions.Source: §1 · p. 2
  • 79.0% success rate with ultra reasoning exceeds reported zero-shot SpatialAnt and Qwen-RobotNav success rates by 13.0 and 6.9 percentage points, respectively.The evaluation uses monocular RGB without navigation-specific fine-tuning, a trained waypoint predictor, or a pre-built scene map; the compared systems use different subsets and configurations.Source: §1 · p. 2
  • GPT-6-Astra grounds landmarks and spatial relations, tracks completed and remaining instruction steps, and revises actions as new observations resolve uncertainty.Successful trajectories connect landmarks and actions into ordered routes through multiple turns and room transitions.Source: §1 · p. 2
  • Among 21 ultra-reasoning failures, 13 end at least 5 m from the goal and seven more than 10 m away, despite plausible local landmark matches.Failures include route deviations, constrained-passage difficulties, and arrival claims conflicting with final position.Source: §1 · p. 2
  • The authors encourage VLN research to build on foundation-model capabilities while identifying limitations and where navigation-specific methods or training could improve them.They call for broader evaluation across diverse instructions, changing environments, and longer routes.Source: §1 · p. 2

2. Evaluation Setup

The setup gives GPT-6-Astra control over when to observe, how to act, and when to stop through a minimal interface, then evaluates it on a 100-episode zero-shot R2R-CE subset. Success and path-efficiency metrics are reported alongside measures of goal distance and reference-path agreement.

2.1. Navigation Interface

  • The interface provides 512×512 monocular RGB observations and primitive actions, while withholding maps, reference trajectories, goal coordinates, global poses, and depth.The model selects observations, action sequences, and when to stop; the harness connects it to the simulator.Source: §2.1 · p. 2
  • observe() returns the current view without advancing the simulator, while step(actions) executes primitive actions and another observation requires a new observe() call.The interface includes forward motion, turns, camera tilts, and STOP; the figure's top-down scene and path are illustrative, not model inputs or executed trajectories.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2. Evaluation Protocol and Metrics

  • Evaluation uses the 100-episode R2R-CE validation-unseen subset, spanning 10 scenes, with no navigation-specific fine-tuning, trained waypoint predictor, or pre-built scene map.The cited protocol notes that trained-method results use the full validation-unseen split, making some comparisons not strictly fair.Source: §2.1 · p. 2, §2.2 · p. 3
  • Medium and ultra reasoning are each run once per episode with the same prompt and tools, under a 500-action budget and 2,400-second time limit.An episode ends when the model issues STOP or either limit is exhausted.Source: §2.2 · p. 3
  • Success requires an explicit STOP within 3 m of the goal; SR and SPL measure completion and path efficiency, while NE, OSR, and nDTW measure goal distance, success-radius entry, and reference-path agreement.Metrics are averaged over all 100 episodes in each setting.Source: §2.2 · p. 3

3. Navigation Performance

GPT-6-Astra performs strongly on the evaluated navigation benchmark using monocular RGB without navigation-specific fine-tuning. Comparisons with zero-shot and trained methods are contextual because evaluation subsets and system settings differ.

3. Navigation Performance

  • 79.0% SR and 69.5% SPL with ultra reasoning compare with 76.0% SR and 67.9% SPL with medium reasoning; ultra has slightly lower OSR and higher NE.Both configurations use monocular RGB and let GPT-6-Astra select observations, primitive actions, and when to stop, without navigation-specific fine-tuning.Source: §3 · p. 3
  • GPT-6-Astra reports higher SR and SPL than zero-shot GC-VLN and SpatialAnt while using monocular RGB without a trained waypoint predictor or pre-built scene map.GC-VLN uses monocular observations; SpatialAnt uses panoramic observations and scene reconstruction.Source: §3 · p. 3
  • Both GPT-6-Astra configurations report numerically higher SR and SPL than trained Image2Nav and Qwen-RobotNav, but the comparisons do not establish superiority under matched conditions.The trained-method results use the full validation-unseen split, while the table also includes subset evaluations.Source: §3 · p. 3
  • The reported results motivate examining how GPT-6-Astra connects instructions with successive observations and actions, and where route execution and goal verification remain unreliable.The section turns from aggregate scores to analysis of behavior behind them.Source: §3 · p. 3

4. From Instructions to Adaptive Navigation

The section examines how GPT-6-Astra grounds spatial instructions, follows routes, and adjusts actions, using aggregate metrics alongside recorded observations and actions. Its findings distinguish reaching a goal from following an instructed route or navigating efficiently.

4. From Instructions to Adaptive Navigation

  • The analysis combines navigation metrics with recorded observations and actions to examine spatial grounding, route following, and action adjustment.Figure 3 presents nDTW and SPL for successful episodes, outcomes for adjusted episodes, and selected backtracking observations from EP42.Source: §4 · p. 3, Figure 3 · p. 5
  • Table 1 covers full-split, R2R-CE-100, and author-sampled results; evaluation settings vary, and several methods' score sources are identified separately.The table defines panoramic versus monocular observations and notes unavailable or unverified entries.Source: Table 1 · p. 4, §4 · p. 4

4.2. Does Reaching the Goal Mean Following the Instructed Route?

  • Among 79 successful episodes, median nDTW is 90.4%, but eight (10.1%) score below 50.0%, showing that success alone does not capture route following.nDTW measures reference-path agreement, not the correctness of individual instruction steps.Source: Figure 3 · p. 5, §4.2 · p. 6

4.3. How Does GPT-6-Astra Adjust Its Actions?

  • Of 16 episodes with documented action adjustments, 10 succeed and six fail; successful cases have median nDTW of 44.3% and SPL of 45.5%.The documented counts do not establish that adjustments caused the outcomes.Source: Figure 3 · p. 5, §4.3 · p. 6

4.1. How Does GPT-6-Astra Ground Spatial Instructions?

  • Order-based selection succeeds on 12/14 tasks (85.7%), versus 8/13 (61.5%) for landmark-based selection; these whole-task scores do not measure individual spatial-relation accuracy.The instruction subsets also differ in scene layouts and route difficulty.Source: Table 2 · p. 5, §4.1 · p. 5
  • EP218 and EP244 illustrate successful order-based and landmark-based selection, with final NE of 2.6 m and 1.8 m, respectively.EP218 passes the first room on the left before selecting the next doorway; EP244 selects the doorway left of the white double doors.Source: §4.1 · p. 5

4.2. Does Reaching the Goal Mean Following the Instructed Route?

  • EP42 succeeds with NE of 0.7 m but nDTW of 27.5% and SPL of 41.2%, indicating a substantial detour.By contrast, EP116 has NE of 0.1 m and nDTW of 89.2%, consistent with close reference-path agreement.Source: §4.2 · p. 6

4.3. How Does GPT-6-Astra Adjust Its Actions?

  • In EP42, GPT-6-Astra backtracks from a bedroom to the kitchen, then follows the instructed kitchen–living-room–bedroom sequence and succeeds.Recorded images and executed actions show the revisitation and route change.Source: Figure 3 · p. 5, §4.3 · p. 6

5. Failure Modes and Reliability

The section examines where route following, execution, and stopping fail, and whether higher reasoning effort resolves those failures. Failures include substantial goal distance, unchanged observations after actions, incorrect stopping, and tasks that remain unsuccessful across reasoning settings.

5. Failure Modes and Reliability

  • On R2R-CE-100, ultra reasoning succeeds in 79 episodes; three others enter the success radius but fail at termination, and 18 never enter it.Figure 4a summarizes the navigation outcomes.Source: §5 · p. 6, Figure 4 · p. 7

5.1. Where Does Route Following Break Down?

  • Of ultra reasoning's 21 failures, 13 end at least 5 m from the goal, including seven over 10 m; in EP513, plausible fireplace and seating matches accompany a reported arrival 14.5 m away.The recorded views do not locate the first route error.Source: §5.1 · p. 6

5.2. Can GPT-6-Astra Recover from Execution Difficulties?

  • Five forward-only action batches across four episodes produce pixel-identical RGB observations, yet three episodes eventually succeed; this indicator shows no visible change, not confirmed collisions.In EP176, eight forward actions produce identical observations before the episode exhausts its 500-action budget at NE 35.4 m.Source: §5.2 · p. 6

5.3. Does GPT-6-Astra Stop at the Right Place?

  • In EP705, GPT-6-Astra stops after reporting a fire extinguisher beside a doorway, despite OSR equal to one and final NE of 12.3 m.OSR alone does not reveal when the trajectory entered the success radius or why it later ended elsewhere.Source: §5.3 · p. 6

5.4. Does Higher Reasoning Effort Resolve These Failures?

  • Of 24 tasks failed with medium reasoning, nine succeed with ultra reasoning while 15 remain unsuccessful (62.5%); six other tasks succeed with medium but fail with ultra.Among the 15 shared failures, ultra reasoning yields lower final NE in eight tasks and higher NE in seven; EP1133 remains unsuccessful despite NE improving from 7.5 m to 5.6 m, above the 3 m threshold.Source: §5.4 · pp. 6–7, Figure 4 · p. 7

6. Implications for Embodied Navigation

General-purpose models offer an alternative route to VLN through a minimal observation-and-action interface. Reliable route execution, goal verification, and broader controlled evaluation remain important priorities.

6.1. Do General-Purpose Models Offer Another Route to VLN?

  • GPT-6-Astra navigates through a minimal observation-and-action interface without additional navigation-specific fine-tuning, suggesting competitive navigation need not require a separately trained policy.Matched observations, action budgets, and task sets are needed to establish when this route is preferable to specialized training.Source: §6.1 · p. 7

6.2. What Should Future VLN Methods Improve?

  • Future methods should preserve route constraints, detect ineffective motion and recover, and verify that the final location satisfies the instruction.Assess these behaviors alongside SR using reference-path agreement, execution feedback, and stopping outcomes.Source: §6.2 · p. 7

6.3. How Should This Route Be Evaluated?

  • The current findings concern R2R-CE-100 and the evaluated monocular interface, so broader controlled evaluations are needed to establish robustness across navigation settings.Comparisons should control observations, action spaces, budgets, and software versions.Source: §6.3 · pp. 7–8
  • Because medium and ultra runs used different Codex versions, their paired outcomes do not isolate the effect of reasoning effort.Recorded observations and actions reveal observable behavior, not the accuracy of individual semantic decisions or internal reasoning.Source: §6.3 · p. 8

7. Conclusion

On R2R-CE-100, GPT-6-Astra reached 76.0% success rate with medium reasoning and 79.0% with ultra reasoning using a minimal monocular interface. The analysis finds both adaptive navigation capabilities and persistent execution and verification difficulties, while motivating broader evaluation and further work.

Conclusion

  • 79.0% SR with ultra reasoning and 76.0% SR with medium reasoning were achieved on R2R-CE-100 without navigation-specific fine-tuning.The evaluation used one run per episode under each reasoning setting.Source: §7 · p. 8
  • The analysis identifies spatial grounding, ordered room transitions, and route revision alongside persistent difficulties in execution and goal verification.Fifteen tasks fail under both reasoning settings.Source: §7 · p. 8
  • The evaluation is limited to R2R-CE-100, and future work will cover additional datasets and report means and standard deviations over repeated runs.The passage names RxR-CE and NavRAG-CE as additional datasets.Source: §7 · p. 8
  • High inference cost and latency constrain practical deployment, while the model's proprietary training data prevent determining whether or how much navigation data appeared in training.Here, zero-shot means no navigation-specific fine-tuning, not that the model never encountered navigation data.Source: §7 · p. 8
Loading 2609.29861v1…