GG-Nav: Goal-oriented Grounding Chain-of-Thought Elicits Reasoning in Navigation Foundation Models

1 Introduction

• Goal-oriented navigation includes ObjectNav, where any target-category instance suffices, and IIGN, where the intended instance must be identified from an underspecified instruction.IIGN also requires maintaining that instance's identity throughout navigation.Source: §1 · p. 1
• Current VLM-based methods may miss target evidence in streaming observations and lack explicit target localization for precise stopping.Sparse key-frame selection can overlook transient evidence, while direct observation-and-instruction-to-action mappings leave grounding implicit.Source: §1 · pp. 1–2
• GG-Nav learns a structured Goal-oriented Grounding CoT mechanism via supervised fine-tuning to localize goal-relevant instances and accumulate grounding evidence.Observation-to-Evidence summarizes intermediate frames between decision steps; Evidence-to-Decision integrates accumulated evidence with the current observation to produce an action or clarification question. Training uses 30K grounding-annotated videos and 60K navigation trajectories, yielding bounding boxes and consistent instance IDs.Source: §1 · p. 2
• GG-Nav outperforms the strongest competing methods by 3.1%, 16.6%, and 5.3%, respectively, across the three benchmarks.The reported gains are on HM3D ObjectNav, VL-LN, and CoIN, respectively.Source: §1 · p. 2
• GG-Nav demonstrates zero-shot capability in unseen real-world environments under both ObjectNav and IIGN settings.The system was deployed on a Unitree Go2 robot.Source: §1 · p. 3

2.1 Goal-oriented Navigation Benchmarks

• ObjectNav accepts any instance of a target category, whereas IIGN resolves underspecified instructions through human interaction to identify a specific instance.Both settings require accumulated visual evidence grounded to candidate targets for reliable decisions.Source: §2.1 · p. 3

2.2 Goal-oriented Navigation Methods

• Classical goal-oriented navigation used end-to-end policies or modular pipelines combining semantic mapping, goal selection, and local planning.Reported approaches include reinforcement and imitation learning, demonstrations, data augmentation, and structured semantic maps.Source: §2.2 · p. 3
• Foundation-model navigation methods use VLMs or LLMs as modular components, fine-tune them on embodied trajectories, or pair sparse VLM reasoning with lightweight controllers.Zero-shot modular approaches can be weakly aligned with navigation and accumulate errors [17] [36] [41] [42]; end-to-end control-loop methods can limit real-time, smooth long-horizon navigation [28].Source: §2.2 · p. 3
• Dual-system methods leave intermediate observations unused and often keep target localization implicit; GG-Nav instead structures reasoning into evidence accumulation and evidence-conditioned decisions.The inference process turns intermediate frames into accumulated grounding evidence, then combines it with the current observation to generate an action or clarification question.Source: §2.2 · p. 3, Figure 2 · p. 4

3 Motivating Study

• 41% (153) of 373 failures occurred when the target appeared only briefly in unsampled intermediate frames, while 20% (75) involved detecting and confirming the target but failing to stop accurately.Table 1 presents the failure analysis of adapted InternVLA-N1 on VL-LN Bench.Source: §3 · p. 4, Table 1 · p. 4
• The target appeared in the agent's RGB observations in 68% (254) of the 373 analyzed failures.The adapted InternVLA-N1 baseline achieved a 25.4% success rate on 500 VL-LN Bench episodes.Source: §3 · p. 4
• The analysis identifies two challenges: discovering goal-relevant evidence in streaming observations, and grounding and reusing accumulated evidence for localization and reliable decisions.Existing methods typically use a small set of uniformly sampled history frames plus dialogue-related key frames.Source: §3 · p. 4

4.1 Task Formulation

• ObjectNav accepts any instance of the instructed category, whereas IIGN requires identifying one intended instance among same-category candidates and may use oracle questions to disambiguate it.The agent accumulates visual evidence during exploration.Source: §4.1 · p. 5

4.2 Goal-oriented Grounding CoT Mechanism

• GG-Nav decomposes each decision-step trace into Observation-to-Evidence, Evidence-to-Decision, and the resulting action.Observation-to-Evidence recovers goal-relevant evidence from intermediate frames; Evidence-to-Decision grounds current candidates against accumulated evidence.Source: §4.2 · p. 5

4.2.1 Observation-to-Evidence

• Observation-to-Evidence processes intermediate frames, producing frame-wise bounding boxes, temporally consistent instance IDs, and summaries of encountered or newly observed instances.Its context includes the instruction, historical frames, grounded key frames, and frames collected since the previous decision.Source: §4.2.1 · p. 5

4.2.2 Evidence-to-Decision

• Evidence-to-Decision grounds visible candidates against accumulated evidence, then generates a navigation action or, in IIGN, a grounded question.Questions refer to observed instances; their answers are stored with grounded frames for later evidence accumulation and decisions.Source: §4.2.2 · p. 5

4.3 Data Collection

• Training combines grounding-annotated videos for Observation-to-Evidence with navigation trajectories for full-task supervision.The collection includes approximately 30K annotated videos and 60K trajectories (20K ObjectNav, 40K IIGN); video samples use Aria Synthetic Environments, while trajectories use Habitat scenes and add rule-based question-answer supervision for unseen IIGN candidates.Source: §4.3 · p. 6

5 Experiments

• The evaluation asks whether GG-Nav improves navigation, reduces missed-evidence failures, improves grounding and stopping, and transfers to unseen real-world environments.SR measures stopping within 0.25 m of a target; SPL also accounts for path efficiency, and IIGN uses NQ to measure interaction cost.Source: §5 · pp. 6–7

5.1 Baseline Comparison

• GG-Nav achieves the best overall performance on HM3D ObjectNav, VL-LN, and all three CoIN splits, including 72.3% SR and 31.12% SPL on HM3D and a 16.6% SR gain over the second-best VL-LN method.On HM3D, it improves over InternVLA-N1 by 3.1% in SR and 3.51% in SPL.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Ablation Study I: Fundamental Designs

• Adding intermediate observations reduces grounding errors by an average of 11.6%, while explicit grounding reduces stopping errors by an average of 49.4%.The full model also improves SR over the action-only baseline from 62.5% to 72.3% on HM3D, 20.8% to 42.0% on VL-LN, and 18.0% to 31.9% on CoIN.Source: §5.2 · pp. 7–8, Table 3 · p. 8

5.3 Ablation Study II: Dialog-frame Retention

• Decision keyframe retention performs best among the tested strategies: a 50-step window reaches 43.4% SR and 24.36% SPL, but raises NQ to 3.32.A 400-step window lowers NQ to 2.74 while SR and SPL fall to 41.0% and 22.62%; more compact memory can also discard disambiguated instances and prompt repeated questions.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 Case Study

• Case studies show that intermediate frames can reveal briefly visible targets, while clarification helps GG-Nav preserve the intended referent among similar nearby instances.Without intermediate observations, a briefly visible target may be missed and the agent may move away; after clarification, it can navigate to the selected chair and stop near it.Source: §5.4 · p. 10, Figure 3 · p. 10

5.5 Real-world Experiment

• On a Unitree Go2, GG-Nav consistently outperforms InternVLA-N1 across ObjectNav and all tested IIGN settings, and qualitative examples show clarification and accurate stopping in ambiguous scenes.The quantitative comparison measures success only when the robot stops within 0.25 m of and faces the intended target instance.Source: §5.5 · pp. 11–12, Figure 5 · p. 11, Figure 4 · p. 11

6 Conclusion

• GG-Nav learns structured reasoning to preserve transient visual evidence and maintain target grounding, improving navigation and stopping reliability across ObjectNav and IIGN benchmarks.The conclusion identifies reliable execution of individual goal-oriented subtasks as a foundation for long-horizon navigation.Source: §6 · p. 12
• Deployment on a Unitree Go2 robot demonstrates applicability to unseen real-world environments.The conclusion presents this deployment as further validation of GG-Nav.Source: §6 · p. 12

A Implementation Details

• The paper provides the training setup, data mixture, and inference configuration needed to reproduce GG-Nav.Source: A Implementation Details · p. 15

A.1 Training Setup

• GG-Nav is initialized from Qwen3-VL-4B-Instruct and fine-tuned for 2.5 epochs using AdamW across the vision encoder, multimodal projector, and language-model backbone.Source: §A.1 · p. 15
• The language-model backbone and projector use a 2 × 10−5 learning rate, while the vision encoder uses 5 × 10−6; training uses batch size 128 and a cosine schedule.The setup also specifies zero weight decay, 0.03 warm-up ratio, maximum gradient norm 1, bfloat16, gradient checkpointing, and DeepSpeed ZeRO-3.Source: §A.1 · p. 15
• Input RGB images are resized to 384 × 384, with minimum and maximum input sizes of 3,136 and 313,600 pixels, respectively.The maximum sequence length is 8,192 tokens.Source: §A.1 · p. 15

A.2 Training Data Mixture

• The 5,722,041-sample training mixture contains 1,000,000 temporal-grounding samples, 1,178,492 ObjectNav samples, and 3,543,549 IIGN samples.These comprise 17.5%, 20.6%, and 61.9% of the mixture, respectively.Source: §A.2 · p. 15

A.3 Inference Configuration

• Inference retains target-instance key frames and dialogue, samples eight historical frames, and passes all observations since the preceding decision to Observation-to-Evidence.Benchmark experiments use Decision-keyframe retention with a 200-decision-step window; action-dependent timing, decoding, and retry handling are also specified.Source: §A.3 · p. 15

B.1 Grounding Output Format

• Each frame-level grounding result is a JSON list of target-category bounding boxes and labels with model-assigned, temporally consistent track IDs.An empty list represents a frame with no detected target-category instance.Source: §B.1 · p. 16

B.2 Prompt and Output Examples

• Observation-to-Evidence grounds recent observations and summarizes instances, while Evidence-to-Decision grounds the current frame and produces an action or clarification question.Questions refer to intermediate or current images as Frame_k or Frame_c; waypoint prediction uses a downward-tilted observation.Source: §B.2 · p. 16

B.3 VLM-based Instance Association

• The VLM assigns temporally consistent instance IDs from grounding records, key frames, and recent observations without an external tracker or post-hoc identity-association module.Dialogue answers are stored with the corresponding grounded frame and track ID.Source: §B.3 · p. 17

C.1 Videos with Grounding Annotations

• Temporal-grounding samples use clips of 1, 2, 3, or 7 frames and context frames containing the selected object category.Instances are retained only when their visible segmentation masks contain at least 200 pixels; boxes and instance IDs supervise the clip and evidence summary.Source: §C.1 · p. 17

C.2 Navigation Trajectories

• Navigation trajectories record observations, poses, actions, boxes, and visible target instances, then retain cases with the target visible in the final observation and no redundant exploration.The retained trajectories supervise both Observation-to-Evidence and Evidence-to-Decision.Source: §C.2 · p. 17

C.3 Interaction Supervision

• Clarification questions are generated from newly observed candidates and the elimination history of previously seen candidates.Templates cover single candidates, multiple unseen candidates, partially eliminated candidates, and the case where only one candidate remains.Source: §C.3 · pp. 17–18
• The oracle answers candidate questions using ground-truth instance segmentation but exposes no privileged mask, coordinate, pose, distance, or directional information.Answers are appended to the dialogue history.Source: §C.3 · p. 18
• The 50,855 candidate episodes include 118,309 question–answer pairs, and approximately 40K IIGN trajectories from the pool enter the final training mixture.The 15 most frequent categories account for 50.97% of episodes, with the most frequent category at 5.26%.Source: §C.3 · p. 18
• 84.24% of IIGN interactions are binary confirmations, while 15.76% involve multiple candidates or rejection of all visible candidates.Single-candidate questions account for 59.47% and one-candidate-left questions after partial elimination account for 24.77%; Figure 6 summarizes the candidate configurations.Source: §C.3 · p. 18, Figure 6 · p. 18
• 60.73% of episodes contain one clarification turn and 13.90% contain two, with a mean of 2.33 turns per episode.The retained multi-turn tail includes repeated clarification and evidence updates.Source: §C.3 · p. 18, Figure 6 · p. 18

D.1 Habitat Agent Configuration

• The Habitat agent uses co-located RGB-D sensors and discrete move, turn, look, and stop actions, with success requiring STOP within 0.25 m of a valid target.ObjectNav accepts any instance of the instructed category; IIGN requires the annotated instance.Source: §D.1 · p. 19

D.2 Baseline Selection and Adaptation

• Five representative baselines span modular and end-to-end methods: FBE/VLFM guide exploration, AIUTA supports interaction, and MTU3D and InternVLA-N1 provide learned comparisons.InternVLA-N1 is adapted for ObjectNav and IIGN; direct comparison with GG-Nav uses the same Habitat shortest-path follower, and interactive methods share oracle access.Source: §D.2 · pp. 19–20

D.3 Stage-wise Failure Attribution

• Failed episodes are automatically assigned in order to exploration, grounding, tracking, reaching, or stopping using instance masks, model records, and final robot-to-target distance.Reaching failures end at least 1 m from the target; stopping failures end between 0.25 and 1 m away.Source: §D.3 · p. 20

E.1 System Architecture and Execution

• On the Go2, RGB-D observations are sent to remote GG-Nav, which uses historical and intermediate frames to emit actions, pixel goals, or clarification questions for onboard execution.Pixel goals are converted to robot-frame waypoints and tracked with MPC; discrete actions use a pose controller, and look-down requests trigger another inference on the fixed camera stream.Source: §E.1 · pp. 20–21

• Easy IIGN requires a navigation distance under 50 m and fewer than five candidate instances.Source: §E.2 · p. 21
• Medium IIGN requires a navigation distance under 50 m and at least five candidate instances.Source: §E.2 · p. 21
• Hard IIGN requires a navigation distance over 50 m and at least five candidate instances.Source: §E.2 · p. 21
• Success requires STOP within 0.25 m of the intended instance while facing it; distance is measured manually and orientation is visually checked.Source: §E.2 · p. 21

• Clarification is limited to target-specific questions about observed candidates; future work proposes open-ended planning for route guidance and richer disambiguation.Source: F Limitations and Future Work · p. 21
• Experiments primarily cover large, common household objects, and the paper does not claim reliable support for small objects such as phones or forks.Broader coverage, finer-grained instance annotations, and higher-resolution or active close-range perception are identified as needs for extending to small objects.Source: F Limitations and Future Work · p. 21