PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperNav: An Agentic Navigation System for Any Task in Any Scene

Jinkai Zhang, Jingyi Xu, Yuanhong Yu, Jiarui Guo, Ruizhen Hu, Hujun Bao, Xiaowei Zhou, Sida Peng

arXiv:2610.12126v1cs.ROcs.CV

TL;DR

General-purpose robots need navigation that handles diverse requests in unfamiliar environments, while existing approaches are constrained by fixed workflows or navigation-training coverage. SuperNav equips a pretrained MLLM with a tool-using harness and visual-point interface, and reports stronger results across evaluated task types alongside category-level evaluation and real-robot deployment.

  • Problem

    General navigation requires adapting to diverse requests and unfamiliar scenes, while modular systems use predefined workflows and end-to-end action models depend on navigation-training coverage.

    Source: §1 · p. 2

  • Method

    SuperNav equips a pretrained MLLM with an agent harness that uses Tools, Navigation Skills, task-progress and context management, and a visual-point interface without navigation-specific fine-tuning.

    Source: §1 · pp. 2–3

  • Results

    78.00% single-object success versus 34.00% for UniNaVid and 59.50% demand-driven success versus 37.50% for OmniNav’s Action Former; category-level evaluation also reports 73.33% SR on HM3D-OVON.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Evaluations across category-level, instance-level, multi-object, and demand-driven navigation, plus real-quadruped deployment, demonstrate applicability across task and scene settings.

    Source: §5 · p. 9

    SuperNav relies on the underlying MLLM’s semantic judgments, and its demand-driven evaluation measures ordered navigation to relevant targets rather than completion of the underlying activities.

    Source: §5 · p. 9

Abstract

from arXiv · show

General-purpose service robots need navigation systems that can handle diverse human requests in unfamiliar environments, combining task generality with scene generality. Some existing methods fine-tune multimodal large language models (MLLMs) to predict navigation actions, making their behavior dependent on the coverage of navigation training data and potentially limiting generalization to new requests and environments. Our key insight is to let the MLLM focus on interpreting requests, understanding scenes, and making decisions while preserving its general-purpose capabilities and delegating motion execution to navigation tools. To realize this idea, we introduce SuperNav, which equips a pretrained MLLM with a specialized agent harness without navigation-specific fine-tuning of the MLLM. Our harness supports these decisions with Navigation Skills, agent-oriented Tools for physical interaction, and task-progress and context management. A unified visual-point interface connects decision-making to motion by allowing the model to specify destinations directly in images and revise its decisions from execution feedback. Together, these components support sustained navigation across different task requirements and environments. SuperNav outperforms four evaluated baselines on instance-level, multi-object, and demand-driven tasks. Category-level evaluation on HM3D and deployment on a real quadruped robot further demonstrate its applicability across environments. Project Page: https://zju3dv.github.io/SuperNav/

1 INTRODUCTION

General navigation must handle varied requests in unseen environments, but existing modular and end-to-end approaches have constraints around predefined workflows and navigation-training coverage. SuperNav instead equips a pretrained MLLM with a tool-using harness and visual-point interface; evaluations report results across task types and scenes.

1 INTRODUCTION

  • General navigation must adapt to diverse requests and unseen scenes, while modular workflows and navigation-trained action models remain constrained by fixed task logic and training-data coverage.Changing from object search to instance identification, multi-target visits, or abstract needs also changes goal interpretation, progress tracking, and completion conditions.Source: §1 · p. 2
  • SuperNav equips a pretrained MLLM with an agent harness for request interpretation and decision-making, using Tools, optional Navigation Skills, task-progress records, and context management without navigation-specific fine-tuning.The harness delegates physical interaction to Tools while the MLLM reasons over requests and visual observations.Source: §1 · pp. 2–3
  • A unified visual-point interface lets the MLLM select destinations in observation images and receive execution feedback, rather than predict low-level actions directly.The same interaction pattern supports different motion backends.Source: §1 · pp. 2–3
  • 78.00% single-object success versus 34.00% for UniNaVid, and 59.50% demand-driven success versus 37.50% for OmniNav’s Action Former.For category-level navigation, SuperNav achieves 73.33% SR and 0.4105 SPL on a 120-episode HM3D-OVON val-unseen subset at a 1 m threshold, without OVON-specific training or adaptation; at 0.25 m it achieves 68.33% SR and 0.3837 SPL.Source: §1 · pp. 2–3
  • The evaluation spans category-level, instance-level, multi-object, and demand-driven tasks, with real-quadruped deployment examining applicability across task and scene settings.Source: §1 · p. 3

2 RELATED WORK

Prior work uses modular pipelines, hierarchical systems, and agentic harnesses to connect semantic reasoning with navigation or other embodied actions. SuperNav focuses on runtime coordination of navigation capabilities by the MLLM.

Modular Navigation Pipelines

  • Modular navigation pipelines assign models specific roles in mapping, exploration, and verification, whereas SuperNav lets the MLLM select and coordinate capabilities at runtime.Examples include semantic exploration and language priors, topological or scene-graph context, and adaptive exploration with multi-frame identification.Source: §2 · p. 3

Hierarchical Navigation Systems

  • Hierarchical systems already combine high-level model reasoning with learned or geometric execution, while SuperNav emphasizes the interfaces and runtime for choosing and combining capabilities across a task.Examples include value-map trajectories, waypoint policies, and classical within-room exploration and motion.Source: §2 · p. 3

Agentic Systems and Harnesses

  • Agentic systems use callable capabilities and feedback; SuperNav studies task and scene generality with a unified visual-point interface, interchangeable motion backends, context pruning, and Navigation Skills.Task-level action selection and semantic completion judgments remain with the MLLM.Source: §2 · p. 3

3 NAVIGATION AGENT HARNESS

SuperNav combines a pretrained MLLM with an agent loop that sustains navigation decisions across requests. Tools connect decisions to physical operations, while Skills provide on-demand procedural guidance and context preserves task progress.

3 NAVIGATION AGENT HARNESS

  • The harness combines a pretrained MLLM with callable Tools and readable Skills, letting the model interpret requests and observations, choose operations, and use returned feedback to decide again.Requests may specify a target, an ordered sequence, or a high-level need.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 AGENT LOOP

  • The agent loop initializes from an observation and repeatedly uses context and task progress to select observation, motion, goal-state updates, or relevant Skill guidance.Tool returns and retrieved instructions enter the next decision context, so execution feedback can change the plan.Source: §3.1 · p. 4
  • Goal records track pending and completed items, while textual history retains inspected areas, candidate clues, and failed attempts; processed image payloads are pruned while paths and task state remain available.Earlier images can be requested again when needed.Source: §3.1 · pp. 4–5

3.2 AGENT-ORIENTED NAVIGATION TOOLS

  • Agent-oriented Tools cover observation, turning, visual-point navigation, progress management, and termination; the image-space destination interface supports both geometric and learned execution backends.The backends use the same decision workflow and return observations with execution feedback.Source: Figure 2 · p. 4, §3.2 · p. 5

3.3 NAVIGATION SKILLS

  • Navigation Skills provide on-demand procedural guidance for search, recovery, and completion, while the MLLM retains responsibility for choosing operations and semantic judgments.Guidance recommends recording inspected areas and alternatives, changing routes or points after failed motion, and checking candidate appearance, relations, and context before recording goals.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

SuperNav is evaluated across instance, demand-driven, category-level, and real-world navigation settings, with experiments also examining executor and harness variants. Results report advantages over baselines in several settings, while the HM3D comparison notes differences in evaluation protocols.

4 EXPERIMENTS

  • The experiments report SR and SPL across four settings using GPT-5.6 Terra with Geo-based or Learned Executor, alongside baseline comparisons.The baselines retain their native inputs and history mechanisms.Source: §4 · p. 6

4.1 INSTANCE NAVIGATION IN HABITAT-GS

  • The InteriorGS benchmark includes 150 single-object tasks and 150 ordered multi-object tasks with two to five targets; success requires explicit STOP.Arrival is measured by navmesh geodesic distance below 1 m to approved target viewpoints; multi-object success requires ordered arrivals.Source: §4.1 · p. 6
  • SuperNav sustains its advantage through later ordered goals; both backends match multi-object success, while Geo-based Executor yields more efficient paths.Figure 3 depicts visits to a washing machine, potted plant, and mahjong table in order.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2 DEMAND-DRIVEN NAVIGATION

  • Demand-driven evaluation uses 200 AI2-THOR tasks adapted to one to eight ordered stages, with each stage requiring entry into its admissible region without object reuse.Success also requires explicit STOP.Source: §4.2 · p. 7
  • Both executors improve on baselines in success rate and SPL for demand-driven navigation, although long sequences remain challenging.The Geo-based Executor progresses furthest.Source: §4.2 · p. 7

4.3 HM3D BENCHMARKS

  • Geo-based Executor is competitive with listed published HM3D methods, but their evaluation subsets and protocols differ from ours.Table 2 reports category-level SR and SPL at indicated thresholds; the comparison is not protocol-matched.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 HARNESS AND MODEL ABLATIONS

  • Removing Navigation Skills reduces SR more than restricting interaction to the front view, especially at the stricter threshold.With Skills retained, front-only interaction also reduces SR and SPL while requiring substantially more explicit turns; external grounding does not improve on direct point selection, and Astra (medium) improves both metrics over Terra (high) with the harness fixed.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 REAL-WORLD NAVIGATION

  • On a Unitree Go2 using four-view RGB and LiDAR-based geometric execution, SuperNav revises routes at blocked passages, verifies targets, and transitions between goals.Figure 5 shows searches for a basketball and another Go2, plus a printer-to-trash-bin sequence.Source: §4.5 · p. 9, Figure 5 · p. 9

5 CONCLUSION

SuperNav combines a pretrained MLLM with a navigation harness and visual-point interface, and experiments support its applicability across task and scene settings. Its scope is bounded by reliance on MLLM judgments, geometric resources, inference cost, and demand-driven scoring that does not assess completion of underlying activities.

5 CONCLUSION

  • SuperNav equips a pretrained MLLM to interpret requests, understand scenes, and decide while tools execute motion, supported by Navigation Skills, task-progress tracking, context management, and a visual-point interface.The MLLM requires no navigation-specific fine-tuning; evaluation spans category-level, instance-level, multi-object, and demand-driven navigation.Source: §5 · p. 9
  • SuperNav relies on the underlying MLLM for semantic judgments, geometric execution needs maps or scene geometry, and inference adds computational cost and variable latency.Demand-driven evaluation measures ordered navigation to relevant targets rather than completion of underlying activities; broader real-world evaluation and reliability on long sequences remain future directions.Source: §5 · p. 9

AI USE STATEMENT

The authors describe AI assistance in manuscript preparation, comparison-paper identification, coding, and experiments, while attributing research ideas and design principles to themselves.

AI USE STATEMENT

  • Codex with GPT-6 assisted manuscript drafting, comparison-paper identification, navigation-tool and evaluation-code implementation, and experiment debugging; authors developed the ideas and tested the AI-assisted code.Generative AI also assisted with scene-based annotations and evaluation instructions, which the authors manually checked.Source: AI USE STATEMENT · p. 9

A.1 EXPERIMENTAL SETUP

The appendix describes geometric and learned execution, the agent runtime and tool interfaces, Skill installation checks, and evaluation criteria. Runtime restrictions rely partly on instructions rather than operating-system isolation.

A.1 EXPERIMENTAL SETUP

  • The geometric executor grounds image points using depth and camera geometry, then plans and follows paths; the learned executor generates motion from RGB observations and image-space goals.Simulation uses a navigation mesh and depth, while the robot backend uses sensed geometry and localization; the learned controller uses visual history and a flow action head.Source: §A.1 · pp. 12–13

A.2 RUNTIME AND TOOL ACCESS

  • The harness runs Codex with task-specific workspaces and exposes environment operations through MCP, with scene access restricted by instructions and a tool whitelist.The shared visual-point contract returns observations and execution feedback; tools vary by backend, and returned task observations include four directional RGB images.Source: §A.2 · p. 13

A.3 SKILL CONFIGURATION

  • A Skill snapshot is checked by file and bundle hashes, and a visibility probe verifies the discovered Skill before startup proceeds.The task prompt references the Skill, and generated instructions require reading its entry file before scene actions.Source: §A.3 · p. 13

A.4 EXECUTION LIMITS AND EVALUATION

  • Evaluation uses task-specific success criteria, including geodesic-distance thresholds, ordered arrival, and explicit STOP; execution timeouts and unsuccessful session closure are recorded separately.Demand-driven scoring matches admissible candidate objects at each stage without reusing a matched candidate.Source: §A.4 · p. 14

B.1 INSTANCE-NAVIGATION BENCHMARK CONSTRUCTION

The benchmark evaluates instance navigation in Habitat-GS using InteriorGS scene assets, with manually reviewed object-level semantic annotations.

B.1 INSTANCE-NAVIGATION BENCHMARK CONSTRUCTION

  • The benchmark uses Habitat-GS with InteriorGS scene assets.Source: §B.1 · p. 14
  • A vision-language model generates object-level semantic annotations, which are manually reviewed.Source: §B.1 · p. 14
  • The final set contains 150 single-object tasks and 150 ordered multi-object tasks, each with two to five targets.Source: §B.1 · p. 14

B.2 INSTANCE- AND DEMAND-DRIVEN NAVIGATION METRICS

This block specifies instance- and demand-driven scoring, ordered-goal progress, baseline execution interfaces, and protocol differences in published HM3D evaluations.

B.2 INSTANCE- AND DEMAND-DRIVEN NAVIGATION METRICS

  • Instance success requires arrival within 1 m geodesic distance of an approved viewpoint and explicit STOP; multi-object success checks ordered trajectory arrivals.Viewpoint-reference SPL uses a shared shortest-route reference, with failed tasks receiving zero SPL.Source: §B.2 · p. 14
  • Demand-driven success requires ordered arrivals within 2 m of admissible object bounding boxes and explicit STOP, under a protocol not directly comparable to DemandAgent’s scores.The protocol omits additional visibility, line-of-sight, and vertical-distance tests.Source: §B.2 · pp. 14–15

B.3 ORDERED GOAL COMPLETION

  • SuperNav leads every multi-object stage and the first five demand-driven stages in ordered-goal progress.Multi-object progress uses a viewpoint geodesic threshold below 1 m; demand-driven progress uses 2 m bounding-box regions.Source: §B.3 · p. 15, Table 4 · p. 15

B.4 OBSERVATION INTERFACES AND BASELINE EXECUTION

  • Baseline configurations retain model-specific observation inputs and native interfaces, without an Agentic controller or evaluator stage-progress signal.SuperNav’s motion backend and geometric resources also differ by task setting.Source: §B.4 · pp. 15–16

B.5 PUBLISHED HM3D PROTOCOLS

  • Published HM3D comparisons use differing thresholds, success rules, episode budgets, and evaluation loops across methods.The described protocol differences include thresholds from 0.2 m to 1 m and varying STOP requirements.Source: §B.5 · p. 16

B.6 OVON EVALUATION AND ABLATION IMPLEMENTATION

This section documents the OVON evaluation and ablation configurations, then describes a learned executor that turns image-space point goals into motion using visual history.

B.6 OVON EVALUATION AND ABLATION IMPLEMENTATION

  • The OVON configurations share a 120-episode manifest, model setting, episode budget, and scoring protocol.Success is scored by final geodesic distance to the goal-viewpoint set; two designated unreachable-goal episodes receive success and SPL contributions of one.Source: §B.6 · p. 16
  • 86 of 120 episodes succeeded at 0.25 m and 95 of 120 at 1 m with GPT-6 Astra, while the navigation harness and execution setup remained fixed.The comparison also changes the stated reasoning setting from high to medium.Source: §B.6 · p. 16
  • The Skills and Tool ablations vary available search guidance and visual access while retaining specified movement and interaction capabilities.Front-only inspection requires turning before side or rear pointing; the adapted Skills retain recovery, approach, verification, and termination guidance.Source: §B.6 · pp. 16–17
  • Grounding Motion Tool uses external language-conditioned localization and candidate confirmation, whereas direct point navigation uses the Agent’s normalized image coordinate.Both variants share the underlying navigation controller.Source: §B.6 · p. 17

C LEARNED EXECUTOR: POINT-GOAL NAVIGATION IN IMAGE SPACE

  • Learned Executor converts an image-space destination into local movements using a marked reference image, shared DINOv2 encoder, visual history, and conditional flow head.The MLLM selects the destination and judges task completion; the executor predicts motion and when to stop the current request.Source: C LEARNED EXECUTOR: POINT-GOAL… · pp. 17–18, Figure 6 · p. 18

C.1 VISUAL GOAL AND HISTORY ENCODING

  • A fixed reference image and normalized pixel specify the destination, while a changing RGB observation is fused with a bounded, age-aware history.The history selection retains initial and current frames and is capped at 12 observations per fusion pass.Source: §C.1 · p. 18

C.2 MOTION REPRESENTATION AND TRAINING

The Learned Executor is trained to map point goals and visual context to displacement sequences, then runs in a feedback loop that replans from fresh observations.

C.2 MOTION REPRESENTATION AND TRAINING

  • Expert Habitat greedy-follower trajectories provide RGB observations and poses; future expert positions are projected into the initial image to create visible point goals.Training also uses brightness augmentation and up to two pixels of goal-marker jitter per axis.Source: §C.2 · pp. 18–19
  • The policy predicts eight robot-frame displacement increments, which are rescaled by 0.25 m and summed into relative waypoints.The training target uses expert ground-plane positions transformed into the robot frame; increments after the endpoint are zero.Source: §C.2 · p. 19
  • A conditional flow head learns increment sequences by interpolating between normalized targets and Gaussian noise and regressing flow velocity.Ten percent of training samples replace the goal token with a learned null token for classifier-free guidance.Source: §C.2 · p. 19
  • Separate heads predict stopping probability and remaining steps, with stopping supervision marking expert observation-index gaps of two or fewer.Training combines flow, stop, and remaining-step losses, and jointly fine-tunes the visual backbone.Source: §C.2 · pp. 19–20

C.3 CLOSED-LOOP MOTION EXECUTION

  • At replanning, the executor samples candidate sequences, selects one using endpoint-direction agreement and path straightness, and normally executes a three-increment prefix.It then obtains fresh RGB observations and predicts again toward the same reference goal; collision recovery and blockage detection run within the executor.Source: §C.3 · p. 20
  • A local stop requires repeated above-threshold stop predictions plus motion-progress conditions; execution status and fresh views return to the MLLM for its next decision.Failure and budget termination are reported separately from confirmed local stops.Source: §C.3 · p. 20

Related papers

Loading 2610.12126v1…