Source-linked AI summary
Scaffolding Foundation Models into Physical-World Agents Pushes the Frontier of Long-Horizon Navigation
Zixing Lei, Gengze Zhou, Xiong-Hui Chen, Jiazhao Zhang, Yiyang Huang, Hang Yin, Haoqi Yuan, Qi Wu, Weixin Li, Siheng Chen
TL;DR
Long-horizon physical-world agents need both high-level evidence-seeking reasoning and reliable closed-loop navigation, but existing VLMs and NFMs provide these capabilities separately. NavMCP scaffolds a VLM reasoning agent with an NFM executor through intent, observation, and memory channels, achieving strong benchmark and real-world results. The framework reaches state-of-the-art performance across three EQA benchmarks and 78.3% success on a Unitree Go2.
Problem
VLMs are brittle and inefficient at repeated navigation grounding, while NFMs execute bounded semantic goals without persistent task-level reasoning.
Method
NavMCP couples a VLM that selects evidence-seeking actions with an NFM that grounds semantic sub-goals, using intent, observation, and memory channels.
Results
NavMCP achieves state-of-the-art results on HM-EQA, MT-HM3D, and EXPRESS-Bench, and reaches 78.3% success on a Unitree Go2.
Takeaways & Limitations
Scaffolding complementary VLM reasoning and NFM execution is an effective route to long-horizon physical-world agents.
Takeaways & Limitations
NavMCP relies on a large VLM, which slows inference, and cross-view duplication impairs counting.
Abstract
from arXiv · showhide
Long-horizon physical-world agents must reason over distant goals while grounding decisions in reliable closed-loop behavior. Today's foundation models split these capabilities: vision-language models (VLMs) infer missing information and adapt high-level plans but remain brittle and inefficient at repeated navigation grounding, while navigation foundation models (NFMs) robustly execute semantic goals but operate as bounded episodes without persistent task-level reasoning. We introduce NavMCP, an agentic scaffolding framework that couples a VLM reasoning agent with an NFM executor for long-horizon exploration. The VLM decides what evidence to seek, where to search, and when to stop, while the NFM grounds each semantic sub-goal into closed-loop navigation. Three channels structure their collaboration: intent translates evidence needs into navigation calls, observation converts rollouts into source-grounded trajectory evidence, and memory accumulates findings, negative evidence, and unresolved goals across calls. This design turns isolated navigation rollouts into persistent embodied interaction without retraining either model. On Embodied Question Answering, NavMCP achieves state-of-the-art results on HM-EQA, MT-HM3D, and EXPRESS-Bench. Under matched agent and executor backbones, it outperforms an episodic interface by 14.9 percentage points on HM-EQA. On a Unitree Go2, NavMCP reaches 78.3% success, with its margin over the strongest baseline growing from 10 to 45 points as the task horizon increases. These results demonstrate the potential of scaffolding complementary foundation models into long-horizon physical-world agents.
1 Introduction
NavMCP organizes complementary VLM reasoning and NFM navigation into persistent long-horizon physical-world agency. Its evidence-centric interface addresses intent mismatch, intermediate observation loss, and missing cross-call memory, producing strong simulated and real-world results.
- VLMs infer missing information and adapt high-level plans, whereas NFMs reliably ground semantic goals but typically lack persistent task-level reasoning.
- NavMCP assigns evidence selection, search location, and stopping decisions to the VLM while the NFM grounds each semantic sub-goal into closed-loop motion.
- Its three channels translate evidence needs into navigation calls, convert rollouts into grounded observations, and preserve findings and unresolved goals across calls.
- 14.9 percentage points: replacing NavMCP with an episodic interface under fixed agent and navigator conditions reduces HM-EQA performance by 14.9 points.
- 78.3% success: NavMCP succeeds on real multi-room search with a Unitree Go2, while its margin over the strongest baseline grows from 10 to 45 points with horizon.
2 Related Work
Related work spans embodied question answering, vision-language navigation and navigation foundation models, and tool-augmented embodied agents with context management. These lines of research motivate combining navigation, reasoning, and memory for long-horizon embodied tasks.
- Embodied Question Answering: EQA requires navigation to answer situated questions, with recent systems addressing exploration, memory, refinement, scene representation, and multi-agent search.
- Vision-Language Navigation and Navigation Foundation Models: Vision-language navigation has progressed from instruction-conditioned navigation through augmentation, pretraining, transformers, video planning, and language-model reasoning.
- Tool-Augmented Embodied Agents and Context Management: Tool-augmented embodied agents use affordance grounding, feedback, code, planning, skills, reasoning-acting, tool use, memory, and prompt compression.
3 Method
NavMCP couples a VLM agent that plans evidence acquisition with a navigation foundation model that executes semantic sub-goals. Intent, observation, and memory channels convert navigation episodes into grounded, persistent evidence for EQA reasoning.
- The VLM maintains hypotheses, evidence needs, and context while the navigation executor handles one semantic sub-goal at a time.
- The executor maps a natural-language sub-goal, egocentric observation, and navigation state to a short-horizon waypoint trajectory.
- Intent Channel: The intent channel converts an evidence need into a semantic navigation call without requiring the agent to micromanage low-level control.
- Observation Channel: The observation channel converts complete rollouts into source-grounded journey evidence, preserving keyframes, uncertainty, and unexplored-region cues.
- Memory Channel: The memory channel stores a compact state Ct = (Ht, Et, Ut), including grounded evidence, searched areas, negative evidence, and unresolved goals.
- Context Maintenance: Raw returns are compressed only after salient information is externalized into source-referenced evidence or unresolved-goal state.
- EQA Instantiation: EQA answers must be supported by current observations, journey artifacts, reviewed keyframes, or notebook entries, with negative answers backed by searched-area records.
4 Experiments
Experiments evaluate NavMCP across three EQA benchmarks, controlled architecture and protocol comparisons, evidence-guided search, and real-robot transfer. Results show state-of-the-art benchmark performance, complementary VLM–NFM capabilities, substantial benefits from the three-channel interface, and increasing gains at longer horizons.
- Overall EQA performance: NavMCP achieves state-of-the-art results across HM-EQA, MT-HM3D, and EXPRESS-Bench, with improved accuracy, answer quality, path efficiency, and normalized agent-step fractions.It reaches 76.7% accuracy on HM-EQA, 54.4% on MT-HM3D, and an LLM Score of 79.27 with an Epath of 33.96 on EXPRESS-Bench.
- Overall EQA performance: 74.0% accuracy is achieved by NavMCP in the matched HM-EQA comparison, exceeding Explore-EQA, ToolEQA, and FAST-EQA by 16.4, 13.2, and 10.5 points.All methods use the same agent, episodes, initial states, and budget, with ToolEQA also receiving the same perception toolset.
- VLM–NFM complementarity: The controlled architecture sweeps show that stronger VLM agents and stronger NFM executors each improve EQA performance, indicating complementary rather than substitutable roles.With Qwen-RobotNav-8B fixed, accuracy rises to 74.0% with the full Qwen3.5 harness and 76.7% with Qwen3.6-Plus; with the full Qwen3.5 agent fixed, executor accuracy rises to 74.0% using Qwen-RobotNav-8B.
- VLM–NFM complementarity: Evidence-guided search uses negative evidence, en-route observations, and non-monotonic movement to eliminate plausible rooms before an object-targeted final request.In the sofa-search episode, the upper layer searches five rooms and records negative evidence while the NFM executes each semantic request.
- Protocol ablations: 14.9 points are lost when the three-channel protocol is replaced by an episodic interface, reducing accuracy from 74.0% to 59.1% with the backbones unchanged.The ablation measures the combined effects of missing intent structure, rollout observations, and cross-call evidence accumulation.
- Protocol ablations: Observation and memory produce the largest component-level ablation effects, while the real-robot system reaches 78.3% success and its margin over the strongest baseline grows from 10 to 45 points across horizon tiers.Observation-only terminal returns cost 5.9 points, removing journey analysis costs 4.4 points, and removing EQA context state costs 4.6 points.
5 Conclusion, Limitations, and Future Work
NavMCP closes the episodic interface gap by combining VLM reasoning with grounded NFM execution through intent, observation, and memory. The framework improves EQA performance while incurring inference costs and facing counting-related limitations.
- Conclusion: NavMCP combines VLM reasoning with NFM execution by preserving evidence across navigation calls.The scaffold separates evidence planning from execution, converts trajectories into reusable journey summaries, and maintains grounded observations across turns.
- Conclusion: NavMCP improves performance across three EQA benchmarks and controlled ablations.The conclusion also includes a matched terminal-only interface baseline isolating the cost of the episodic interface gap.
- Limitations: Large-VLM inference slows NavMCP, while cross-view duplication impairs counting.The paper identifies smaller agents or distillation as possible ways to reduce inference cost and stronger cross-view association as future work.
- Future Work: Agent-directed exploration could extend NavMCP beyond EQA to collect embodied data for training spatial intelligence.The proposed data would couple high-level reasoning with grounded observations and navigation trajectories.
A.1 Per-Category Results
NavMCP performs unevenly across EQA categories: object presence and identification are strongest, while counting is hardest because repeated views can cause over-counting. On MT-HM3D, relationship and comparison questions remain more challenging.
- HM-EQA: Over 80% accuracy on existence and identification questions indicates effective grounding after active search.These categories benefit when answers depend on object presence or category recognition.
- HM-EQA: 77.2% accuracy on state questions shows strong verification-oriented performance.The agent inspects candidate evidence rather than answering from a single observation.
- HM-EQA: 65.0% accuracy makes counting the hardest category because overlapping keyframes can produce repeated observations.The paper motivates stronger cross-view object association and uncertainty tracking.
- MT-HM3D: 63.9% relationship accuracy and 59.6% comparison accuracy are reported on MT-HM3D.Relationship questions benefit from spatial perception tools, while comparison questions benefit from source-recorded target observations.
A.2 Detailed Ablation Analysis
Ablations show that learned navigation, multi-mode execution, visual sampling, and structured interaction contribute to NavMCP’s performance. Executor capability has the largest reported gain, while interface and context changes produce narrower effects.
- Executor capability: 69.3% accuracy with StreamVLN exceeds Random Walk at 60.9% and Frontier Exploration at 65.3%.This improvement supports language-conditioned learned navigation over heuristic movement.
- Executor capability: 73.3% accuracy with Qwen-RobotNav-4B is 4.0 percentage points above StreamVLN.Scaling from 4B to 8B adds only 0.7 percentage points, reported as a scaling trend because estimates use three stochastic hosted-LLM runs.
- Protocol ablations: 68.1% accuracy with terminal-only observation return isolates a 5.9-point cost from discarding en-route observations.Intent and memory remain unchanged in this ablation.
- Protocol ablations: Reducing keyframe density costs 2.8 percentage points, while restricting the executor to one navigation mode costs 2.0 points.These changes indicate that visual sampling and complementary search modes support performance.
- Protocol ablations: Context compaction changes accuracy by 0.9 percentage points, and removing zoom_in_object costs 0.6 points.Context compaction mainly supports efficient, stable long-horizon context management rather than acting as a stand-alone accuracy module.
B Evaluation Protocol and Statistics
The evaluation reports how baselines and matched comparisons are constructed, while accounting for stochastic hosted-LLM variation. Cross-paper differences are treated as contextual rather than paired statistical comparisons.
- Evaluation protocol: Baseline values in Table 1 are quoted from corresponding papers, with ∗ marking results reproduced and reported by FAST-EQA.These cross-paper differences are excluded from paired significance tests because several implementations and per-episode predictions are unavailable.
- Statistics: Main results are reported as mean ± standard deviation over three independent runs.Variation comes from stochasticity in the hosted LLM API rather than explicit random seeds.
- Matched comparison: Matched-task comparisons use the same Qwen3.5 reasoning model, benchmark episodes, initial states, and interaction budget across NavMCP and reproduced baselines.ToolEQA also uses NavMCP’s perception and verification toolset, while each method retains its specified navigation interface.
C Real-Robot Evaluation Details
The real-robot evaluation measures evidence-gathering across task horizons using fixed queries, starting poses, and environment configurations. NavMCP is compared with reactive and frontier-based alternatives under tier-specific time limits.
- Evaluation Protocol: The study records RGB observations, poses, explored maps, keyframes, searched regions, negative observations, and answer support without privileged object locations or unseen geometry.Episodes begin from natural-language queries and permit multiple navigation calls.
- Evaluation Protocol: 60 real-robot episodes per method span low-, medium-, and high-difficulty tiers, with two starting poses repeated five times for each task.Each tier contains two tasks, yielding 20 episodes per method and tier.
- Evaluation Protocol: Success requires autonomous evidence gathering, a correct observation-supported answer, and completion within a 5-, 10-, or 15-minute tier limit.Timeouts and incorrect answers count as failures.
- Baselines and Ablations: The reactive baseline removes multi-step deliberation, while the frontier baseline replaces the learned navigator with heuristic coverage.These ablations test memory-driven orchestration and query-conditioned semantic navigation under real odometry drift and local recovery.
- Results: 35% cross-room success for the reactive variant rises to 85% with NavMCP, while long-horizon success is 0% for reactive, 15% for frontier, and 60% for NavMCP.Single-room tasks achieve 70–90% across methods, whereas longer tasks expose the need for persistent deliberation and orchestration.
D Qualitative Analysis
Qualitative traces show NavMCP treating navigation calls as evidence-gathering steps, preserving checked regions and adapting search modes before answering. Real-robot and ultra-long-horizon examples illustrate repeated planning, execution, memory updates, and verification.
- Ultra-Long-Horizon Deployment: A physical deployment exceeds 50 m by decomposing the request into landmark-conditioned sub-goals and visually verifying a green umbrella.Cotti Coffee serves as an intermediate anchor while progress is recorded in memory.
- Persistent Evidence Gathering: The agent searches broad room categories, records checked regions and negative observations, and uses notebook entries to reduce repeated exploration.A stored entry such as a searched kitchen with absent coffee maker conditions later sub-goals.
- Adaptive Navigation Modes: The agent switches from coarse instruction-guided search to object navigation or visual verification after observing an object anchor.Figure 6 applies this pattern to locating a bunk bed and counting pillows from a closer view.
- Failure Analysis: Precise counting and fine-grained state verification concentrate failures because overlapping keyframes can be treated as distinct evidence.The paper identifies cross-view object association and uncertainty tracking as future improvements.
- Evaluation Context: The real-robot protocol uses two tasks per tier, fixed starting poses, and five repeated episodes per pose.This produces ten episodes per task and twenty episodes per method and tier.
E Tool Specification
The tool specification exposes navigation, panoramic reading, detection, depth, cropping, and historical inspection tools to support evidence acquisition and verification. Perception prompts emphasize scene transitions, fine-grained attributes, and contextual interpretation.
- Tool Interface: The complete tool schema is exposed through the tool-calling interface, while an optional cross-episode retrieval store is disabled in all reported experiments.Tool availability does not imply selection in every episode.
- Perception Backends: Reader tools send task-specific visual prompts to the VLM, whereas detector tools provide open-vocabulary segmentation, detection, and metric depth estimates.Single-frame tools inspect the current front view; panoramic tools provide broader scene coverage.
- Panoramic Inspection: Panoramic scene reading combines front, right, back, and left views into a 360 degree context and asks for visible paths and room transitions relevant to navigation.The prompt also distinguishes natural from artificial illumination for light-state questions.
- Fine-Grained Verification: zoom_in_object sends both a padded target crop and its full source view, giving global context priority when crop and full-view judgments disagree.The crop supports color, pattern, subtype, and text inspection.
F Implementation Details
Implementation details define a fixed navigation executor, closed-loop adapter, trajectory and journey artifacts, evidence arbitration, and semantic intent calls. The protocol preserves grounded observations and task state across repeated exploration cycles.
- Executor and Adapter: Qwen-RobotNav is a fixed pretrained 8B executor that converts semantic sub-goals and egocentric observations into short-horizon waypoint rollouts.The adapter exposes instruction-guided and object-search navigation modes.
- Trajectory Recording: Each navigation call returns execution status, path information, journey analysis, persisted images, and sampled keyframes for downstream reasoning.Keyframes are sampled every four simulator steps and capped at 16 per call.
- Deployment: The deployment uses Qwen3.6-Plus for main results and a fixed open-source agent for ablations, with shared executor and controlled serving settings across variants.Simulation and local perception run on one RTX 4090, while hosted APIs introduce run-to-run stochasticity.
- Agent–Executor Protocol: The VLM agent uses tool calls and repeated turns to emit either a candidate answer or an evidence need, then continues exploration when support is insufficient.The protocol initializes context state, translates evidence needs into intent calls, and loops until a supported answer or the turn limit.
- Intent Interface: Intent calls specify navigation mode, semantic sub-goal, step budget, and constraints, while the executor observes four RGB views, pose, explored map, and collision state.Execution stops on STOP, budget exhaustion, or simulator completion; repeated collisions trigger recovery through is_stuck.
- Grounded Evidence: Journey artifacts link object and room mentions to source keyframes, viewpoints, location hints, and confidence labels while requiring explicit uncertainty.The summarizer reports only visible evidence and avoids treating absence from one view as definitive negative evidence.
- Evidence Arbitration: analyze_status assesses visibility and sufficiency, may request one supplementary inspection, and then authorizes an answer, records negative evidence, or directs further exploration.Negative answers require coverage of the relevant area rather than mere failure to observe the target.
- Efficiency Accounting: The compensated efficiency accounting charges additional steps for rollout distance beyond 3 m, with one extra step for (3, 6] m and two for (6, 9] m.The accounting includes detours, repeated motion, and collision recovery, while outer-agent reasoning and API latency dominate wall-clock time.