Source-linked AI summary
HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness
Yang Chen, Lirong Che, Zhenyu Huang, Wenbo Fu, Chuang Wang, Xu Cao, Daqi Liu, Yuzhe Yang, Jian Su, Lan-Zhe Guo
TL;DR
Embodied navigation needs to connect visual and language reasoning with spatially grounded, state-aware execution, but existing training-free systems often lack this coordination. HarnessVLN uses a unified Agent Harness with memory, spatial evidence, proposal validation, recovery, and executable navigation, achieving higher success across four benchmarks and demonstrating humanoid applicability.
Problem
Training-free navigation systems often lack mechanisms to reconcile proposed actions with spatial evidence, task progress, and execution failures.
Method
HarnessVLN coordinates perception, retrieval, grounding, navigation, recovery, and termination through an Agent Harness with event memory, a persistent Spatiotemporal Graph, and validated execution.
Results
HarnessVLN achieves 60.8%, 53.9%, 76.0%, and 59.3% success on R2R, RxR, HM3D-v2, and HM3D-OVON, respectively, and demonstrates applicability on a humanoid robot.
Takeaways & Limitations
A shared harness can support both instruction-following and object-goal navigation while maintaining consistency between semantic reasoning and physical execution.
Abstract
from arXiv · showhide
Embodied navigation requires agents to interpret visual observations, accumulate spatial knowledge, and execute actions to follow instructions or locate objects. Training-based methods face generalization challenges, while training-free methods exploit multimodal large language models (MLLMs) but often lack mechanisms to reconcile proposed actions with spatial evidence, task progress, and execution failures. We present HarnessVLN, a zero-shot, training-free framework whose Agent Harness coordinates perception, retrieval, grounding, navigation, recovery, and termination through a unified tool interface. The Harness validates planner proposals against spatial evidence, geometric feasibility, and subgoal consistency, incorporating structured tool feedback into subsequent decisions. Hierarchical event memory tracks task progress and execution history, while a persistent Spatiotemporal Graph maintains reusable spatial evidence and failure annotations for verification and recovery. A replaceable Navigation Executor converts validated targets into executable motions, allowing the same Harness protocol to support instruction-following and object-goal navigation. HarnessVLN achieves success rates of 60.8%, 53.9%, 76.0%, and 59.3% on R2R, RxR, HM3D-v2, and HM3D-OVON, respectively, surpassing prior training-free SOTA results. Humanoid deployment further demonstrates its applicability to both tasks in real-world environments. The project page is: https://harnessvln.netlify.app/.
1 Introduction
HarnessVLN addresses the gap between semantically plausible navigation plans and grounded execution by coordinating perception, memory, validation, recovery, and termination in one training-free harness. It supports both instruction-following and object-goal navigation, improving benchmark success and extending to humanoid deployment.
- MLLM planners can propose semantically plausible actions without sufficient spatial evidence or execution support, while stale observations and failures can derail later decisions.Reliable navigation therefore requires runtime validation linking spatial evidence, task progress, and execution outcomes.
- HarnessVLN coordinates perception, retrieval, grounding, navigation, recovery, and termination through a unified Agent Harness for instruction-following and object-goal navigation.The framework integrates an MLLM planner, hierarchical event memory, a persistent Spatiotemporal Graph, and specialized execution tools.
- The Harness validates proposals against supporting evidence, geometric feasibility, active subgoals, and failure history before dispatch, then feeds structured execution feedback into subsequent decisions.A replaceable Navigation Executor converts validated targets into executable paths, while stop validation checks task completion.
- The framework maintains spatial and temporal context by combining hierarchical event memory with a persistent Spatiotemporal Graph that preserves reusable evidence and failure links.These mechanisms support task-conditioned retrieval and failure-aware recovery.
- HarnessVLN achieves success rates of 60.8%, 53.9%, 76.0%, and 59.3% on R2R, RxR, HM3D-v2, and HM3D-OVON, respectively, improving over prior training-free state-of-the-art results.The improvements are 5.8, 12.1, 1.6, and 9.1 percentage points, respectively.
2 Related Work
Prior work develops zero-shot navigation, navigation foundation models, and agent harnesses, but commonly organizes perception, memory, planning, and control in task-specific systems. HarnessVLN instead provides a shared runtime with explicit execution validation and recovery across navigation tasks.
- Zero-Shot Embodied Navigation: Zero-shot navigation methods use pretrained language and vision models for object-goal perception, mapping, exploration, instruction decomposition, progress tracking, and target selection.
- Zero-Shot Embodied Navigation: These zero-shot systems typically integrate perception, memory, planning, and control through task-specific pipelines, whereas HarnessVLN uses shared execution validation and recovery.
- Navigation Foundation Models: Navigation foundation models learn general policies from large-scale trajectory data, while agent harnesses coordinate pretrained models, tools, memory, and execution feedback beyond monolithic policies.
3 HarnessVLN: A Harness-Mediated Embodied Navigation Agent
HarnessVLN coordinates planning, memory, spatial retrieval, grounded execution, recovery, and termination through a shared harness-mediated decision process. Its validation and feedback mechanisms connect semantic proposals to executable navigation while preserving task progress and reusable spatial evidence.
- Harness-Mediated Decision Making: HarnessVLN coordinates planning, memory, retrieval, grounding, navigation, recovery, and termination through a unified tool interface.The interface supports instruction following and ObjectNav without changing the underlying Harness protocol.
- Harness-Mediated Decision Making: Before execution, the Harness checks evidence provenance and recency, geometric reachability, active-subgoal consistency, and applicable failure history.Structured tool feedback reports execution status, measurements, and failure evidence for subsequent state and graph updates.
- Hierarchical Event Memory: Hierarchical event memory records short-term context, subgoal states, and execution outcomes, with completion requiring pose-aligned observation or successful tool results.Working memory is bounded, while reflection memory preserves detailed failure traces such as collisions, unreachable targets, and rejected stopping requests.
- Harness-Managed Spatiotemporal Graph: The persistent ST Graph stores time-indexed spatial structure, observations, waypoints, entities, relations, and failure annotations for retrieval, replanning, and recovery.Place nodes form a bidirectional navigable topology, while entity provenance links targets to viewpoints and observation times.
- Harness-Managed Spatiotemporal Graph: Task-conditioned retrieval ranks places by semantic relevance, recency, spatial salience, and prior-failure penalties, then augments the top-K set with nearby evidence.This produces a compact retrieval set without increasing MLLM context with trajectory length.
- Grounded Navigation Execution: The Navigation Executor converts validated targets into routes and local control, while termination requires evidence of semantic identity, geometric validity, and task progress.Rejected stopping requests trigger further observation, refinement, approach, or backtracking rather than direct environment-level stopping.
4 Experiments
HarnessVLN is evaluated across instruction-following and object-goal benchmarks, with cumulative ablations isolating memory, graph retrieval, and stop validation. It improves over training-free methods while revealing efficiency and termination trade-offs for stop validation.
- Instruction-Following Navigation: HarnessVLN has the highest SR among listed training-free methods on both R2R and RxR.It exceeds AgenticNav by 5.8 SR points on R2R and HSGM by 12.1 SR points on RxR.
- Object-Goal Navigation: 76.0% SR is highest among training-free methods on HM3D-v2, while HM3D-OVON reaches 59.3% SR and 36.6% SPL, highest among listed methods.On HM3D-OVON, HarnessVLN exceeds DRIVE-Nav by 9.1 SR points and 4.0 SPL points, although HM3D-v2 SPL remains lower.
- Ablation Studies: Event memory adds 8.0 and 7.0 SR points, graph retrieval adds 6.0 and 1.0, and stop validation adds 4.0 and 2.0 on R2R and HM3D-OVON subsets.The full Harness improves SR by 18.0 and 10.0 points over the base agent.
- Ablation Studies: Stop validation increases completion but does not uniformly improve efficiency or termination behavior across the two tasks.It lowers HM3D-OVON SPL from 34.2% to 33.0% and increases its OSR–SR gap from 18.0 to 19.0 points.
5 Real-World Deployment
HarnessVLN is deployed on a humanoid robot using a shared Harness protocol for instruction following, open-vocabulary search, and a combined route-and-search task. The system tracks subgoals, grounds targets, retains evidence, and validates stopping during these scenarios.
- Robot Platform and System Configuration: All deployment tasks use the same Harness protocol without training task-specific policies in the deployment environment.The deployment therefore covers both navigation task types under a shared runtime protocol.
- Robot Platform and System Configuration: The humanoid platform uses head-mounted stereo cameras for primary visual sensing, with dense metric depth estimated from synchronized stereo images.The robot is an AgiBot A3U equipped with LiDAR and multiple RGB-D and fisheye cameras.
- Real-World Navigation Experiments: Three real-world settings test sequential instruction following, open-vocabulary search, and a combined route-following and object-search task.The scenarios include stopping at landmarks or objects before continuing to subsequent targets.
- Real-World Navigation Experiments: The shared Harness tracks subgoal progress, grounds landmarks and targets, and validates stopping with semantic and spatial evidence.The ST Graph retains landmark and target evidence across navigation stages.
6 Conclusion
HarnessVLN unifies instruction-following and object-goal navigation through a shared Agent Harness that coordinates planning, memory, spatial evidence, and validated execution. Benchmark improvements and humanoid deployment support its applicability, while future work targets self-evolving orchestration.
- Conclusion: HarnessVLN unifies instruction-following and object-goal navigation through a shared Agent Harness.The framework coordinates MLLM planning, hierarchical memory, spatial evidence, and validated tool execution.
- Conclusion: Benchmark results show improved task completion across both navigation tasks, and humanoid deployment demonstrates real-world applicability.The current Harness relies on predefined orchestration and validation; future work will explore self-evolving mechanisms.
1 Additional Experimental Settings
The additional settings specify shared benchmark, observation, retrieval, distance-validation, and model-comparison protocols. They also define controlled evaluation choices for assessing configuration effects and model sensitivity.
- Benchmarks and Model: A single pretrained MLLM performs language and vision reasoning functions, including decomposition, planning, grounding, stop verification, and recovery.These functions use different prompts while sharing the same model within each experiment.
- Simulation and Context: Four-direction observations use headings of 0°, 90°, 180°, and 270°, while a full panorama requires twelve 30° rotations counted in the step budget.Forward-view history is sampled every two steps and thinned to fit the image budget.
- Memory Retrieval and Generation: ST Graph retrieval expands the top three seed places by one hop and returns at most five places, with subgoal-conditioned failure penalties capped against permanent exclusion.Semantic relevance uses local IDF-weighted cosine similarity.
- Distance Measurements: Geometric stopping uses fresh depth or projected-target distance with thresholds of 2.5 m and 1.0 m, respectively.Unreliable depth triggers the projected-target fallback, and these measurements do not replace benchmark geodesic distance.
- Model Sensitivity and Framework Comparison: Holding the Harness protocol and tools fixed across models measures model sensitivity, while shared-model framework comparisons use matched planner settings.The comparison is conducted on a fixed 100-episode HM3D-OVON subset.
2 Unified Model Prompts
The unified model prompts coordinate planning, progress tracking, grounding, and stop decisions through structured inputs and execution checks. They distinguish instruction-following endpoint verification from object-category verification before the Harness issues actions.
- Navigation planning: TODO items require stable identifiers and concrete observed evidence for completion, while blocked items retain an explicit reason.Only one item remains active, and earlier items can be revisited when later subgoals are complete.
- Navigation planning: The navigation prompt updates task progress before selecting one navigation or backtracking action from the revised context.It receives the instruction, history, directional views, TODO list, retrieved memory, feedback, and available actions.
- ObjectNav stop verification: Object grounding prompts reject look-alikes, require depth-based distance measurement, and return CONTINUE when category evidence is insufficient.Confident detections return a selected view and a tight normalized bounding box around the object.
- Execution contract: Stop decisions remain Harness requests: route verification checks the instruction endpoint, whereas ObjectNav verification checks the target category.The Harness applies geometric checks before issuing the simulator STOP action.
- Execution contract: Bounding boxes use [x1, y1, x2, y2], with normalized coordinates converted to pixels by xpx = Wx/1000 and ypx = Hy/1000.The conversion applies to a W × H image.
3 Simulation Visualizations
The simulation visualizations connect observations, maps, Harness state, and progress memory to recorded navigation outcomes. They illustrate both successful verification and failures where target proximity does not establish task completion.
- Simulation visualizations: Six recorded rollouts across four benchmarks illustrate execution behavior and failure modes, not aggregate performance estimates.GPT-5.5 is used for R2R/RxR and GPT-5.6-luna for ObjectNav, with one shared model across reasoning functions in each rollout.
- Reading the visualizations: The runtime dashboard links RGB observations, navigation maps, Harness state, task actions, and TODO memory for each episode step.Its panels also expose stop checks, ST Graph counts, executor counters, trajectories, targets, obstacles, and explored area.
- Reading the visualizations: The visualizations distinguish geometric maps and stop-check distances from evaluator diagnostics and terminal benchmark goal distances.The header goal distance is not an agent input, and verification views may differ from displayed forward RGB views.
- Success cases: Successful cases show projected-target or fresh-depth checks preceding termination across R2R, RxR, and HM3D-v2.Reported terminal goal distances are 2.39 m, 0.43 m, and 0.05 m respectively.
- Success and recovery cases: Fresh-depth verification rejects an apparently close ObjectNav stop at 4.65 m and later accepts a target at 1.53 m before success.HM3D-OVON 1297 succeeds at step 430 with a terminal goal distance of 0.13 m.
- Failure cases: A projected-target check can pass while the episode still fails because proximity to a selected target does not establish the correct instruction endpoint.In R2R 839, the check passes at 0.998 m, but termination remains 6.48 m from the goal.