Source-linked AI summary
AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation
Long Giang Vu, Chengkai Yao, Yuxin Liu, FNU Aryan, Rajath Chandrashekar Aralikatti
TL;DR
VLMs support open-vocabulary grounding but struggle to produce reliable metric quantities and spatial comparisons from images. AnchorVLN separates semantic proposals from geometric measurements through a phrase-and-handle tool server. It reports instruction-following performance and object-reference hits on the CMU VLN Challenge 2026.
Problem
VLMs can ground open-vocabulary expressions but may invent metric ranges or misjudge comparative spatial relations, while existing remedies require pipeline changes or model-generated waypoints.
Source: §I · p. 1Method
AnchorVLN uses a VLM for semantic proposals and a phrase-and-handle MCP server whose geometry supplies metrics without accepting client-authored distances or bearings.
Source: §I · p. 2Results
The system scores 64.4% on instruction following and records 22% vs. 0% 3D-IoU hits when geometry rather than the model emits object coordinates.
Source: §I · p. 2Takeaways & Limitations
The tool schema keeps model-authored metric coordinates out of the robot command path while leaving the downstream autonomy stack unmodified.
Source: §I · p. 2Evaluation uses one frontier model and the development set, and each model call plus triggered step costs roughly half a minute.
Source: §V · pp. 5–6
Abstract
from arXiv · showhide
Vision-Language Navigation (VLN) in unseen indoor environments is useful in real-world robotics, where an agent must follow natural-language instructions, locate objects, and answer spatial questions without a pre-built map or fixed object vocabulary. Multimodal vision-language models (VLMs) provide strong open-vocabulary grounding and zero-shot reasoning, but struggle to emit reliable metric quantities such as range, bearing, and comparative spatial relations directly from images. Existing approaches address this by folding geometry into hand-engineered pipelines or asking models to output waypoints, requiring changes to the control stack for different robots, tasks, or vocabularies. We introduce AnchorVLN, an open-vocabulary VLN system built on a simple rule: the VLM proposes semantics; geometry decides metrics. It is realised as EMBODIED-NAV-MCP, a Model Context Protocol (MCP) server driven by a VLM agent through a compact set of callable tools. Since no tool accepts distance in metres or bearing in radians, the schema enforces the semantic-geometry boundary without modifying the downstream autonomy stack. We benchmark both tasks of the CMU Vision-Language Navigation Challenge 2026: 30 instruction-following questions over 15 scenes and a frozen 45-question object-reference set. The full system achieves 64.4 percent on instruction following, dropping by 13.3 percentage points without controller modeling (t = 2.77). On object reference, geometric anchoring clears the challenge overlap threshold on 10 of 45 questions, versus 0 of 45 for direct coordinate estimation, reducing median center error from 3.37 m to 2.48 m.
I. INTRODUCTION
The introduction frames metric grounding as the remaining difficulty for VLM-based navigation: models handle open-vocabulary semantics but can invent ranges or misjudge spatial comparisons. AnchorVLN separates semantic proposals from geometric measurements through a tool interface and evaluates that design on the CMU VLN Challenge 2026.
I. INTRODUCTION
- Earlier geometric stacks used metric maps and closed-vocabulary detectors, leaving free-form referring expressions and compositional phrases out of reach without extra supervision,,.VLN reframed navigation as following natural-language instructions in unseen buildings, [5].Source: §I · p. 1
- VLMs can ground open-vocabulary expressions, but still invent metric ranges or misjudge comparative spatial relations.Existing remedies either require per-robot pipeline changes or leave waypoint output to the model.Source: §I · p. 1
- AnchorVLN treats metric failure as an interface problem: the VLM proposes semantics while geometry decides metrics.The MCP schema accepts phrases and opaque handles, not metric distances or bearings, so model-authored coordinates cannot reach the robot.Source: §I · p. 2
- A VLM agent composes EMBODIED-NAV-MCP tools at inference time instead of relying on hand-scripted per-tick planners.The server exposes semantic grounding and geometric operations through a compact tool set.Source: §I · p. 2
- 64.4% on instruction following and 22% vs. 0% 3D-IoU hits when geometry, rather than the model, emits object coordinates.The evaluation uses the CMU VLN Challenge 2026 development set.Source: §I · p. 2
II. RELATED WORK
Related work spans geometric mapping, VLN, object-goal navigation, and VLM-driven planning. AnchorVLN draws on prior semantic-planning designs but keeps its own controller-independent stack focused on ordered trajectories and object-level referring.
A. Vision-Language Navigation
- Geometric mapping with closed-vocabulary detectors,, [16]–, underlies earlier navigation; VLN, [5] extends the setting to natural-language instructions and continuous control.Related lines include zero-shot ObjectNav [19], [20] and referential 3D benchmarks VLA-3D [21] and IRef-VLA [22].Source: A. Vision-Language Navigation · p. 2
B. Vision-Language Models for VLN
- VLM navigation ranges from landmark grounding and explicit reasoning, [24], [6] to action prediction, [25] and learned-controller systems [9], [26],.AO-Planner [11] also avoids model-generated 3D coordinates, while AnchorVLN takes metres from LiDAR and does not own the controller.Source: B. Vision-Language Models for VLN · p. 2
C. Agentic Systems for Real-World Tasks
The paper uses an agentic client–server tool pattern, with a phrase-and-handle interface separating language choices from measurements and robot motion. The server grounds referents, resolves spatial instructions, and hands waypoints to an autonomy stack it does not own.
C. Agentic Systems for Real-World Tasks
- The Model Context Protocol standardizes a client–server tool schema for agents, and EMBODIED-NAV-MCP uses phrase or handle arguments rather than numbers.This follows the broader pattern of language-model agents invoking tools and skills instead of outputting motor commands –.Source: C. Agentic Systems for Real-World Tasks · p. 2
A. The interface
- Every tool input is a phrase or an opaque handle, which names a server-held record carrying measurements rather than a client-authored location.The interface exposes ten tools; the client reads instructions and chooses calls, while most tools perform arithmetic over existing object records.Source: A. The interface · pp. 2–3, Table I · p. 3
- Instructions are processed as ordered destinations, keep-outs, and passages; the server writes a Pose2D waypoint and reads arrival from odometry.The platform stack supplies sensor data, while publish returns a box based on geometry committed to the server’s object graph.Source: A. The interface · p. 3
B. Perception
Perception uses VLMs to identify image regions, then LiDAR and stored object records to ground those regions geometrically and answer spatial comparisons.
B. Perception
- ground unwraps the panorama into overlapping gnomonic faces, obtains candidate boxes from the VLM, converts box centres to bearings, and estimates range from nearby LiDAR returns.It uses median range within a narrow cone to avoid averaging foreground and background surfaces into a plausible but wrong distance.Source: B. Perception · p. 3
- Three checks can reject a metric lift: an elevation floor, an implied-size inconsistency, or too few LiDAR returns in the cone.A rejected lift still returns an unmeasured handle; drive_to can move a bounded distance along its known bearing, changing the geometry for a later attempt.Source: B. Perception · p. 3
- Nearest, farthest, and between resolve comparisons over persistent identity-keyed object records, which update across LiDAR sweeps and viewpoints.The object graph stores the 3D boxes used to answer object-reference questions.Source: B. Perception · p. 3
C. Motion
Motion tools adapt to an external controller while exploration and orbit improve what the scene graph knows and measures. Evaluation covers 15 scenes and two tasks, with trajectory-based scoring for instruction following and box-based metrics for object reference.
C. Motion
- The server predicts the external controller’s resting place and publishes the waypoint expected to land nearest the target; arrival is checked using odometry.Porting to another platform requires replacing this controller model, while the tools above it remain unchanged.Source: C. Motion · p. 3
- Exploration drives toward unseen-space frontiers when a phrase remains unresolved, while orbit adds viewpoints around a reached object to refine its geometry.Exploration expands the known object set; orbit can reveal occluded faces and tighten an object’s extent.Source: C. Motion · pp. 3–4
A. Experimental Setup
- The scene graph is built from open-vocabulary detections, segmented masks, and registered LiDAR, with filtering thresholds applied to detections and masks.The detector and segmenter run at 2 Hz; boxes require at least 10 LiDAR returns, and nearby viewpoints are gated to suppress redundant detections.Source: A. Experimental Setup · p. 4
- The development set contains 15 indoor scenes, each with two navigation instructions, and questions allow 10 minutes with navigation cut off at 540 seconds.The robot starts from a fixed pose without a prior map; the final 60 seconds are reserved for committing an answer.Source: A. Experimental Setup · p. 4
- Instruction following is scored on driven trajectories, while object reference is scored on published oriented boxes using distinct task-specific metrics.Instruction metrics include Challenge Score, nDTW, trajectory-length ratio, and closest approach; object-reference metrics include hit rate, mean IoU, and centre distance.Source: A. Experimental Setup · p. 4
- Object-reference evaluation reports mean IoU and threshold-free centre distance, while instruction-following scores include destination and path-constraint adherence.Challenge Score awards six points across destinations and path constraints, with a 1.25 m arrival tolerance.Source: A. Experimental Setup · p. 4
B. Instruction Following
Controller modeling materially affects instruction-following scores, while trajectory distance can reveal differences that the score misses. Ablations also show that aim-point accuracy and actual stopping location need not move together.
B. Instruction Following
- 13.3 points are lost when the platform model is removed, whereas removing the lift costs 0.8 points, consistent with zero at the reported resolution.The platform-model removal has t = 2.77; the lift’s scan-range median absolute error is 0.16 m versus 2.80 m for the model’s distance estimate.Source: B. Instruction Following · p. 4
- The platform model’s waypoint replanning reduces a median 1.47 m aim difference to 0.15 m in stopping location, making the lift’s measured return fewer grounding calls rather than higher score.The lift saves 1.45 grounding calls per question (t = 2.86).Source: B. Instruction Following · p. 4
- A 1.00× trajectory-length ratio after removing both components is indistinguishable from correct distance, yet the score is 10.0 points below the full system.Without the platform model, the robot is 0.52 m further from destinations on 24 of 30 questions; the arrival tolerance is 1.25 m.Source: B. Instruction Following · p. 4
C. Object Reference
Geometric anchoring produces threshold-clearing object boxes on some questions, while direct model coordinate estimates produce none. Removing the MCP structure changes hit rate modestly, but also removes tools that support exploration and vocabulary refinement.
C. Object Reference
- 22% of 45 referring questions clear the 3D-IoU threshold with the full loop, with mean IoU 0.060 and a mean centre error of 36 cm on hits.The result is described as modest coverage but geometrically tight when the system commits.Source: C. Object Reference · p. 5
- Removing MCP structure changes object-reference hit rate from 22% to 18% with mean IoU essentially unchanged, but the naive arm also loses exploration and vocabulary tools.Without exploration, its graph is smaller on 30 of 45 questions; without vocabulary priming it substitutes a coarser label in the cited example.Source: C. Object Reference · p. 5
- 0/45 answers clear the overlap threshold when the model emits the final coordinates directly, and its default cube leaves no extent for the overlap metric to credit.For the bedroom-light query, the grounded system lands 0.23 m from truth while direct estimation misses by 6.0 m.Source: C. Object Reference · p. 5
D. Qualitative Analysis
A qualitative episode shows the model handling two ordered destinations from successive viewpoints. A separate exploration example contrasts object selection from a larger scene graph with selection from a sparse one.
D. Qualitative Analysis
- The two-clause instruction is completed in order: the model drives to the coffee table with the kettle, then to the dining table near the big picture, and stops.At the first stop, the table is partly occluded and the model skips a verification view before grounding the next clause.Source: D. Qualitative Analysis · p. 5
- For the same table-nearest-to-dish query, the reach↔explore controller builds a 25-object graph and selects the low tatami table, while the naive arm gathers 10 objects and chooses incorrectly.Source: Figure 2 · p. 5
V. DISCUSSION AND LIMITATIONS
The conclusion reports instruction-following and object-reference results for EMBODIED-NAV-MCP, while the discussion bounds the evidence to a simulator-based development-set evaluation with substantial deployment and evaluation caveats.
V. DISCUSSION AND LIMITATIONS
- The evidence is bounded by roughly half-minute model-call-and-step latency, one frontier model, one simulator and robot, and no real-hardware evaluation.The authors say deployment outside simulation would require a faster ground-role model, a client that commits to several steps per call, or both; porting the controller is asserted but untested.Source: §V · p. 5
- Question-driven vocabulary expansion grounds a console-table nautilus shell sculpture absent from the fixed 100-class prior, where the naive arm selects a wall picture.Source: Figure 3 · p. 6
- Every reported number is from the development set, and the closed evaluator leaves arrival tolerance and instance matching to the authors.Source: §V · pp. 5–6
VI. CONCLUSION
- 64.4% across all 30 instruction-following questions on 15 scenes, and the system clears the object-reference overlap threshold on 10 of 45 questions.Source: §VI · p. 6
APPENDIX
The appendix quantifies which arm differences are resolvable and reports that removing both components produces a resolved score loss, while the proposed distance-correction mechanism does not explain that loss.
APPENDIX
- The appendix reports the design’s resolvable effects and omitted arms; Table IV covers all five arms on the same 30 questions and flags two significant arms.Source: APPENDIX · p. 8, Table IV · p. 8
A. What this design can resolve
- At 30 questions, the smallest resolvable difference between single-pass arms is near 9.4 points; Naive clears that floor, but w/o Memory does not.A repeated identical configuration differed by as much as 5.8 points, with roughly a quarter of corpus-mean variance attributed to run-to-run noise.Source: A. What this design can resolve · p. 8
B. Removing both components
- Naive scores 54.4% against an additive prediction of 50.3%, but its +3.3-point gap over w/o Platform Model is unresolved at t = 0.83.Source: B. Removing both components · p. 8
- 10.0 points below the full system at t = 2.38, the combined-fault arm is resolved on 8 of the 9 questions where the two differ.The four added questions carried the comparison over the resolution threshold rather than changing the arm’s result; the loss occurs whether or not the range reaching the waypoint was measured.Source: B. Removing both components · p. 8
- Path length reaches 1.00× the reference after adding the model’s long range to w/o Platform Model’s 0.79× path, but score gains do not track shortfall.The per-question relationship is r = 0.27, t = 1.45; Naive is also 0.27 m farther from the referenced destination than the full system (t = 2.25).Source: B. Removing both components · p. 8
C. Removing the prompt’s memory
Removing the prompt’s list of previously occupied poses does not measurably change the four predicted movement quantities, and the score difference is too small to support a benefit or harm claim. The drive loop’s separate refusal to revisit a spent bearing remains unchanged.
C. Removing the prompt’s memory
- The ablation removes only a prompt instruction to prefer unvisited poses; the drive loop’s separate refusal to revisit a spent bearing remains untouched.The prompt block lists poses the robot has already occupied.Source: C. Removing the prompt’s memory · p. 8
- Revisit rate is 27 % against 28 %, positional spread 2.64 m against 2.72 m, steps per question 8.9 against 8.8, and target-visible steps 94 % against 94 %.Paired t values are −0.10, −0.64, 0.11, and 0.12, with every sign test near-tied.Source: C. Removing the prompt’s memory · p. 8
- 3.9 points in favour of removing the block (t = 1.13) is smaller than the 5.8-point spread between identical-configuration passes.The ablation perturbs 6 of 30 questions, versus 8 to 9 for a plain re-drive; the authors conclude only that removing the block does not hurt.Source: C. Removing the prompt’s memory · p. 8
- Reference trajectories double back over 15 % of their length on average, and 21 of 30 return within 0.5 m of a point left at least a metre earlier.The instruction to prefer unvisited ground therefore argues against behavior present in the organisers’ reference trajectories.Source: C. Removing the prompt’s memory · p. 8