Source-linked AI summary
Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation
Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah, Omkar Patil, Thao Nguyen, Nakul Gopalan
TL;DR
Metric-semantic language grounding must jointly resolve referents, spatial relations, and metric constraints in a 3D map, while existing evaluations lack such embodied queries. MAPG decomposes these components and composes their grounded outputs into continuous, planner-ready goals; on MAPG-Bench it reports improved metric and spatial consistency over comparison systems.
Problem
Metric-semantic grounding requires jointly reasoning over referent semantics, spatial predicates, and metric scale, while existing spatial benchmarks center on images and HM-EQA lacks metric-semantic predicate queries.
Source: §I · p. 1, §IV · p. 4Method
MAPG couples online 3D scene graphs with analytically defined continuous spatial kernels to produce planner-ready distributions for metric-semantic instructions.
Source: §I · p. 2Results
Across MAPG-Bench, MAPG produced more metrically accurate and spatially consistent outputs; against GraphEQA, it reduced O–W error from 5.82 m to 0.07 m.
Source: §V · p. 5, §VII · p. 8Takeaways & Limitations
Composed spatial belief maps yield goal likelihoods from which planners can extract waypoints in free space.
Source: §III · p. 4, §VI · p. 7
Abstract
from arXiv · showhide
Robots collaborating with humans must convert natural language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the right of the fridge" requires grounding semantic references, spatial relations, and metric constraints within a 3D scene. While recent vision language models (VLMs) demonstrate strong semantic grounding capabilities, they are not explicitly designed to reason about metric constraints in physically defined spaces. In this work, we empirically demonstrate that state-of-the-art VLM-based grounding approaches struggle with complex metric-semantic language queries. To address this limitation, we propose MAPG (Multi-Agent Probabilistic Grounding), an agentic framework that decomposes language queries into structured subcomponents and queries a VLM to ground each component. MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space. We evaluate MAPG on the HM-EQA benchmark and show consistent performance improvements over strong baselines. Furthermore, we introduce a new benchmark, MAPG-Bench, specifically designed to evaluate metric-semantic goal grounding, addressing a gap in existing language grounding evaluations. We also present a real-world robot demonstration showing that MAPG transfers beyond simulation when a structured scene representation is available.
I. INTRODUCTION
Metric-semantic instructions require jointly grounding referents, spatial relations, and measurable constraints in a map. MAPG decomposes these queries and composes probabilistic outputs into actionable goals, alongside a benchmark for evaluating metric-semantic grounding.
I. INTRODUCTION
- Metric-semantic grounding requires jointly reasoning over referent semantics, spatial predicates, and metric scale to identify a continuous region anchored to an object pose.Source: §I · p. 1
- MAPG decomposes queries into structured clauses, grounds referents against a 3D scene graph and egocentric view, then composes analytic kernels into a planner-queryable goal density.The planner can use the resulting density to generate executable waypoints.Source: §I · pp. 1–2
- MAPG is presented as a multi-agent probabilistic 3D reasoning framework coupling online scene graphs and continuous spatial kernels for metric-semantic instructions.Source: §I · p. 2
- The introduced HM3D goal-grounding benchmark contains 30 indoor scenes and 100 annotated metric-semantic queries for object-to-world grounding.Source: §I · p. 2
- MAPG reports 0.07 m distance error and angular errors of 0.3° yaw and 3.8° pitch in goal grounding.The work also provides a failure taxonomy for reproducible comparison.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
Prior work spans embodied language grounding, persistent 3D spatial memory, multimodal spatial reasoning, and probabilistic inference. MAPG targets the missing interface between structured spatial memory and planner execution for metric-semantic queries.
II. BACKGROUND AND RELATED WORK
- Embodied systems often commit to a single action or discrete target, while scene graphs provide spatial memory but lack metric information needed for planner-ready navigation goals.Prior VLN work uses backtracking and progress-based recovery to mitigate trajectory drift from greedy decisions.Source: §II · p. 2
- VLMs and LLMs show inconsistent spatial competence, and VLA policies typically predict actions rather than explicitly infer metric goals in a shared 3D frame.Source: §II · p. 2
- MAPG addresses the missing interface between structured 3D spatial memory and planner execution with a map-aligned, controllable mechanism for translating language into navigation targets.Source: §II · p. 2
III. METHODOLOGY
MAPG parses spatial language into symbolic clauses, grounds referents in a scene graph, and composes spatial and metric likelihoods into a continuous goal density. This density supports planner waypoint selection in free space.
III. METHODOLOGY
- The framework’s goal density represents locations satisfying semantic referents, spatial predicates, and metric constraints, and can support navigation or object selection.Source: Figure 2 · p. 3, Figure 3 · p. 3, Figure 4 · p. 4
- The Orchestrator parses instructions into symbolic clauses binding spatial predicates to referents, such as an anchor object, relation, and metric constraint.Multiple clauses form constraints composed downstream.Source: §III · p. 3
- The Grounding Agent resolves referents to scene-graph instances using label similarity, CLIP image-crop similarity, and a spatial-salience prior.A belief distribution over referents is updated on the shared memory ledger.Source: §III · p. 3
- The Spatial Agent represents each predicate as a continuous 3D probability density, with kernel parameters learned using VLMs.The framework uses von Mises or radial Gaussian kernels, a domain-specific choice reported to work well in its experiments.Source: §III · p. 4
- Directional and metric likelihoods are composed in log-space; compound constraints are warped into global coordinates and normalized into a density over free space.Aligned kernels reinforce one another, while conflicting kernels distribute mass between satisfying regions.Source: §III · p. 4
- A planner extracts navigation targets from the composed goal likelihood map by peak estimation or importance sampling, with top-k waypoints passed to an RRT* planner.Source: §III · p. 4
IV. DATASETS
Existing spatial-reasoning benchmarks do not test metric-semantic predicate grounding in embodied settings. MAPG-Bench evaluates metric grounding, predicate interpretation, and anchor disambiguation in navigable indoor environments.
Benchmark gap
- MAPG-Bench targets metric-semantic predicate grounding, which embodied QA benchmarks such as HM-EQA lack.Image-centric benchmarks also limit evaluation of viewpoint consistency, partial observability, and exploration.Source: §IV · p. 4
Benchmark scope
- MAPG-Bench tests allocentric localization, viewpoint variation through exploration, and compositionality through multi-anchor relations.It is a human-annotated dataset for metric grounding, predicate interpretation, and anchor disambiguation.Source: §IV · p. 4
Dataset overview
- Figure 5 shows the distributions of query categories and anchor-object labels in MAPG-Bench.The left panel shows query categories and the right panel shows anchor-object labels.Source: Figure 5 · p. 5
Construction
- The benchmark uses 30 HM3D indoor scenes and queries with sampled predicates and distances from 0.5 to 4.0 meters.Annotators explored scenes and recorded anchor instances and agent poses; object-to-world answers are navigable-surface points representing actionable waypoints.Source: §IV · p. 5
V. EXPERIMENTAL RESULTS
On MAPG-Bench, MAPG reduces object-to-world localization and directional errors relative to GraphEQA, with strong task and anchor-pick success. Ablations, occlusion tests, HM-EQA evaluation, and a small real-world demonstration characterize its performance and scope.
Evaluation protocol
- Evaluation tests planner-usable waypoints or object selections for anchor resolution, frame consistency, and metric constraints under viewpoint changes and partial observability.Comparisons include GraphEQA, SRGPT, and generalist VLMs; metrics cover localization, direction, selection, task success, anchor picking, and trajectory length.Source: §V · p. 5
MAPG-Bench results
- MAPG (Claude Opus 4.6) reaches 0.07 m O–O error, an 86% reduction from SRGPT’s 0.50 m.The passage attributes the improved metric satisfaction to spatial kernels anchored to resolved 3D object instances.Source: §V · p. 5
Model variation
- Model choice changes the operating point, but MAPG variants consistently show low metric error and improved directional consistency versus the scene-graph planner baseline.The authors identify pipeline structure—decomposition plus composition—as the primary driver of spatial-grounding performance.Source: §V · p. 5
MAPG-Bench results
- 5.82 m→0.07 m O–W error, 13.5°→1.9° yaw, and 27.9°→4.4° pitch; MAPG also reaches 0.98 Task SR and 0.98 anchor-pick SR.These comparisons are against GraphEQA; the reported average trajectory length is 1.3 m.Source: §V · p. 5, Table I · p. 6
HM-EQA
- GraphEQA achieves 0.67 HM-EQA accuracy, while MAPG variants are competitive; HM-EQA is a complementary diagnostic, not the primary metric-semantic grounding evaluation.It tests object selection and semantic state inference rather than metrically grounded waypoint prediction.Source: §V · pp. 5–6
Ablations
- Removing the explicit spatial reasoner drops object-selection SR from 0.42 to 0.20, while full MAPG improves over GraphEQA from 0.34 to 0.42.The authors interpret this as evidence that explicit composition of metric and predicate constraints is not recovered by prompting alone.Source: §V · p. 6
Occlusion robustness
- Under occlusion, the explicit spatial reasoner raises object-selection SR from 0.30 to 0.50 when the anchor eventually enters the map.The authors connect this result to maintaining intermediate anchor beliefs and deferring commitment.Source: §V · p. 6
Robot demonstration
- MAPG grounds three real-world queries with a structured scene graph, while commonsense QA is not the current design’s target strength.A systematic real-world benchmark would require a larger dataset with consistent scene-graph annotations and ground-truth spatial labels.Source: §V · p. 6
VI. OBSERVATIONS AND DISCUSSIONS
MAPG substantially improves metric-semantic grounding over GraphEQA, with the discussion attributing gains to query decomposition and probabilistic composition. The section also reports results on HM-EQA, a real-world example, and limitations tied to scene representation.
Real-world demonstration
- In the real-world example, MAPG places the grounded area 1 meter right of the blue trash can, in the general region of the bicycle.The example uses an offline scene graph built from Robotis AI Worker observations.Source: Figure 6 · p. 7
Overall performance
- MAPG (GPT-5.2) cuts O–W error from 5.82 m to 0.07 m and raises Task SR from 0.78 to 0.98 versus GraphEQA, with trajectories changing from 1.32 m to 1.3 m.Source: §VI · p. 7
Ablations
- Removing the explicit spatial reasoner reduces object-selection SR from 0.42 to 0.20, while under occlusion explicit reasoning raises it from 0.30 to 0.50.The table captions identify the comparisons as replacing the spatial reasoner with CoT and evaluating occlusion, respectively.Source: §VI · p. 7, Table III · p. 7, Table IV · p. 7
Multi-view grounding
- Multi-view anchor disambiguation delays commitment until evidence accumulates; MAPG reaches Anchor Pick SR up to 0.98 and Task SR 0.98 with a 1.3 m average trajectory.Source: §VI · p. 7
Composed belief maps
- Composing anchor, metric, and predicate likelihoods over free space yields planner-ready waypoints; MAPG (CO 4.6) achieves 0.07 m O–O error versus SRGPT’s 0.50 m.The composed density supports waypoint extraction as an allocentric coordinate for a motion planner.Source: §VI · p. 7
HM-EQA
- On HM-EQA, MAPG (Claude 4.6 Opus) scores 0.71, 0.04 above GraphEQA-Gemini-2.5Pro’s 0.67, while MAPG (OpenAI GPT-5.2) scores 0.60.Source: §VI · p. 8
Limitations
- Failures arise from incomplete scene graphs, frame-of-reference ambiguity, and misalignment between the free-space map and scene graph.Occluded objects may be absent from the map, and misalignment can blur the composed belief map and reduce waypoint precision.Source: §VI · p. 8
VII. CONCLUSION
The conclusion presents MAPG as a compositional probabilistic framework for converting metric-semantic language into planner-ready 3D goals, alongside a benchmark designed for this grounding task. It reports improved grounding results over baselines, a real-world transfer demonstration, and limits associated with scene representation.
Results
- Across MAPG-Bench, MAPG reduces object-to-world grounding error from 5.82 m to 0.07 m and improves task success rate from 0.78 to 0.98 in its best setting.Source: §VII · p. 8
Approach
- MAPG decomposes queries into anchor, predicate, and metric components, grounds them against an online 3D scene graph and egocentric observations, and composes them into a continuous goal distribution.Source: §VII · p. 8
Benchmark and scope
- MAPG-Bench covers 30 HM3D scenes and 100 annotated queries, while the real-world zero-shot demonstration suggests transfer beyond simulation.Source: §VII · p. 8