PaperlayerView exact revision on arXiv

Source-linked AI summary

Map2Route: Benchmarking Compositional Language-Grounded Route Planning over Semantic Maps

Muyi Bao, Hang Xu, Jingfan Tang, Zihan Liu, Yuxin Cai, Chen Lv, Wenshan Wang, Ji Zhang

arXiv:2609.17910v1cs.RO

TL;DR

Map2Route addresses the limited joint evaluation of compositional language grounding, route requirements, and soft preferences over complete routes. It introduces a human-curated benchmark and Grounding2Route, which substantially outperforms adapted baselines, though a substantial gap to human demonstrations remains.

  • Problem

    Existing work rarely evaluates compositional grounding, multi-stage route requirements, and scoped preferences jointly over complete routes.

    Source: §I · p. 2

  • Method

    The paper introduces Map2Route and proposes Grounding2Route, which uses executable code-as-grounding and scope-aware planning to produce routes from language over semantic maps.

    Source: §I · p. 2

  • Results

    57.9% hard-requirement satisfaction is achieved by Grounding2Route, compared with 29.4% for the strongest baseline, and it outperforms adapted baselines across metrics.

    Source: §I · p. 2

  • Takeaways & Limitations

    Map2Route remains far from saturated, with a substantial gap between Grounding2Route and human demonstrations indicating headroom for future progress.

    Source: §VI · p. 8

    The benchmark isolates grounding and planning from online perception, covers 40 scenes annotated by two experts, and excludes dynamic or multi-floor settings.

    Source: §VI · p. 8

Abstract

from arXiv · show

We introduce Map2Route, a human-curated benchmark for compositional language-grounded route planning over pre-built semantic maps. Map2Route contains 1,000 episodes across 40 scenes, where instructions use relational, comparative, and nested descriptions to identify route-relevant objects and regions, while specifying ordered must-pass regions, must-avoid requirements, five categories of soft preferences, and spatial and route-stage scopes, which is partially tested by existing works. Alongside Map2Route, we propose Grounding2Route, which combines executable code-as-grounding with verification-guided repair and scope-aware planning.Across seven representative adapted baselines, Grounding2Route substantially outperforms existing methods in all metrics. Despite these gains, a substantial gap to human demonstrations remains, highlighting the difficulty of Map2Route and the considerable headroom for future progress. Additional qualitative results and resources are available on https://anonymous.4open.science/w/Map2Route-F05F/.

I. INTRODUCTION

Map2Route tests whether language-grounded route planning over semantic maps can jointly handle compositional references, ordered requirements, and scoped preferences. Grounding2Route is a structured reference method, and evaluation shows substantial gains over adapted baselines while leaving a large gap to human demonstrations.

I. INTRODUCTION

  • The task requires resolving relational, comparative, and nested references while following route preferences and complex multi-room, ordered, or conditional requirements.For example, instructions can specify a target through several linked spatial relations or impose an order on destinations.Source: §I · pp. 1–2
  • Map2Route jointly evaluates compositional grounding, multi-stage route requirements, scoped soft preferences, and path efficiency over complete routes.Existing work has often studied these capabilities separately and with limited compositional complexity.Source: §I · p. 2
  • Grounding2Route combines executable code-as-grounding with a Route Semantic Intermediate Representation that is deterministically compiled into a route through scope-aware planning.The framework is presented as a two-stage structured language-to-route method.Source: §I · p. 2
  • 57.9% hard-requirement satisfaction is achieved by Grounding2Route, versus 29.4% for the strongest baseline, with Grounding2Route outperforming all adapted baselines across metrics.The comparison is reported on Map2Route alongside shortest-feasible A* and human demonstrations.Source: §I · p. 2
  • The paper contributes a human-curated benchmark, a reference planning framework, and empirical analysis of adapted methods, task components, difficulty, and real-world trials.The study reports that seven adapted methods struggle substantially on the task.Source: §I · p. 2

II. RELATED WORK

Prior work addresses different parts of language-guided planning over pre-built maps, while Map2Route evaluates their combination over complete routes. Online language-guided navigation instead studies interpretation and decision-making from observations without assuming a pre-built semantic map.

A. Language-Guided Planning over Pre-built Maps

  • Prior methods separately emphasize grounding, task-level reasoning, temporal hard requirements, or behavioral preferences, whereas Map2Route jointly tests these capabilities over complete routes.The cited formulations differ in how naturally they address graded route-geometry preferences and complex compositional grounding.Source: A. Language-Guided Planning over… · p. 2

B. Online Language-Guided Navigation

  • Online language-guided navigation uses egocentric observations for perception, grounding, and sequential decisions, while Map2Route assumes an available semantic map and plans a complete route over it.The cited online tasks include VLN, ObjectNav, REVERIE, and SOON.Source: B. Online Language-Guided Navigation · p. 2

III. MAP2ROUTE

Map2Route asks for a complete route over a semantic map that realizes a natural-language instruction. Its benchmark combines diverse compositional instructions and requirements, with systematically harder episodes and documented scale and complexity.

A. Task Formulation

  • The task maps a semantic map, initial position, and instruction to a complete route that faithfully realizes the instruction without explicit goal coordinates.The semantic map contains aligned traversability, object, and room layers.Source: A. Task Formulation · p. 2

1) Semantic Map Representation:

  • Each environment uses aligned traversability, object, and room maps at 0.05 m/grid resolution, derived offline from ProcTHOR annotations and AI2-THOR metadata.Object footprints come from projected 3D bounding boxes, and room labels from rasterized room polygons.Source: 1) Semantic Map Representation · p. 2

2) Compositional Route Instructions:

  • Instructions combine ordered must-pass and must-avoid regions with five soft-preference types, and may assign spatial or route-stage scopes.The five types are Near, Far, Relative, Path-Shape, and Clearance; the decomposition is used for annotation and evaluation, while methods receive the original instruction.Source: 2) Compositional Route Instructions · p. 3

B. Benchmark Construction

  • The evaluation set contains 1,000 episodes across 40 scenes, and Hard episodes are more complex than Easy episodes across all reported dimensions.The benchmark spans 96 object categories and 4 room types; two experts annotated and cross-checked episodes, followed by quality control.Source: B. Benchmark Construction · p. 3

C. Benchmark Metrics

Map2Route evaluates route planning through hard-requirement satisfaction, soft-preference adherence, and path efficiency. Its hard score measures ordered must-pass completion and treats violations as consequential to credit.

C. Benchmark Metrics

  • The benchmark reports hard-requirement score, soft-preference adherence, and path efficiency as three complementary evaluation dimensions.The hard score covers language-specified destinations and ordered requirements; soft adherence checks preferences within their intended scopes.Source: C. Benchmark Metrics · p. 3

1) Hard Constraint Score (HCS):

  • HCS is the ratio of correctly reached must-pass regions in the specified order to the total number of must-pass regions.A region is reached when the route intersects it; a route-stage violation stops further credit, while a global violation yields HCS = 0.Source: 1) Hard Constraint Score (HCS) · p. 3

C. Benchmark Metrics

  • Five route-shaping preference types are measured separately because they differ in scale and optimization direction, with valid measurements averaged within episodes and then across episodes.The types cover distinct route preferences and are selected to avoid highly redundant evaluation.Source: 2) Soft Preference Metrics · p. 3

2) Soft Preference Metrics:

This section specifies selected soft-preference scoring rules and route-efficiency reporting through path length and SPL.

2) Soft Preference Metrics:

  • Relative means staying closer to reference A than B, measured using their mean distances, DA and DB.Source: 2) Soft Preference Metrics · p. 3
  • For Near, Far, Relative, and Path-Shape, failure to enter the target region or complete the segment receives the metric-specific worst value and remains in the denominator; Clearance is always global.Source: 2) Soft Preference Metrics · p. 4
  • 0.99 on Easy and Hard: independently drawn human trajectories receive nearly identical tolerance-aware nDTW Path-Shape scores, supporting use of a single canonical reference.Source: 2) Soft Preference Metrics · p. 4

3) Path Efficiency:

  • Path efficiency reports PL and SPL; SPL uses transition success and segment length against the A* shortest feasible segment under the same hard constraint.Source: 3) Path Efficiency · p. 4

IV. GROUNDING2ROUTE

Grounding2Route separates route-semantic grounding from planning: executable map programs produce a RouteIR that is verified and repaired, then compiled into a complete route.

IV. GROUNDING2ROUTE

  • Grounding2Route uses grounding and planning stages: executable map APIs and verification-guided repair produce grounded RouteIR, which scope-aware sequential planning compiles into a complete route.Source: §IV · p. 4

A. Executable Code-as-Grounding

  • Code-as-grounding composes predefined read-only semantic-map APIs and helper logic into a program; execution yields provisional grounding, with map-dependent bindings computed during execution.Source: A. Executable Code-as-Grounding · p. 4
  • The API spans six primitive categories; for Path-Shape, planning enforces waypoint order because additive spatial costs cannot reliably preserve topological ordering.Source: A. Executable Code-as-Grounding · p. 4
  • The sandbox bars raw-grid access, arbitrary coordinates, planner-cost access, and direct trajectory output, keeping entity selection and geometry within a deterministic interface.Source: A. Executable Code-as-Grounding · p. 4

B. Route Semantic Intermediate Representation

  • RouteIR separates ordered grounded destinations from grounded constraints, which carry spatial and route-stage scopes and encode Near, Far, Relative, PathShape, or Avoid semantics; Clearance is global.Source: B. Route Semantic Intermediate… · p. 5

C. Grounding Verification and Bounded Repair

Grounding2Route verifies generated grounding specifications, repairs detected errors within a bounded budget, then plans route segments sequentially while applying constraints only within their scopes.

C. Grounding Verification and Bounded Repair

  • Verification checks reference validity and cardinality, constraint signatures, segment and scope consistency, and map-dependent relations, but does not certify semantic equivalence to the instruction.Detected errors return structured feedback for repair; the repaired program is re-executed and re-verified until verification succeeds or the repair budget is exhausted.Source: C. Grounding Verification and Bounded… · p. 5

D. Scope-Aware Sequential Route Planning

  • For each segment, deterministic cost-aware A* connects an ordered sequence of targets, including Path-Shape waypoints and the grounded destination; consecutive trajectories form the complete route.Planning proceeds sequentially over the route segments.Source: D. Scope-Aware Sequential Route Planning · p. 5
  • Hard avoidance defines segment-specific traversability masks, while Near, Far, and Relative preferences contribute spatial costs only within their grounded route-stage and spatial scopes.Clearance is applied globally; if a stage is infeasible, planning terminates without changing RouteIR or triggering further LLM repair.Source: D. Scope-Aware Sequential Route Planning · p. 5

V. EXPERIMENT RESULTS

The experiments assess benchmark difficulty, method components, complexity effects, and real-world execution, using adapted baselines and evaluations across Easy and Hard episodes.

V. EXPERIMENT RESULTS

  • The experiments address baseline difficulty, Grounding2Route comparisons, component ablations, benchmark complexity, and real-world execution.Results are evaluated on the full benchmark and reported separately for Easy and Hard episodes.Source: §V · p. 5, A. Experiment Setup · p. 5

A. Experiment Setup

  • LLM-based methods use Gemini-3.5-Flash; Grounding2Route’s preference weights are tuned on the development set and fixed for benchmark evaluation.The setup includes a real-world evaluation using a mobile robot, semantic maps, and local collision avoidance and path following.Source: A. Experiment Setup · p. 5
  • Seven representative methods are adapted to Map2Route while retaining their original reasoning representations and planning pipelines; all receive the same map, traversability grid, instruction, and start pose.The comparison measures performance under the unified Map2Route setting, not on the methods’ original tasks.Source: A. Experiment Setup · p. 5
  • Table II compares performance on Map2Route across Easy and Hard difficulty levels.The table’s comparison is conducted alongside shortest-feasible A* and human demonstrations.Source: §V · p. 5, Table II · p. 6

B. Main Benchmark Performance

Grounding2Route outperforms adapted baselines across benchmark dimensions, while controlled grounding-strategy ablations favor executable program composition over iterative tool calling.

B. Main Benchmark Performance

  • The best baseline achieves only 0.368/0.124 HCS on Easy/Hard, while its best Path-Shape scores are 0.02/0.01 versus 1.00 for humans.These results indicate difficulty resolving compositional references and satisfying soft constraints.Source: B. Main Benchmark Performance · p. 6
  • Grounding2Route achieves 0.667/0.375 HCS and 0.53/0.29 SPL on Easy/Hard, beats seven baselines on five soft-preference metrics, yet trails humans on Hard (0.375 HCS, 0.08 Path-Shape).On Hard, human demonstrations score 1.00 HCS and 0.99 Path-Shape.Source: B. Main Benchmark Performance · p. 6

1) Grounding strategy:

  • CaG achieves 0.579/0.46 HCS/SPL, versus 0.338/0.27 for Schema, 0.474/0.39 for Direct ID, and 0.539/0.44 for ToolCall.The controlled comparison finds executable program composition more effective than iterative tool calling for compositional route grounding.Source: 1) Grounding strategy · p. 6

2) Intermediate representation:

The ablations examine the intermediate representation, verification-guided repair, scope-aware soft-constraint modeling, and planning with oracle grounding.

2) Intermediate representation:

  • Replacing RouteIR with LTL reduces HCS/SPL from 0.579/0.46 to 0.439/0.36, indicating RouteIR better represents Map2Route’s heterogeneous route semantics and scopes.Source: 2) Intermediate representation · p. 6

3) Verification and repair:

  • Increasing the repair budget from R = 0 to R = 3 raises HCS from 0.395 to 0.579, while Relative improves from 0.56 to 1.11.The reported gains extend to grounding soft preferences and their scopes, not only hard requirements.Source: 3) Verification and repair · p. 6

4) Soft-constraint modeling:

  • Globalizing temporal or spatial scopes leaves HCS nearly unchanged but lowers Far from 2.52 to 1.85–1.91 and Relative from 1.11 to 0.84–0.87; removing soft costs improves efficiency but reduces soft metrics.The qualitative real-world examples also show indoor execution of compositional instructions with scoped Far preferences and must-avoid requirements.Source: 4) Soft-constraint modeling · pp. 6–7, Figure 4 · p. 7

5) Planner over Oracle Grounding:

  • With oracle grounding, the planner reaches all targets and scores 2.09 / 3.33 / 2.06 on Near / Far / Relative, improving over shortest-path A* and recovering 74% of the A*-to-human gap on average.The results identify grounding as the dominant source of full-pipeline error, with a smaller but non-negligible gap in soft-constraint-aware planning.Source: 5) Planner over Oracle Grounding · p. 7

D. Benchmark Difficulty Analysis

Grounding2Route remains stronger than OSG-LLM as reference depth, instance ambiguity, and hard-constraint complexity increase, though both methods degrade. Real-world evaluation and the remaining gap to human performance underscore the benchmark’s difficulty and scope.

D. Benchmark Difficulty Analysis

  • HCS decreases for Grounding2Route and OSG-LLM as reference depth, instance ambiguity, and hard-constraint complexity increase; Grounding2Route remains stronger but also declines in the hardest settings.Degradation is larger under deeper references and more hard constraints.Source: D. Benchmark Difficulty Analysis · p. 7, Figure 5 · p. 8

E. Real-World Experiments

  • Across 48 episodes in 8 indoor scenes, Grounding2Route achieves 72.22% HCS; semantic-map construction errors account for 73.75% of HCS loss.Grounding2Route failures in grounding or route planning account for 22.5% of HCS loss, and SLAM issues account for 3.75%.Source: E. Real-World Experiments · p. 8

VI. CONCLUSION

  • Grounding2Route consistently improves hard-requirement satisfaction, soft-preference adherence, and path efficiency, yet a large gap to human performance remains.The conclusion also reports additional validation in real-world environments and substantial headroom for future research.Source: §VI · p. 8
  • Map2Route isolates grounding and planning from online perception using preconstructed semantic maps, and does not cover dynamic or multi-floor settings.Its 40 scenes and two expert annotators may limit scene diversity and the diversity of linguistic styles and annotation strategies.Source: §VI · p. 8

Related papers

Loading 2609.17910v1…