Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

1 INTRODUCTION

• City- and regional-scale reasoning involves heterogeneous geographic features and topological, directional, metric, and ordering relations across larger extents and multiple scales.Prior work on world models, embodied AI, and multimodal spatial reasoning has primarily evaluated confined indoor or local scenes and short trajectories.Source: §1 · p. 1
• Existing geospatial evaluations leave unclear whether models can both recognize spatial relations and use them to modify configurations under constraints.A correct QA answer may not show that a relation can guide a scene change, while editing has usually been evaluated separately from QA.Source: §1 · p. 2
• GeoSpatialBench combines QA and constrained spatial editing over 28 relations and 8,400 real-world instances from six cities, using GeoJSON scenes or maps rendered from the same geometry.The relations span topological, directional, metric, and ordering families.Source: §1 · p. 2
• GeoJSON generally supports stronger geospatial reasoning than map images, highlighting a challenge for precise reasoning from maps.The benchmark compares both representations using the same underlying geometry.Source: §1 · p. 2

2.1 SPATIAL RELATIONS AND SPATIAL REASONING OF AI SYSTEMS

• Evaluations of spatial reasoning in AI have largely centered on local scenes with limited spatial extent, leaving broad formal relations among georeferenced features less explored.The unresolved scope includes deriving geometric quantities and using relations to modify spatial configurations.Source: §2.1 · p. 2

2.2 BENCHMARKS FOR GEOSPATIAL REASONING IN MLLMS

• Existing geospatial benchmarks vary in representations, relation sets, and task formulations, and few compare GeoJSON with a map rendered from the same geometry across broad formal relations.Recent benchmarks already use real geographic data and map, imagery, or structured representations.Source: §2.2 · p. 3

2.3 SPATIAL EDITING AND RELATION-GROUNDED EVALUATION

• GeoSpatialBench evaluates QA and constrained editing on separate instance pools, scores by relation family, and verifies answers and edits with GIS operations.This design tests whether spatial reasoning transfers to editing geographic geometry.Source: §2.3 · p. 3

3.1 TASK DEFINITION

• The benchmark evaluates spatial reasoning through QA and constrained geometric editing across 28 relations involving points, lines, and polygons.Relations are grouped into topology, direction, metric, and order; QA responses can be categories, numbers, feature IDs, or orderings.Source: §3.1 · p. 3, §3 · p. 3

3.2 BENCHMARK CONSTRUCTION

• Six cities contribute 700 QA and 700 editing instances each, using OpenStreetMap features and separate QA and editing feature pools.Canonical scenes retain geometries, roles, attributes, and provenance; a geometric oracle supplies reference answers, and editing instances are retained only when a valid solution exists.Source: §3.2 · pp. 4–5

3.3 INPUT MODALITIES

• GeoJSON, annotated maps, and their combination represent the same QA scenes, while editing interfaces differ between structured transformations and pixel-coordinate geometry.Together uses the GeoJSON editing interface plus a map; image editing changes both input representation and output interface, so its comparison with GeoJSON is not a pure input-only ablation.Source: §3.3 · p. 5

3.4 EVALUATION PROTOCOL

• QA uses deterministic scoring with answer-specific normalization and tolerances, and geometry-only scoring also reports an information-sufficient score excluding 25 vertical-order questions per city.Distance and perimeter answers allow one-meter tolerance, bearings one-degree circular tolerance, and density 0.001 in stated units; unparseable answers count as incorrect.Source: §3.4 · p. 5
• GeoJSON and Together edits use strict CSR, while map-image edits use raster-tolerant success; a corrective retry is allowed for undecodable or unchanged edits.CSR is the proportion of edits that are valid and satisfy all constraints; image predictions are mapped back to task coordinates and may also pass within four pixels of the reference under Hausdorff distance.Source: Figure 1 · p. 4, §3.4 · pp. 5–6

4.2 OVERALL PERFORMANCE

• Under GeoJSON, GPT-6 Astra leads with 96.5% QA accuracy and 97.6% editing success, followed by KIMI K3 at 90.7% and 85.1%, respectively.The relative model order is not identical between QA and editing.Source: §4.2 · p. 6

4.3 PERFORMANCE ACROSS RELATIONS AND CITIES

• Under GeoJSON, topology leads QA (74.4% open-ended; 77.9% multiple-choice), while metric is lowest (38.6%; 53.4%); editing instead favors topology and direction (65.1%; 63.4% CSR).Source: Figure 3 · p. 7, §4.3 · p. 7

4.2 OVERALL PERFORMANCE

• MiMo-V2.6-Pro's mean QA accuracy falls from 67.3% with GeoJSON to 45.2% with map image; Together's effect on editing varies across models.GeoJSON has lower family-level error across all four relation families, while Together does not consistently improve over GeoJSON alone.Source: §4.2 · p. 7, §4.4 · p. 8

4.3 PERFORMANCE ACROSS RELATIONS AND CITIES

• Across multiple-choice relations, Equality reaches 98.4% and Layer Hierarchy 23.7%; within-model open-ended QA ranges 1.00–5.71 percentage points across the six cities.Layer Hierarchy depends on a road-layer attribute omitted from geometry-only GeoJSON input; pooled accuracy differs by 67.03 percentage points between the highest- and lowest-performing models.Source: §4.3 · pp. 7–8

4.4 ERROR ANALYSIS AND DISCUSSION

• Representative cases include map-image errors on Touching and Boundary Distance QA, and a KIMI K3 map-image edit yielding 42.8 m against an 8–22 m Euclidean Distance range.Editing errors also include moving a target in the wrong direction with GeoJSON.Source: §4.4 · p. 8, Figure 4 · p. 9, Figure 5 · p. 9
• The results suggest that explicit geometric and relational representations may support precise reasoning, but the experiments do not isolate a specific architectural cause.The authors identify spatial tokenization as one possible direction for bridging recognition, geometric reasoning, and intervention.Source: §4.4 · pp. 8–9
• 61.4% open-ended metric error persists even with GeoJSON, and Minimum Distance, Hausdorff Distance, Total Perimeter, and POI Density each exceed 70% error.Map-image QA is less reliable for precise boundary and metric estimates; explicit coordinates reduce representational ambiguity but do not ensure reliable continuous-quantity computation.Source: §4.4 · p. 8, Figure 4 · p. 9
• Touching and Disjointness have low GeoJSON MCQ-4 error rates (7.3% and 8.4%) but editing error rates of 71.2% and 47.3%, respectively.Equality remains easy in both settings, with 1.6% MCQ-4 error and 3.3% editing error; direction editing CSR (63.4%) is close to MCQ-4 accuracy (62.0%).Source: §4.4 · p. 8, Figure 5 · p. 9

• GeoSpatialBench evaluates spatial relation understanding, geometric computation, and constrained spatial editing in urban scenes using GeoJSON and rendered map images.Source: §5 · p. 9

• The authors state that evaluation code and supporting benchmark materials will be provided upon request.Source: REPRODUCIBILITY STATEMENT · p. 10

• The authors used ChatGPT (5.6 Sol) for code development, data processing, language editing, and LATEX formatting, and state that they reviewed and tested AI-assisted outputs.Source: AI USE STATEMENT · p. 10

• 1.00%–5.71% is the reported range of differences between cities.Source: A ADDITIONAL RESULTS · p. 13
• Table 1 jointly reports GeoJSON, Image, and Together performance for nine models, ordered by six-city GeoJSON open-ended QA.Image editing uses raster-tolerant success; GeoJSON and Together editing use strict CSR.Source: A ADDITIONAL RESULTS · p. 13, Table 1 · p. 13
• Table 2 reports open-ended GeoJSON QA across six cities, with 700 questions per city and 4,200 questions pooled overall.Its Range is the maximum minus minimum city score in percentage points.Source: Table 2 · p. 13

• Figure 6 and Tables 3 and 4 summarize relation-level error rates for the nine reported models.Source: §B.1 · p. 14
• Figure 6 breaks out error rates by spatial relation, input, and task, with lower percentages indicating better performance.Source: Figure 6 · p. 14
• Table 3 reports error rates by relation family, pooled over nine models, with GeoJSON and Map Image covering six cities for every model.Source: Table 3 · p. 15
• Table 4 reports error rates for all 28 spatial relations.Source: Table 4 · p. 15