Source-linked AI summary
GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks
Amirhossein Vahdat, Thierry Badard, Luc Lamontagne
TL;DR
Geospatial data requirements elicitation is rarely evaluated, even though natural-language geo-analytic requests often leave needed datasets and constraints implicit. GeoFitBench-LLM assesses this capability with two reasoning dimensions, a structured provider-independent schema, and evaluations of resource identification and attribute specification. Results show performance declines as inferential demands increase, and the authors identify broader benchmark coverage and validation as future priorities.
Problem
Evaluation of LLMs for geospatial tasks rarely tests whether they can infer the datasets and constraints implicit in natural-language geo-analytic requests.
Source: §1 · pp. 1–2Method
Results
Performance varies across models and declines with increasing inferential demand, with models more reliable when resources are explicitly recoverable than when inferred from analytical intent.
Source: §5 · p. 7Takeaways & Limitations
GeoFitBench-LLM provides a systematic basis for assessing a foundational yet under-evaluated GeoAI capability and identifying models better suited to retrieval-oriented geospatial tasks.
Source: §5 · p. 7Future work identifies broader task and domain coverage, additional specialist involvement in item design and validation, and evaluation under alternative prompting strategies as priorities.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints. GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language. The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks. Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification. Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable, with GPT-5.4 Thinking achieving the best result by recovering the complete set of resources in 66.67% of benchmark requests. Results further indicate that accurate fitness-for-use specification remains more difficult as inferential demands increase.
1. Introduction
Geospatial requests often state an analytical objective while leaving the needed datasets and constraints implicit, making data-requirement elicitation a foundational but under-evaluated GeoAI capability. GeoFitBench-LLM assesses whether LLMs can infer those requirements in a provider-independent form.
1. Introduction
- Geospatial retrieval benchmarks have largely left implicit whether models can infer the data requirements and constraints needed for a task.A plausible workflow cannot be executed meaningfully if necessary datasets are missing, misidentified, or insufficiently specified.Source: §1 · pp. 1–2
- Geo-analytic requests commonly express intended analytical outcomes rather than the datasets needed to achieve them, so the challenge is recovering latent data needs rather than matching keywords.This difficulty is especially apparent when domain specialists who are not GIS experts formulate requests around decision problems.Source: §1 · p. 2
- GeoFitBench-LLM evaluates whether LLMs can infer input and intermediate resources from natural-language tasks and specify them in a provider-independent, retrieval-oriented form.It isolates requirement elicitation from workflow execution, portal-specific retrieval, API use, and tool orchestration.Source: §1 · p. 2
2. Towards Fitness-for-Use-Aware Geospatial Data Retrieval from Natural Language Requests
Fitness-for-use-aware retrieval requires inferring implicit data requirements from analytical requests. GeoFitBench-LLM organizes this challenge around spatio-temporal relations and semantic interpretation, with criteria covering the varied reasoning needed for retrieval.
2. Towards Fitness-for-Use-Aware Geospatial Data Retrieval from Natural Language Requests
- The benchmark distinguishes Spatio-Temporal Relational challenges, where requirements depend on spatial or temporal structure, from Semantic Grounding challenges involving lexical, conceptual, or referential interpretation.The dimensions reflect challenges identified in geospatial information retrieval, geoportal discovery, geocoding, and natural-language access to spatial-data systems.Source: §2 · p. 2
- Spatio-temporal criteria cover spatial relations, temporal alignment, comparable evidence across places or periods, aggregation, and chained dependencies.The required resources can depend on how the request structures the analysis across space and time.Source: §2 · pp. 2–3
- Retrieval criteria also include reasoning across nested spatial or temporal levels and distinguishing network-based accessibility, flow, or distance from geometric proximity.The criteria table summarizes the benchmark's reasoning dimensions and evaluation criteria.Source: §2 · pp. 2–3, Table 1 · p. 3
- Other semantic challenges include metadata-language mismatches, indirect spatial meaning, literal interpretations that misalign resources, and ambiguous geographic names or extents.Each can affect whether the intended resource is discovered or selected.Source: §2 · pp. 2–3
- Semantic criteria include resolving context-dependent terms, abbreviations, quantities and units, relative dates, and requests that name a property instead of its dataset class.These distinctions affect which resource or operational constraint is required.Source: §2 · pp. 2–3
3. GeoFitBench-LLM
GeoFitBench-LLM is a 48-item benchmark for evaluating retrieval-oriented geospatial requirement inference across two reasoning dimensions and three levels of difficulty. It uses provider-independent structured requirements and evaluates both resource identification and attribute specification.
3. GeoFitBench-LLM
- The benchmark contains 48 items across three difficulty levels, from direct dataset identification to spatio-analytical inference and domain-conceptual inference.D2 requires inferring data themes, supporting layers, or companion resources; D3 requires inferring conceptual inputs for modeled, derived, or domain-mediated phenomena.Source: §3 · p. 3
- Its two reasoning dimensions yield 21 Spatio-Temporal Relational items and 27 Semantic Grounding items, with each criterion instantiated across all three difficulty levels.This supports comparisons across reasoning challenges and increasing inferential demand.Source: §3 · p. 4
- Ground truth uses a normalized, provider-independent schema aligned with ISO 19115-1 and ISO 19115-2 to specify resources and task-relevant attributes.Fields include resource role and theme, area and time coverage, data type, resolution, reference system, format, and additional mandatory attributes or constraints.Source: §3 · p. 4
- Candidate responses are generated with a minimal prompt requiring schema-valid JSON, while a separate LLM judge applies parsing, resource-alignment, and counting rules.Resource name and role primarily determine alignment; theme, area of interest, time coverage, and data type mainly serve as contradiction checks.Source: §3 · p. 5
- Dataset Identification measures resource recovery, while Attribute Accuracy measures aligned-resource specification; binary pass rates and precision, recall, and F1 provide strict and partial-credit views.This separates identifying the right resources from specifying attributes that determine fitness for the intended use.Source: §3 · p. 5
4. Experiments
Under minimal prompting, GPT-5.4 Thinking leads the overall benchmark comparison, while criterion-level strengths and weaknesses vary and performance declines as difficulty increases. The evaluation covers seven models across 48 items.
4. Experiments
- Seven models were evaluated on the 48-item benchmark under minimal prompting, without worked examples or decomposition guidance.The evaluation reports Dataset Identification for the full benchmark and each reasoning subset, and Attribute Accuracy for resource-level specification.Source: §4 · p. 5
- GPT-5.4 Thinking achieves the strongest overall performance, followed by GPT-5.4 and Gemini 3.1 Pro; GPT-OSS 20B is the strongest evaluated open-source model.Table 2 reports DI_binary, DI_F1, and AA_F1 overall, plus DI_binary for both reasoning subsets.Source: Table 2 · p. 5, §4 · p. 5
- Comparative and temporal-relation reasoning are among the weakest Spatio-Temporal Relational criteria, while aggregation and network reasoning are handled more reliably.In Semantic Grounding, place-name and boundary disambiguation, feature-property distinction, and temporal reference and recency interpretation are weaker; synonymy and units and thresholds are more reliable.Source: §4 · p. 6, Figure 2 · p. 6
- Performance declines from D1 to D3 across the Spatio-Temporal Relational subset, Semantic Grounding subset, and combined benchmark.Models are more reliable when data needs are explicitly recoverable than when requirements must be inferred or are domain-conceptual.Source: §4 · p. 6, Figure 3 · p. 6
5. Conclusions and Future Work
GeoFitBench-LLM structures evaluation of geospatial requirement inference across reasoning dimensions and difficulty levels. Results show that elicitation remains challenging, and the authors propose broader coverage, prompting studies, and error analysis.
5. Conclusions and Future Work
- The benchmark evaluates two reasoning dimensions across three difficulty levels, enabling structured assessment of retrieval-oriented requirement inference.The levels range from direct dataset identification to advanced domain-level conceptual inference.Source: §5 · pp. 6–7
- Performance varies substantially across models and declines as inferential demands rise; resources are more reliably identified when explicit than inferred from analytical intent.The authors describe the benchmark as a basis for assessing this under-evaluated capability and identifying models better suited to retrieval-oriented geospatial tasks.Source: §5 · p. 7
- Planned extensions broaden task and domain coverage with specialist validation, test alternative prompting, and analyze which schema attributes and requirement inferences drive errors.Prompting strategies include few-shot prompting, Decomposition (DECOMP), and chain-of-thought prompting.Source: §5 · p. 7