Source-linked AI summary
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
Yuhan Ji, Song Gao, Ying Nie, Ivan Majić, Krzysztof Janowicz
TL;DR
The paper examines whether LLMs can represent and reason about geospatial geometries and spatial relations encoded as WKT, including vernacular descriptions of topological relationships. It evaluates embedding-based, prompt-engineering, and everyday-language approaches across three spatial reasoning tasks, finding promising but limited performance for geospatial reasoning.
Problem
LLMs have limited ability to represent and reason with vector geometries and natural-language descriptions of complex spatial relations, which are not usually expressed as text for model consumption.
Method
The study evaluates WKT geometry representations using LLM embeddings and direct prompt engineering, alongside conversion of vernacular spatial descriptions into predefined topological predicates.
Results
Over 0.6 average accuracy was achieved for identifying topological spatial relations, while LLM-generated geometries improved retrieval and GPT-4 few-shot prompting exceeded 0.66 accuracy for topological relation inference.
Takeaways & Limitations
The results provide insights for refining LLMs with geographical knowledge toward GeoAI foundation models capable of qualitative spatial reasoning.
Takeaways & Limitations
The study focuses on in-context learning and does not explore fine-tuning approaches that could potentially improve performance.
Abstract
from arXiv · showhide
Applying AI foundation models directly to geospatial datasets remains challenging due to their limited ability to represent and reason with geographical entities, specifically vector-based geometries and natural language descriptions of complex spatial relations. To address these issues, we investigate the extent to which a well-known-text (WKT) representation of geometries and their spatial relations (e.g., topological predicates) are preserved during spatial reasoning when the geospatial vector data are passed to large language models (LLMs) including GPT-3.5-turbo, GPT-4, and DeepSeek-R1-14B. Our workflow employs three distinct approaches to complete the spatial reasoning tasks for comparison, i.e., geometry embedding-based, prompt engineering-based, and everyday language-based evaluation. Our experiment results demonstrate that both the embedding-based and prompt engineering-based approaches to geospatial question-answering tasks with GPT models can achieve an accuracy of over 0.6 on average for the identification of topological spatial relations between two geometries. Among the evaluated models, GPT-4 with few-shot prompting achieved the highest performance with over 0.66 accuracy on topological spatial relation inference. Additionally, GPT-based reasoner is capable of properly comprehending inverse topological spatial relations and including an LLM-generated geometry can enhance the effectiveness for geographic entity retrieval. GPT-4 also exhibits the ability to translate certain vernacular descriptions about places into formal topological relations, and adding the geometry-type or place-type context in prompts may improve inference accuracy, but it varies by instance. The performance of these spatial reasoning tasks offers valuable insights for the refinement of LLMs with geographical knowledge towards the development of geo-foundation models capable of geospatial reasoning.
1. Introduction
Foundation models enable conversational GIS interactions, but current AI systems still struggle to represent and reason with topological spatial relations. This study evaluates whether WKT geometry representations and language descriptions can support LLM-based geospatial reasoning.
- Foundation models support broad applications through large-scale training and adaptable downstream use, including conversational GIS analysis.LLMs such as GPT are generative foundation models that return synthetic outputs such as natural-language answers or imagery.
- Early GeoAI research has explored spatial-data representation, map generation, place semantics, GIS automation, code generation, and spatial inference.
- Topological spatial relations remain a prominent weakness across foundation models, including both LLMs and text-to-image systems.
- The study investigates whether WKT representations of points, polylines, and polygons can enable LLMs to perform GIS operations and improve geospatial reasoning.The authors also evaluate natural-language descriptions of vague spatial configurations, although accurate geometries and their relations are not usually expressed in natural language.
- The workflow assesses topological reasoning through geometry embeddings, direct WKT prompt engineering, geometry retrieval, and conversion of vernacular descriptions into formal relations.The contributions include predicting relations between two geometries, reconstructing a second geometry, and extracting formal relations from everyday language.
2. Related work
Related work formalizes spatial relations through qualitative language, topology-based models, and conceptual neighborhoods, while LLM research increasingly examines symbolic and vector representations of spatial information. These foundations motivate evaluating whether textual geometry representations preserve topological structure for reasoning and retrieval.
- Spatial relations: Spatial relations connect objects through geometric properties and support navigation, information retrieval, geographic question answering, spatial reasoning, disaster management, and robotics.
- Spatial relations: Binary spatial relations are commonly represented as {subject, predicate, object}, with qualitative predicates combined with contextual cues to describe complex spatial layouts.Everyday descriptions can express relative position through prepositions, verbs, phrases, or implicit text.
- Topological formalisms: RCC-8 and intersection models such as 4-IM and 9-IM formalize topological relations using region, boundary, interior, and exterior intersections.DE-9IM extends the 9-intersection model by considering geometry dimensionality.
- Conceptual neighborhoods: Conceptual neighborhoods represent smooth transitions between topological relations, making some relations closer or more similar than others.For example, polygon objects may transition from disjoint to touching before overlapping.
- Conceptual neighborhoods: Topological distance measures relation similarity, with smaller distances indicating greater similarity, and is used to compare LLM errors with human-perceived neighborhoods.
- LLMs and geospatial representations: Recent LLM studies examine symbolic spatial representations and vector data, but language models also exhibit human-like misconceptions and spatial distortions.This study hypothesizes that LLMs can encode WKT strings while preserving crucial geometric information for semantic comparison and retrieval.
3. Methodology
The methodology represents geospatial objects as WKT and evaluates LLMs on topological relation qualification, spatial query processing, and vernacular-to-formal relation conversion. It combines embedding-based representations with prompt-based reasoning and formal DE-9IM relation definitions.
- Research workflow: The framework retrieves spatial objects and place descriptions, then evaluates LLMs on topological relations between geometric objects.The overall workflow combines spatial-database geometries with textual place descriptions containing vernacular relations.
- Topological relation model: The study uses seven DE-9IM predicates and interprets within and contains as excluding equals to preserve unique relations.The modified decision process also restricts which geometry-type combinations support each predicate.
- Geometry representation: WKT strings are converted into sentence embeddings so spatial information can be compared and retrieved through semantic search.The embedding approach hypothesizes that LLM-based text encoders preserve useful geometric information from points, polylines, and polygons.
- Topological relation qualification: Task 1 predicts a topological predicate from WKT(A) and WKT(B), returning a tuple of the two geometry types and their relation.A correct output preserves the geometry types and matches the ground-truth predicate.
- Topological relation qualification: Task 1 compares embedding-based classification with prompt-based LLM reasoning for identifying relations between two geometries.The embedding workflow concatenates geometry embeddings for a random-forest classifier, while prompting passes WKT directly to an LLM.
- Spatial query and vernacular conversion: Task 2 retrieves a subject geometry from a query geometry and topological predicate, while Task 3 converts vernacular descriptions into formal predicates.Task 2 uses WKT(B) and a relation as input; Task 3 concerns everyday descriptions of relations between geographic entities.
4. Data and Experiments
The experiments use real-world geospatial data from Madison and DBpedia-derived place descriptions to evaluate geometry-based and language-based spatial reasoning. Sampling, filtering, contextual grouping, and model-specific settings define the experimental comparisons.
- Datasets: The Madison study area combines road networks, points of interest, land parcels, and census block groups for Tasks 1 and 2.The evaluation focuses on Point, LineString, and Polygon objects, with computations performed in GeoPandas.
- Tasks 1 and 2: Each geometry-type, predicate, and geometry-type combination contributes 200 triplets, with 160 for random-forest training, 40 for evaluation, and 25 extra few-shot examples.The sampling strategy addresses imbalanced real-world distributions of topological relations.
- Data construction: The dataset construction creates or filters examples to preserve target relations while avoiding trivial identification from coordinate matching or large spatial separation.Disjoint cases are restricted to nearby entities, and equals cases are manually constructed.
- Tasks 1 and 2: Task 2 excludes disjoint relations and yields 40×26=1040 triplets for geographic-entity retrieval.The exclusion reflects the high frequency of disjoint relationships among real-world entities.
- Task 3: Task 3 begins with 1078 DBpedia-based triplets from Wisconsin, Texas, and California, removing 212 directional or distance relations to retain topological cases.The remaining records are used to evaluate GPT-4 on vernacular-to-formal relation conversion.
- Task 3: Task 3 tests invariant, place-type, geometry-type, and place-name contexts for mapping vernacular descriptions to formal predicates.Context-conditioned pairs are filtered by occurrence thresholds before evaluation.
- Models and settings: The experiments evaluate GPT-3.5-turbo, GPT-4, and DeepSeek-R1-14B alongside OpenAI text-embedding models with 1536- and 3072-dimensional outputs.GPT models are primarily evaluated for few-shot learning, while DeepSeek-R1-14B emphasizes zero-shot capabilities.
5.1. Topological spatial relation qualification
LLMs can identify topological spatial relations from WKT geometries, but performance remains challenging and depends strongly on prompt design and reasoning behavior.
- 5.1.1. Validity of the output: GPT-4 and GPT-3.5-turbo generally followed the required output format, while DeepSeek-R1-14B exceeded 0.9 validity accuracy but adhered less reliably to formatting instructions.Few-shot DeepSeek results were excluded because the model largely ignored the examples.
- 5.1.2. Classification metrics: Over 0.6 accuracy was achieved by both embedding-based classification and GPT geospatial question answering for topological predicate identification.Failures may reflect information loss during tokenization and confusion among semantically similar predicates.
- 5.1.2. Classification metrics: 0.66 accuracy was achieved by GPT-4 with few-shot prompting, the best performance among the evaluated prompt types for GPT-3.5-turbo and GPT-4.The few-shot examples paired geometries with their topological relationships.
- 5.1.2. Classification metrics: Chain-of-thought prompting did not improve spatial relation classification and could reduce accuracy through cascading errors in intermediate reasoning steps.The observed errors included failing to determine whether geometry interiors intersected at the beginning of the reasoning process.
- 5.1.2. Classification metrics: DeepSeek-R1-14B showed overreliance on superficial, linear interpretations of coordinate information rather than holistic topological reasoning across the plane.
5.2. Spatial query processing
LLM-generated geometries and spatial query reformulation can support geospatial retrieval, although retrieval remains difficult and benefits depend on query direction and geometry context.
- 5.2.1. Direct query: Specifying the subject geometry type improved retrieval performance by narrowing the mapping space to the same geometry type.The workflow therefore assumed that user queries include geometry-type information, such as interpreting a street as a LineString.
- 5.2.2. Geometry generation: GPT-4 generated valid WKT geometries and preserved the reference object's topological relation even in the zero-shot setting.The generated geometry was expected to remain topologically related to the reference object while being close in embedding space.
- 5.2.3. Expanded query on subject retrieval: Over 23% of target subjects ranked within the top 5 when an LLM-generated geometry was added to expanded spatial queries.Adding additional synthetic geometries did not provide further improvements.
- 5.2.3. Expanded query on subject retrieval: Subject retrieval from embeddings of expanded spatial queries remained challenging despite the use of query expansion.
- 5.2.4. Object retrieval: Reversing the topological relation and treating the object as the subject matched subject-retrieval performance, whereas original subject-to-object directionality performed worse.This result highlights the importance of query structures aligned with everyday language patterns.
5.3. Vernacular topological relation conversion
GPT-4 can convert some vernacular place descriptions into formal topological relations, but accuracy and consistency vary substantially by description and contextual information.
- 5.3.1. Conversion pairs invariant to context: Six vernacular descriptions consistently mapped to the same topological relationship in the dataset, but conversion accuracy varied across descriptions.“Is an enclave of” was interpreted as within rather than touches or disjoint, while “has part of the population in” yielded overlaps with multiple alternatives.
- 5.3.1. Conversion pairs invariant to context: GPT-4 inferred an overlaps relation from “has part of the population in” in all ten experiments, but sometimes supplied multiple predicate alternatives.The model's uncertainty was reflected in its alternative answers.
- 5.3.1. Conversion pairs invariant to context: The higher entropy of “halfway” than “midway” indicated greater randomness in the corresponding conversion.
- 5.3.2. Conversion pairs conditioned on place types or geometry types: Place-type and geometry-type context did not consistently improve accuracy or reduce ambiguity across conversion pairs.Some instances produced the same incorrect answer, including pairs with 0 accuracy.
- 5.3.3. Place-name context: Mentioning place names did not necessarily improve conversion accuracy or guide GPT-4 toward a preferred answer.GPT-4 focused on general geographic locations or boundaries rather than specific place knowledge.
6. Discussion
The discussion examines how LLM confusion patterns reflect conceptual neighborhoods, geometry-type effects, directional asymmetries, and mismatches between formal geometries and vernacular place descriptions.
- Discussion scope: The discussion evaluates whether observed predicate confusions align with conceptual neighborhoods across qualification, geometry generation, and vernacular conversion.The comparison uses confusion matrices and conceptual-neighborhood analysis.
- Topological relation qualification: GPT-4 frequently confuses point–line or point–polygon touching with within, whereas confusion is less common between same-dimensional geometries.These patterns indicate that geometry type and dimensionality affect recognition of formal boundaries.
- Geometry generation: Directional wording strongly affects geometry generation: GPT-4 handles generating a LineString crossing a Polygon better than generating a Polygon crossing a LineString.The model similarly handles contains queries better than reversed within queries.
- Geometry generation: GPT-4 more successfully generates geometries for contains relations than for the corresponding within relations.The asymmetry is attributed to extracting coordinates from the query geometry to construct the second geometry.
- Vernacular description conversion: Vernacular-to-formal conversion is constrained by differences between crisp dataset geometries and vague human perceptions of place boundaries.Examples include differing interpretations of “is along,” boundary sensitivity to marked points, and official boundaries that differ from perceived place extents.
7. Conclusion and future work
The conclusion evaluates LLM reasoning over WKT-based topological relations through qualification, geometry retrieval, and vernacular conversion tasks. Results show useful but imperfect spatial reasoning, while future progress requires methods beyond in-context learning.
- Study scope: The study evaluates GPT-3.5, GPT-4, and DeepSeek-R1-14B on WKT-based qualification, geometry generation and retrieval, and vernacular relation conversion.The workflow tests whether textual geometry representations retain information needed for formal topological reasoning.
- Conclusion: Over 0.6 accuracy on average was achieved by random forest and GPT-based models for identifying most topological relations in Task 1.Some relations remained confounding, and few-shot prompting improved GPT-3.5-turbo and GPT-4 performance.
- Conclusion: LLM-generated geometries reached up to 0.76 accuracy in preserving topological relations or their conceptual neighbors.Customized queries and geometry augmentation improved retrieval, although Task 2 still exposed challenges in replacing spatial queries with semantic search.
- Conclusion: Task 3 produced limited improvement in reducing ambiguity, although generated outputs usually fell within the conceptual neighborhood of the ground-truth predicate.Response changes across contexts indicate reasoning about commonsense knowledge and typical spatial configurations.
- Conclusion: Adding geometry-type context generally improved performance more than place-type context, but context effects varied by instance.The conclusion frames tailored prompting and specialized GIScience knowledge as targeted strategies for geospatial reasoning.
- Future work: The study focuses on in-context learning and does not explore fine-tuning, while RAG and neurosymbolic approaches are identified as future directions.These approaches would require reliable spatial data, formal query generation, and improved translation from natural language into symbolic representations.
Notes on contributors
The paper’s contributors are Yuhan Ji, a PhD student, and Song Gao, an associate professor, in GIScience at the University of Wisconsin-Madison.
- Contributors: Yuhan Ji is a University of Wisconsin-Madison GIScience PhD student whose research includes transportation, geospatial data science, and GeoAI.
- Contributors: Song Gao is a University of Wisconsin-Madison GIScience associate professor whose research includes GeoAI, geospatial data science, and human mobility.