PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence

Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, Chris Cundy, Ziyuan Li, Rui Zhu, Ni Lao

arXiv:2304.06798v1cs.AIcs.CLcs.CV

TL;DR

Foundation models have not yet been developed for GeoAI despite their success in language and vision, raising questions about their suitability for geospatial tasks. The paper evaluates existing models across seven tasks and proposes a multimodal framework based on geospatial alignment. LLMs outperform task-specific supervised models on several language-only tasks, whereas existing foundation models underperform on especially multimodal tasks.

  • Problem

    Foundation models have not yet been developed for GeoAI, whose tasks involve multiple geospatial data modalities and modality-specific structures.

  • Method

    The paper evaluates existing foundation models against fully supervised task-specific models across seven geospatial tasks and proposes a multimodal framework using geospatial alignments.

  • Results

    LLMs can outperform task-specific fully supervised models on several language-only geospatial tasks, while existing foundation models underperform on other tasks, especially multimodal ones.

  • Takeaways & Limitations

    GeoAI foundation models must address geospatial multimodality, with geospatial alignment proposed as a basis for reasoning across data types.

  • Takeaways & Limitations

    Foundation-model development remains constrained by increasing model size and the need for improved computational efficiency in training and fine-tuning.

Abstract

from arXiv · show

Large pre-trained models, also known as foundation models (FMs), are trained in a task-agnostic manner on large-scale data and can be adapted to a wide range of downstream tasks by fine-tuning, few-shot, or even zero-shot learning. Despite their successes in language and vision tasks, we have yet seen an attempt to develop foundation models for geospatial artificial intelligence (GeoAI). In this work, we explore the promises and challenges of developing multimodal foundation models for GeoAI. We first investigate the potential of many existing FMs by testing their performances on seven tasks across multiple geospatial subdomains including Geospatial Semantics, Health Geography, Urban Geography, and Remote Sensing. Our results indicate that on several geospatial tasks that only involve text modality such as toponym recognition, location description recognition, and US state-level/county-level dementia time series forecasting, these task-agnostic LLMs can outperform task-specific fully-supervised models in a zero-shot or few-shot learning setting. However, on other geospatial tasks, especially tasks that involve multiple data modalities (e.g., POI-based urban function classification, street view image-based urban noise intensity classification, and remote sensing image scene classification), existing foundation models still underperform task-specific models. Based on these observations, we propose that one of the major challenges of developing a FM for GeoAI is to address the multimodality nature of geospatial tasks. After discussing the distinct challenges of each geospatial data modality, we suggest the possibility of a multimodal foundation model which can reason over various types of geospatial data through geospatial alignments. We conclude this paper by discussing the unique risks and challenges to develop such a model for GeoAI.

1 INTRODUCTION

This paper examines whether existing foundation models can support GeoAI and develops a framework for multimodal geospatial foundation models. It evaluates models across seven tasks and discusses their opportunities, challenges, risks, and future directions.

  • Background: Foundation models transfer knowledge across domains through fine-tuning, few-shot, or zero-shot adaptation, reducing reliance on task-specific training data.Examples include language, vision, multimodal, and reinforcement-learning foundation models.
  • Motivation: GeoAI has received little attention in foundation-model development, while its inherently multimodal data create a central technical challenge.Relevant modalities include text, images, trajectories, knowledge graphs, and geospatial vector data, each with distinct structures and representations.
  • Motivation: Existing foundation models cannot be applied straightforwardly across GeoAI tasks because combining modality-specific representations requires appropriate inductive biases and careful design.The paper identifies this challenge as a barrier to using existing models across geospatial data modalities.
  • Evaluation: The paper systematically compares popular foundation models with fully supervised task-specific models across seven tasks spanning geospatial semantics, health geography, urban geography, and remote sensing.The tasks include toponym and location-description recognition, dementia forecasting, urban-function and noise classification, and remote-sensing scene classification.
  • Evaluation: The study examines foundation-model performance on toponym recognition, location-description recognition, dementia forecasting, POI-based urban-function classification, street-level noise classification, and remote-sensing scene classification.These tasks cover multiple geospatial domains and data modalities.
  • Contributions: The paper proposes a multimodal foundation-model framework for GeoAI, discusses modality-specific development challenges, and identifies risks requiring attention during development and evaluation.The stated contributions include establishing foundation-model baselines for future GeoAGI research and discussing risks of general-purpose geospatial models.

2 RELATED WORK

Related work traces foundation models from large-scale language modeling and task-specific vision models toward multimodal systems that jointly reason over different data types. It motivates multimodal GeoAI because geospatial tasks commonly combine complementary modalities.

  • Language Foundation Models: Large language models learn from vast text corpora through self-supervised next-token prediction and can transfer their capabilities to downstream tasks.Scaling model and data size has been associated with improved downstream quality.
  • GeoAI Motivation: The paper analyzes the potential and limitations of GPT and other language models across different geospatial domains.This establishes the study’s connection between existing foundation-model research and GeoAI.
  • Vision Foundation Models: Vision foundation models extend large-scale pretraining beyond individual tasks, including image generation, recognition, segmentation, and other visual applications.Examples include transformer, convolutional, GAN, diffusion, and segmentation-based systems.
  • Multimodal Foundation Models: Multimodal reasoning is especially relevant to GeoAI because geospatial data are ubiquitous across modalities, whose views can complement one another.The paper frames multimodal data as supporting a more holistic understanding of geospatial information.
  • Multimodal Foundation Models: Multimodal foundation models learn joint representations or reasoning across modalities such as images, text, audio, and video.Examples include CLIP, BLIP, BEiT-3, Florence, KOSMOS-1, Flamingo, and GPT-4.

3 EXPLORATION OF THE EFFECTIVENESS OF EXISTING FMS ON VARIOUS GEOSPATIAL DOMAINS

The paper asks how existing foundation models compare with fully supervised, task-specific models across geospatial tasks. It evaluates models across four geospatial subdomains to identify their strengths and weaknesses.

  • Research Question: The central question is whether cutting-edge foundation models can match state-of-the-art fully supervised task-specific models on varied geospatial tasks.The comparison spans multiple geospatial subdomains rather than a single benchmark.
  • Evaluation Scope: The evaluation covers Geospatial Semantics, Health Geography, Urban Geography, and Remote Sensing.These subdomains represent distinct types of geospatial problems.
  • Evaluation Scope: The study uses experiments with various foundation models on tasks from each selected geospatial subdomain.The paper discusses both advantages and weaknesses of existing foundation models.

3.1 Geospatial Semantics

The geospatial-semantics experiments test GPT-family models on toponym and location-description recognition against supervised and other task-specific baselines. LLMs perform strongly on some tasks with few-shot prompts, but output-format and prompt-design limitations remain.

  • Toponym Recognition: Except for the smallest GPT2 model, the other LLMs consistently outperform fully supervised baselines on Hu2014 without additional training.The finding is based on natural-language instructions and the shared evaluation setup.
  • Toponym Recognition: 8.7% performance improvement over TopoCluster is reported for GPT-3 on the Hu2014 toponym-recognition dataset.The result is reported relative to the previous state of the art.
  • Limitations: ChatGPT performance decreases on the representative toponym datasets, while larger parameter counts generally correspond to better LLM performance.The paper hypothesizes that chatbot optimization may reduce flexibility for new tasks such as toponym recognition.
  • Location Description Recognition: GPT-3 achieves the best recall on HaveyTweet2017, but all LLMs have low precision and consequently low F1-scores.The authors attribute this pattern to converting recognition into a natural-language generation problem, which may produce tokens absent from the input.
  • Overall Findings: LLMs outperform fully supervised task-specific models on well-defined geospatial-semantics tasks using only a small number of few-shot examples.The comparison includes toponym recognition and location description recognition across benchmark datasets and baseline models.
  • Limitations: Developing appropriate prompts for particular geospatial-semantics tasks remains an open challenge.The paper identifies prompt design as requiring further investigation.

3.2 Health Geography

The health-geography experiments forecast dementia deaths at state and county levels using zero-shot GPT models and compare them with persistence and ARIMA baselines. InstructGPT performs especially strongly, while GPT2 models perform poorly and the reasons for InstructGPT’s success remain uncertain.

  • Task and data: The models are evaluated against persistence and fully supervised ARIMA baselines in a zero-shot prompting setting.Persistence uses the 2019 count as the 2020 prediction, while ARIMA models are trained on each state’s 1999–2019 series.
  • Results: GPT2 models perform poorly on every evaluation metric and underperform the simple persistence model, suggesting difficulty with zero-shot time-series forecasting.County-level examples indicate that GPT2 often repeats information from the prompt rather than generating new predictions.
  • Interpretation: The authors propose two possible explanations for InstructGPT’s success: learned autoregressive ability or exposure to relevant dementia time-series data during pre-training.They state that the available evidence cannot determine which explanation is primary.

3.3 Urban Geography

The urban-geography experiments test foundation models on POI-based urban-function classification and street-view noise classification. Results show useful zero- or one-shot visual and language capabilities, but multimodal and spatial-structure limitations remain substantial.

  • Tasks and datasets: Urban-geography tasks predict regional functions from POIs and neighborhood characteristics such as noise intensity from street-view imagery.These tasks combine point, text, and image modalities, so different foundation models are used for each.
  • POI-based classification: The UrbanPOI5K dataset contains 5,344 labeled regions divided into six urban-function categories and split 60%:20%:20% into training, validation, and test sets.The categories include residential, commercial, industrial, education/health/civic/governmental/cultural, transportation, and outdoors/natural.
  • POI-based classification: LLMs receive POI frequency statistics converted into text and are evaluated in zero-shot and one-shot settings against supervised neural baselines.The one-shot prompt includes a paragraph-question-answer demonstration, whereas the zero-shot prompt removes it.
  • POI-based classification: In zero-shot classification, GPT-3 achieves the best GPT precision but still underperforms the supervised models, while the smallest GPT2 gains accuracy and recall by predicting mostly Residential.This pattern reflects a strong class bias rather than balanced urban-function recognition.
  • POI-based classification: In one-shot classification, ChatGPT (Raw.) reaches 52.4% precision, only 4.4% below HGI, and shows reasonably good performance across urban-function classes.GPT2-XL has the highest one-shot accuracy but remains highly biased according to its confusion matrix.
  • Limitations: POI-based prediction is constrained because LLMs cannot access POI spatial distributions, while missing POI categories can also affect results in industrial-oriented Shenzhen.The authors argue that GeoAI foundation models need neural architectures able to directly encode geospatial vector data.
  • Street-view perception: For street-view noise classification, zero-shot BLIP achieves the best visual-language-model accuracy and weighted F1, comparable to AlexNet but slightly below ResNet18.BLIP is also the only evaluated visual-language model described as producing balanced predictions across all four noise-intensity classes.
  • Street-view perception: Overall, visual-language models show some ability to recognize neighborhood characteristics, but their performance remains generally weaker than language models on language-only tasks.The results emphasize the distinct challenges of foundation-model research for GeoAI.

3.4 Remote Sensing

The study evaluates CNNs and visual-language foundation models on remote sensing image scene classification using AID. Results show that prompt semantics and remote-sensing exposure strongly affect foundation-model performance, while task-specific CNNs remain competitive.

  • Dataset and models: The experiment uses AID, a 10K-scene aerial image dataset with 30 scene classes, split into 80% training and 20% testing.The split uses stratified sampling to preserve scene-type label distributions.
  • Dataset and models: Four CNNs and four visual-language foundation models are compared, with CNNs fine-tuned and foundation models evaluated zero-shot.CNNs use ImageNet-V1 initialization and supervised fine-tuning; the foundation-model set includes OpenCLIP-L, OpenCLIP-B, BLIP, and OpenFlamingo-9B.
  • Prompt semantics: Ambiguous AID labels such as “center” and “commercial” can prevent models from assigning images to those classes when used unchanged.The study therefore tests updated prompt names such as “theater” and “commercial area.”
  • Results: AlexNet achieves the best accuracy and F1 score among CNNs, while OpenCLIP-L (Updated) achieves the best accuracy and F1 score among visual-language foundation models.Table 7 reports accuracy and F1 score for the compared models and prompt variants.
  • Results: 0.206 accuracy is achieved by OpenFlamingo-9B, showing that larger models do not necessarily perform better on this task.The authors suggest that limited exposure to remote-sensing imagery during pre-training may contribute to this result.
  • Implications: The study highlights pre-training on diverse visual inputs, including remote-sensing images, as important for improving performance on remote-sensing tasks.Remote-sensing images differ visually from common internet photographs used in much vision-language pre-training.
  • Prompt semantics: Changing “center” to “theater” enables OpenCLIP to find relevant remote-sensing scenes, demonstrating the importance of expressive prompts.The revised label only partially captures the class semantics, but still improves interpretation.

4 A MULTIMODAL FOUNDATION MODEL FOR GEOAI

This section compares existing foundation models across geospatial tasks and examines why their capabilities vary across modalities. It argues that effective GeoAI foundation models should align multimodal representations through geographic relations to support spatial reasoning and knowledge transfer.

  • Cross-task findings: Existing foundation models outperform fully supervised task-specific models on several language-only tasks, including toponym recognition, location-description recognition, and dementia time-series forecasting.These results are obtained with large language models used as zero-shot or few-shot learners.
  • Cross-task findings: Existing foundation models underperform task-specific models on the two Urban Geography tasks and one remote-sensing task, especially when multiple modalities are involved.The evaluated multimodal examples include POI-based urban-function classification, street-view noise classification, and remote-sensing scene classification.
  • Multimodality: Geospatial tasks commonly combine text, knowledge graphs, remote-sensing images, street-view images, trajectories, and vector data, creating a major challenge for GeoAI foundation-model development.The section discusses modality-specific challenges before proposing a multimodal framework.
  • Geo-text reasoning: LLMs can recognize toponyms while predicting coordinates more than 500 miles from ground truth, and their coordinate probabilities do not follow Tobler’s First Law of Geography.The geoparsing examples show that textual recognition does not ensure real-world spatial reasoning.
  • Geo-text reasoning: Geospatial knowledge graphs can complement LLMs by providing factual information and spatial footprints for explicit spatial-relation computations.The proposed uses include integrating knowledge into training, enriching prompts, and supporting geographic question answering.
  • Proposed framework: The proposed multimodal foundation model aligns representations of different modalities through geospatial relations, enabling spatial reasoning and knowledge transfer across modalities.Geographic footprints provide a basis for aligning geospatial knowledge graphs and other representations.
  • Remote sensing: Remote-sensing foundation models should be sensor-agnostic, spatiotemporally aware, and environmentally invariant.These properties target differing sensor resolutions, geospatial metadata, image geolocalization, object tracking, and changing environmental conditions.

5 RISKS AND CHALLENGES

The paper identifies general foundation-model risks alongside GeoAI-specific challenges involving geographic fidelity, geographic and temporal bias, spatial scale, and spatial generalizability. These issues constrain reliable deployment because geospatial outputs must remain faithful across places, times, modalities, and scales.

  • General risks: Increasing model size raises the need for more computationally efficient training and fine-tuning.Additional general challenges include limited access to closed-model parameters and increasing privacy concerns in remote deployment.
  • Geographic fidelity: Geographically faithful outputs are especially important because inaccurate text and generated remote-sensing images can misrepresent places and geographic layouts.The paper shows examples involving incorrect place answers and remote-sensing images whose feature layouts do not correspond to real cities.
  • Geographic and temporal bias: Large-scale training can amplify inherited geographic bias in adapted models, motivating geographic debiasing frameworks.The paper also notes that complex model structures make interpretation and debiasing more difficult.
  • Geographic and temporal bias: Geographic bias causes models to interpret less-popular place names incorrectly relative to more frequent names in training corpora.Examples include San Jose, Batangas and Washington, New York being confused with more popular alternatives.
  • Geographic and temporal bias: Temporal bias causes GPT-4 to mishandle historical place names because training data contain more current than historical geographic entities.The cited examples involve historical names that are now used for different nearby places.
  • Spatial scale and generalizability: GeoAI foundation models must represent the same phenomenon at different spatial scales, such as a city as a polygon for traffic forecasting or a point for geoparsing.The model must infer the appropriate spatial scale for each downstream task.
  • Spatial scale and generalizability: An open problem is learning general spatial trends while preserving location-specific details without introducing intrinsic bias or overly complex global prediction surfaces.The paper states that large-scale training can amplify this tension and requires care.

6 CONCLUSION

The paper explores existing foundation models for GeoAI across seven tasks and uses the results to motivate a multimodal model aligned through geospatial relations. It concludes that language-only tasks are more favorable for current models, while multimodal tasks expose a central development challenge alongside distinctive risks.

  • Study scope: The study compares zero-shot or few-shot LLMs and visual-language foundation models with fully supervised task-specific models across seven tasks and multiple geospatial subdomains.The subdomains include Geospatial Semantics, Health Geography, Urban Geography, and Remote Sensing.
  • Main findings: LLMs can outperform fully supervised task-specific models on some language-only geospatial tasks, whereas existing foundation models underperform on other tasks involving multiple data modalities.The multimodal examples include POI-based urban-function classification, street-view noise classification, and remote-sensing scene classification.
  • Main findings: The paper identifies the multimodal nature of geospatial tasks as a major challenge for developing GeoAI foundation models.This conclusion follows the contrast between language-only successes and multimodal performance gaps.
  • Proposed direction: The proposed vision is a multimodal foundation model pretrained by aligning different data modalities through their geospatial relations.The paper presents this alignment as the basis for its proposed GeoAI framework.
  • Risks and challenges: The paper closes by discussing unique challenges and risks associated with developing such a multimodal GeoAI foundation model.These include the limitations and risks developed in the preceding discussion.

A.1 The Full Prompts Used in Various Experiment

The experiments use prompt-based questions and answers for recognizing location descriptions, named places, and geographic locations. The prompts include examples from addresses, historical text, flooding reports, news paragraphs, and geoparsing tasks.

  • Location description recognition: Location description prompts ask models to identify words representing location descriptions and separate different descriptions with semicolons.Examples include an address and place-related descriptions such as “Clay Rd & Queenston” and “west Houston.”
  • Place name recognition: Place name recognition prompts ask models to identify named places in a paragraph and separate them with semicolons.One example labels “Clay Rd & Queenston” and “west Houston” as named places.
  • Prompt examples: The prompts use news-style and historical paragraphs containing locations such as Alabama, Greenville, Interstate 65, Lowndes County, Xenia, and Bourbon County.These paragraphs provide the text contexts that the recognition and localization questions operate on.
  • Geographic localization: Geographic localization prompts first identify named places, then request geographic coordinates for each place with five-decimal precision.The examples include Greenville, Alabama, Lowndes County, San Jose, and New Mexico.
Loading 2304.06798v1…