Foundation Models for Spatio-Temporal Data Science: A Tutorial and Survey

1 Introduction

• ST data science senses, manages, and mines data to reveal patterns, understand complex systems, and predict future dynamics.The paper connects these activities to decision-making and innovation across urban environments and the planet.Source: §1 · p. 1
• STFMs include LLM- and PFM-based approaches with perception, reasoning, and optimization capabilities, extending beyond mining to sensing and management.The survey describes LLMs and cross-domain PFMs as complementary approaches to adaptable ST solutions.Source: §1 · p. 1, Figure 1 · p. 1
• Existing STFM surveys focus mainly on mining and numerical problems, leaving sensing, management, and inferential applications less examined.The survey addresses these gaps across the ST data science workflow.Source: §1 · p. 2
• The survey categorizes STFMs as LLMs used zero-shot or few-shot and PFMs pretrained from scratch on cross-domain ST data.It examines PFM architectures, pretraining, and adaptation across locations, trajectories, events, rasters, and graphs.Source: §1 · p. 2
• The survey presents a vision of STFMs, reviews methodologies for key capabilities, and identifies future research directions.Its stated contribution is a comprehensive, modern survey spanning sensing, management, and mining.Source: §1 · p. 2

2 Background

• Foundation models are deep neural networks trained on large datasets to acquire cross-domain knowledge and adapt to tasks with relatively little task-specific data.The background cites ST examples including Time-MoE, Chronos, UniST, and UniTraj.Source: §2 · p. 2
• A location is a fixed spatial point or object represented by latitude and longitude coordinates in R2.Locations may be profiled using satellite imagery, street-view imagery, and descriptions.Source: §2 · p. 2
• A trajectory is a time-ordered sequence of locations describing an object's movement through geographic space.Each point pairs a location with its time.Source: §2 · p. 2
• An event sequence is a series of timestamped events describing the progression of actions or occurrences.Each event pairs an event representation with a timestamp.Source: §2 · p. 2
• An ST raster organizes signals across evenly distributed locations and time, while an ST graph adds explicit spatial correlations for nonuniform locations.The raster has H by W spatial positions and T time steps; the graph represents locations as nodes and their relationships as edges.Source: §2 · p. 3

3 The Workflow Perspective

• ST data sensing acquires data that varies over space and time from sources such as sensors, satellites, and social media.The survey also includes synthetic data generation as a way to enhance data diversity and quantity.Source: §3 · p. 3
• ST data management stores, indexes, and organizes large-scale heterogeneous datasets, with FMs supporting cleaning, retrieval, and integration.Distributed architectures are among the strategies discussed for efficient retrieval and integration.Source: §3 · p. 3
• ST data mining analyzes data across space and time to uncover patterns, trends, and relationships using DM, DL, or STFMs.The survey describes STFMs as bringing perception, optimization, and reasoning capabilities to this stage.Source: §3 · p. 3
• Downstream applications use insights from ST data for numerical and inferential problems, including formulating informed actions and policies.The workflow proceeds from acquisition toward high-level services and solutions.Source: §3 · p. 3

3.1 Spatio-Temporal Data Sensing

• FMs support real-world ST sensing and synthetic data generation as complementary aspects of data acquisition.Synthetic generation can increase data diversity and quantity.Source: §3.1 · p. 3
• LLMs can process citizen reports, classify and prioritize reported issues, and trigger responses to conditions such as congestion or environmental hazards.The passage frames this as active sensing for urban and environmental monitoring.Source: §3.1.1 · p. 3

3.1.1 Real-World Data Sensing.

• FMs can schedule crowdsensing participants by region, time, proximity, availability, and past contributions under budget constraints.The passage describes reducing redundant collection and coordinating sensing tasks across citizens or devices.Source: §3.1.1 · p. 3
• LLMs can generate vehicle and human trajectories from textual descriptions, personas, memory modules, or routine profiles.The survey notes synthetic ST data can help compensate for costly, privacy-sensitive, missing, or sparse real-world data.Source: §3.1.2 · p. 3

3.2 Spatio-Temporal Data Management

• After acquisition, management must address data-quality problems and support retrieval and integration.The passage names missing values and missing views as examples of quality issues.Source: §3.2 · p. 3

3.2.1 Data Cleaning.

• For cleaning, LLMs can impute missing values and augment missing views or information in ST datasets.The survey cites PLMTrajRec for recovering sparse trajectories and notes tests across varied sampling intervals.Source: §3.2.1 · p. 4

3.2 Spatio-Temporal Data Management

• LLMs can improve contextual retrieval by translating urban queries into subtasks or retrieving planning documents with RAG.The survey also describes multimodal LLMs for cross-modal information retrieval.Source: §3.2.2 · p. 4

3.2.3 Data Integration.

• For data integration, LLMs extract relational triplets and use geospatial reasoning to complete knowledge graphs from heterogeneous urban sources.UrbanKGent is described as an agent framework for automating urban knowledge-graph construction.Source: §3.2.3 · p. 4

3.3 Spatio-Temporal Data Mining

• ST data mining models spatial and temporal dependencies, while STFMs integrate perception, optimization, and reasoning.The survey directs readers to Section 3.4 for applications of these capabilities.Source: §3.3 · p. 4

3.3.1 Perception.

• Perception includes understanding an agent’s surroundings and extracting ST patterns from sensor data for prediction.The survey notes that generalization remains challenging: research is often in-domain, and UniST’s training and testbeds are predominantly transportation-focused.Source: §3.3.1 · p. 4

3.3.2 Optimization.

• Optimization adapts models to task objectives and actionable decisions; examples include LLM-agent frameworks for traffic control, analysis, and planning.The cited examples also include mobility prediction and zero-shot geospatial question answering.Source: §3.3.2 · p. 4

3.3.3 Reasoning.

• The survey identifies common-sense, numerical, and causal reasoning as underdeveloped capabilities needed to move beyond pattern recognition toward ST general intelligence.Examples include counterfactual video question-answering, quantitative forecasting, and causal-structure learning.Source: §3.3.3 · p. 5

3.4.1 STFMs for Numerical Problems.

• Numerical ST tasks include forecasting, imputation, and anomaly detection, where models capture dependencies and learn system dynamics.The survey also lists time-series classification, geospatial prediction, traffic-speed inference, and socioeconomic-indicator prediction.Source: §3.4.1 · pp. 5–6
• Forecasting approaches range from task-specific STGNNs to universal models, including methods that pretrain on observations or combine ST inputs with LLMs.The survey describes UniST as unifying traffic prediction tasks and UrbanGPT as using ST instruction tuning.Source: §3.4.1 · p. 5
• For imputation and anomaly detection, surveyed approaches use language models, contrastive learning, specialized tokenization, and signal-to-text pipelines.The survey also describes benchmark-based zero-shot reasoning and anomaly-context enrichment for anomaly detection.Source: §3.4.1 · p. 5

3.4.2 STFMs for Inferential Problems.

• Inferential problems ask what happened, where it happened, what to do, or what might happen, requiring ST understanding beyond numerical prediction.The survey emphasizes location, movement, and environmental context in these tasks.Source: §3.4.2 · p. 6
• For event analysis, LAMP uses abductive reasoning to propose causes and retrieve evidence, while LEAP frames event detection as question answering.LEAP also predicts missing event components and future relations.Source: §3.4.2 · p. 6
• Physical-grounding applications use multimodal and map information for localization, map updates, personalized routing, and navigation.The survey describes ChinaTravel as a benchmark for evaluating constraint satisfaction and preference optimization in real-world Chinese travel planning.Source: §3.4.2 · p. 6
• Decision-making examples include traffic optimization, autonomous-vehicle policies, and disaster response using spatio-temporal data.For autonomous driving, STFMs contribute directly through policy methods and indirectly through behavior prediction and multimodal perception.Source: §3.4.2 · p. 6
• Scenario simulation uses multi-agent systems to model interactions in urban planning, social dynamics, and games.The survey reports land-use plans with improved accessibility and ecological outcomes compared with human expert solutions.Source: §3.4.2 · p. 6

4 The Methodology Perspective

• The method-centric taxonomy distinguishes LLMs used zero-shot or fine-tuned across the ST workflow from PFMs pretrained from scratch, mainly for ST data mining.Figure 5 presents the taxonomy, with a full version in Figure 7.Source: §4 · p. 7, Figure 5 · p. 7

4.1 Large Language Models (LLM)

• The survey broadly includes standard LLMs, vision-language models, and multimodal LLMs, used as augmenters, predictors, or agents.It groups LLM zero-shot use and fine-tuning as two methodological approaches.Source: §4.1.1 · p. 7

4.1.1 Zero-shot Learner.

• Zero-shot methods shape outputs through prompt construction, prompt learning, chain-of-thought prompting, and agent design.Agent design covers role assignment, memory, tool definition, and multi-agent systems; prompts can include instructions, tokenization, and examples.Source: §4.1.1 · p. 7

4.1.2 Supervised Fine-Tuning for LLMs.

• LLM fine-tuning updates all or part of model parameters, or adds trainable components, with trade-offs in data needs, compute, and preservation of general knowledge.Add-on approaches keep original weights frozen; input-tokenization layers and task-specific prediction heads are also described.Source: §4.1.2 · p. 7

4.2 Pretrained Foundation Models (PFM)

• PFMs are trained from scratch on cross-domain ST data to model ST dependencies without constraints from linguistic priors.The survey examines their architectures, pretraining methods, and adaptability.Source: §4.2 · p. 8

4.2.1 Neural Architecture.

• PFM architectures include transformer, diffusion, graph, state-space, and CNN-based models.The survey associates transformers with sequential modeling, diffusion with generation and prediction, and graph models with spatially structured data.Source: §4.2.1 · p. 8

4.2.2 Pretraining Scheme.

• PFM pretraining objectives are generative, contrastive, or hybrid, with contrastive learning aligning augmented views and heterogeneous modalities.Hybrid pretraining combines generative and contrastive objectives.Source: §4.2.2 · p. 8

4.2.3 Data Modality.

• PFMs are tailored to location, trajectory and event, ST raster, and ST graph data, with comparatively limited work on ST graphs.The survey gives examples including location embeddings, trajectory transfer, climate forecasting, and graph-based traffic forecasting.Source: §4.2.3 · p. 8, Figure 6 · p. 8

• The survey reviews foundation models across sensing, management, and mining, covering diverse downstream tasks and datasets.Source: §5 · p. 8
• It identifies perception, reasoning, and optimization as key capabilities and establishes a taxonomy of STFMs’ methodologies.Source: §5 · p. 8
• Generalization, interpretability, and efficiency remain challenges, and the survey outlines future directions for scalable, adaptive STFMs.Source: §5 · p. 8

Limitations and Future Opportunities

• Directly using LLMs for numerical tasks such as forecasting is non-trivial.Source: A Limitations and Future Opportunities · p. 13
• Fine-tuning LLMs or training STFMs from large cross-domain ST data can come at the cost of interactive capabilities, hindering prediction interpretability.Source: A Limitations and Future Opportunities · p. 13
• Large FMs’ superiority over smaller expert models is context-dependent, and expert models can outperform them with sufficient domain-specific data.Source: A Limitations and Future Opportunities · p. 13
• Current FMs typically support specific stages rather than the full workflow from sensing and management through mining and downstream applications.Source: A Limitations and Future Opportunities · p. 13
• Although current STFMs excel on structured ST data, their ability to integrate and reason over text, images, video, and sensor data remains underdeveloped.Source: A Limitations and Future Opportunities · p. 13

Zero-shot Utilization of LLMs

• LLM-as-Augmenter uses pretrained LLMs to improve data understanding or model performance by enriching inputs or adding a frozen model component [40] [79].Source: B Zero-shot Utilization of LLMs · p. 13
• LLM-as-Predictor employs LLMs directly for tasks, using prompt engineering or patching and tokenization to address the modality gap [53].Source: B Zero-shot Utilization of LLMs · p. 13
• LLM-as-Agent equips LLMs to route tasks to plug-and-play domain models or coordinate multi-agent systems [144] [174] [185].Source: B Zero-shot Utilization of LLMs · p. 13

Comparison between LLMs and PFMs

• PFMs excel in numerical reasoning but often struggle with common-sense understanding.Source: C Comparison between LLMs and PFMs · p. 13
• The section says there is no free lunch and recommends choosing between LLMs and PFMs according to the downstream application.Source: C Comparison between LLMs and PFMs · p. 13
• Table 3 compares LLMs and PFMs across perception, optimization, and reasoning.Source: Table 3 · p. 13, C Comparison between LLMs and PFMs · p. 13