PaperlayerView exact revision on arXiv

Source-linked AI summary

UrbanGPT: Spatio-Temporal Large Language Models

Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, Chao Huang

arXiv:2403.00813v3cs.CLcs.AIcs.CY

TL;DR

Spatio-temporal models often depend on abundant labeled data, despite data scarcity in practical urban sensing and the need to generalize across downstream tasks. UrbanGPT integrates a spatio-temporal dependency encoder with instruction tuning, and experiments report consistent superiority over state-of-the-art baselines, especially in zero-shot settings. The paper therefore supports large language models as a promising approach for spatio-temporal learning when labeled data is scarce.

  • Problem

    Existing spatio-temporal methods depend heavily on labeled data, while urban sensing faces data scarcity and requires generalization across diverse downstream scenarios.

  • Method

    UrbanGPT integrates a spatio-temporal dependency encoder, including multi-level temporal convolution, with instruction tuning to align urban context and language-model knowledge.

  • Results

    Experiments consistently show UrbanGPT outperforming state-of-the-art baselines across public datasets and spatio-temporal prediction tasks, including zero-shot scenarios.

  • Takeaways & Limitations

    The findings highlight the potential of large language models for generalizable spatio-temporal learning when labeled data is scarce, particularly in zero-shot settings.

  • Takeaways & Limitations

    The authors identify the need to collect a more diverse range of urban data in future studies.

Abstract

from arXiv · show

Spatio-temporal prediction aims to forecast and gain insights into the ever-changing dynamics of urban environments across both time and space. Its purpose is to anticipate future patterns, trends, and events in diverse facets of urban life, including transportation, population movement, and crime rates. Although numerous efforts have been dedicated to developing neural network techniques for accurate predictions on spatio-temporal data, it is important to note that many of these methods heavily depend on having sufficient labeled data to generate precise spatio-temporal representations. Unfortunately, the issue of data scarcity is pervasive in practical urban sensing scenarios. Consequently, it becomes necessary to build a spatio-temporal model with strong generalization capabilities across diverse spatio-temporal learning scenarios. Taking inspiration from the remarkable achievements of large language models (LLMs), our objective is to create a spatio-temporal LLM that can exhibit exceptional generalization capabilities across a wide range of downstream urban tasks. To achieve this objective, we present the UrbanGPT, which seamlessly integrates a spatio-temporal dependency encoder with the instruction-tuning paradigm. This integration enables LLMs to comprehend the complex inter-dependencies across time and space, facilitating more comprehensive and accurate predictions under data scarcity. To validate the effectiveness of our approach, we conduct extensive experiments on various public datasets, covering different spatio-temporal prediction tasks. The results consistently demonstrate that our UrbanGPT, with its carefully designed architecture, consistently outperforms state-of-the-art baselines. These findings highlight the potential of building large language models for spatio-temporal learning, particularly in zero-shot scenarios where labeled data is scarce.

1 INTRODUCTION

UrbanGPT targets spatio-temporal prediction under scarce labeled data by aligning urban time-space dependencies with large language models. Its instruction-tuned architecture is designed for broad zero-shot generalization across urban tasks.

  • Motivation: Spatio-temporal neural networks often require abundant labeled data, while practical urban sensing faces scarcity because citywide sensor deployment is costly.This motivates models that can generalize beyond datasets with extensive supervision.
  • Motivation: UrbanGPT aims to develop a spatio-temporal large language model that generalizes across diverse urban prediction tasks and previously unseen scenarios.The goal includes comprehending and forecasting complex spatial and temporal patterns.
  • Approach: UrbanGPT integrates a spatio-temporal dependency encoder with instruction tuning to align textual information and urban spatio-temporal context.The encoder uses multi-level temporal convolution to capture dynamics at different time resolutions.
  • Evidence: In zero-shot traffic-flow prediction, UrbanGPT outperforms LLaMA-70B and STGCN by combining domain-specific spatio-temporal knowledge with language modeling.The comparison highlights limitations of LLaMA on numeric dependencies and of pre-trained spatio-temporal models on source-dataset overfitting.
  • Evidence: Experiments on three benchmark datasets provide evidence that UrbanGPT generalizes effectively in zero-shot spatio-temporal learning with limited or absent prior target-data training.The stated contribution covers diverse urban phenomena across datasets.

2 PRELIMINARIES

Spatio-temporal prediction uses historical observations to forecast future urban data across regions, times, and features. Zero-shot learning addresses downstream datasets or tasks not encountered during training.

  • Spatio-Temporal Data: Spatio-temporal data is represented as a tensor whose entries encode features at time intervals and spatial regions.The representation is X ∈ R^R×T×F, with each element corresponding to one feature, time interval, and region.
  • Spatio-Temporal Prediction: Prediction commonly forecasts the next P time steps from the preceding H steps using a trained spatio-temporal function.The framework supports forecasting future trends from historical data.
  • Task Types: Spatio-temporal tasks include regression of continuous quantities such as traffic flow and classification of events such as crime occurrence.Different scenario characteristics determine the loss functions used to optimize the prediction model.
  • Spatio-Temporal Zero-Shot Learning: Spatio-temporal zero-shot learning forecasts downstream data from previously unseen datasets or tasks without training the prediction function specifically on target data.This setting tests generalization across downstream spatio-temporal scenarios.

3 METHODOLOGY

UrbanGPT combines a spatio-temporal dependency encoder with instruction tuning to align urban time-space information with language-model representations. Its methodology captures temporal dependencies at multiple levels, injects cross-level correlations, incorporates textual context, and converts forecasting-token representations into predictions.

  • 3.1 Spatio-Temporal Dependency Encoder: UrbanGPT begins with an initial embedding E_r obtained by applying a linear layer to the original data X.A slice of the embedding is used for residual operations to address gradient vanishing.
  • 3.1 Spatio-Temporal Dependency Encoder: The spatio-temporal encoder uses gated dilated convolution and multi-level correlation injection to capture and preserve temporal dependencies at different granularities.The gated convolution captures dependencies across multiple time steps, while correlation injection incorporates relationships between temporal levels.
  • 3.1 Spatio-Temporal Dependency Encoder: The encoder produces final spatio-temporal dependency representations and is designed without graph-structure dependence for diverse zero-shot urban scenarios.The resulting representations are obtained after multiple encoding layers and nonlinear merging of the encoder outputs.
  • 3.2.1 Spatio-Temporal Dependency Encoding: UrbanGPT aligns textual and spatio-temporal information by projecting spatio-temporal representations into the language model’s hidden dimension and inserting them as special instruction tokens.The token markers identify the beginning and end of spatio-temporal content, while the placeholder represents the projected spatio-temporal token.
  • 3.2.2 Spatio-Temporal Prompt Instructions: The instructions encode multi-granularity time information and regional details, including day, hour, city, administrative areas, and nearby POIs.These inputs help UrbanGPT identify patterns across regions and timeframes for zero-shot reasoning.
  • 3.2.3 Spatio-Temporal Instruction-Tuning: Because numerical forecasts differ from language-model inputs, UrbanGPT generates forecasting tokens whose hidden representations are mapped to predictive values through a regression layer.The forecasting-token representations retain spatio-temporal contextual attributes used for prediction.
  • 3.3 Model Optimization: UrbanGPT uses task-dependent optimization with language-model, regression, and classification losses to support language modeling, regression, and classification tasks.The model applies different losses according to the task inputs, including regression loss for traffic-flow prediction.

4 EVALUATION

UrbanGPT is evaluated across zero-shot, supervised, ablation, and robustness settings to test generalization across regions, cities, datasets, and spatio-temporal patterns. Results attribute its performance to semantic context, instruction-tuning, the spatio-temporal encoder, and regression prediction components.

  • Evaluation settings: UrbanGPT is evaluated on four datasets spanning taxi, bike, and crime data, with experiments covering zero-shot and supervised prediction scenarios.The evaluation includes NYC-taxi, NYC-bike, NYC-crime, and CHI-taxi, with unseen regions or cities used for zero-shot testing and training regions reused for supervised testing.
  • Zero-shot prediction: In cross-region zero-shot prediction, UrbanGPT surpasses baseline models on regression and classification tasks across multiple datasets.The reported advantage is attributed to spatio-temporal-text alignment and the preservation of transferable spatio-temporal knowledge through instruction-tuning.
  • Zero-shot prediction: UrbanGPT consistently outperforms the comparison method at every time step on unseen CHI-taxi data, maintaining an advantage in both short- and long-term prediction.The authors associate this cross-city performance with transferable spatio-temporal patterns combined with geographical and temporal information.
  • Supervised prediction: UrbanGPT shows advantages over baselines in long-term supervised forecasting and uses geographic and POI information to capture correlations among spatially similar areas.The supervised experiments include broader time intervals, such as training on 2017 and evaluating on 2021.
  • Ablation study: Ablations show that removing spatial-temporal context, diverse datasets, or the spatio-temporal encoder reduces performance, while textual regression produces suboptimal numerical predictions.The encoder aligns urban dependency dynamics with the LLM, whereas a regression predictor improves precision for continuous forecasting values.
  • Robustness study: UrbanGPT improves performance in high-variance regions, especially the (0.75, 1.0] interval where the baseline struggles with inaccurate zero-shot predictions.Most models perform well in lower-variance regions with relatively stable patterns, while high-variance areas represent more diverse spatio-temporal behavior.
  • Case study: The case study reports that UrbanGPT captures universal spatio-temporal patterns and generalizes across diverse zero-shot forecasting scenarios.The comparison emphasizes the difficulty of directly inferring spatio-temporal patterns from numeric geo-series data with general-purpose LLMs.

5 RELATED WORK

Deep spatio-temporal prediction models combine temporal dependency modeling with spatial correlation encoding, while later work explores multi-scale and multi-granularity temporal learning. Most existing studies focus on supervised settings, leaving zero-shot forecasting comparatively underexplored.

  • Deep spatio-temporal models typically combine temporal dependency modeling with spatial correlation encoding.
  • Multi-scale and multi-granularity temporal learning capture long-term, short-term, and periodic temporal correlations.
  • Most existing studies target supervised contexts, whereas zero-shot spatio-temporal forecasting has received limited research attention.

6 CONCLUSION

UrbanGPT is presented as a spatio-temporal large language model that aligns contextual signals with LLMs through instruction tuning. The conclusion reports promising effectiveness while identifying broader data coverage and interpretability as future priorities.

  • UrbanGPT uses spatio-temporal instruction tuning to align contextual signals with LLMs and learn transferable patterns across urban data.
  • Extensive experiments and ablation studies demonstrate the effectiveness of UrbanGPT’s architecture and key components.
  • Future work will collect more diverse urban data to expand UrbanGPT across a broader range of urban computing domains.
  • Future research will focus on improving UrbanGPT’s interpretability and explainability.

A APPENDIX

The appendix documents the experimental setup and supplements the main paper with implementation details, baselines, and a zero-shot case study. It is intended to support a comprehensive understanding of the evaluation.

  • The appendix provides detailed information about datasets, hyperparameters, instruction-tuning settings, test settings, and baselines.
  • It includes experimental settings for both the instruction-tuning phase and the test phase.
  • The appendix presents a case study demonstrating UrbanGPT’s effectiveness for zero-shot spatio-temporal prediction.

A.1 Experimental Details Description

The experiments use New York City taxi, bicycle, and crime data represented over grids with task-specific spatial resolutions and sampling intervals. Taxi and bicycle data are sampled every 30 minutes, while crime data are sampled daily.

  • The NYC-taxi dataset contains 263 regions, each measuring approximately 3km x 3km, with 30-minute sampling intervals.
  • The NYC-bike and NYC-crime datasets each contain 2162 regions represented by 1km x 1km grids.
  • NYC-bike uses 30-minute sampling, whereas NYC-crime uses 1-day sampling.

A.1.1 Dataset Details.

The time encoder uses a dilation convolution kernel with specified input and output dimensions and a dilation factor.

  • The time encoder’s dilation convolution kernel uses a dilation factor of 1.
  • The passage also identifies an additional output-dimension parameter for the kernel, without stating its complete notation.

A.1.2 Hyperparameters Settings.

The prediction setup uses 12 historical steps to forecast the next 12 steps, with configured projection and regression-layer dimensions.

  • The model predicts the next 12 data steps from the previous 12 steps.
  • Both the history length H and prediction length P are set to 12.
  • The projection layer uses d set to 64 and d_L set to 4096, while the regression hidden layer uses d′ set to 128.

A.1.3 Further Experimental Setup Descriptions.

Instruction tuning uses 80 randomly selected New York City regions across three datasets, with dataset-specific training intervals.

  • Instruction tuning randomly selects 80 regions from three New York City datasets as training data.
  • Region indices are kept consistent between the NYC-bike and NYC-crime datasets.

A.1.4 Baseline Details.

The experiments compare categorized spatio-temporal baselines and evaluate zero-shot LLM predictions for bicycle and taxi flow in New York City.

  • Baseline categories: Baselines are grouped into RNN-based, attention-based, and graph neural network-based spatio-temporal prediction models.
  • RNN-based methods: Examples of RNN-based baselines include ST-LSTM, AGCRN, and DMVSTNET, which model temporal dependencies in spatio-temporal data.
  • UrbanGPT: UrbanGPT integrates spatio-temporal contextual signals with LLM reasoning through spatio-temporal instruction tuning.
  • Zero-shot evaluation: Zero-shot tests examine LLM predictions for NYC bicycle flow and taxi flow, with the taxi experiment omitting spatial information.
Loading 2403.00813v3…