PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Large Language Models for Human Mobility Prediction under Public Events

Yuebing Liang, Yichao Liu, Xiaohan Wang, Zhan Zhao

arXiv:2311.17351v1cs.AIcs.CL

TL;DR

Public-event demand is difficult to predict because event descriptions are rich but hard to incorporate, while event examples are sparse. LLM-MPE formats event text, decomposes mobility histories, and prompts LLMs to forecast and explain demand; the paper reports that it outperforms existing models in its Barclays Center case study.

  • Problem

    Public-event descriptions are difficult for conventional models to use, while sparse and irregular events complicate prediction and existing text methods may miss full-description semantics.

    Source: §1 · p. 2, §2.1 · p. 5

  • Method

    LLM-MPE standardizes online event descriptions, separates regular mobility patterns from event-related deviations, and prompts LLMs to predict and explain travel demand.

    Source: §1 · p. 3

  • Results

    LLM-MPE outperformed existing machine-learning and deep-learning models in the Barclays Center case study, and the study reports value from incorporating event descriptions.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The framework offers interpretable demand predictions intended to provide insights for managing public events.

    Source: §1 · p. 4

    LLMs can produce incorrect information and may lack current knowledge, and the paper identifies spatial relationships as a challenge for LLMs.

    Source: §6 · pp. 19–20

Abstract

from arXiv · show

Public events, such as concerts and sports games, can be major attractors for large crowds, leading to irregular surges in travel demand. Accurate human mobility prediction for public events is thus crucial for event planning as well as traffic or crowd management. While rich textual descriptions about public events are commonly available from online sources, it is challenging to encode such information in statistical or machine learning models. Existing methods are generally limited in incorporating textual information, handling data sparsity, or providing rationales for their predictions. To address these challenges, we introduce a framework for human mobility prediction under public events (LLM-MPE) based on Large Language Models (LLMs), leveraging their unprecedented ability to process textual data, learn from minimal examples, and generate human-readable explanations. Specifically, LLM-MPE first transforms raw, unstructured event descriptions from online sources into a standardized format, and then segments historical mobility data into regular and event-related components. A prompting strategy is designed to direct LLMs in making and rationalizing demand predictions considering historical mobility and event features. A case study is conducted for Barclays Center in New York City, based on publicly available event information and taxi trip data. Results show that LLM-MPE surpasses traditional models, particularly on event days, with textual data significantly enhancing its accuracy. Furthermore, LLM-MPE offers interpretable insights into its predictions. Despite the great potential of LLMs, we also identify key challenges including misinformation and high costs that remain barriers to their broader adoption in large-scale human mobility analysis.

1. Introduction

The introduction frames public-event mobility prediction as a problem of irregular demand, sparse examples, and underused event text. It presents LLM-MPE as a framework that combines event descriptions and mobility histories while producing prediction rationales.

Motivation

  • Public-event surges make advance demand forecasts relevant to congestion, crowding, and safety, and can inform changes to taxi dispatches and transit service.The paper also names feeder buses and early warnings as possible responses.Source: §1 · p. 2

Research gap

  • Historical mobility methods can miss event timing and attributes, while existing event studies often reduce online information to numerical or categorical features.Text descriptions may convey themes, participants, and background details that those features omit.Source: §1 · p. 2

Research challenges

  • Event descriptions vary from sparse titles to lengthy narratives, creating challenges in interpreting semantics and separating relevant details from extraneous text.The introduction gives “WALK THE MOON” as a title that may require contextual interpretation.Source: §1 · pp. 2–3
  • Infrequent events leave limited historical examples, and event-specific impacts can differ substantially, while conventional models also provide limited human-readable explanations.The paper identifies interpretability as important for checking reliability and informing transport decisions.Source: §1 · p. 3

Contributions

  • LLM-MPE standardizes event descriptions, separates regular mobility from event-related deviations, and prompts LLMs to predict demand and explain their reasoning.The paper reports a Barclays Center case study and contributions involving prediction performance, chain-of-thought explanations, and analysis of LLM limitations.Source: §1 · pp. 3–4

2. Literature Review

Prior public-event forecasting research uses surveys, historical mobility, and online event indicators, but often omits rich textual descriptions. The review positions LLMs as a potential predictor for event-related mobility, beyond their prior urban-language applications and routine-movement forecasts.

2.1. Human Mobility Forecasting for Public Events

  • Survey-based event planning can model travel indicators but requires labor-intensive data collection, limiting scalability to the many smaller urban events.The cited examples include venue surveys in Phoenix, Taipei, and Istanbul.Source: §2.1 · p. 4
  • Historical-mobility anomaly methods can detect demand fluctuations but struggle to identify event occurrence, precise timing, and specific event characteristics.Examples include approaches using station flows, ARIMA/GARCH, transition models, and collective mobility patterns.Source: §2.1 · p. 5
  • Online event studies commonly encode categories, timing, or social-media activity, while unstructured descriptions remain difficult to integrate into conventional models.Prior text methods use topic distributions or word embeddings, which may miss paragraph-level semantics in messy or incomplete descriptions.Source: §2.1 · p. 5

2.2. Large Language Models

  • Urban LLM research has emphasized language tasks and human-machine interaction; an existing mobility-prediction study addressed routine movement, leaving public-event prediction unexplored.The review also describes LLM capabilities including few-shot learning and step-by-step reasoning.Source: §2.2 · p. 6

3. Methodology

The methodology defines venue-level demand and formulates prediction using recent mobility and event information. LLM-MPE then formats event text, decomposes mobility patterns, and prompts LLMs to forecast demand with a human-readable reasoning process.

3.1. Definitions and Problem Statement

  • For a venue, travel demand is inflow and outflow trips; the task predicts upcoming demand from recent mobility, past events, and scheduled events, focusing on daily forecasts.The paper notes that the methodology can be adapted to other time intervals.Source: §3.1 · pp. 6–7

3.2. Overview

  • LLM-MPE has three stages: event feature formatting, mobility feature decomposition, and prediction with explanations using the processed features.Figure 1 depicts the framework.Source: §3.2 · p. 7, Figure 1 · p. 8

3.3. Event Feature Formatting

  • LLMs turn event titles and available descriptions into concise, standardized summaries, condensing clutter and sometimes identifying event categories omitted from titles.The paper notes that event descriptions can be sparse or lengthy and inconsistent.Source: §3.3 · p. 7

3.4. Mobility Feature Decomposition

  • Mobility decomposition estimates regular demand from comparable non-event periods, then defines irregular deviations as actual demand minus that baseline.For daily forecasts, the baseline uses corresponding weekdays without events; both components become LLM inputs.Source: §3.4 · pp. 7–8

3.5. Human Mobility Prediction

  • The prediction task is challenging because event effects are complex and historical event data is sparse, motivating LLMs’ reasoning and few-shot capabilities.The paper describes these capabilities as potential support for inferring event-induced demand from limited historical events.Source: §3.5 · pp. 8–9
  • Prompts provide instructions, historical event and mobility features, next-event details, and estimated regular mobility, then request demand predictions and step-by-step reasoning.The requested outputs are inflow and outflow values; the reasoning is intended to make predictions interpretable.Source: §3.5 · pp. 9–10

4. Case Study

The Barclays Center case study uses taxi trips and event records to examine mobility around public events. Its setup compares LLM-MPE with statistical and machine-learning baselines using historical mobility and event features.

4.1. Data Description

  • From 2013-10-01 to 2015-06-30, taxi trips beginning or ending within 220 m of Barclays Center were aggregated into daily intervals.The radius was chosen to encompass adjacent roads.Source: §4.1 · p. 10
  • Figure 3 presents a prediction prompt template whose red placeholders are adapted for specific venues and dates.Source: Figure 3 · p. 11
  • Event days show markedly higher taxi use than non-event days, while pickup counts exceed dropoffs and ridership varies substantially across events.These patterns motivate modeling event-specific characteristics.Source: §4.1 · p. 11, Figure 5 · p. 12
  • The event dataset contains 350 events across 285 days, with descriptions ranging from absent to more than 400 words.The authors standardized event data to address variation in description format and length; limited historical records at other venues motivated the single-venue focus.Source: §4.1 · p. 12

4.2. Experiment Setups

  • The comparison uses GPT-4 at temperature zero and LR, GBDT, FNN, and RNN baselines, evaluated separately on 149 event days and 216 event-free days.Deep-learning baseline scores average 10 runs; baselines represent event time and text with occurrence and LDA topic distributions, respectively.Source: §4.2 · pp. 13–14
  • All models use a 28-day historical window; conventional models train on 2013-10-01–2014-06-30, while LLM-MPE runs on the 2014-07-01–2015-06-30 test set.Source: §4.2 · p. 13

5. Results

LLM-MPE outperforms baseline models, with the largest reported gains on event days; feature ablations highlight the value of event information and mobility decomposition. Its generated rationales can expose useful event comparisons, but may also contain incorrect calculations.

5.1. Performance Comparison with Baseline Models

  • Compared with GBDT, LLM-MPE improves event-day RMSE by 13.9% and MAE by 17.8%, and non-event-day RMSE by 4.1% and MAE by 13.7%.LLM-MPE achieves the best performance on all evaluation metrics in all circumstances, especially on event days.Source: §5.1 · pp. 14–15, Table 2 · p. 15
  • FNN and RNN underperform GBDT in this case, which the authors attribute to limited training data, including only 350 recorded public events.The paper notes that deep learning models typically need over 10,000 data points for accurate predictions.Source: §5.1 · p. 15
  • Figure 7 shows LLM-MPE tracking actual demand more closely than XGBoost during event-day variations, especially at major events, with occasional peak-time over- or underprediction.The figure illustrates actual demand and predictions from LLM-MPE and XGBoost.Source: §5.1 · p. 15, Figure 7 · p. 16

5.2. Performance Comparison with Different Input Features

  • Adding the next day’s event count reduces LLM-MPE RMSE and MAE by 27.7% and 30.3%, respectively, on event days.Event count also reduces GBDT RMSE and MAE by 16.2% and 17.6%, respectively.Source: §5.2 · p. 16, Table 3 · p. 17
  • Raw event descriptions reduce LLM-MPE RMSE by 22.0% and MAE by 24.9% on event days.Formatted descriptions further improve performance; relying on original demand rather than decomposed mobility features increases LLM-MPE RMSE by 11.1% and MAE by 16.1%.Source: §5.2 · p. 17, Table 3 · p. 17

5.3. Interpretability of LLMs

  • LLM-MPE’s generated rationales identify weekly regularity, daily trends, and patterns from identical or comparable past events when explaining predictions.For unseen events, it can compare them with related historical events, such as comparing the NBA All-Star Event with regular NBA games.Source: §5.3 · pp. 17–18, Table 4 · p. 18
  • In one example, LLM-MPE makes incorrect averages, showing that its rationales can contain inaccurate and misleading information.The authors advise users to be cautious when using these models.Source: §5.3 · p. 18

6. Discussion: Limitations of LLMs

LLM use in human mobility prediction remains an initial exploration rather than a complete real-world solution. The discussion identifies reliability, currency, cost, efficiency, and spatial-representation constraints, alongside possible mitigation strategies.

6. Discussion: Limitations of LLMs

  • The authors characterize the work as an initial exploration, not a complete real-world solution for large-scale human mobility applications.They frame the limitations discussion as input for future research.Source: §6 · p. 19
  • LLMs remain constrained by hallucinations, outdated knowledge, usage costs, slow API calls, and unmodeled spatial interactions between venues.The study reports 12.7 seconds for one LLM-MPE test prediction, compared with 2.4 seconds for GBDT predictions across all test samples.Source: §6 · p. 19
  • Suggested mitigations include reviewing outputs, developing open-source models, and using conventional models on non-event days while reserving LLMs for event days.The proposed hybrid strategy draws on conventional models’ comparable effectiveness on non-event days.Source: §6 · p. 19
  • Spatial dependencies could be represented through natural-language descriptions or by using LLM-derived text representations as node attributes in graph neural networks.The latter approach combines LLMs with domain-specific deep neural networks such as GNNs.Source: §6 · p. 20

7. Conclusion

The paper presents LLM-MPE for predicting mobility under public events and reports better performance than existing machine-learning models, particularly on event days. It also identifies limits to real-world use and directions for extending evaluation and application.

7. Conclusion

  • LLM-MPE surpasses existing machine-learning models, particularly on event days, and ablations identify event-description text as critical to its performance.The study also analyzes LLM-generated explanations, reporting clear and detailed justifications for predictions.Source: §7 · p. 20
  • LLM-MPE standardizes online event descriptions, separates regular mobility patterns from irregular deviations, and prompts LLMs to predict demand and explain their reasoning.The case study uses New York City taxi trips and online event information from Barclays Center.Source: §7 · p. 20
  • The study identifies inaccurate and outdated information, high operating costs, efficiency constraints, and difficulty representing spatial relationships as barriers to large-scale use.The authors state that further progress is needed before LLMs provide a fully viable real-world solution.Source: §7 · p. 20
  • Further evaluation could test other venues, while extending prediction beyond inflow and outflow to origin-destination flows, travel time, transport mode, and individual decisions.The proposed transfer to other venues presumes access to human mobility data and event information.Source: §7 · p. 20
  • The authors propose exploring LLMs to identify unforeseen events from text, as a potential way to improve real-time mobility-flow predictions in those scenarios.They note that unpredictable event timing and location make this task more challenging.Source: §7 · pp. 20–21

Related papers

Loading 2311.17351v1…