Leveraging Language Foundation Models for Human Mobility Forecasting

1 INTRODUCTION

• Numerical forecasting typically maps historical mobility values to future values, modeling numeric data while making seamless integration with natural-language assistants difficult.The paper identifies customer-flow prediction as one practical application of mobility forecasting.Source: §1 · p. 1
• Pretrained language models operate on sentences, while mobility forecasting uses sequential numerical data; the paper identifies direct application of these models to mobility prediction as unexplored.The mismatch separates language-model pretraining from the downstream prediction format.Source: §1 · p. 2
• Mobility prompting converts historical mobility observations into natural-language inputs for an encoder-decoder that predicts future mobility.The paper says prompts can represent numerical sequences, textual context, and semantic information.Source: §1 · p. 2
• AuxMobLCast adds auxiliary POI-category classification associated with the encoder’s [CLS] token, without modifying the foundation-model structure.The authors say this design permits direct use of existing pretrained weights.Source: §1 · p. 2
• The paper presents a first attempt to fine-tune existing pretrained language foundation models for human mobility forecasting and empirically examines multiple models.The introduction reports that AuxMobLCast can further improve prediction performance.Source: §1 · p. 2

2.1 Language Foundation Models

• Foundation models are pretrained on large-scale data and adapted to downstream tasks; the section describes BERT’s masked-language-modeling pretraining and notes applications beyond NLP.The paper distinguishes its focus on sequential numbers from prior language-model prediction of separate numbers and.Source: §2.1 · p. 2

2.2 Time-series Forecasting

• Time-series and mobility forecasting work includes numerical sequence models such as RNNs and Transformer-based methods.The cited discussion names ST-RNN and DeepMove among numerical-paradigm approaches.Source: §2.2 · p. 2
• Existing forecasting methods cannot directly leverage pretrained language models because large-scale time-series pretraining data are lacking and task-specific architecture changes impede direct application.The passage connects the scarcity of data to privacy concerns in mobility-data collection.Source: §2.2 · p. 3
• The paper shifts from modifying model structures to converting aggregated mobility sequences into language descriptions through prompts, enabling direct use of pretrained language foundation models.The AuxMobLCast illustration shows numerical and contextual tokens interacting in prompt embeddings alongside auxiliary POI-category classification.Source: §2.2 · p. 3, Figure 2 · p. 3

3.1 Problem Definition

• For each POI, the forecasting task predicts next-day visits from a history of daily customer visits.The method targets aggregated human mobility for POIs in an area such as a city.Source: §3.1 · p. 3

3.2 Mobility Prompting

• Mobility prompting converts visit sequences into language descriptions containing mobility, temporal, and POI information for pretrained language models.The prompts include history timestamps and the prediction target time; Prompt B removes the POI ID but retains category information, while Prompt C adds a category target.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 Proposed AuxMobLCast

• AuxMobLCast uses an encoder-decoder pipeline for token generation and an auxiliary classifier that predicts POI category from the encoder’s [CLS] embedding.The auxiliary classification operation does not change the encoder structure, allowing pretrained encoder and decoder weights to initialize the pipeline.Source: §3.3 · pp. 4–5

3.4 Training Objective

• AuxMobLCast combines sequence-generation cross-entropy with an auxiliary POI-category classification loss during end-to-end fine-tuning.The two losses are weighted by factors whose sum is 1.Source: §3.4 · p. 5

• RQ1 compares language-based encoder-decoder forecasting with mobility prompts against conventional numerical-paradigm forecasting methods.Source: §4 · p. 5
• RQ2 asks whether different pretrained language foundation models can serve as encoders in AuxMobLCast and how their forecasting performance compares.Source: §4 · p. 5
• RQ3 tests whether auxiliary POI-category classification yields further mobility-forecasting gains in AuxMobLCast.Source: §4 · p. 5

• The three city datasets contain 479 POIs and 39 categories, 1,374 POIs and 65 categories, and 1,007 POIs and 51 categories, respectively.Dallas has the most POIs and categories, while New York City has the fewest.Source: §4.1 · p. 5
• The evaluation uses aggregated and anonymized SafeGraph daily visit counts and POI category information collected for three representative cities.The data were accessed through SafeGraph Data for Academics.Source: §4.1 · p. 5
• Each city dataset is split into 70% training, 10% validation, and 20% testing, with a 15-day history observation window.Numerical-format splits are also maintained for comparisons with numerical-paradigm methods.Source: §4.1 · p. 5

• Encoder and decoder configurations are initialized with HuggingFace pretrained weights and fine-tuned on generated mobility prompts for up to 50 epochs with early stopping.Source: §4.2 · p. 5
• The loss weights are λCE=0.8 and λPOI=0.2 for New York City and Miami, and λCE=0.9 and λPOI=0.1 for Dallas.Source: §4.2 · p. 5
• Models use Adam with an initial learning rate of 5 × 10^-5 and weight decay of 5 × 10^-4, plus ReduceLROnPlateau with patience 6 and cooldown 2.These parameters were selected based on validation-set performance.Source: §4.2 · p. 5

4.3 Evaluation

• RMSE and MAE measure the closeness of predicted values to ground truth.The metrics are calculated from the predicted and actual values.Source: §4.3 · p. 5
• Results report average performance and standard deviation across five runs with different random seeds, excluding linear regression.Source: §4.3 · p. 5

4.4 Results and Analysis

• The comparison includes numerical forecasting baselines and GRUAtt and Transformer encoder-decoder models trained with Prompt A or Prompt B.Numerical baselines include GRU [5], GRUAtt, Transformer, Reformer, Informer, and Autoformer.Source: §4.4 · pp. 5–6
• Among numerical methods using temporal embeddings, Informer achieves the best RMSE and MAE; Transformer with temporal embeddings improves RMSE over Transformer without them.Transformer, Reformer, Informer, and Autoformer are evaluated with temporal embeddings; Table 3 reports the results.Source: §4.4 · p. 6, Table 3 · p. 6
• With the same backbone, mobility-prompt methods can outperform their non-prompt counterparts; vanilla Transformer-A/B are comparable to Informer and Autoformer with temporal embeddings, with worse average RMSE but better MAE.Prompt A and Prompt B have very close performance.Source: §4.4 · pp. 6–7
• Prompt B is preferable because its performance is close to Prompt A while omitting sensitive POI-id information.Source: §4.4 · p. 7

4.5 Performance of AuxMobLCast and Ablation Study

• AuxMobLCast uses GPT-2 as decoder and evaluates BERT, RoBERTa, and XLNet as encoders using Prompt C, with or without the auxiliary POI-category target.When the auxiliary task is disabled, the POI-category target is removed and only the mobility target remains.Source: §4.5 · p. 7

4.5.1 Different Encoders.

• BERT-encoder AuxMobLCast outperforms Informer and Transformer-B on average; BERT reduces Dallas RMSE by 9.5% relative to RoBERTa, while XLNet is slightly worse than Transformer-B on average.BERT and RoBERTa are not the best on Miami, but improve on Dallas against the Table 3 baselines.Source: §4.5.1 · p. 7

4.5.2 With or Without Auxiliary Classification.

• Auxiliary classification improves average performance with BERT or RoBERTa, but effects vary by encoder and dataset; BERT gains substantially on Dallas and Miami RMSE.With XLNet, the auxiliary task improves Miami but worsens both RMSE and MAE on Dallas; with RoBERTa, Dallas RMSE increases while MAE decreases.Source: §4.5.2 · p. 7

4.5.3 With or Without Date Information in Prompts.

• Removing temporal date information from prompts decreases performance for BERT, RoBERTa, and XLNet; the authors suggest missing date-number relationships such as weekly patterns may explain the reduction.The simplified prompt retains the sequence of visitor counts but omits temporal information.Source: §4.5.3 · p. 8

4.6 Zero-shot Performance

• AuxMobLCast achieves four top performances across six zero-shot train/test configurations.The comparison includes numerical methods with temporal embeddings and AuxMobLCast using XLNet, BERT, or RoBERTa encoders.Source: §4.6 · p. 8
• Numerical methods perform better when Miami is the test set for models trained on either NYC or Dallas; AuxMobLCast shows promising, but not uniformly superior, generalization.Source: §4.6 · p. 8

5 DISCUSSION

• Mobility prompts and the AuxMobLCast encoder-decoder pipeline, with auxiliary POI-category classification, improve human mobility forecasting performance.The prompts transform numerical mobility data into sentences for model inputs and output targets.Source: §5 · p. 8
• Transforming numerical sequences into language prompts may help Transformers model both relationships across time steps and links between values and contextual information.The discussion connects this approach to Transformers’ attention mechanisms for discovering relationships among sequence tokens.Source: §5 · pp. 8–9
• The reported improvement is not very large, while the authors suggest language-model forecasting could inform work on weather and demand forecasting.They also mention integrating spatio-temporal forecasting into multimedia assistants as a possible direction.Source: §5 · p. 9
• Mobility prompt generation remains a limitation; the authors identify prompt-learning techniques, automatic transformation of diverse sequences, and multivariate forecasting as future work.They describe automatic transformation of diverse sequential numerical behavior and time-series data as beneficial for exploring forecasting ability.Source: §5 · p. 9