Do LLMs Know Your Neighborhood? Auditing LLM Priors for Neighborhood-Level Mobility Prediction and Structural Alignment

Saad Mohammad Abrar; Eesha Kurella; Arnav Dadarya; Naman Awasthi; Kazi Tasnim Zinat; Vanessa Frias-Martinez

Source-linked AI summary

Do LLMs Know Your Neighborhood? Auditing LLM Priors for Neighborhood-Level Mobility Prediction and Structural Alignment

Saad Mohammad Abrar, Eesha Kurella, Arnav Dadarya, Naman Awasthi, Kazi Tasnim Zinat, Vanessa Frias-Martinez

arXiv:2609.00345v1Aug 31, 2026cs.LGcs.CY

TL;DR

Fine-grained mobility data are difficult to access, motivating tests of whether zero-shot LLMs can infer aggregate neighborhood mobility from urban context. Using anonymized Cuebiq data across four U.S. metropolitan areas, the paper compares LLM predictions with supervised baselines and audits directional alignment with empirical relationships. LLMs recover nontrivial mobility signal but remain below supervised models and often rely on stable, coarse predictor priors rather than consistently structurally aligned relationships.

Problem
Fine-grained mobility data are proprietary, access-restricted, and privacy-sensitive, while LLMs’ ability to infer aggregate neighborhood-level mobility remains unclear.
Method
The study evaluates zero-shot LLM and supervised CBG-level predictions from sociodemographic and built-environment context, alongside directional alignment against empirical relationships.
Results
Supervised baselines average 0.580 accuracy versus 0.435 for the best LLM, while spatial extent is most predictable and has the largest LLM–baseline gaps.
Takeaways & Limitations
LLMs can partially recover aggregate mobility patterns, but their predictions require empirical-alignment and bias auditing before being treated as structurally grounded.
Takeaways & Limitations
Two CBGs corresponding to military locations are excluded under Cuebiq’s data-sharing policies.

Abstract

from arXiv · show

Human mobility is central to urban planning, transportation, public health, and emergency response, yet fine-grained trajectory data are often proprietary, restricted, and privacy-sensitive. Large language models (LLMs) offer a potential alternative by generating plausible mobility traces and predicting individual movement, but their ability to infer aggregate neighborhood-level mobility remains unclear. We evaluate zero-shot LLMs on Census Block Group-level mobility prediction across four U.S. metropolitan areas using anonymized Cuebiq data to construct point-level, trajectory-level, and temporal mobility outcomes, paired with sociodemographic and built-environment predictors. We compare LLM predictions with supervised baselines and introduce a directional alignment analysis to test whether LLM-implied predictor effects agree with empirical OLS and Jonckheere-Terpstra trends. Supervised models achieve 0.580 average accuracy, compared with 0.435 for the best LLM, with spatial extent outcomes showing the strongest predictability but also the largest LLM-baseline gaps. Directional analysis shows that LLMs often rely on coarse, stable predictor-level priors that remain similar across outcomes and cities, including asymmetric treatment of protected-group predictors. Overall, LLMs can partially recover aggregate mobility patterns from urban context, but their predictions should not be treated as structurally grounded without auditing empirical alignment and potential bias.

1 Introduction

The introduction motivates neighborhood-level mobility prediction as a response to restricted trajectory data and evaluates whether LLMs can recover aggregate mobility patterns and their empirical structure.

Fine-grained smartphone mobility data offer detailed behavioral coverage but remain proprietary, access-restricted, and privacy-sensitive.
Aggregate mobility predictions are more directly relevant to transportation agencies, planners, emergency managers, and public-health officials than individual trajectories.
Prediction accuracy alone may reflect heuristics, stereotypes, or memorization rather than the specific contextual relationships that structure mobility.
The paper asks whether LLMs can predict CBG-level mobility from built-environment and sociodemographic context and whether their implied relationships align with empirical patterns.
The evaluation uses anonymized Cuebiq traces aggregated to Census Block Groups and constructs point-level, trajectory-level, and temporal-level mobility indicators.

2 Related Work

Prior work demonstrates broad mobility and geographic capabilities in LLMs, while leaving neighborhood-level aggregate prediction and structural alignment comparatively underexamined.

Existing studies use LLMs for individual trajectory modeling, activity-chain generation, travel simulation, recommendation, and urban prediction.
Aggregate-oriented examples include regional taxi and bike-flow prediction, but most evaluations still target individual trajectories, diaries, or next-location prediction.
Geographic-knowledge studies find that LLMs can predict some socioeconomic indicators but struggle with complex semantic reasoning over long GPS trajectories.
Evaluation research reports geographic and demographic biases, including more favorable subjective judgments of wealthier regions and biased predicted POI visits.
This paper extends evaluation-oriented work by testing empirically meaningful neighborhood-mobility priors from built-environment and sociodemographic context.

3.1 Problem Formulation

The problem formulation treats each CBG–outcome pair as a three-class prediction task and separately evaluates predictive accuracy and directional agreement with empirical relationships.

For CBG i, contextual predictors x_i and mobility outcome m define an outcome-specific task to infer a discretized mobility label.
The predictor F^(m) can be either a zero-shot LLM reasoning over textual neighborhood context or a supervised model trained on labeled CBG examples.
The formulation tests whether sociodemographic and built-environment context alone can recover structured variation in aggregate mobility across neighborhoods.
RQ1 measures whether models correctly classify CBG mobility levels, while RQ2 measures whether model-implied predictor directions agree with directions estimated from observed data.
The evaluation pipeline constructs CBG outcomes from individual traces, applies LLM and supervised predictors, and compares model-implied relationships with empirical patterns.

3.2 Mobility Outcome Construction

The study converts anonymized annual mobility traces into aggregate CBG outcomes spanning spatial footprint, travel behavior, and temporal organization, then defines metrics for each family.

Mobility Outcome Construction: Anonymized 2021 traces yield user-level annual measures of spatial extent, movement structure, and temporal variability before CBG aggregation.
Mobility Outcome Construction: Users are linked to home CBGs, and their mobility measures are summarized into typical resident mobility profiles for each CBG.
Mobility Outcome Construction: Aggregation aligns targets with contextual predictors, preserves privacy through behavioral summaries, and retains CBGs with sufficient user support.
Mobility Outcome Families: The outcome families are point-level, trajectory-level, and temporal-level, covering visited-location organization, travel sequences, and temporal regularity.
Point-level outcomes: Stay-point entropy captures how evenly dwell time is distributed across locations, with higher values indicating more even distribution.
Point-level outcomes: Radius of gyration measures characteristic distance from the activity center, while convex hull diameter measures maximum activity-space extent.
Trajectory- and temporal-level outcomes: Trajectory measures include total travel length, travel entropy, and average duration; temporal measures include daily temporal fragmentation.

3.3 Mobility Outcome Prediction Framework

The framework predicts three-class, outcome-specific mobility labels for Census Block Groups from textual neighborhood context, then compares zero-shot LLM inference with supervised classifiers on the same held-out data.

Each Census Block Group is represented by contextual predictors x_i, and each mobility outcome is discretized into low, neutral, and high classes.
LLM-Based Prediction: Outcome-specific prompts combine the metropolitan area, mobility-outcome definition, CBSA-specific tertile thresholds, and a textual profile of contextual predictors.
LLM-Based Prediction: Percentile-based profiles express each predictor through its raw value, within-CBSA percentile rank, and quintile bucket, emphasizing relative urban context.
LLM-Based Prediction: LLM prompts ask for one of three class labels plus brief reasoning and a ranked assessment of influential predictors.
Supervised Baselines: Supervised baselines include multinomial logistic regression, random forest, decision tree, histogram gradient boosting, and XGBoost trained on the same predictors and labels.
Evaluation: LLMs and supervised models are evaluated on the identical held-out 20% test split for direct comparison.

3.4 Directional Alignment Framework

The directional alignment framework compares LLM-implied predictor effects with empirical OLS directions using ordered predictor buckets and corrected one-sided Jonckheere–Terpstra tests.

Classification accuracy is supplemented by directional alignment analysis because correct predictions may still rely on misleading or weakly grounded reasoning.
Empirical Direction: OLS regressions estimate empirical relationships between contextual predictors and continuous mobility outcomes, with coefficient signs defining ground-truth directions.
LLM-Implied Direction: LLM outputs encode each ranked predictor’s influence as negative, neutral, or positive using p_ijm ∈ {−1, 0, +1}.
Alignment Testing: Across very_low-to-very_high predictor buckets, increasing LLM pushes should follow positive OLS coefficients and decreasing pushes should follow negative coefficients.
Alignment Testing: OLS-oriented transformation makes larger transformed values consistently represent stronger agreement with the empirical direction.
Alignment Testing: For each CBSA–predictor–outcome combination, corrected one-sided JT tests assess increasing alignment or movement in the opposite direction.
Alignment Labels: Corrected JT results classify combinations as aligned_increasing, opposite_trend, or no_clear_trend.

4 Experiments

Across four U.S. metropolitan areas, neighborhood context contains meaningful signal for CBG-level mobility prediction, but zero-shot LLMs recover less structure than supervised baselines. LLM performance and directional behavior vary by mobility dimension, with strong spatial-extent prediction but coarse, stable predictor priors across outcomes and cities.

Predictive performance: 0.580 average accuracy for supervised baselines exceeded 0.435 for the best zero-shot LLM across 32 city–outcome pairs.The average best-baseline–best-LLM gap was 0.144 accuracy points, ranging from 0.059 to 0.231.
Predictive performance: 0.653, 0.671, and 0.670 supervised accuracy for convex hull diameter, ellipse area, and radius of gyration exceeded corresponding best-LLM accuracies of 0.495, 0.510, and 0.484.The average LLM–baseline gap across these spatial-extent outcomes was 0.168, larger than gaps for line-level and temporal outcomes.
Predictive performance: 0.394 and 0.342 were the best-LLM averages for stay-point entropy and travel entropy, making entropy-based outcomes the weakest predictions from static neighborhood context.Travel entropy remained near the three-class chance level of 0.333 in every CBSA, while supervised baselines were also weakest on the two entropy measures.
Model variation: Gemma3-27B was the top LLM in 20 of 32 city–outcome pairs, while Claude-Sonnet-4.5 was strongest for daily temporal fragmentation in all four CBSAs.Gemma3-27B led spatial-extent outcomes across all four CBSAs, so the strongest LLM depended on the mobility dimension.
Directional alignment: Gemma-3-27B often captured broad context–mobility signs but used coarse predictor-level directions that did not fully reflect metropolitan or outcome-specific variation.The directional comparison used empirical OLS relationships and LLM-implied Jonckheere–Terpstra directions.
Directional alignment: Median income, land-use mix, transit access, connectivity, density, and young-adult share generally received positive directions across outcomes, with land-use mix positive in 27 of 28 city–outcome pairs.The model reused broad directions once predictors were salient rather than adapting them separately to each mobility feature.
Directional alignment: Black population share received negative directions in 25 of 28 clear cases, contrasting with generally non-negative directions for Asian and Hispanic/Latino population shares.This pattern reflects asymmetric treatment of protected-group predictors in the model’s implied directional priors.
Directional alignment: Directional priors were largely stable across ATL, LA, MIA, and SF, including positive median-income directions in all clear cases and negative Black-population-share directions in all clear cases.Land-use mix, transit access, connectivity, and density also showed highly consistent directions across metropolitan areas.

5 Conclusion

The paper evaluates whether zero-shot LLMs can infer aggregate CBG-level mobility from urban context and whether their implied relationships align with empirical mobility patterns. LLMs recover nontrivial signal but remain below supervised baselines, while their directional priors are often coarse, stable across outcomes and cities, and asymmetrical for protected-group predictors.

Conclusion: The study tests zero-shot LLM prediction of CBG-level mobility outcomes and structural alignment with empirical context–mobility relationships across four U.S. metropolitan areas.The evaluation uses fine-grained mobility trajectories and examines whether model-implied predictor effects agree with observed relationships.
Conclusion: LLMs recover nontrivial neighborhood-level mobility signal but remain substantially below supervised baselines trained on local data.The paper frames LLMs as potentially useful in data-scarce settings, while reporting a persistent performance gap relative to supervised models.
Conclusion: LLMs often rely on coarse, stable predictor-level priors that persist across mobility outcomes and metropolitan areas.The alignment analysis identifies directional patterns that remain similar across contexts rather than adapting consistently to specific empirical relationships.
Conclusion: Asymmetric directional patterns for protected-group predictors further motivate auditing LLM predictions for empirical alignment and potential bias.The authors caution that predictive performance alone does not establish structural grounding for urban mobility analysis or policy-support settings.

A Appendix

The appendix provides the prompts used for RQ1 and RQ2, reports parse-success and sample-size information, and documents mobility abbreviations and baseline-comparison tables.

Prompts: The RQ1 system prompt accompanies a prediction setup in which contextual CBG information is used to classify a mobility outcome.The prompt materials are identified as Figure 6, while the prediction formulation specifies the CBG context and mobility classification task.
Prompts: The directional-alignment materials include a user prompt and system prompt identified as Figures 7 and 8.These figures correspond to the RQ2 directional-alignment procedure described in the paper.
Evaluation details: Table 3 reports parse-success rates and excludes models below 95% from downstream analysis.The threshold is an explicit inclusion rule for subsequent evaluation.
Data description: Figure 9 shows sample-size distributions for CBGs across the four CBSAs.The appendix identifies the figure as a distributional description of the study’s CBG samples.
Notation: The appendix defines abbreviations for mobility outcomes and MNLR, including HD, EA, RoG, SE, AD, TL, TE, and TF.The listed abbreviations cover spatial, entropy, duration, travel-length, and temporal-fragmentation measures; MNLR denotes multinomial logistic regression.
Performance tables: Table 4 compares the best zero-shot LLM and supervised baseline by CBSA and mobility outcome, with the gap defined as baseline accuracy minus LLM accuracy.This table organizes performance comparisons by metropolitan area and outcome.