Source-linked AI summary
Applying foundation model embeddings towards urban livability evaluation
Ayush Khot, Wen Zhou, Shaowen Wang
TL;DR
The paper addresses limited socioeconomic observations by studying whether geospatial foundation embeddings can support urban livability evaluation from widely available geospatial data. It integrates AlphaEarth, AnySat, and TerraMind into a multimodal Transformer and finds that embeddings improve prediction and reduce degradation when modalities are missing, with stronger practical gains from AlphaEarth and multi-embedding combinations in the reported settings.
Problem
Urban livability is difficult to measure in data-scarce regions, while existing approaches may rely on manually selected indicators and costly data integration.
Method
The paper adds AlphaEarth, AnySat, and TerraMind embeddings to a multimodal Transformer that combines imagery, POI text, and embedding features for multitask livability prediction.
Results
Embeddings improve livability prediction and mitigate performance degradation when individual modalities are unavailable; with null POI, AME RMSE improves from 0.048 to 0.040.
Takeaways & Limitations
AlphaEarth is the most reliable single-embedding choice, while rural or mixed regions require multi-embedding combinations to reliably recover and improve upon baseline performance.
Takeaways & Limitations
All evaluations use the Netherlands LBM dataset, so transfer to other regions and out-of-distribution areas remains unvalidated.
Abstract
from arXiv · showhide
While accurate measurement of socioeconomic indicators remains challenging in data-scarce regions, which limits policy interventions and resource allocation, high-resolution geospatial data is widely available and can contain information on various livability statistics. We investigate which physical features are encoded within foundation model embeddings, such as AlphaEarth, AnySat, and TerraMind, and provide a systematic framework for identifying the most predictive geospatial indicators. By analyzing how different types of geospatial data influence urban livability predictions, our approach enables researchers to prioritize the most informative features for their specific applications. Additionally, we demonstrate how to leverage foundation model embeddings to enhance prediction performance for these outcomes. This work contributes a principled methodology for extracting actionable information from satellite imagery while accounting for complex spatial dependencies, with applications in predicting urban livability in regions with limited observation data.
1 Introduction
Urbanization is increasing the need to measure and improve city quality of life within a shared sustainability framework. Existing livability methods rely on manually selected indicators, while geospatial data offers a broader basis for deep-learning evaluation.
- 1 Introduction: Urban population growth makes understanding and improving urban quality of life increasingly important.The paper situates livability assessment within Sustainable Development Goal 11’s emphasis on inclusive, safe, resilient, and sustainable cities.
- 1 Introduction: Existing statistical livability methods are not generalizable because their indicators are selected manually.Prior deep-learning work uses multiple geospatial data sources, but these sources require substantial integration effort and may omit valuable datasets.
2 Related Work
Prior livability research uses manually selected indicators and heterogeneous administrative or map data, whereas this paper examines geospatial foundation embeddings as another source of livability information. It evaluates AlphaEarth, AnySat, and TerraMind, which differ in architecture, modality coverage, embedding size, and spatial resolution.
- 2 Related Work: Earlier approaches assess livability with selected indicators, including multidimensional categories and accessibility- and economy-based measures.Examples use analytic hierarchy processes, five livability categories, or government and OpenStreetMap data.
- 2 Related Work: The paper investigates AlphaEarth, AnySat, and TerraMind as three geospatial foundation embeddings.These models provide different representations, including 64-, 1536-, and 384-dimensional embeddings at 10m or 160m spatial resolution.
3 Data
The study combines Dutch 100m-grid livability labels with raster, POI, and foundation-embedding inputs extracted at target-grid and contextual spatial scales. It uses held-out cities to evaluate generalization beyond locations seen during training.
- 3 Data: The LBM dataset measures overall livability and five domains on 100m × 100m grids across Dutch cities.The domains are physical environment, housing availability, amenities, social cohesion, and nuisance and insecurity, aggregating 47 environmental characteristics.
- 3 Data: DSM and livability labels use the 100m × 100m target grid, while RS, NLRS, and POI use a broader 500m × 500m context window.The dataset adds a 200m buffer outside each evaluation cell; TerraMind’s 500m window is resized to 224×224 inputs.
- 3 Data: The model inputs are RS, DSM, NLRS, POI, and AlphaEarth, AnySat, and TerraMind embeddings from 2020 geospatial data.These modalities represent features such as car density, population density, building height, burglary, and facility use.
- 3 Data: Testing holds out Eindhoven, Hengelo, Dordrecht, and Beesel, so reported performance measures generalization to entirely unseen cities.The remaining nine cities are divided between training and validation.
4 Model
The model extends a multimodal Transformer regression architecture by adding learned geospatial embeddings alongside imagery and POI text. Modality-specific feature extractors feed a shared encoder, which predicts overall livability and five domain scores using a multitask objective.
- 4 Model: The model extends TMTMR to integrate learned geospatial embeddings with raster imagery and POI text.This addresses limited prior attention to combining embeddings with geospatial imagery or text.
- 4 Model: Pretrained DenseNet extracts RS, DSM, and NLRS features, while pretrained BERT extracts POI features.Images are normalized and resized before DenseNet processing, and POI text is tokenized for BERT.
- 4 Model: Dedicated convolutional and pooling layers extract features from high-dimensional embedding pixels before fusion.Additional convolutional extractors can be appended when more embeddings are added as inputs.
- 4 Model: All modality features are concatenated in a Transformer Encoder, then decoded into overall livability and five domain scores.The model uses a multitask MAE objective so shared features can capture relationships across tasks and modalities.
5 Experiments
Experiments show that foundation-model embeddings generally improve multimodal urban-livability prediction, with AlphaEarth providing the most consistent gains and robustness to missing inputs. Benefits vary by livability domain, city type, and available modalities, while embedding combinations may add redundant capacity or signals.
- 5.1 Results on adding embeddings: AlphaEarth consistently improves livability prediction, while TerraMind offers smaller gains and AnySat performs similarly to the baseline.AlphaEarth lowers RMSE for LIV and AME, with additional AME gains when combined with other embeddings.
- 5.1 Results on adding embeddings: Embedding-only probes underperform multimodal models on LIV and NUI, although probe-aef beats the baseline on PHY and AME.This pattern indicates that multimodal inputs contribute information beyond embeddings alone, while PHY- and AME-relevant signals are concentrated in AlphaEarth.
- 5 Experiments: Embedding combinations should not be treated as clean evidence of complementary information because shared pretraining sources, added capacity, or noisy duplicate signals may explain improvements.The experiments compare embedding variants within the authors’ implementation rather than the published baseline, which trails by roughly 5 to 11% RMSE.
- 5.1 Results on adding embeddings: Embedding benefits vary geographically: AlphaEarth performs best in Eindhoven, but single embeddings worsen LIV performance in rural Beesel.The aef+as+tm combination recovers and improves performance in Beesel, whereas no combination dominates every score in Hengelo or Dordrecht.
- 5.1 Results on adding embeddings: With POI absent, aef+tm performs best and AME RMSE improves from 0.048 to 0.040, compared with 0.056 to 0.045 where POI is present.All models perform worse in areas with POI, while embeddings improve performance across both POI conditions.
- 5.2 Results on replacing modalities: Adding embeddings improves every single-modality ablation, and AlphaEarth often recovers performance close to full-data models when DSM, NLRS, or POI is missing.Removing RS remains harder: aef improves over the ablated baseline but remains worse than full-data aef.
- 5.2 Results on replacing modalities: Without RS, DSM, NLRS, or POI, aef+tm matches the full-data baseline on all metrics and exceeds it on PHY and AME.Adding AnySat to aef+tm hurts RMSE, suggesting potentially redundant or noisy information in that setting.
- 5.3 Results on missing modality: AlphaEarth-containing models are more robust to missing modalities, while POI can introduce heterogeneity that complicates overall livability prediction.RS carries the most unique information, and zeroing POI lowers base LIV RMSE from 0.106 to 0.099.
6 Discussion
The discussion uses attention, entropy, spatial maps, and Full Grad-CAM to interpret how AlphaEarth changes multimodal livability prediction. AlphaEarth can redistribute attention, compensate for missing remote-sensing information, and alter spatial feature contributions, but attention weights require cautious interpretation.
- 6.1 Attention: Zeroing RS increases LIV RMSE less for aef than the baseline, falling from 0.156 to 0.112, which supports partial compensation by AlphaEarth.The authors present this as indirect support because attention weights alone cannot establish modality importance.
- 6.1 Attention: POI receives the highest attention in both models despite contributing less predictive information, likely because heterogeneous listings require substantial internal variation to be resolved.POI attention may also reflect its direct relationship to the AME output rather than broad importance across all livability dimensions.
- 6.1 Attention: Across Eindhoven and Hengelo, difference maps show small consistent regions where aef improves predictions, while Dordrecht is mixed near its historical center.Figure 3 compares ground truth, base predictions, aef predictions, and their L1-score differences across four cities.
- 6.1 Attention: High-livability areas distribute attention more broadly across modalities, with entropy gaps of +0.012 for base and +0.022 for aef.Both differences are statistically consistent, with 95% confidence intervals [0.008, 0.016] and [0.019, 0.026], respectively.
- 6.2 Full Grad-CAM: Full Grad-CAM removes ReLU to visualize both positive and negative convolutional feature contributions when comparing base and aef spatial attention.The analysis examines changes across RS, DSM, and NLRS branches; in one Dordrecht example, AlphaEarth strengthens relative attention in a park.
7 Conclusion
The study integrates foundation-model embeddings into urban livability evaluation, improving performance and mitigating missing-modality degradation, while revealing limits in rural transfer and RS substitution.
- Foundation-model embeddings improve the original livability model and substantially mitigate degradation when individual modalities are unavailable.AlphaEarth provides especially reliable gains, while combinations can approach full-data performance under broader modality loss.
- AlphaEarth does not fully replace high-resolution remote sensing because fine-grained spatial information remains incompletely captured by embeddings.
- Single embeddings can worsen rural baseline performance, whereas multi-embedding combinations reliably recover and improve upon the baseline in rural or mixed regions.The authors associate this deployment risk with a mismatch between predominantly urban pretraining data and rural geospatial patterns.
- Attention analysis indicates that high-livability areas distribute attention broadly across modalities, while low-livability areas are dominated by POI.
- The framework’s generalizability gains are lower than expected, and the authors identify rural validation, spatial-bias mitigation, and cross-region evaluation as future work.The study also proposes temporal embeddings for longitudinal analysis.
Ethics and Privacy Statement
The study uses publicly available, aggregated geospatial and livability data rather than personal information, but warns that downstream deployment could entrench inequality.
- The study uses non-personal data aggregated to 100 m × 100 m grid cells and collects no human-subjects data.
- Potential misuse in real estate, insurance, or resource allocation could reinforce inequality, especially given rural–urban performance disparities and embedding-driven biases.The authors position the system for human-supervised policy and planning analysis rather than automated eligibility or pricing decisions.
A Training Details
Training uses dedicated convolutional branches to convert each foundation-model embedding into tokens compatible with the existing multimodal Transformer architecture.
- Models were trained for up to 12 epochs with AdamW, learning rate 5e-5, weight decay 0.1, batch size 16, and early stopping patience 5.
- MAE was selected instead of MSE because it performed slightly better for most outputs in the baseline comparison.
- Each AlphaEarth, AnySat, and TerraMind embedding uses a separate three-layer convolutional branch before pooling and projection to a shared token shape.
- The resulting embedding tokens are concatenated with DenseNet and BERT branches before entering a Transformer Encoder.The encoder uses 12 self-attention heads, 768 hidden units, and GELU activation.
B Comparison to Published TMTMR Baseline
The reimplemented base model underperforms the published TMTMR baseline, with the authors attributing the gap to differences in training configuration and loss choice.
- The reimplemented base model trails the published base-mse baseline by roughly 5 to 11% RMSE across all six livability scores.The comparison is reported in Table 7.
- The authors attribute this gap to unreproduced training differences, including 20 versus 12 epochs and MSE versus MAE loss.