Source-linked AI summary
CoST: Semantic-Aware Urban Understanding via Spatial-Temporal Alignment
Yutian Jiang, Jiabo Liu, Xixuan Hao, Yuxuan Liang
TL;DR
Existing satellite-based representation learning struggles with cross-region generalization and semantic interpretability because it relies on region-specific auxiliary data or lacks explicit semantic alignment. CoST jointly models spatial neighborhoods and multi-temporal semantic transitions, achieving superior performance across diverse downstream tasks and unseen scenarios. Its results support learning generalizable, interpretable geospatial representations from satellite imagery.
Problem
Current methods struggle with cross-region generalization and semantic interpretability because auxiliary data are region-specific and visual pre-training lacks explicit high-level geo-semantic grounding.
Method
CoST aligns spatial context with multi-temporal semantics by modeling spatial neighborhoods, encoding temporal semantic transitions, and coupling both signals in a shared latent space.
Results
CoST consistently outperforms strong baselines across urban indicator prediction, land-use classification, and change detection, including an F1 score of 0.6425 on LEVIR-CD.
Takeaways & Limitations
CoST learns generalizable and interpretable geospatial representations that bridge visual features with high-level geo-semantics across diverse downstream tasks.
Abstract
from arXiv · showhide
Geospatial representation learning from satellite imagery is a fundamental problem for large-scale urban analysis and real-world applications. Despite recent advances, current methods struggle with cross-region generalization and semantic interpretability due to their reliance on region-specific auxiliary data and the neglect of semantic alignment within multi-temporal urban imagery. Therefore, we present CoST, a novel \underline{Co}ntrastive-based \underline{S}patial-\underline{T}emporal framework that aligns spatial context with multi-temporal semantics to extract universal geographic regularities shared across regions. Specifically, CoST explicitly models spatial correlations to capture transferable geographic structures and exploits multi-year urban change semantics to align learned representations with high-level geo-semantics. Extensive experiments demonstrate that CoST consistently achieves superior performance across various downstream tasks and in unseen scenario, yielding an average relative gain of 8.7\% over the strongest competing methods across eight city-indicator settings. The code is available in \href{https://github.com/Arandinglv/CoST}{this repo}.
1 Introduction
Current geospatial representation methods struggle to generalize across regions while remaining semantically interpretable. CoST addresses these challenges by jointly modeling spatial structure and multi-temporal semantic transitions.
- Satellite imagery provides geographically consistent observations for large-scale geospatial representation learning.Such representations support urban perception and socio-economic assessment.
- Multimodal alignment methods depend on auxiliary data whose availability, quality, and distribution vary across cities and countries.This dependence limits scalability and weakens transferability.
- Self-supervised visual pre-training captures invariant visual patterns but provides limited explicit grounding in high-level geo-semantics.Its features can remain less informative for semantically demanding urban analysis.
- Universal geospatial representations must address cross-region distribution shifts and semantic interpretability under weak supervision.They should generalize across differing visual styles, development patterns, and functional compositions while reflecting urban transitions.
- CoST couples spatial context with multi-temporal semantics to improve cross-region generalization and semantic interpretability.It models spatial neighborhoods, encodes temporal semantic transitions, and aligns spatial proximity with temporal consistency in a shared latent space.
2 Related Works
Related work follows multimodal semantic alignment or self-supervised visual pre-training, but both paradigms leave transferability or semantic grounding limitations.
- Multimodal alignment injects external semantic knowledge from POIs, human mobility, and textual descriptions into satellite imagery.These modalities provide complementary information about land use, human activity, and urban dynamics.
- Region-specific external knowledge can limit scalability and transferability when signals are sparse, noisy, or unavailable in new regions.
- Visual pre-training learns generic image representations using self-supervised objectives such as contrastive learning and masked image modeling.
3 Methodology
CoST learns transferable and interpretable geospatial representations by jointly modeling spatial neighborhood structure and multi-year semantic transitions. Its spatial-temporal objectives preserve geographic context while aligning representation distances with the direction and magnitude of urban change.
- Preliminaries: CoST represents each region through multi-temporal satellite observations and its geographic neighborhood of nearest neighboring regions.This spatial-temporal formulation supplies both within-region temporal evolution and cross-region spatial context.
- Spatial Neighborhood Modeling: Spatial neighborhood modeling replaces independent-instance contrastive learning with hierarchical supervision from an anchor, its augmented view, and nearby regions.The target distribution assigns stronger supervision to the anchor and auxiliary supervision to spatial neighbors, reflecting shared local context.
- Spatial Neighborhood Modeling: Geographically proximate regions are encouraged to occupy nearby embedding locations, preserving shared spatial semantics and improving transferability to unseen regions.The spatial loss uses normalized embeddings and a temperature parameter to align predicted similarities with proximity-aware targets.
- Cumulative Semantic Signal Construction: CoST extracts yearly semantic change vectors from land-use maps, accumulates intermediate transitions with linear complexity O(T), and projects them into a semantic latent space.The pipeline captures cumulative transitions such as barren → construction → building instead of relying only on endpoint comparisons.
- Temporal Semantic Guidance: Temporal Semantic Alignment maps cumulative semantic-change magnitude to soft similarity labels, pulling stable temporal views together and separating substantially changed views.The resulting latent distances are intended to reflect the magnitude of real-world semantic changes.
- Temporal Semantic Guidance: Temporal Instance Discrimination adds InfoNCE training with hard temporal negatives so representations remain stable, discriminative, and sensitive to significant urban changes.The combined temporal objectives address instability from relying solely on a BCE-style objective and noisy foundation-model embeddings.
4 Experiments
CoST is evaluated across urban indicator prediction, classification, change detection, interpretability, ablation, dynamic profiling, and domain adaptation. The experiments assess performance, cross-region transfer, semantic structure, and the contributions of spatial and temporal components.
- Experimental Setup: The evaluation covers urban indicator prediction, land-use classification, change detection, semantic interpretability, component ablations, dynamic profiling, and domain adaptation.The experiments use multiple cities and datasets, including a held-out Chicago dataset for cross-city generalization.
- Overall Performance: CoST-R achieves R^2 scores of 0.664 for GDP and 0.698 for population in New York City, outperforming prior methods by over 5%.On held-out Chicago, CoST achieves GDP and population R^2 scores of 0.646 and 0.691, respectively.
- Overall Performance: CoST-R and CoST-V achieve 97.13% and 98.73% top-1 accuracy on UCM, while CoST-R reaches an F1 score of 0.6425 on LEVIR-CD.CoST remains competitive on BigEarthNet and surpasses specialized bi-temporal methods for change detection.
- Representation Interpretability: Embedding arithmetic produces predictable semantic transformations, indicating that CoST learns semantically disentangled and interpretable representations.Subtracting built-up areas and adding farmland changes the target semantics while preserving surrounding context.
- Ablation Study: Removing spatial neighborhood modeling decreases R^2 by 15–30% across cities, while removing temporal or spatial-temporal alignment also reduces performance.The ablation results identify spatial aggregation, temporal semantics, and spatial-temporal alignment as important components for socio-economic prediction.
- Qualitative Analysis: Temporal representations outperform static snapshots across Beijing, Guangzhou, and New York, with CoST-R reaching R^2 scores of 0.62, 0.57, and 0.72, respectively.Cross-city retrieval also aligns unseen Chicago regions with semantically consistent New York regions and matching GDP profiles.
5 Conclusion and Future Work
CoST learns generalizable and interpretable geospatial representations from satellite imagery by bridging visual features and high-level geo-semantics. The authors report state-of-the-art performance across diverse downstream tasks and identify predictive urban change modeling and image generation as future directions.
- Conclusion: CoST is designed to learn generalizable and interpretable geospatial representations that bridge visual features and high-level geo-semantics.The framework is presented as a unified approach for satellite imagery.
- Conclusion: CoST achieves state-of-the-art performance across diverse downstream tasks.The conclusion points to continuous multi-year imagery as a basis for future work.
- Future Work: Future research could explore predictive representations for urban change prediction and image generation to support urban planning and scenario simulation.These directions build on the continuous multi-year imagery used in the work.
A Experiment Results
CoST remains effective across Shanghai and Shenzhen, with its variants achieving strong GDP and population prediction results. Comparisons with remote-sensing foundation models further show lower errors across multiple cities, including unseen Chicago.
- CoST remains effective across both Shanghai and Shenzhen and both evaluated urban indicators.
- CoST-V achieves a GDP R2 of 0.442 in Shanghai, improving over SatMAE (0.404).
- CoST-R leads Shanghai population prediction with an R2 of 0.434.
- In Shenzhen, CoST-V achieves the highest GDP R2 of 0.273, while CoST-R obtains the best population R2 of 0.463 and MSE of 0.496.
- CoST-R achieves lower GDP and population MSE than Prithvi-100M and DOFA in Beijing, Guangzhou, and unseen Chicago.
A.1 Pre-training Details
The pre-training setup uses frozen BERT-derived urban land-use semantics and specified optimization settings. Evaluation includes comparisons with remote-sensing foundation models using GDP and population MSE.
- A frozen BERT encoder generates embeddings for 11 urban land-use categories.
- Pre-training uses Adam with β=(0.9, 0.999), weight decay 1 × 10^-4, and a target learning rate of 3 × 10^-4.
- The Shenzhen evaluation is reported in Table 5 as a performance comparison between CoST variants and other methods.
- Figure 9 compares CoST-R, Prithvi-100M, and DOFA using GDP MSE and population MSE.
- Training uses 32 groups per GPU, 64 groups total, with four elements per group, num_neg=6, τ=0.07, and ϵ=0.1.
A.2 Fine-tuning details
Fine-tuning freezes the pre-trained encoder and trains lightweight task-specific heads for classification, urban indicator prediction, and change detection. The implementation also includes temporal urban-indicator modeling with a Visual-LSTM.
- Fine-tuning: Linear probing freezes the pre-trained encoder while training task-specific heads for downstream evaluation.
- Classification: Classification uses a linear projection head with Cross-Entropy Loss for single-label and Multi-Label Soft Margin Loss for multi-label tasks.
- Static Urban Indicator Prediction: Static urban indicator prediction uses a 3-layer MLP regressor to predict GDP and population density from frozen features with MSE loss.
- Dynamic Urban Indicator Prediction: Dynamic urban indicator prediction projects 2010–2020 features into 256 dimensions and processes them with a single-layer LSTM of hidden dimension 256.
- Change Detection: Change detection uses a lightweight U-Net head for pixel-level segmentation from multi-stage bi-temporal features.
B Temporal Semantic Guidance
Temporal Semantic Guidance converts multi-year satellite imagery into interpretable semantic change signals. It accumulates yearly transitions, embeds land-use categories, and maps aggregate change magnitude to a soft similarity target.
- Step 1: Change Extraction: Temporal Semantic Guidance uses frozen Grounded-SAM to generate semantic maps assigning pixels to urban land-use categories.
- Step 2: Semantic Accumulation: Yearly transition vectors are accumulated across intervals, reducing complexity from O(T^2) to O(T) while capturing intermediate development stages.
- Step 3: Semantic Projection and Signal Generation: A frozen BERT encoder embeds 11 urban categories, whose weighted aggregation forms the final change embedding.
- Step 3: Semantic Projection and Signal Generation: The tanh function maps the L2 norm of the semantic change embedding to a soft similarity target.
C Baselines
CoST is evaluated against both general-purpose self-supervised learning frameworks and specialized remote sensing foundation models. The baseline suite includes SimCLR, MoCoV3, and DINOv2.
- The evaluation compares CoST with general-purpose self-supervised learning frameworks and specialized remote sensing foundation models.
- SimCLR learns representations by maximizing agreement between augmented views of the same sample.
- MoCoV3 uses momentum-based contrastive learning optimized for Vision Transformers to improve training stability and feature quality.
- DINOv2 is a vision foundation model trained with self-distillation to generate high-quality, discriminative features.
GenAI Usage Disclosure
The authors disclose that AI-assisted technologies were used strictly for language polishing and grammar correction. They retain responsibility for the work’s integrity, results, and references.
- AI-assisted technologies were used strictly for language polishing and grammar correction.
- The authors take full responsibility for the integrity of the work, including experimental results and references.