Source-linked AI summary
SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery
Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, Marc Rußwurm
TL;DR
Location modeling needs representations that capture relevant ground conditions without expensive imagery processing and that generalize beyond observed regions. SatCLIP learns such representations by contrastively matching satellite-image features with geographic coordinates, and it outperforms alternatives across diverse tasks while improving geographic generalization. Its main scope boundaries are reliance on Sentinel-2 imagery and limited spatial scales set by the encoder configuration.
Problem
Coordinate-based location modeling can suffer geographic distribution shift, while extracting ground conditions from imagery can require expensive processing; existing encoders also lack broad task and global coverage.
Method
SatCLIP contrastively trains coordinate and satellite-image encoders to place corresponding locations and visual representations near each other in a shared embedding space.
Results
SatCLIP outperformed other approaches on globally distributed downstream tasks and was the best model tested under geographic domain generalization, while regional datasets favored or matched GeoCLIP.
Takeaways & Limitations
The results support using compact SatCLIP location embeddings for global prediction tasks spanning social and environmental domains, including out-of-sample geographic prediction.
Takeaways & Limitations
The current implementation relies solely on Sentinel-2 imagery, and its pretrained weights cover limited spatial scales controlled by the location encoder's L parameter.
Abstract
from arXiv · showhide
Geographic information is essential for modeling tasks in fields ranging from ecology to epidemiology. However, extracting relevant location characteristics for a given task can be challenging, often requiring expensive data fusion or distillation from massive global imagery datasets. To address this challenge, we introduce Satellite Contrastive Location-Image Pretraining (SatCLIP). This global, general-purpose geographic location encoder learns an implicit representation of locations by matching CNN and ViT inferred visual patterns of openly available satellite imagery with their geographic coordinates. The resulting SatCLIP location encoder efficiently summarizes the characteristics of any given location for convenient use in downstream tasks. In our experiments, we use SatCLIP embeddings to improve prediction performance on nine diverse location-dependent tasks including temperature prediction, animal recognition, and population density estimation. Across tasks, SatCLIP consistently outperforms alternative location encoders and improves geographic generalization by encoding visual similarities of spatially distant environments. These results demonstrate the potential of vision-location models to learn meaningful representations of our planet from the vast, varied, and largely untapped modalities of geospatial data.
1 Introduction
SatCLIP addresses limitations of coordinate-based location modeling by learning globally covered location embeddings from satellite imagery. It is designed for diverse downstream tasks and geographic generalization across unseen areas.
- Motivation: Geographic coordinates can amplify distribution shift, making many location-informed models poorly suited to cross-regional generalization.Such models are often limited to interpolation when evaluation regions are absent from training data.
- Research gap: Existing work leaves gaps in evaluating location encoders across tasks and achieving global coverage with approximately equal performance.SatCLIP is introduced to address both challenges.
- Approach: SatCLIP distills spatially varying visual patterns from globally distributed satellite data into an implicit neural location representation.The encoder maps latitude and longitude to vectors matched with visual representations from CNN or ViT encoders.
- Evaluation: The study compares SatCLIP with existing location encoders and other geographic feature approaches on nine diverse downstream tasks.The tasks range from temperature prediction to population density estimation, with prediction and geographic generalization as evaluation concerns.
2 Satellite Contrastive Location-Image Pretraining
SatCLIP jointly trains coordinate and image encoders so location embeddings capture both spatial effects and satellite-observed ground conditions. The resulting embeddings are compact and designed for efficient downstream use.
- Motivation: SatCLIP aims to combine spatial effects and ground conditions in location embeddings while avoiding the cost of processing full downstream satellite images.Raw coordinates omit ground conditions, whereas full images require image downloads and large vision models.
- Encoders: Each coordinate c_i and multispectral image I_i is mapped by separate encoders into a shared d-dimensional latent space.The location encoder takes latitude/longitude pairs, while the image encoder processes multispectral imagery.
- Pretraining objective: The CLIP objective matches each coordinate with its corresponding image and contrasts that pair against other images in the batch.A complementary image-to-coordinate loss trains both encoders simultaneously.
- Encoder architectures: The location encoder uses spherical-harmonic positional encodings with a Siren network, while L controls representation smoothness and spatial resolution.Larger values correspond to finer spatial resolution.
- Implementation: SatCLIP uses pretrained ResNet18, ResNet50, or ViT16 Sentinel-2 vision encoders and trains on the S2-100K dataset.S2-100K is split into 90% pretraining data and 10% validation data; batch size 8k was found useful for finer-grained representations.
3 Experimental Setup
The experiments test SatCLIP across diverse tasks and geographically separated evaluation settings against pretrained location, image-only, and coordinate-only alternatives. The setup emphasizes global coverage and generalization beyond training regions.
- Research questions: The study evaluates SatCLIP across diverse downstream tasks and unseen geographic areas, including spatial interpolation and geographic domain generalization.The experiments address performance across tasks and geographic regions, while also examining why the method works and its limitations.
- Pretraining data: S2-100K contains 100,000 globally distributed 256×256-pixel, 12-channel Sentinel-2 image tiles with associated centroid locations.The dataset is designed for multi-task applicability and geographic generalization and is nearly uniformly distributed across global land mass.
- Downstream tasks: The nine downstream datasets span socioeconomic and environmental applications, using latitude/longitude coordinates transformed into location embeddings.Targets include variables such as air temperature and elevation.
- Comparison methods: SatCLIP is compared with GPS2Vec, CSP, GeoCLIP, globally precomputed MOSAIKS features, and raw-coordinate Identity models.The comparisons separate pretrained location embeddings from image-only and location-only baselines.
- Evaluation protocol: Downstream MLPs use location embeddings or raw coordinates to predict continuous or discrete outcomes, with MSE for regression and cross-entropy for classification.Hyperparameters are tuned on an independent validation set, and results are reported on an unseen test set.
4 Results
SatCLIP performs strongly across diverse downstream tasks and geographic regions, with especially consistent performance across continents and in geographic adaptation. Its embeddings capture fine-grained and spatially smooth environmental patterns, while lower-resolution models favor adaptation and higher-resolution models favor interpolation.
- Downstream Task Performance: SatCLIP embeddings achieve the best prediction scores by a large margin on seven of nine downstream tasks.The exceptions are California Housing and Median Income, where GeoCLIP is competitive.
- Downstream Task Performance: SatCLIP performs well across all continents for biome classification and population density estimation.CSP and GPS2Vec perform better in Europe and North America than in underrepresented continents, while GeoCLIP is closer on population density but worse on biome classification.
- Geographic Adaptation: Under held-out-continent evaluation, SatCLIP models are often better than comparison approaches, with L = 10 winning 8 tasks versus 5 for L = 40.The experiments hold out Africa or Asia and use no or few samples from the held-out continent.
- Geographic Adaptation: For African Ecoregion prediction, SatCLIP with L = 10 is closest to ground truth, while MOSAIKS is too fine-grained and CSP-iNat too coarse.The figure includes sparse African training locations whose nearest neighbors are 480km apart on average.
- Location Embedding Analysis: SatCLIP embeddings provide fine-grained representations that capture global patterns such as climate zones and spatial similarities between environmentally similar locations.PCA and cosine-similarity analyses examine the 256-dimensional embedding space and compare spatial smoothness across models.
- Ablation Analysis: Vision encoder choice changes performance by less than 1%, whereas location-encoder scale affects results more: L = 40 favors interpolation and L = 10 favors zero- and few-shot adaptation.The comparison includes ViT-16, ResNet-18, and ResNet-50 image encoders.
5 Discussion
SatCLIP provides useful information across varied tasks and performs well across continents, while transferring spatial patterns from satellite imagery to support geographic generalization. The location encoder can be applied directly to any point on the globe without downloading additional imagery.
- Task Generalization: SatCLIP is useful across a wide range of downstream tasks, although its concrete benefit varies by task.It outperforms other approaches on globally distributed datasets, while GeoCLIP remains competitive on regional datasets such as California Housing.
- Geographic Robustness: SatCLIP performs well across continents and is less prone to geographic bias than GPS2Vec or CSP.The comparison methods' performance degrades outside Europe or North America.
- Geographic Generalization: Transferring spatial patterns from Sentinel-2 imagery into the location encoder enables generalization across geographic areas.Similarity between distant locations in embedding space reflects visual similarity in their satellite images and helps explain generalization with little or no training data.
- Practical Use: SatCLIP location encoders can be applied directly to any point on the globe at test time without downloading additional imagery.The encoder summarizes location characteristics in its learned representation.
6 Conclusion
SatCLIP learns global visual location representations by contrastively matching satellite images with coordinates and supports prediction across social and environmental tasks. Its current scope is bounded by Sentinel-2-only context, limited spatial scales, and no direct time embedding.
- 6 Conclusion: SatCLIP matches satellite images with coordinates to learn an implicit neural representation of visual patterns on the globe.The method uses a contrastive location-image pretraining objective.
- 6 Conclusion: SatCLIP supports global prediction tasks across social and environmental domains, including interpolation and out-of-sample geographic prediction.The conclusion compares SatCLIP with existing location encoders and with image-only and location-only prediction.
- 6 Conclusion: 100,000 openly available satellite images and a single contrastive loss provide a comparatively simple implementation.The pretraining samples are uniformly distributed across the globe, supporting globally distributed downstream tasks.
- 6 Conclusion: The current implementation uses Sentinel-2 imagery as its sole source of contextual grounding.Future extensions could integrate other location-specific modalities such as acoustic sensor audio or geolocated social-media text.
- 6 Conclusion: SatCLIP’s pretrained weights have limited spatial scales controlled by the location encoder’s L parameter, and time is not directly embedded.The training data spans two years and includes seasonal image differences, but the model marginalizes over time.
A S2-100K Dataset Overview
The S2-100K dataset is sampled from qualifying Sentinel-2 Level-2A scenes worldwide. Random 256×256-pixel patches are retained when they contain sufficiently little nodata, producing multispectral patches at native UTM resolution.
- A S2-100K Dataset Overview: S2-100K samples Sentinel-2 Level-2A scenes captured from January 1, 2021 through May 17, 2023.Eligible scenes have less than 20% reported cloud cover and overlap land masses.
- A S2-100K Dataset Overview: 2,359,972 Sentinel-2 Level-2A scenes meet the dataset’s date, cloud-cover, and land-mass criteria.The scene count is identified using the Microsoft Planetary Computer.
- A S2-100K Dataset Overview: 100,000 random 256×256-pixel patches are retained after rejecting patches with 10% or more nodata pixels.A random scene and patch are repeatedly sampled until the target count is reached.
- A S2-100K Dataset Overview: Each retained patch contains all available listed Sentinel-2 bands resampled to 10m/px in native UTM coordinates.The passage lists bands including B01, B02, B03, B04, B06, B07, B08, B08A, B09, B11, and B12.
B.1 (Pre-)training Data
The reviewed location encoders differ in their geographic coverage, contextual data, coordinate representations, and contrastive objectives. Their pretraining datasets are often geographically clustered or specialized toward particular visual or social dimensions.
- B.1 (Pre-)training Data: iNat 2018 contains over 400,000 image-location pairs covering more than 8,000 plant and animal species, but its locations are concentrated mainly in North America and Europe.The dataset is less descriptive of socioeconomic processes than other contextual sources described in the passage.
- B.1 (Pre-)training Data: FMoW contains over 363,000 satellite image-location pairs annotated across 63 functional or land-use categories, with fewer natural scenes and Western-country skew.Its design emphasizes detection of functional objects and human-built infrastructure.
- B.1 (Pre-)training Data: YFCC100M contains approximately 48 million Flickr image-tag-location triplets spanning diverse scenes, but is less representative of natural processes and mostly covers Western countries.Its imagery is more indicative of social dimensions and physical infrastructure.
- B.1 (Pre-)training Data: MP-16 contains 4.27 million Flickr images with geotags and overrepresents Western countries because it clusters where Flickr activity is higher.The passage describes geographic clustering as a property of the dataset.
- B.1 (Pre-)training Data: CSP combines a location encoder with image and semantic-tag context using CLIP, random-location-negative, and SimCSE contrastive objectives.The additional objectives help balance the location and context encoders.
- B.1 (Pre-)training Data: GPS2Vec trains location encoders specific to UTM zones containing training data and does not learn a context encoder.It extracts text and image features directly and uses contextual features as labels through KL-divergence minimization.
- B.1 (Pre-)training Data: GeoCLIP transforms coordinates with equal earth projection and random Fourier features, then trains separate MLPs with a CLIP objective and additional location negatives.Its image encoder uses a CLIP-pretrained ViT16, frozen except for the last two linear projection layers.
C.1 Training Details
SatCLIP training uses large batches, frozen pretrained Sentinel-2 image encoders, validation-loss model selection, and a spherical-harmonics location encoder whose L parameter controls embedding resolution.
- C.1 Training Details: SatCLIP models use batch sizes of 8k after experiments found the most effective training at 8k and 16k.Traditional CLIP image-text pretraining is described as optimal at 32k, but SatCLIP training differs in this respect.
- C.1 Training Details: ViT16, ResNet18, and ResNet50 image encoders pretrained on Sentinel-2 imagery are kept frozen during SatCLIP training.Only a final projection component is trained, according to the implementation passage.
- C.1 Training Details: Models are selected according to minimum validation loss after using 10% of the data for validation.Figure 7 compares validation loss across the three vision backbones with L = 10 and L = 40.
- C.1 Training Details: The location encoder’s Legendre-polynomial count L controls spatial resolution, with smaller L favoring large-scale patterns and larger L favoring small-scale patterns.Smaller L values are also computationally more efficient.
D.1 SatCLIP Embeddings in Different Biomes
The analyses separate geographic-coordinate effects from learned SatCLIP weights and show that trained embeddings organize locations by environmental conditions across biomes. Additional visualizations examine embedding scale and variance across models and datasets.
- Coordinate versus learned representations: Pure spherical-harmonic coordinate embeddings form geographic manifolds, separating northern and southern hemispheres and arctic from antarctic tundra.
- Coordinate versus learned representations: Trained SatCLIP weights shift the embedding organization toward dryness and temperature rather than pure geographic location.
- Biome structure: In tropical biomes that are geographically intertwined, only pretrained SatCLIP embeddings reveal a clear dry-to-wet progression across biome types.
- Interpretation: The supplementary analyses support interpreting SatCLIP embeddings as an implicit representation of environmental and societal ground conditions visible in Sentinel-2 imagery.
- Latent-space scale: SatCLIP models with L = 40 resolve positional embeddings more finely than models at other scales.
E.5 Additional Results: Predictive Performance
Additional evaluations show SatCLIP embeddings improve predictive performance across natural and socioeconomic tasks and perform best in few-shot geographic adaptation. The visualizations report signed errors for selected prediction tasks and compare adaptation across embeddings.
- Predictive performance: SatCLIP embeddings provide meaningful features for prediction in both natural and socioeconomic settings.
- Predictive performance: SatCLIP embeddings are more informative for iNat classification than the location encoder pretrained on iNat, CSP-iNat.
- Evaluation protocol: The experiments report average test-set MSE and accuracy across 10 independently initialized MLP runs, with results expressed as mean ±1 standard deviation.
- Geographic adaptation: In few-shot Ecoregions prediction on held-out Africa, SatCLIP performs best, followed by GeoCLIP and MOSAIKS.
E.7 Additional Results: Combining Embeddings
The study tests whether concatenating embeddings from different location encoders improves downstream performance. Across the reported regression comparisons, combinations do not outperform the best individual embedding.
- Combination strategy: Concatenated embeddings are fed to downstream learners to test whether combining representations improves performance.
- Results: Embedding combinations do not improve performance over the respective best single embedding.
- Evaluation: Table 9 reports average test-set MSE ±1 standard deviation across 10 independently initialized MLP runs on regression tasks.