CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations

1. Introduction

• Geo-tagged images are plentiful and inexpensive to obtain, while object, bounding-box, and land-use labels are scarce and costly to collect.Existing labeled geospatial datasets can also have limited size and geographic coverage.Source: §1 · p. 1
• Distinct geographic distribution patterns can help distinguish visually similar species such as Arctic foxes and bat-eared foxes.The figure contrasts the species’ similar appearances with their different image-location patterns.Source: Figure 1 · p. 1, §1 · p. 2
• CSP separately encodes locations and images, then contrasts them to learn spatial representations from unlabeled geo-tagged images for later supervised use.The encoders can be fine-tuned on a small amount of labeled data and used jointly for inference.Source: §1 · pp. 2–3
• CSP explores three strategies for forming contrastive positive and negative examples and three self-supervised losses; combining the strategies and using MC usually performs best.The losses compared are MSE, NCE, and MC.Source: §1 · p. 3
• 10–34% relative improvement on iNat2018 few-shot settings, with CSP also improving performance on both datasets when trained with the full training set.The few-shot results use stratified samples of 5%, 10%, and 20% of the training data.Source: §1 · p. 3

2. Related Work

• Tile2Vec, Geo-SSL, SeCo, and GeoKR use spatial relations, co-located imagery, or land-cover information to pre-train image encoders, but do not directly use geo-locations as model inputs.Geo-SSL and SeCo use spatially aligned remote-sensing images from different times; GeoKR predicts land-cover distributions.Source: §2 · p. 3
• Unlike these prior approaches, CSP explicitly encodes and contrasts location-image pairs and retains the location encoder for joint image-and-location prediction.The related-work discussion notes that prior methods ignore location during fine-tuning and inference.Source: §2 · p. 3
• Many existing location-encoding approaches are supervised, while Zhai et al. (2018) bins locations for cross-entropy supervision rather than leveraging their continuity; CSP trains location encoders self-supervised from unlabeled geotagged images.The paper positions CSP as a way to use massive unlabeled geographic data.Source: §2 · p. 3
• CSP contributes to spatially explicit AI by learning multi-scale location representations from unlabeled images, alongside prior work on spatial dependency, heterogeneity, temporal patterns, and Earth’s spherical geometry.The cited spatial principles include temporal continuity and periodicity.Source: §2 · p. 4

3.1. A Dual-Encoder for Geo-Tagged Images

• CSP encodes locations and images separately, projecting image-network features into a shared space for location–image contrastive learning.The location encoder maps geo-locations to vectors, while a projection layer maps image features to the contrastive space.Source: §3.1 · p. 4

3.2. Contrastive Spatial Pre-Training(CSP)

• Three pair-sampling schemes use mismatched in-batch pairs, randomly sampled locations, or dropout-based paired location embeddings for contrastive training.The original location-image pair is positive for the in-batch and random-location schemes; the dropout scheme pairs two embeddings of the same location.Source: §3.2 · p. 5
• CSP combines three pair-sampling loss components under NCE or multi-class objectives, while the tested regression loss performs less well.The pretrained image network is locked during CSP; the authors motivate this choice by the image encoder's much larger parameter count relative to the location encoder.Source: §3.2 · pp. 4–6

3.3. Supervised Fine-Turning

• For supervised transfer, the image and location encoders are fine-tuned separately on labeled samples, with presence-absence supervision for the location encoder.The location objective includes labeled negatives and uniformly sampled negative locations.Source: §3.3 · p. 6

3.4. Model Inference

• At inference, CSP combines the fine-tuned image and location predictions, with the joint prediction based on conditional independence of image and location given the class.The paper estimates the joint class probability as proportional to the product of the image-based and location-based probabilities.Source: §3.4 · p. 6

4. Experiments

• Experiments test the dual-encoder in few-shot settings and use separate unlabeled pre-training, labeled fine-tuning, and held-out evaluation datasets.The evaluation covers fine-grained species recognition and satellite image classification.Source: §4 · pp. 6–7

4.1. Models and Baselines

• Baselines include image-only and supervised dual-encoder models, while CSP variants share grid location encoders and training procedures but differ in contrastive loss.The baselines also include MSE pre-training; CSP-NCE-BLD and CSP-MC-BLD use NCE and MC losses, respectively.Source: §4.1 · p. 7

4.2. Fine-Grained Species Recognition

• On iNat2018, image-only training has the lowest performance across labeled-data ratios, grid outperforms wrap, and CSP-MC-BLD ranks above CSP-NCE-BLD and MSE.CSP-MC-BLD also beats Sup. Only (grid) at λ% = 100%, and each tested loss component contributes to performance.Source: §4.2 · p. 7
• With iNat2018, embedding dimension d = 256 performs best at 5%, 10%, and 20% labeled data, while d = 1024 performs best when fully supervised.CSP-MC-BLD remains best with either Inception V3 or ViT as the image encoder at λ% = 5%.Source: §4.2 · p. 7, Table 4 · p. 8
• 10.4%, 34.3%, and 16.6% relative improvements over Sup. Only at λ% = 5%, 10%, and 20%, respectively, show CSP-MC-BLD’s iNat2018 few-shot gains.The comparison is on the iNat2018 validation dataset; Table 1 reports Top1 accuracy across training-data ratios.Source: §4.2 · p. 7, Table 1 · p. 8
• Pre-trained iNat2018 location embeddings show regional clusters resembling Köppen climate zones, while US clusters are smaller where training data are denser.Figure 4 contrasts embeddings after CSP pre-training with those after supervised fine-tuning.Source: §4.2 · pp. 7–8, Figure 4 · p. 9

4.3. Satellite Image Classification

• CSP-MC-BLD outperforms all models on fMoW and is superior or comparable to CSP-NCE-BLD, though its margins are smaller than on iNat2018.The authors attribute the fMoW improvements mainly to the location encoder’s spatial-distribution modeling.Source: Table 5 · p. 8, §4.3 · p. 8
• On fMoW, pre-training yields more fine-grained US clusters, and fine-tuning makes location representations more detailed; regional effects are less clear than on iNat2018.The authors note that land-use distributions are complex and difficult to distinguish, a challenge beyond this paper’s scope.Source: §4.3 · p. 9, Figure 5 · p. 9

5. Conclusion and Discussion

• CSP improves performance on both iNat2018 species recognition and fMoW satellite image classification across different labeled-data sampling ratios.The conclusion describes a dual encoder that separately encodes locations and images, contrasting their representations during pre-training.Source: §5 · p. 9
• 10–34% relative improvement on iNat2018 at few-shot sampling ratios, with performance also improving at λ = 100%.The few-shot ratios are 5%, 10%, and 20%.Source: §5 · p. 9
• The study evaluates location-image pre-training using existing geo-tagged datasets and single coordinates; larger datasets and more complex geometries remain future directions.The authors also identify location- or time-alignment with other modalities, such as text, as future work.Source: §5 · p. 9

6. Ethics Statements

• The authors report finding no negative societal impact from this research.Source: §6 · p. 9

A.1. A Detailed Version of Figure 2b

• ImageNet pretraining trains the image encoder on the ImageNet dataset before supervised fine-tuning for image classification.Source: Figure 6 · p. 14
• Tile2Vec pretrains the image encoder with a triplet loss that makes embeddings of spatially nearby image tiles more similar than distant tiles.Source: Figure 6 · p. 14
• Geo-SSL and SeCo use locations to form spatially aligned remote-sensing images from different timestamps as positive samples for MoCo-v2-style contrastive pretraining.Source: Figure 6 · p. 14
• GeoKR trains the image encoder in a teacher-student network by minimizing KL loss between image representations and spatially aligned auxiliary data such as land-cover maps.Source: Figure 6 · p. 14
• In the illustrated prior approaches, location is auxiliary information for image-encoder pretraining and is ignored during supervised learning.Source: Figure 6 · p. 14

A.2.1. TRAINING DETAILS OF IMAGE ENCODER f()

• For iNat2018, the image features come from ImageNet-pretrained InceptionV3 or ViT; for fMoW, they come from a Geo-SSL-pretrained ResNet50.The paper uses off-the-shelf networks to encode images as feature vectors before CSP pretraining.Source: §A.2.1 · pp. 14–15
• During CSP pretraining, a projection layer maps image features to dimension d so they can be contrasted with location embeddings.Source: §A.2.1 · p. 15
• For supervised classification, the projection layer is replaced by a classification head that maps image features to logits over the classes.Source: §A.2.1 · p. 15
• The iNat2018 models are fine-tuned end-to-end, whereas fMoW uses linear probing because its image encoder was self-supervised-pretrained on that dataset.The authors report that end-to-end fine-tuning performed better than linear probing for iNat2018.Source: §A.2.1 · p. 15

• Space2Vec grid encodes locations for both datasets, except Img. Only uses no location encoder and Sup. Only (wrap) uses a wrap location encoder.Source: §A.2.2 · p. 15
• After pretraining the image encoder, CSP locks it and uses it to pretrain the location encoder; location-encoder fine-tuning is described in Section 3.3.Source: §A.2.2 · p. 15

A.2.3. MODEL IMPLEMENTATION DETAILS

• The implementation uses PyTorch on a Linux machine with 252GB memory and two GeoForce CUDA cores, and the code, data, and pretrained models are available online.Source: §A.2.3 · p. 15

A.3. Implementation Details of SimCSE

• SimCSE uses two shared-parameter location-encoder towers with independently sampled dropout masks; the same location forms a positive pair, while a different batch location forms a negative pair.Source: §A.3 · p. 15

A.4. Model Hyperparameter Tuning

• Sup. Only tuning supplied location-encoder settings reused for CSP and MSE; pretrained encoders selected unchanged settings, while pretraining rates and loss-specific weights, temperatures, and sampling counts were tuned separately.The authors report that deeper location-encoder MLPs did not necessarily improve performance.Source: §A.4 · p. 16

A.5. The Baseline Selection Criteria

• For iNat2018, the authors omitted Tile2Vec and Geo-SSL because their remote-sensing assumptions do not fit species images, and excluded GeoKR because it adds land-use-map information.Source: §A.5 · p. 16
• For fMoW, the selected image-only baselines were Tile2Vec and Geo-SSL; GeoKR requires additional global land-use maps, while Tile2Vec’s nearby-image assumption is weak for geographically separated samples.The authors cite prior results to choose Geo-SSL over ImageNet as the stronger baseline and say Table 5 confirms their Tile2Vec assumption.Source: §A.5 · p. 16

A.6. Additional Related Work

• Prior spatial methods either require labels or ignore spatial data at inference; CSP applies contrastive learning to image-location alignment with a frozen image encoder, scaling to batch size 8192 without multi-batch training.Source: §A.6 · pp. 17–18

A.7. The Spatial Distribution of iNat2018 and fMoW dataset

• Figure 8 depicts the spatial distributions of training, few-shot fine-tuning, and validation data for iNat2018 and fMoW.Source: Figure 8 · p. 18