Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment

• Satellite image analysis models are often specialized to predefined concepts, limiting open-ended analysis of large earth-observation datasets.The paper notes that these models can also be complex, reducing accessibility to experts outside AI.Source: §1 · p. 1
• Satellite VLM development is constrained by the lack of image-text annotations; existing remote-sensing datasets are about 10k pairs versus 400 million used to train CLIP.Curating satellite captions is expensive and requires expertise.Source: §1 · p. 1
• GRAFT uses geotagged, co-located ground-satellite pairs to train a satellite encoder into CLIP’s image-feature space, linking satellite images to text without textual annotations.The training uses a contrastive loss on ground-satellite image pairs.Source: §1 · p. 2, Figure 1 · p. 2
• Because satellite images cover more area than ground images, GRAFT also aligns satellite patches with CLIP ground-image features for text-to-patch retrieval and localization.The patch representation can support text-to-image segmentation with foundational models such as SAM.Source: §1 · p. 2
• GRAFT reports up to 20% relative improvement over baseline for text-to-image retrieval and more than 80% for text-to-segmentation.Source: §1 · p. 2
• The authors collect million-scale datasets at two resolutions and report open-world image- and pixel-level understanding, plus zero-shot VQA using an extended ViperGPT framework.The datasets use 1m NAIP and 10m Sentinel-2 imagery.Source: §1 · p. 2

• Existing remote-sensing foundation models can be effective image encoders but cannot perform open-vocabulary recognition.Open-vocabulary interfaces such as retrieval or question answering can make satellite data more accessible to non-AI experts.Source: §2 · p. 3
• Prior satellite VLMs for captioning or retrieval use far less image-text data than internet models because satellite captions must be curated rather than crowd-sourced.The passage compares thousands of satellite pairs with billions for internet counterparts.Source: §2 · p. 3
• Earlier ground-image work linked ground and remote-sensing imagery for geo-localization, whereas this work uses ground images to connect remote sensing with language.Source: §2 · p. 4

3.1 GRAFT: GROUND REMOTE ALIGNMENT FOR TRAINING VLMS

• GRAFT uses ground images as an intermediary to train remote-sensing encoders in the representation space of a pretrained internet vision-language model.The paper uses CLIP and geotagged ground-satellite image pairs, without textual annotations.Source: §3 · p. 4, §3.1 · p. 4

3.1.1 IMAGE-LEVEL VLMS

• Because satellite images cover larger areas and may match multiple ground images, the image-level objective pulls each satellite embedding toward all its associated ground images and away from other batch images.The loss uses cosine similarity and a temperature parameter; with one ground image per satellite, it reverts to a CLIP-like contrastive loss.Source: §3.1.1 · pp. 4–5

3.1.2 PIXEL-LEVEL VLMS

• Pixel-level models use ground-image geotags to align satellite patch representations with the corresponding ground-image representation.Training signals are sparse: the loss is computed only at satellite locations with at least one ground image, which the authors report suffices empirically.Source: §3.1.2 · p. 5

3.2 COLLECTING GROUND-SATELLITE IMAGE PAIRS

• Collection uses geographically distributed, accurately geotagged outdoor Flickr images; Sentinel-2 imagery is temporally matched and limited to at most 1% cloud cover.Satellite sampling avoids high overlap, and images with more than 25 associated ground images are randomly subsampled.Source: §3.2 · pp. 5–6
• The paired datasets contain 10.2 million NAIP pairs at 1-meter resolution and 8.7 million Sentinel-2 pairs at 10-meter resolution.Ground images are assigned to satellite images according to their geotags.Source: §3.2 · pp. 5–6

3.3 ENHANCING GRAFT VLMS WITH FOUNDATIONAL MODELS

• For segmentation, GRAFT selects high-scoring patches as point prompts for SAM; for more nuanced VQA, it replaces ViperGPT’s GLIP detector with pixel-level GRAFT.The VQA pipeline thresholds pixel-level scores, forms connected components, and refines instances with SAM.Source: §3.3 · p. 6

3.4 IMPLEMENTATION DETAILS

• The models use ViT-B/16, are initialized with the CLIP image encoder, and share hyperparameters selected on a NAIP validation set.The same selected hyperparameters are used for Sentinel-2 models.Source: §3.4 · p. 6
• The Sentinel-2 image-level model is significantly better on almost all classification and retrieval metrics in the zero-shot benchmarks.The table also compares it with one-shot remote-sensing models that lack language capabilities.Source: Table 1 · p. 7

4.1 IMAGE-LEVEL UNDERSTANDING

• Almost 10 points (20% relative) on EuroSAT classification and 30 points on SAT-6: GRAFT outperforms nearly all comparison models.GRAFT VIT-B/32 also exceeds 1-shot Satlas by ∼13 mAP on BigEarthNet without annotated images.Source: §4.1 · pp. 7–8, Table 2 · p. 8

4.2 PIXEL-LEVEL UNDERSTANDING

• GRAFT nearly doubles per-class accuracy on NAIP and improves Sentinel-2 by more than 50% relatively; SAM refinement adds only a minor gain.NAIP masks are evaluated without SAM, while Sentinel-2 results are reported both with and without SAM.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 VISUAL QUESTION ANSWERING

• With GRAFT's pixel-level model, ViperGPT gains almost 6 points in zero-shot accuracy over its GLIP variant across question types except presence.The evaluation subsamples 500 unique questions from the high-resolution RSVQA test set.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 ABLATIONS

• 74.8% mAP falls to 69.3% mAP with random satellite-image sampling, supporting the strategy of spacing image centers at least 112 meters apart.The ablation uses a held-out NAIP validation set annotated with OpenStreetMap.Source: §4.4 · p. 9
• Replacing ground-image embeddings with alt-text embeddings drops performance from 74.8% mAP to 65.4% maP; adding text as an auxiliary loss yields 70% mAP.The authors suggest noisy alt-text may contribute to the lower performance.Source: §4.4 · p. 9

4.5 EXAMPLE APPLICATIONS

• Open-world queries produce regional maps for cities, roads, and farmland, and the farmland map matches the ground-truth agricultural pattern.The examples use NAIP images from Massachusetts.Source: §4.5 · p. 9, Figure 5 · p. 9

5 DISCUSSION AND CONCLUSION

• Low satellite revisit rates make dynamic-object alignment difficult, although GRAFT can still retrieve locations where such objects might be found.The paper identifies daily satellite revisits as a possible way to address this limitation.Source: §5 · p. 9, Figure 6 · p. 9
• GRAFT reports state-of-the-art zero-shot accuracy for classification, retrieval, segmentation, and VQA, with gains up to 20% on classification and 80% on segmentation.The conclusion describes the method as connecting satellite images with text through internet images as an intermediary.Source: §5 · pp. 9–10

6 ETHICS

• Privacy risks include detecting environmental alterations, and CLIP-derived race, culture, and gender biases may persist or worsen in sensitive applications.Access to the data and trained models is limited to users complying with an ethics agreement, who must provide their intended use case.Source: §6 · p. 10

A TRAINING DATASET DETAILS

• The NAIP dataset contains 10.2 million pairs, while Sentinel-2 contains 8.7 million ground-satellite pairs.NAIP has 2.0 million unique satellite images and 7.9 million unique ground images; Sentinel-2 has 1.9 million and 7.6 million, respectively.Source: A TRAINING DATASET DETAILS · p. 14

A.1 SPATIAL AND TEMPORAL ALIGNMENT

• Pairs can contain multiple ground views at mapped locations; Sentinel-2 matches use the closest cloudless imagery, averaging 21-day gaps, with street-level geotags and patches spanning 14m or 140m.For Flickr images predating Sentinel-2 imagery, the authors sample seasonally aligned satellite images; they note that some geotags may still be incorrect.Source: §A.1 · pp. 14, 16, Figure 8 · p. 15

B PROMPTING

• GRAFT uses multiple prompts describing internet imagery, while CLIP baselines use satellite-image prompts; GRAFT performs better with internet-style prompts.GRAFT prompts include “A photo of a {label}” and “I took a photo from a {label}”; the authors average text representations from multiple prompts.Source: B PROMPTING · p. 16

C.1 TRAINING DETAILS

• All models train for 10 epochs with AdamW and CLIP initialization, using cosine learning-rate schedules after linear warmup.The image-level peak learning rate is 1e-5 and the pixel-level peak is 5e-5; the temperature is τ = 0.07.Source: §C.1 · p. 16

C.2 INTERNAL VALIDATION DATASET

• Image-level model selection uses a 14-category, 2,632-image internal OSM validation set, while pixel-level selection uses a 32-category, 17k-image NAIP-OSM subset.Image-level performance is measured by mean average precision; pixel-level models are selected using zero-shot patch-prediction accuracy.Source: §C.2 · p. 16

C.3 IMAGE-LEVEL UNDERSTANDING EVALUATION DETAILS.

• Sentinel-2 image-level evaluation uses EuroSAT’s 5,400 single-label images and BigEarthNet’s 104k images with 19 multilabel classes.EuroSAT images are 64×64 with 10 classes; BigEarthNet test images are 120×120.Source: §C.3 · pp. 16–17
• NAIP evaluation uses SAT-4 and SAT-6 plus a 33-category NAIP-OSM dataset of 1.7 million 224×224 images; SAT-4’s “None” category is excluded.SAT-4 and SAT-6 test sets contain 64k and 81k images, respectively; Table 5 lists the NAIP-OSM categories.Source: §C.3 · p. 17, Table 5 · p. 17
• For image-level benchmarks, the authors report top-1 accuracy, mean average precision, and retrieval mAP@100 and mAP@20, selecting the best tested preprocessing for each model.The tested transforms include zero padding, reflection padding, and resizing; the chosen transform varies by model.Source: §C.3 · p. 17

C.4 SEGMENTATION

• Segmentation evaluation covers eight of eleven land-cover classes on about 39k NAIP images and 2,183 Sentinel-2 images, each processed as four 224×224 crops.For Sentinel-2, standalone GRAFT upsamples logits; the SAM variant assigns each bottom-up segment its majority-pixel class, and the cited experiments find a minor performance improvement with SAM.Source: §C.4 · p. 18

C.5 VQA

• VQA evaluation uses 500 unique questions, yielding 2,985 image-question-answer tuples across presence, area, comparison, and count questions.The tuple counts are 576 presence, 1,161 area, 590 comparison, and 658 count.Source: §C.5 · p. 18

D ADDITIONAL RESULTS

• GRAFT retrieves fine-grained and open-world concepts such as campgrounds and hedge mazes, and can retrieve solar farms, water treatment facilities, and wind farms.Figure 10 gives further retrieval examples, including a water treatment facility and a hedge maze.Source: D ADDITIONAL RESULTS · p. 18, Figure 10 · p. 19
• 53.78 EuroSat zero-shot classification accuracy is reported for the multispectral model, a 1.26% improvement over the RGB model’s 52.52%.The model was trained on 100k multispectral Sentinel-2–ground-image pairs, with its patch embedding expanded from 3 to 12 input channels.Source: D ADDITIONAL RESULTS · p. 18

E.1 ABLATIONS: ALT-TEXT ALIGNMENT

• Ground-satellite image alignment produces better-performing models than alt-text alignment, which the authors attribute to ground images providing better semantic features.Flickr captions can be filenames or descriptions that do not capture the image’s scene; the authors note that LAION-5B’s much larger scale may still allow learning from noisy text.Source: §E.1 · p. 20

E.2 ABLATIONS: CLIP INITIALIZATION

• 42.46% EuroSat zero-shot accuracy for training from scratch is below CLIP at 53.59% and GRAFT at 63.76%.The authors initialize GRAFT from CLIP, reporting better learning and faster convergence; they note that compared baselines also use CLIP initialization.Source: §E.2 · p. 20

E.3 ABLATIONS: LOSS FORMULATIONS

• The alternative L2 loss performs on par with GRAFT’s default formulation on the internal validation set.The authors report this result for the NAIP high-resolution dataset.Source: §E.3 · p. 20
• The conventional contrastive loss underperforms GRAFT, while the L2 results support keeping satellite embeddings close to associated ground images and away from other satellites’ ground images.The authors identify both the positive association and the separation from other satellite-associated ground images as important.Source: §E.3 · p. 20

E.4 ADDITIONAL ABLATIONS: SCALING

• The authors conjecture that NAIP plateaus because its coverage is limited to the United States and a small image sample suffices to learn a good embedding.Source: §E.4 · p. 21
• NAIP-model performance plateaus after 2 × 105 examples.Figure 11 reports this result for a ViT-B/16 model evaluated on the internal validation set.Source: Figure 11 · p. 21, §E.4 · p. 21
• Low-resolution Sentinel-2 model performance continues to scale as more data is used.Figure 11 reports this result for a ViT-B/16 model evaluated on EuroSAT classification.Source: Figure 11 · p. 21, §E.4 · p. 21

F FUTURE WORK

• GRAFT and other CLIP-like VLMs lack text-generation capabilities, though GRAFT can be combined with ViperGPT for VQA.Source: F FUTURE WORK · p. 21
• ClipCap is proposed as a possible framework for adding captioning capabilities to GRAFT.Source: F FUTURE WORK · p. 21
• Because the text encoder is not fine-tuned, it may retain ground-image concept bias, including directional terms that may not map meaningfully to satellite images.Source: F FUTURE WORK · p. 21
• Despite possible bias, GRAFT outperforms other VLMs, including some text-supervised models, and is robust to it for most remote-sensing recognition tasks.Source: F FUTURE WORK · p. 21
• Fine-tuning with a small amount of satellite image-caption data is suggested as one possible way to create stronger models.Source: F FUTURE WORK · p. 22