PaperlayerView exact revision on arXiv

Source-linked AI summary

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Delyan Boychev, Luc Van Gool, Danda Pani Paudel

arXiv:2603.11804v3cs.CVcs.LG

TL;DR

Remote-sensing VLMs need costly, scarce domain-specific annotations, while teacher-based pseudo-labeling depends on expensive external models and inherits their errors. OSMDA pairs satellite images with rendered OSM maps so a capable base VLM can generate geographically enriched captions, then fine-tunes on satellite images alone. Across six zero-shot benchmarks, OSMDA-VLM achieves the highest average score and is identified as the strongest overall performer.

  • Problem

    Remote sensing has abundant satellite and aerial imagery but scarce structured annotations, while synthetic labeling can be expensive, biased, and limited by teacher-model errors.

  • Method

    OSMDA renders crowd-sourced OpenStreetMap data alongside satellite images, uses the base VLM to generate OSMDA-Captions, and fine-tunes with satellite images alone.

  • Results

    46.1 average zero-shot score: OSMDA-VLM leads Teacher-VLM at 43.6 and prior remote-sensing VLMs at 43.0 at best across six benchmarks.

  • Takeaways & Limitations

    OSMDA suggests that structured crowd-sourced geographic data can provide an effective, reproducible alternative to relying primarily on increasingly capable teacher models.

  • Takeaways & Limitations

    OSMDA depends on a base model with strong OCR and map-comprehension abilities, so its gains are not backbone-agnostic in practice.

Abstract

from arXiv · show

Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce. Prevailing pseudo-labeling pipelines address this gap by distilling knowledge from large frontier models, but this dependence on large teachers is costly, limits scalability, and caps achievable performance at the ceiling of the teacher. We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency. Our key insight is that a capable base VLM can serve as its own annotation engine: by pairing aerial images with rendered OpenStreetMap (OSM) tiles, we leverage optical character recognition and chart comprehension capabilities of the model to generate captions enriched by OSM's vast auxiliary metadata. The model is then fine-tuned on the resulting corpus with satellite imagery alone, yielding OSMDA-VLM, a domain-adapted VLM that requires no manual labeling and no stronger external VLM teacher. We conduct exhaustive evaluations spanning six zero-shot and five in-distribution benchmarks across vision-language tasks, where OSMDA leads to substantial improvement. We further compare against nine competitive baselines, demonstrating that our method achieves superior overall performance, while being substantially cheaper to train than teacher-dependent alternatives. These results suggest that, given a strong foundation model, alignment with crowd-sourced geographic data is a practical and scalable path towards remote sensing domain adaptation. Dataset and model weights will be made publicly available upon acceptance.

1 Introduction

Remote-sensing VLM adaptation is constrained by scarce, costly annotations and teacher-dependent synthetic labeling. OSMDA uses crowd-sourced OSM data to provide self-contained geographic supervision, producing OSMDA-Captions and a model reported as the strongest overall performer.

  • Remote sensing offers abundant imagery but lacks scalable, structured, task-specific annotations.Manual curation is costly, while synthetic pipelines can be expensive, biased, and bounded by teacher-model errors.
  • OSMDA leverages large-scale crowd-sourced OpenStreetMap annotations as geographic supervision without dedicated annotation effort.The framework is motivated as an alternative to increasingly sophisticated pseudo-label generation.
  • OSMDA-Captions is produced without human annotators or an external VLM, then combined with five remote-sensing datasets to create OSMDA-VLM.The resulting model is evaluated across six zero-shot benchmarks spanning varied vision-language tasks and settings.
  • OSMDA-VLM is identified as the strongest overall performer in an evaluation of nine competitive models.The study reports failure cases among competing systems and presents the comparison as unusually thorough.
  • The contributions include a self-contained adaptation framework, over 200k detailed OSMDA-Captions pairs, superior zero-shot performance, and unified evaluation across eleven benchmarks.The framework is described as reducing annotation cost and eliminating dependence on external teacher models.

2 Related Works

Remote-sensing VLM adaptation has progressed through domain instruction tuning and increasingly automated pseudo-labeling, often using stronger external models. OSM-related efforts already provide geographic supervision, but prior pipelines differ in scale, filtering, and label-generation strategy.

  • Domain adaptation commonly combines continued pretraining on domain image-text pairs with instruction tuning on curated question-answer data.When labeled remote-sensing data is scarce, these stages rely on manual annotation or automatically generated pseudo-labels.
  • Knowledge distillation is widely used but inherits teacher errors and imposes a ceiling on student performance.
  • Remote-sensing VLMs now support grounded dialogue, instruction tuning, multimodal understanding, and pixel-level perception across many systems.
  • Automated dataset construction increasingly relies on external models, ranging from Vicuna reformatting and rule-based templates to GPT-4o and Gemini prompting.
  • Prior remote-sensing VLM comparisons track student architecture, teacher model, generated-sample count, generation cost, and weak-annotation sources.
  • SkyScript and ChatEarthNet use OSM or related geographic labels to ground large image-text datasets at million-scale or Sentinel-2 scale.SkyScript converts OSM tags into short descriptions, while ChatEarthNet grounds captions in land-cover labels.

3 Method. OSM-based domain adaptation

OSMDA curates geographically diverse satellite images with OSM annotations, renders readable co-registered maps, and uses paired image-map inputs to generate captions. Fine-tuning then uses satellite images alone, producing a map-free domain-adapted model.

  • OSM-based domain adaptation: The pipeline comprises data curation, semantically rich co-registered map rendering, caption generation, and satellite-only fine-tuning.The final model does not require maps at inference time.
  • Data curation: OSMDA starts from approximately 1.5 million georeferenced satellite images whose footprints enable spatial retrieval of corresponding OSM objects.
  • Data curation: Visibility and anonymization filters remove non-visible, identifying, and commercially sensitive OSM information before caption generation.
  • Data curation: Qwen2.5 converts unique OSM tag sets into brief descriptive labels, yielding 48k semantic labels versus 29k from SkyScript’s rule-based heuristics.
  • Distribution balancing: Inverse-frequency sampling and DINOv3 K-means clustering address skew and redundancy, producing a curated dataset of 200k satellite images with OSM annotations.
  • Map Rendering: Mapnik renders filtered semantic OSM layers and standardized labels into visually structured tiles aligned to each image.Label placement prioritizes higher-ranked features and suppresses overlapping lower-priority labels, while the rendering supports OCR without exposing identifying text.
  • OSMDA-Captions Corpus: A base VLM generates 200k varied geographic captions from paired satellite images and maps, but fine-tuning exposes it only to satellite imagery.Stochastic generation at temperature T = 1.0 is used to vary equivalent scene descriptions and avoid mode collapse.
  • OSMDA-Captions Corpus: OSMDA is reported to reduce hallucination and improve descriptions of spatial and visual layout relative to Base and Base-FT examples.

4 Experiments

Experiments use controlled mixtures, standardized zero-shot evaluation, and ablations against teacher-based and alternative data sources. OSMDA-VLM achieves the highest reported average zero-shot score and strong per-benchmark results.

  • Training setup: RS-Instruct combines equal numbers of training examples from five remote-sensing datasets and serves as the fixed base mixture for experiments.This setup supports controlled comparisons of additional datasets and reports in-distribution testing results.
  • Evaluation: Zero-shot evaluation spans six datasets covering classification, captioning, single-image VQA, temporal VQA, high-resolution perception, and change-detection reasoning.Prompt templates are standardized and excluded from training and hyperparameter selection.
  • Metrics: Classification uses macro-averaged F1, VQA uses dimension-averaged accuracy, and open-text answers use G-Eval, with all scores normalized to 0–100.
  • Ablations: Ablations replace rendered maps with object-count dictionaries, substitute SkyScript captions, or pretrain on OSMDA-Captions before RS-Instruct fine-tuning.The last variant is evaluated in-distribution on RS-Instruct testing sets.
  • Training setup: Training uses LoRA with rank 16, 0.05 dropout, bfloat16 mixed precision, 16 NVIDIA H200 GPUs, batch size 32, learning rate 1 × 10^-4, AdamW, and one epoch.

5 Results

OSMDA-VLM achieves the strongest overall zero-shot performance across the evaluated benchmarks, while cartographic OSM conditioning preserves high-resolution quality and improves targeted in-distribution performance. Ablations show that the proposed cartographic representation outperforms alternative representations across the task suite.

  • Zero-shot performance: 46.1 average score makes OSMDA-VLM the top zero-shot model, ahead of Teacher-VLM at 43.6 and prior remote-sensing VLMs at 43.0 at best.It also ranks in the top three across all reported columns.
  • Zero-shot performance: 51.4 XLRS-Bench VQA accuracy and 41.4 GEOBench captioning score are OSMDA-VLM's best task-level results.It also reports 44.9 EuroSAT F1, 50.4 Million-AID F1, 50.7 SkyScript-Bench F1, and 44.5 XLRS-Bench captioning g_eval.
  • Pseudo-labelling ablation: The cartographic representation is the only configuration that improves over Base on five of six tasks without sacrificing high-resolution quality.In XLRS-Bench, it preserves captioning at 44.5 versus Base's 44.9 g_eval and achieves the best VQA score at 51.4 versus Base's 50.9.
  • Targeted downstream performance: +1.5 points in RSVQA LR is the largest reported in-distribution gain from OSMDA pre-training, followed by +1.2 in UCM captions, +0.6 in RSVQA HR, and +0.25 in VRS-Bench.NWPU Captions is the only reported degradation, at 0.2.

6 Discussion and conclusions

OSMDA-VLM achieves strong performance without increasingly large teacher models, but its benefits depend on the coverage and representational biases of OpenStreetMap and the capabilities of the base backbone.

  • 6 Discussion and conclusions: Across eleven evaluation settings, OSMDA-VLM achieves the highest overall performance while costing substantially less to train than teacher-dependent alternatives.The model preserves or improves performance across diverse tasks, including challenging captioning and VQA benchmarks.
  • 6 Discussion and conclusions: Map rendering provides spatial context that textual tag lists alone cannot recover, contributing to the observed gains.The ablations attribute gains not merely to richer captions but to the cartographic representation itself.
  • 6 Discussion and conclusions: OSMDA-Captions can serve as an intermediate training stage, giving downstream fine-tuning a better initialization than direct fine-tuning of the base model.This effect is reported for Base-OSMDA-FT, which achieves higher downstream performance after first training on OSMDA-Captions.
  • Map-Induced Biases: OSMDA-VLM performs well for distinct, well-labeled infrastructure and clear boundaries, but performance drops in sparse or crowded mixed-use environments.Examples include gains for barns and prisons, versus weaker results for commercial and industrial zones and overlapping commercial-building and parking semantics.
  • Limitations: The method presupposes a base model with strong OCR and map-comprehension abilities, so its gains are not backbone-agnostic in practice.The authors report that rendered maps improve caption generation for InternVL3.5-8B but not for weak-OCR LLaVA.
  • Limitations: The main experiments use a single backbone, while direct evaluation across additional model families and OSM annotation densities remains limited.Most public benchmarks discard the georeferencing required for the proposed approach, constraining geographically aware evaluation.
  • 6 Discussion and conclusions: The authors suggest structured, crowd-sourced geographic data as an alternative scaling strategy whose bottleneck is the quality and coverage of openly available geospatial data.They propose incorporating richer GIS layers as a potentially more scalable direction than relying solely on progressively larger teacher models.

A.1 OSMDA components and direct teachers’ performance

The direct-teacher evaluation requires georeferenced imagery, and OSMDA’s map representation combines semantic labels, spatial relations, and cartographic structure. This configuration outperforms simpler map ablations, while weak OCR and image comprehension undermine the framework.

  • Direct-teacher evaluation requires georeferenced satellite imagery to acquire co-registered OpenStreetMap data.
  • OSMDA combines object labels, spatial relations, and cartographic knowledge to convey hierarchical relations, functionality, and topology.
  • 45.1 average score vs 44.7 for the RGB-only baseline is achieved by the raw OSM map configuration across four benchmarks.
  • OSM-Text describes image content but loses spatial relations, while OSM-Overlay can introduce label clutter in low-resolution, high-density scenes.
  • All LLaVA configurations perform worse than the blind control, indicating that weak innate OCR and image-comprehension abilities are insufficient for OSMDA.

A.2 Trainings without RS-Instruct

Training only on OSMDA-Captions, Teacher-Captions, or RS-Instruct can produce overfitting-related deterioration on some benchmarks. Joint training with RS-Instruct and OSMDA-Captions achieves the highest average score.

  • Joint training on RS-Instruct and OSMDA-Captions effectively regularizes both datasets and achieves the highest average score.
  • Training only on OSMDA-Captions or Teacher-Captions deteriorates GEOBench and XLRS captioning scores, despite improving some datasets.
  • Base-Teacher improves Million-AID, whereas Base-OSMDA improves XLRS VQA, showing dataset-specific gains under single-source training.
  • Base-FT trained on RS-Instruct reduces average score, providing another sign of overfitting to the training format.
  • Qualitative XLRS-Bench examples show baseline format failures and frequent airport hallucinations during captioning.

C OSMDA-captions Details

OSMDA-Captions is constructed by filtering and relabeling OpenStreetMap objects, balancing and deduplicating co-registered imagery, rendering semantic maps, and prompting a VLM to caption paired aerial images. The maps guide caption generation but are omitted during model training.

  • Data pipeline: OSMDA-Captions uses OpenStreetMap data through acquisition, heuristic filtration, relabeling, distribution balancing, map rendering, and captioning.
  • Filtration: Objects are filtered when geometries are below pixel-scale visibility or represent underground, covered, or otherwise invisible infrastructure.
  • Relabeling: Qwen2.5-72B-Instruct converts remaining object tags and geometry hints into labels of at most three words with the primary physical object type last.
  • Balancing: The curated dataset contains 200514 images, 17.6 million objects, and 47984 unique semantic tags after balancing and visual deduplication.
  • Map rendering: OpenStreetMap-carto renders dense semantic maps after replacing names, removing toponyms and addresses, and labeling previously unnamed objects.
  • Captioning: InternVL3.5-8B receives the RGB aerial image and map as captioning context, while the map is completely omitted during subsequent training.
  • Captioning: Captions are sampled with temperature T = 1.0 to preserve linguistic diversity and mitigate mode collapse.

D.1 Benchmarks

The benchmarks target urban-focused remote-sensing scene understanding across varied image sizes, resolutions, instruction formats, and tasks. Evaluation combines task-specific metrics into normalized aggregate scores, with prior fine-tuning and zero-shot status tracked explicitly.

  • Benchmark scope: The benchmark suite covers image sizes from 64x64 to 10000x11500 and resolutions from 0.1 to 153 meters per pixel.
  • Benchmark scope: The study evaluates urban-focused scene understanding across varied instruction formats and tasks, with some benchmarks used for fine-tuning and others for evaluation.
  • Benchmark scope: VRSBench and XLRS-Bench include visual grounding, but that task is omitted from the study.
  • Benchmark scope: RSVQA images are georeferenced, enabling evaluation of base models with maps as OSMDA teachers.
  • Metrics: RSVQA aggregation reverses, normalizes, and clips MAE before averaging accuracy and nMAE across four tasks.
  • Evaluation protocol: Table 7 records whether each model was fine-tuned or evaluated zero-shot on each benchmark, while noting that Intern-S1-mini training data is not public.

geval metric.

The evaluation uses G-Eval to score open-ended outputs on a normalized 0–1 scale, alongside deterministic, task-specific prompts and generation limits. Prompt formats vary across VQA, captioning, and classification benchmarks to enforce task-appropriate responses.

  • G-Eval metric: G-Eval converts evaluator-model logits for scores 1–5 into a continuous score from 1.0 to 5.0, then normalizes it to [0,1].Qwen2.5-32B-Instruct serves as the evaluator, with one-forward-pass prediction omitting chain-of-thought reasoning.
  • Caption evaluation: Caption evaluation checks object accuracy, counts, hallucinations, and spatial relationships against the ground truth under the same 1–5 scoring structure.A perfect score requires matching object types, counts, and spatial layout without hallucinations.
  • VQA evaluation: VQA evaluation compares predicted and ground-truth answers for factual correctness, completeness, and hallucinations using a strict 1–5 rubric.Scores range from critical failure at 1 to exact, precise, hallucination-free answers at 5.
  • Generation protocol: Models generate deterministic predictions by selecting highest-likelihood tokens under custom dataset-task prompts and task-specific maximum token limits.The evaluation configurations are provided to support reproducibility across benchmark and task combinations.
  • Task-specific prompts: Prompt constraints require concise formats for VQA, descriptive paragraphs for captioning, and single-word classifications for multiple-choice datasets.Examples include 16-token limits for RSVQA-LR/HR answers, 512 tokens for long captions, and 16 tokens for EuroSAT and SkyScript-Bench classification.
  • Task-specific prompts: Caption prompts request comprehensive single-paragraph descriptions, while short-caption and classification prompts impose stricter output formats.VRSBench and OSMDA-Captions use up to 512 tokens; NWPU-Captions and UCM-Captions use up to 128 tokens; EuroSAT and SkyScript-Bench require one classification word.

F.1 Standard errors

The backbone-controlled evaluation reports standard errors and tests whether OSMDA-VLM outperforms Teacher-VLM across average scores. The paired bootstrap analysis finds a statistically significant positive difference favoring OSMDA-VLM.

  • Standard errors: Standard errors accounting for per-data-sample variation are reported for backbone-controlled evaluations in Table 9.Table 9 covers OSMDA-VLM across six benchmarks and their tasks against prior fixed-backbone alternatives.
  • Statistical significance: Δ= +2.47 [2.14, 2.80] points with p = 0.0001 indicates higher average OSMDA-VLM performance than Teacher-VLM.The comparison uses a two-sided paired percentile bootstrap test and a 95% confidence interval.
Loading 2603.11804v3…