PaperlayerView exact revision on arXiv

Source-linked AI summary

SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing

Zhecheng Wang, Rajanie Prabha, Tianyuan Huang, Jiajun Wu, Ram Rajagopal

arXiv:2312.12856v1cs.CVcs.AIcs.LG

TL;DR

Remote sensing lacks the large, semantically diverse image-text data needed for versatile VLMs, and such data cannot be efficiently collected from the public Internet. SkyScript links open remote sensing images to OpenStreetMap semantics, producing a 2.6-million-pair dataset whose continual pre-training improves zero-shot scene classification and supports transfer to other tasks.

  • Problem

    Remote sensing lacks large, semantically diverse image-text datasets because its imagery and associated descriptions cannot be efficiently collected from the public Internet at scale.

  • Method

    SkyScript uses geo-coordinates to connect open, unlabeled remote sensing images with rich OpenStreetMap semantic information and derives image-text pairs for continual pre-training.

  • Results

    SkyScript-derived CLIP outperforms original CLIP and other baselines across zero-shot scene classification, fine-grained classification, and cross-modal retrieval.

  • Takeaways & Limitations

    SkyScript supports remote sensing VLM development for open-vocabulary classification, retrieval, captioning, and text-to-image generation.

  • Takeaways & Limitations

    SkyScript has limited geographic representativeness because unrestricted imagery is concentrated in the U.S. and Europe and OSM annotations are less complete in developing countries.

Abstract

from arXiv · show

Remote sensing imagery, despite its broad applications in helping achieve Sustainable Development Goals and tackle climate change, has not yet benefited from the recent advancements of versatile, task-agnostic vision language models (VLMs). A key reason is that the large-scale, semantically diverse image-text dataset required for developing VLMs is still absent for remote sensing images. Unlike natural images, remote sensing images and their associated text descriptions cannot be efficiently collected from the public Internet at scale. In this work, we bridge this gap by using geo-coordinates to automatically connect open, unlabeled remote sensing images with rich semantics covered in OpenStreetMap, and thus construct SkyScript, a comprehensive vision-language dataset for remote sensing images, comprising 2.6 million image-text pairs covering 29K distinct semantic tags. With continual pre-training on this dataset, we obtain a VLM that surpasses baseline models with a 6.2% average accuracy gain in zero-shot scene classification across seven benchmark datasets. It also demonstrates the ability of zero-shot transfer for fine-grained object attribute classification and cross-modal retrieval. We hope this dataset can support the advancement of VLMs for various multi-modal tasks in remote sensing, such as open-vocabulary classification, retrieval, captioning, and text-to-image synthesis.

Introduction

Remote sensing has not fully benefited from versatile VLMs because large, semantically diverse image-text datasets are difficult to obtain. SkyScript addresses this gap by linking remote sensing imagery with OpenStreetMap semantics.

  • Contribution: SkyScript is over two orders of magnitude more semantically diverse than existing remote sensing image-text datasets.Figure 1 compares large classification, detection, and segmentation datasets with image-text datasets.
  • Motivation: Remote sensing has not fully benefited from task-agnostic VLMs because large, semantically diverse image-text collections are not readily available.Remote sensing imagery cannot be crawled from the web at scale, and available images are usually standalone rather than surrounded by relevant text.
  • Motivation: Existing remote sensing datasets rarely reach millions of images and contain no more than 150 semantic classes.Human annotation is challenging because annotators need domain familiarity.
  • Contribution: SkyScript connects open, unlabeled remote sensing images with rich OpenStreetMap semantics using geo-coordinates.This produces a global dataset with 2.6 million image-text pairs and 29K distinct semantic tags.
  • Results: Continual pre-training with SkyScript yields a CLIP model that outperforms original CLIP and other baselines across three zero-shot remote sensing tasks.The tasks are scene classification, fine-grained attribute classification, and cross-modal retrieval.

Related work

Prior remote sensing foundation-model research has mainly learned image representations through image similarity or masked-patch reconstruction. Existing datasets remain limited in scale and semantic diversity compared with the needs of versatile VLMs.

  • Vision-language pre-training: Large-scale image-text pre-training has produced VLMs that generalize across domains and remain robust to distribution shifts.This broader VLM development motivates image-text approaches for remote sensing.
  • Resources: The SkyScript dataset and associated models are publicly available.The cited repository is github.com/wangzhecheng/SkyScript.
  • Remote sensing foundation models: One research direction learns remote sensing representations by comparing images from different geo-locations or multiple views of the same object or location.This approach establishes similarity metrics across related images.
  • Remote sensing foundation models: Another research direction trains vision models to reconstruct masked patches of the input image.
  • Dataset limitations: Only a few remote sensing datasets contain million-scale images, and they cover fixed semantic class sets of no more than 150 classes.The passage identifies this as a challenge for constructing large and semantically diverse datasets.

Dataset

SkyScript is constructed by linking open remote sensing imagery from GEE with semantically rich OSM tags through geo-coordinates. The resulting dataset applies tag filtering and image selection to provide globally distributed, semantically diverse image-text pairs for remote sensing.

  • Data sources and pairing: SkyScript links large-scale unlabeled remote sensing images with geo-tagged OSM semantic information to construct image-text pairs.GEE supplies the imagery, while OSM supplies rich semantic tags connected through geographic coordinates.
  • Data sources and pairing: The image pool combines multiple sources and resolutions, with RGB imagery spanning 0.1 m/pixel to 30 m/pixel GSD.Additional spectral bands beyond RGB are left for future research.
  • Semantic filtering: A two-stage tag classifier predicts whether an OSM tag is visually groundable and, if so, the maximum GSD at which it remains visually groundable.Both stages use CLIP tag embeddings; the first uses binary logistic regression and the second predicts among discrete GSD thresholds.
  • Data selection: Data selection first samples objects from globally distributed OSM grids, then selects images and filters image-text pairs by similarity.The approach is designed to support global representativeness and semantic diversity.
  • Dataset scale and content: 2.6 million filtered image-text pairs cover 29K distinct tags, including 100K single-object captions and 1.2 million multi-object captions.The retained pairs are the top 50% by similarity from 5.2 million unfiltered pairs; sampled tag accuracy is 96.1%.
  • Coverage and diversity: SkyScript covers all continents except Antarctica, with especially dense object coverage in the U.S. and Europe and richer semantics than previous remote sensing datasets.It includes broad categories plus fine-grained attributes such as crop type, road surface, and power-plant source and capacity.
  • Applications: SkyScript supports remote sensing tasks including open-vocabulary classification, retrieval, captioning, and text-to-image generation.Potential applications include infrastructure monitoring, illegal-mining identification, land-use tracking, and renewable-energy mapping.

Experiments

SkyScript-derived SkyCLIP is evaluated across zero-shot scene classification, fine-grained attribute classification, and cross-modal retrieval, generally outperforming web-based and supervised baselines. Results also examine layer freezing and the trade-off between dataset size and pair quality.

  • Experimental setup: SkyCLIP is continually pretrained from web-based CLIP weights on SkyScript using image-text contrastive learning.Two CLIP backbones, ViT-B-32 and ViT-L-14, are considered.
  • Zero-shot scene classification: 59.93% average top-1 accuracy gives SkyCLIP-50 with ViT-L-14 a 6.17% gain over original CLIP/ViT-L-14 and a 2.06% gain over CLIP-laion-RS.The evaluation covers seven unseen remote sensing benchmark datasets.
  • Zero-shot scene classification: +9.98% average accuracy over RemoteCLIP and +4.52% over human-curated captions show SkyCLIP's advantage in zero-shot scene classification.The comparison supports the importance of semantic diversity in remote sensing image-text datasets.
  • Ablation and model adaptation: SkyCLIP-50 outperforms SkyCLIP-20 and SkyCLIP-30 on some datasets but not others, reflecting a trade-off between pair quantity and predicted pair quality.The comparison is reported for MillionAID, PatternNet, and RESISC45 among other benchmarks.
  • Ablation and model adaptation: Freezing more image- or text-encoder layers degrades accuracy, with image-encoder freezing performing worse than text-encoder freezing.The result indicates that both modalities require adaptation, with greater adjustment needed for image representations.
  • Fine-grained classification: 46.83% is the highest roof-shape accuracy for SkyCLIP with ViT-L-14, while ViT-B-32 reaches 38.00% for road smoothness and 67.73% for surface materials.SkyCLIP improves over its CLIP counterpart by +6.37% to +24.77%, except for a -5.5% ViT-B-32 roof-shape decrease.
  • Cross-modal retrieval: +2.57% to +6.48% mean-recall gains over CLIP and CLIP-laion-RS occur on three unseen retrieval benchmarks.SkyCLIP's mean recall is comparable to or higher than supervised models trained on those benchmarks, including larger gains on UCM-Captions.

Conclusion

The paper presents SkyScript and derives a remote-sensing CLIP model that outperforms baselines across three zero-shot tasks. Geographic representativeness remains a limitation, while future work includes additional multimodal tasks.

  • Conclusion: SkyScript is a large and semantically diverse image-text dataset for remote sensing images.The paper uses it to derive a remote-sensing-specialized CLIP model.
  • Conclusion: The derived model outperforms baseline models across zero-shot scene classification, fine-grained classification, and cross-modal retrieval.These are the three downstream tasks highlighted in the conclusion.
  • Conclusion: Geographic representativeness is identified as a limitation, and future work may explore image captioning and text-to-image generation.These directions extend beyond the three evaluated downstream tasks.

A.1. Tag classification

The tag-classification pipeline uses CLIP tag embeddings with two logistic-regression stages: visual groundability first, then the maximum usable ground sampling distance.

  • Pipeline: Each tag-classification stage takes a CLIP embedding of the tag as input, produced by the pretrained CLIP/ViT-L-14 text encoder.The two-stage approach determines whether a tag should describe an image.
  • Stage I: visual groundability: The first binary logistic-regression stage predicts whether a tag can be visually grounded in remote sensing images.It is developed using labels for 2,000 tags.
  • Stage I: visual groundability: The first-stage model achieves a best 5-fold cross-validation F1 score of 0.88.F1 is defined as the harmonic mean of precision and recall.
  • Stage II: resolution suitability: The second multi-class logistic-regression stage predicts the maximum GSD at which a visually groundable tag can be grounded.It uses PCA-reduced CLIP embeddings and is applied only after a positive first-stage prediction.
  • Stage II: resolution suitability: The second-stage model achieves a best 5-fold cross-validation accuracy of 0.53 across cutoff options from 0.1 m to 10 m.The model is trained using labels for 514 tags.

A.2.1. Image collection selection.

Image collection selection uses predicted tag-specific resolution limits to determine which GEE collections can visually ground OSM objects.

  • Image collection selection: A GEE image collection is suitable for point or polyline OSM objects when its GSD is smaller than the tag's predicted maximum allowable GSD.The predicted resolution threshold comes from the second-stage tag-classification model.

A.2.2. Image boundary selection.

Image boundaries are centered on the mapped object, then randomly varied in position and size to increase image diversity before downloading imagery from GEE.

  • Point and polyline objects are centered in 224 × 224 image boundaries.Polylines use a randomly selected node as the center.
  • Polygon boundaries are selected using the object’s extent and randomized dimensions.Polygon images use dimensions from 150 to 1,500 pixels with aspect ratios between 0.5 and 2.
  • Object centers are moderately displaced while remaining within the middle 1/3 of the image.This variation applies to point and polyline objects.
  • Image height and width for point and polyline objects are randomly re-selected between 168 and 300 pixels.The final boundaries are used to download images from GEE.

A.3. Assembling caption

SkyScript converts OSM tags into captions through renaming, connective-word rules, and composition of phrases for single- and multi-object descriptions.

  • Captions are assembled in three stages: renaming keys or values, connecting key-value phrases, and connecting multiple tags.The rules adapt OSM terminology and structure tag combinations into readable descriptions.
  • OSM names are normalized by replacing terms such as “highway,” “aeroway,” and “lit” with more commonly used descriptions.Examples include “road,” “airport,” and “light.”
  • Key-value phrases use spaces, “is,” “under,” or “of” depending on whether the key is an adjective, attribute, construction value, or other case.For example, “smoothness” becomes “smoothness is good,” while construction becomes “under construction.”
  • Single-object captions join tag phrases with commas, while multi-object captions connect object descriptions with “with.”A power pole example is paired with surrounding power-line attributes in the multi-object caption.
  • Simple caption assembly may make trained VLMs behave like bag-of-words because it lacks compositional structure and stronger alignment with human language.More natural and complicated captions are identified as future work.

A.4. SkyScript classification dataset

The supplementary evaluation uses separate remote-sensing datasets and benchmark partitions to assess scene classification, fine-grained attributes, retrieval, filtering, and unseen-class transfer.

  • SkyScript classification dataset: SkyScript-classification is an in-domain scene-classification test set whose objects are excluded from the main SkyScript dataset.It contains 70 classes, with 100 images per class except several rare classes.
  • Evaluation protocol: Benchmark partitions used for scene classification are not seen during CLIP continual pre-training, although training partitions are used when test images or labels are unavailable.This applies across the benchmark datasets listed in Table A1.
  • Fine-grained classification: The fine-grained test set evaluates roof shape, road smoothness, and road surface material using images collected through Google and Bing Maps APIs.Objects already included in SkyScript are excluded, and the examples are illustrated in Figure A1.
  • Scene classification: Increasing SkyScript’s training size improves average top-1 scene-classification accuracy despite adding lower-similarity image-text pairs.Table A2 compares SkyCLIP variants selected by descending pairwise similarity.
  • Cross-modal retrieval: SkyCLIP surpasses original CLIP and LAION-RS on most cross-modal retrieval metrics across RSICD, RSITMD, and UCM-caption.It outperforms previously trained-on benchmarks on RSICD and UCM-caption, while remaining comparable on RSITMD.
  • Unseen-class transfer: Removing ten semantic classes from training still yields average accuracies of 96.1% on MillionAID and 94.0% on PThe passage reports these values for unseen-class evaluation, but the second dataset name is truncated.
Loading 2312.12856v1…