PaperlayerView exact revision on arXiv

Source-linked AI summary

RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Zilun Zhang, Tiancheng Zhao, Yulong Guo, Jianwei Yin

arXiv:2306.11300v5cs.CVcs.AIcs.CLcs.MM

TL;DR

The paper tackles the lack of large-scale remote-sensing image-text data and the challenge of adapting general VLMs to remote-sensing tasks. It introduces the DVLM framework and constructs RS5M by filtering paired datasets and captioning label-only remote-sensing datasets. GeoRSCLIP trained on RS5M improves performance across zero-shot classification, cross-modal retrieval, and semantic localization.

  • Problem

    General VLMs are trained on common objects, while remote-sensing transfer is limited by small paired datasets and large datasets with only class-level labels.

  • Method

    The paper develops a DVLM framework and constructs RS5M by filtering public image-text pairs and generating captions for large label-only remote-sensing datasets.

  • Results

    GeoRSCLIP improves over baseline or state-of-the-art models by 3%–20% in ZSC, 3%–6% in RSCTIR, and 4%–5% in SeLo.

  • Takeaways & Limitations

    RS5M and the DVLM framework successfully transfer pre-trained VLMs to the remote-sensing domain across several downstream tasks.

  • Takeaways & Limitations

    The current PEFT and full fine-tuning methods do not account for interaction between image and text modalities, motivating more complex DVLMs.

Abstract

from arXiv · show

Pre-trained Vision-Language Models (VLMs) utilizing extensive image-text paired data have demonstrated unprecedented image-text association capabilities, achieving remarkable results across various downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. In this paper, we propose a new framework that includes the Domain pre-trained Vision-Language Model (DVLM), bridging the gap between the General Vision-Language Model (GVLM) and domain-specific downstream tasks. Moreover, we present an image-text paired dataset in the field of remote sensing (RS), RS5M, which has 5 million RS images with English descriptions. The dataset is obtained from filtering publicly available image-text paired datasets and captioning label-only RS datasets with pre-trained VLM. These constitute the first large-scale RS image-text paired dataset. Additionally, we fine-tuned the CLIP model and tried several Parameter-Efficient Fine-Tuning methods on RS5M to implement the DVLM. Experimental results show that our proposed dataset is highly effective for various tasks, and our model GeoRSCLIP improves upon the baseline or previous state-of-the-art model by $3\%\sim20\%$ in Zero-shot Classification (ZSC), $3\%\sim6\%$ in Remote Sensing Cross-Modal Text-Image Retrieval (RSCTIR) and $4\%\sim5\%$ in Semantic Localization (SeLo) tasks. Dataset and models have been released in: \url{https://github.com/om-ai-lab/RS5M}.

I. INTRODUCTION

The paper addresses the limited scale of remote-sensing image-text data and the challenge of adapting general vision-language models to domain-specific tasks. It introduces the DVLM framework and RS5M dataset to transfer general knowledge with remote-sensing priors.

  • Motivation: Remote-sensing applications are constrained by expensive expert labeling and small-scale labeled datasets.Paired text offers natural supervision and can represent modalities such as SAR, hyperspectral, and multi-satellite imagery.
  • Framework: DVLM connects a GVLM with domain-specific data and downstream task models, transferring general knowledge enriched with remote-sensing prior knowledge.The framework positions DVLM between the general model and downstream task model.
  • Motivation: Existing remote-sensing image-text datasets are too small for effectively transferring or fine-tuning large-scale VLMs.Large remote-sensing datasets often provide only class-level labels, leaving a shortage of richly paired image-text data.
  • Contributions: RS5M is introduced as the first large-scale remote-sensing image-text paired dataset, built by filtering paired datasets and captioning label-only datasets.The dataset construction combines publicly available image-text data with automatically generated descriptions for remote-sensing images.
  • Results: GeoRSCLIP improves over baseline or state-of-the-art models by 3%–20% in ZSC, 3%–6% in RSCTIR, and 4%–5% in SeLo.These results support the framework and dataset across three remote-sensing vision-language tasks.

A. Filter Large-Scale Image-Text Paird Datasets

RS5M combines filtered public image-text pairs with captioned remote-sensing datasets, using deduplication, model-based filtering, and caption-quality selection. The pipeline also applies rotation-invariant caption selection to reduce hallucinated descriptions while retaining image-content alignment.

  • Source collection: PUB11 is formed from 11 public image-text datasets, yielding 3 million image-text pairs dominated by aerial imagery.The collection includes both aerial-view and satellite-view images.
  • Filtering: The filtering pipeline uses remote-sensing keywords, invalid-image checks, URL deduplication, and clustering to remove duplicate images.For each duplicate cluster, one image is retained and the others are discarded.
  • Filtering: A CLIP similarity score and an RS-image detector jointly filter image-text pairs, retaining pairs above thresholds corresponding to the top 90% similarity and top 80% detector scores.The detector estimates the probability that an image is a remote-sensing image.
  • Captioning remote-sensing datasets: BLIP2 generates captions for BigEarthNet, FMoW, and MillionAID, producing 2,062,377 images for the RS3 subset.The selected data use training images from FMoW and BigEarthNet and test images from MillionAID.
  • Caption selection: The captioning process generates and reranks candidate captions, integrates geospatial metadata, and applies a rotation-invariant criterion across 12 image rotations.The selected captions tend to be broader and exclude hallucinated descriptions, though detailed content may sometimes be omitted.

IV. DATASET DESCRIPTION

RS5M captions show strong aerial and satellite-oriented content, while the dataset’s geographic coverage is uneven across UTM zones. Location information extracted from captions supplements geolocation analysis and supports potential GeoAI uses.

  • “Aerial view” is predominant in RS5M captions, with “satellite,” “field,” “building,” “road,” and “farm” also frequent.
  • RS5M has a long-tailed distribution of images per UTM zone, with sparse coverage in several African, Australian, Asian, American, Canadian, and Russian regions.High image density is observed in North America, Europe, and most parts of Asia and South America.
  • The geographic analysis uses geolocation information from 1,079,370 images drawn from Fmow, BEN, and YFCC.
  • Named-entity recognition found location information in 880,354 captions, mostly as country or city names rather than UTM zones or latitude/longitude.City names can be converted to UTM zones, whereas country-only captions provide coarser spatial information.
  • RS5M captions include real-world geographic and seasonal information that could help GeoAI tasks such as geolocation and GeoQA.

A. Experiment Setup

The experiments evaluate RS5M-tuned domain vision-language models across zero-shot classification, cross-modal retrieval, and semantic localization. Results compare untuned CLIP, supervised approaches, full fine-tuning, and parameter-efficient adapters.

  • The study evaluates domain generalizability on zero-shot classification, remote sensing cross-modal text–image retrieval, and semantic localization.
  • Evaluation Tasks: Zero-shot classification selects the most relevant category for an image, with accuracy as the evaluation metric.
  • Evaluation Tasks: Remote sensing cross-modal retrieval retrieves paired images from text or paired text from images, using Recall@1/5/10 and mean recall.
  • Evaluation Tasks: Semantic localization uses cross-modal information to locate semantically similar regions in large-scale remote-sensing scenes without bounding-box training annotations.
  • Models: The DVLM implementations use full fine-tuning or LoRA, Pfeiffer, Prefix-tuning, and UniPELT adapters, with models tuned on RS5M named GeoRSCLIP.
  • Main Results: Fine-tuned CLIP models generally outperform the baseline across zero-shot classification and semantic localization, while RS5M-tuned models generally improve retrieval over most baselines and supervised methods.
  • Efficiency: PEFT methods reduce hyperparameter-tuning requirements and computational overhead but do not consistently match comprehensive full fine-tuning.
  • Main Results: GeoRSCLIP retains strong retrieval performance when further tuned on RSICD, RSITMD, or RET-2 training data, achieving state-of-the-art results on each retrieval task.

C. Ablation Study

The ablation study compares RS5M’s PUB11, RS3, and Geometa subsets and examines how their composition affects downstream performance. The full RS5M dataset generally performs best across classification, localization, and retrieval.

  • Per Subset Analysis: RS5M consists of PUB11 aerial-image data, RS3 satellite-image data, and a Geometa subset containing 1,036,734 geo-information-rich image-text pairs.Geometa data include class labels, image locations, and acquisition dates.
  • Per Subset Analysis: RS5M-trained models outperform models trained on PUB11, RS3, or Geometa across most zero-shot classification, semantic localization, and image-text retrieval comparisons.
  • Per Subset Analysis: RS5M leads in most semantic-localization metrics, including Rsu, Rda, and Rmi.
  • Per Subset Analysis: RS5M consistently performs best across image-to-text and text-to-image retrieval Recall metrics.
  • Interpretation: The authors attribute RS5M’s robustness to combining PUB11’s zero-shot classification strength with RS3’s retrieval capability.
  • Interpretation: Geometa underperforms, possibly because its limited prompt-template diversity provides insufficient linguistic variation for CLIP training.
  • Interpretation: Generated captions and web data address the linguistic-diversity issue associated with captions based solely on geographic information.

2) Influence of RS5M Scale:

Increasing RS5M’s training size consistently improves performance across zero-shot classification, retrieval, and semantic localization. Additional analyses find only marginal effects from normalization choices and examine PUB11 noise levels.

  • Influence of RS5M Scale: Models trained on the full RS5M dataset and progressively larger subsets show consistent performance gains as training data increases.The comparison uses the full dataset and random subsets of one-half, one-quarter, and one-eighth of RS5M.
  • Influence of RS5M Scale: Performance improves with dataset size across AID, RESISC45, and EuroSAT zero-shot accuracy and RSITMD and RSICD recall metrics.
  • Influence of RS5M Scale: Semantic localization’s Rmi metric also improves significantly as the training-data volume increases.
  • Normalization: RS5M normalization parameters provide a slight advantage, but differences among normalization settings are extremely marginal.
  • Noise Level: PUB11 noise levels are varied by changing VLM-filter and RS-image-detector thresholds before evaluating fine-tuned CLIP on ZSC, VLR, and SeLo.

5) Influence of Freezing Encoder:

Freezing-encoder and scaling experiments examine how CLIP components and model size affect remote-sensing downstream performance. Jointly training both encoders generally performs best for classification and retrieval, while larger models improve most metrics but not semantic localization’s Rmi indicator.

  • 5) Influence of Freezing Encoder:: Non-frozen encoders perform best on RESISC45 and EuroSAT, indicating cooperative learning between text and image encoders.
  • 5) Influence of Freezing Encoder:: Across RSITMD and RSICD, the non-frozen model outperforms frozen variants on all recall metrics, while freezing the image encoder causes the larger drop.
  • 6) Influence of Model size:: Increasing model size generally improves classification and retrieval metrics after RS5M fine-tuning.The comparison uses CLIP ViT-B-32, ViT-B-16, ViT-L-14, and ViT-H-14 encoders.
  • 6) Influence of Model size:: SeLo’s Rmi indicator shows little variation with model size, suggesting task-specific limits to gains from increased model complexity.
  • Influence of Batch Size: Batch-size experiments with ViT-B-32 evaluate sizes from 64 to 700 and report generally improving performance as batch size increases.
  • VII. CONCLUSION, LIMITATION, AND FUTURE WORK: The framework remains limited by the lack of image-text interaction modeling in both PEFT and full fine-tuning, motivating more complex DVLMs.

APPENDIX

The appendix describes established remote-sensing image-text and image-only datasets alongside general vision-language models and parameter-efficient tuning methods. These resources provide context for the paper’s dataset construction, model choices, and retrieval or vision-language applications.

  • Remote-sensing image-text datasets: UCM Captions contains 2,100 RGB aerial images from 21 classes, with five captions per image and 2,032 unique captions.Images are 256 × 256 pixels at 1 ft spatial resolution.
  • Remote-sensing image-text datasets: Sydney Captions contains 613 RGB images from Sydney, with 3,065 captions and 1,109 non-duplicate captions.Images are 500 × 500 at 1 ft spatial resolution.
  • Remote-sensing image-text datasets: RSICD contains 10,921 remote-sensing RGB images, each annotated with five captions, but its language is described as repetitive and low-detail.
  • Remote-sensing image-text datasets: RSITMD is designed for fine-grained image-text matching, with detailed relational captions and one-to-five keyword attributes per image.It contains 23,715 captions for 4,743 images across 32 scenes.
  • Vision-language models: CLIP aligns visual and textual information in a shared semantic space through contrastive learning, with performance linked to large-scale data.The related-work discussion also describes image-text matching, masked language modeling, and other VLM objectives.
  • Parameter-efficient tuning: Adapters freeze pretrained weights while training substantially fewer adapter parameters, offering a more efficient alternative to full fine-tuning.

6) Parameter-Efficient Tuning for Vision-Language Models:

The section surveys parameter-efficient adaptation strategies for vision-language models and related remote-sensing pretraining. It covers adapters, prompt learning, remote-sensing pretraining findings, and large image-text sources used in dataset construction.

  • Adapters: CLIP-Adapter adds a two-layer bottleneck MLP with residual connections to the text and vision encoders.
  • Adapters: Tip-Adapter performs training-free CLIP adaptation by constructing a key-value cache from few-shot training examples.
  • Prompt-based learning: CoOp learns prompt tokens in the text encoder, while CoCoOp generates image-conditional tokens through a lightweight neural network.
  • Remote-sensing pretraining: MillionAID pretraining improved scene recognition but showed limitations on other remote-sensing tasks in the cited study.
  • Remote-sensing pretraining: Self-supervised remote-sensing models use unlabeled, geographically or temporally related imagery to learn transferable representations.
  • Large-scale image-text data: The dataset-construction sources include large public collections such as LAION2B-en, LAION400M, LAIONCOCO10, COYO700M, CC3M, CC12M, and YFCC15M.

4) Remote Sensing Binary Classification Dataset:

The paper constructs and filters remote-sensing data using binary classification, duplicate and invalid-image removal, VLM filtering, and captioning. The resulting collections are predominantly aerial or satellite imagery, with caption quality varying across models.

  • Binary classification dataset: A binary classifier trained on 2,500 MillionAID satellite images, 2,500 LAION2B aerial images, and 5,000 ImageNet-1k negatives reaches 97.55% test accuracy.The data are split 7:1:2 into train, validation, and test sets.
  • Dataset filtering: Invalid-image checking removes files that cannot be opened or have zero size, while duplicate filtering uses URL and fastdup-based processing.
  • Captioning: CoCa and BLIP2 generate meaningful remote-sensing captions, whereas BLIP-Large and GIT-Large can produce repetitive or nonsensical captions.Examples include recognition of churches, airports, and farms in satellite views.
  • Captioning: The captioning examples include beam-search and nucleus-sampling outputs for airport imagery, with both satellite and aerial descriptions.

8) Rotational Invariance:

RS5M enriches generated captions with structured geographic, temporal, image, and dataset metadata, then evaluates caption quality through a five-level, three-perspective rating system. The resulting captions receive strong relevance, detail, and hallucination scores.

  • Caption construction: RS5M captions combine model-generated descriptions with metadata such as location, season, timestamps, class labels, and ground sample distance.The metadata is integrated into readable sentences to provide a more comprehensive image description.
  • Caption construction: Created UTC timestamps from one source were excluded because they represented blog creation rather than image capture.This preserves the distinction between publication metadata and acquisition metadata.
  • Caption construction: The dataset records metadata coverage per source dataset, including geographic, temporal, class-label, and image-specific information.Some sources also include UTM zones, cloud cover, scan direction, target azimuth, and off-nadir information.
  • Caption evaluation: The five-level rating system evaluates captions from three perspectives: relevance and detail, hallucination, and fluency and conciseness.Scores range from 1, indicating poor quality, to 5, indicating excellent quality.
  • Caption evaluation: 4.53/5.00 relevance and detail and 4.73/5.00 hallucination scores indicate good overall caption quality after adding metadata.The FMoW subset achieves relevance and detail of 4.85 and hallucination of 4.87, with the lowest reported standard deviations.

12) Visualization:

The visualization presents sampled PUB11 image-text pairs across satellite, aerial, and outlier categories, alongside image-dimension statistics. The collection is predominantly aerial-view imagery and includes varied image sizes.

  • Image-text examples: PUB11 visualizations divide sampled image-text pairs into satellite-view, aerial-view, and outlier groups.The satellite examples occupy the left four columns and aerial examples the middle four columns of Figure 21.
  • Image-text examples: Some PUB11 captions are not highly informative, but they remain related to their corresponding images.Longer captions were truncated for visual presentation.
  • Image-text examples: Outlier examples include meteorological satellite images, illustrative figures, space images, artwork, and a town model photograph.These examples have connections to remote sensing but do not strictly belong to conventional aerial or satellite imagery.
  • Image statistics: PUB11 image dimensions average 402.87 pixels in height and 522.53 pixels in width.Figure 22 reports the width and height statistics for the collection.

13) Outlier & Misfiltered Image Analysis:

The analysis quantifies residual outliers and over-filtered RS images, while also documenting dataset licensing and the RS-SD generative-model extension. Filtering errors persist, but RS-SD improves remote-sensing image generation over vanilla Stable Diffusion.

  • Outlier analysis: 0.8% of sampled RS5M images are outliers, while 3.4% of the removed collection consists of remote-sensing images.Outliers are mainly maps, illustrations, and weather imagery; misfiltered RS images are mainly low-altitude captures.
  • Outlier analysis: The confusion-matrix analysis samples 5,000 images from RS5M and 5,000 from the removed collection to examine filtering errors.These samples represent 0.1% of RS5M and 0.5% of the removed collection.
  • License: RS5M redistribution is limited by source licensing, with planned academic-only use and possible removal of data raising rights concerns.The authors plan to release metadata first for PUB11 and metadata plus image-text archives for eligible RS3 sources.
  • Stable Diffusion: RS-SD is Stable Diffusion tuned with 1% of RS5M because training from scratch on 5M pairs was impractical.The tuning uses DreamBooth, 512-pixel resolution, batch size 50, and 50,000 steps, with the text encoder also trained.
  • Stable Diffusion: FID improves from 36.86 for vanilla Stable Diffusion to 28.32 for RS-SD on remote-sensing generation.RS-SD also produces more realistic, caption-aligned satellite and aerial imagery in qualitative comparisons.
Loading 2306.11300v5…