PaperlayerView exact revision on arXiv

Source-linked AI summary

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, Jun Zhou

arXiv:2306.11029v4cs.CV

TL;DR

Remote sensing foundation models lack rich language understanding, while scarce pre-training data constrains vision-language specialization. RemoteCLIP addresses both issues by scaling heterogeneous annotations into image-caption data and incorporating UAV imagery, achieving strong downstream performance across retrieval and zero-shot tasks.

  • Problem

    Existing remote sensing foundation models primarily learn low-level features, lack language understanding, and face insufficient pre-training data for large vision-language models.

  • Method

    RemoteCLIP scales remote sensing data by converting segmentation masks and detection boxes into unified image-caption pairs, adding UAV imagery, and aligning image-text representations with InfoNCE loss.

  • Results

    RemoteCLIP achieves superior downstream performance, including state-of-the-art results across retrieval benchmarks and approximately 10% retrieval and 15% zero-shot improvements over prior pre-training techniques.

  • Takeaways & Limitations

    RemoteCLIP demonstrates that data-centric scaling can support general-purpose vision-language foundation modeling for remote sensing across multiple downstream benchmarks.

  • Takeaways & Limitations

    Rule-based annotation conversion limits caption diversity, and RemoteCLIP currently supports limited modality diversity beyond RGB imagery.

Abstract

from arXiv · show

General-purpose foundation models have led to recent breakthroughs in artificial intelligence. In remote sensing, self-supervised learning (SSL) and Masked Image Modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion. By further incorporating UAV imagery, we produce a 12 $\times$ larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, $\textit{k}$-NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images. Evaluation on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets. Project website: https://github.com/ChenDelong1999/RemoteCLIP

I. INTRODUCTION

Remote sensing foundation models have advanced through SSL and MIM, but often lack language alignment and require annotated fine-tuning. RemoteCLIP addresses these gaps with scaled image-caption pretraining and a vision-language foundation model for diverse downstream tasks.

  • MIM-based remote sensing foundation models primarily learn occlusion-invariant features, which may be less suited to unobstructed aerial imagery.
  • Existing remote sensing foundation models require annotated data and fine-tuning because they lack jointly aligned vision-language representations.
  • RemoteCLIP learns robust semantic visual features and aligned text embeddings for downstream tasks across domains.
  • 12× larger pretraining data are created by unifying heterogeneous annotations through M2C and B2C strategies and adding UAV imagery.
  • RemoteCLIP is evaluated on retrieval, classification, and object counting, including the new RemoteCount benchmark.

II. RELATED WORK

Related work spans SSL and MIM-based remote sensing foundation models, image-text retrieval systems, and CLIP-based vision-language models. These approaches establish the technical context for RemoteCLIP’s multimodal design.

  • Remote sensing foundation models mainly use SSL and MIM to learn visual representations from large satellite-image datasets.
  • Contrastive learning adaptations use spatial neighbors, rotations, or geographic information as remote-sensing-specific augmentations.
  • MIM-based models extend the framework with scale, temporal, and temporal-invariance properties, while also increasing model scale.
  • Remote sensing image-text retrieval began with CNN-LSTM systems and later incorporated dynamic fusion and multilingual language encoders.
  • CLIP aligns image and text representations through contrastive learning on large-scale Internet image-text pairs and supports broad vision-language applications.

III. REMOTECLIP

CLIP trains image and text encoders to align paired representations while separating mismatched examples. This alignment supports retrieval, grouping supports recognition, and their combination enables zero-shot classification.

  • InfoNCE aligns paired image-text samples and pushes apart mismatched samples.
  • CLIP encodes images and texts into latent representations and optimizes a bidirectional InfoNCE objective within each batch.
  • Representation alignment produces high similarity for paired samples and supports cross-modal retrieval performance.
  • Representation grouping clusters semantically similar unimodal samples and separates dissimilar ones, supporting recognition tasks such as linear classification.
  • Joint alignment and grouping, combined with sufficient open-set concepts, can support strong zero-shot classification.

1) Large CLIP is also a strong model for remote sensing tasks:

Large pretrained CLIP models are strong remote sensing retrieval models despite lacking domain-specific design. Larger models perform better, and large CLIP models outperform representative single-tower alternatives.

  • The evaluation covers RSITMD, RSICD, and UCM using CLIP models ranging from ResNet-50 to ViT-G-14.
  • Larger CLIP models consistently yield better remote sensing retrieval performance than smaller models.
  • ViT-G-14 zero-shot retrieval surpasses previous remote-sensing-specific methods except Rahhal et al.’s fine-tuned CLIP model.
  • Large CLIP models outperform ALBEF and BLIP by a large margin on the evaluated retrieval benchmarks.

2) Continual pretraining on small CLIP further improves the performance:

Continual pretraining adapts CLIP to remote sensing, but small image-text datasets limit larger models and motivate annotation unification for scalable training.

  • Continual pretraining on remote sensing image-text data improved CLIP performance and established state-of-the-art results on three retrieval benchmarks.
  • 4.6% improvement was achieved over a model using the same ViT-Base-32 architecture through continual pretraining on the RET-3 collection.
  • ViT-Large-14 exhibited severe over-fitting because the RET-3 dataset contains only 13k samples, far fewer than typical CLIP pretraining datasets.
  • Crowdsourcing additional captions is expensive, and its annotation quality and diversity are difficult to guarantee.
  • Annotation unification scales training by converting bounding-box and segmentation annotations into image-caption data through B2C and M2B.

1) Box-to-Caption (B2C) Generation:

B2C converts object-detection annotations into varied natural-language captions that encode object locations, categories, and counts.

  • B2C uses a rule-based approach to generate five distinct captions from object-detection bounding boxes and labels.
  • The first two captions describe whether objects are located at the image center or outside it.
  • Three additional captions describe the number of different object categories represented in the image.
  • When an object appears more than ten times, B2C substitutes general terms such as “many” or “a lot of” for exact counts.
  • Segmentation masks are converted into bounding boxes before B2C generates corresponding captions.

3) Sample De-duplication:

The dataset combines retrieval, detection, and segmentation sources, removes duplicate images, and analyzes the resulting data distribution.

  • RemoteCLIP removes duplicate images using p-Hash-based blockwise local detection to reduce possible test-set contamination.
  • 40 to 3k duplicate samples are removed across different datasets.
  • The final dataset combines RET-3 retrieval data, DET-10 detection data, and SEG-4 segmentation data translated through M2B and B2C.
  • DET-10 provides diverse satellite and UAV imagery with average object counts ranging from 1 to 70 per image.
  • The data-scaling approach produces a diverse sample distribution associated with strong performance across downstream domains.

IV. EXPERIMENTS

RemoteCLIP is implemented at three visual-backbone scales with augmentation and standard optimization choices for training on remote sensing image-text data.

  • The model uses ResNet-50, ViT-Base-32, and ViT-Large-14 backbones with 38M, 87M, and 304M parameters, respectively.
  • The training dataset contains 165,745 images paired with five captions each, yielding 828,725 image-text pairs.
  • Data augmentation uses random crops, horizontal flips, and rotations of 0°, 90°, 180°, and 270°.
  • Training uses automatic mixed precision, Adam optimization, linear warm-up, and a cosine learning-rate scheduler.

B. Benchmarking RemoteCLIP

RemoteCLIP is evaluated on cross-modal retrieval and zero-shot object counting, with results indicating strong retrieval performance and improved counting robustness over CLIP.

  • Cross-modal Retrieval: RemoteCLIP achieves state-of-the-art performance on all three remote sensing retrieval benchmarks.The benchmarks are RSITMD, RSICD, and UCM, without dataset-specific fine-tuning or re-ranking.
  • Cross-modal Retrieval: 9.14% and 8.92% are RemoteCLIP’s margins over the previous state-of-the-art method on RSITMD and RSICD, respectively.These margins are reported for the ViT-Large-14 RemoteCLIP model.
  • Cross-modal Retrieval: RemoteCLIP outperforms CLIP-CL on all three retrieval benchmarks, supporting the effectiveness of data scaling.CLIP-CL uses only the existing RET-3 data.
  • Object Counting: RemoteCount evaluates zero-shot counting from 1 to 10 using 947 image-text pairs across 13 remote sensing categories.The benchmark is mainly selected from the DOTA validation set and manually verified.
  • Object Counting: RemoteCLIP produces higher counting accuracy than CLIP, with a clear confusion-matrix diagonal and an advantage maintained through top-6 accuracy.It is also more robust when number words are replaced by digits.

3) Zero-shot Image Classification:

RemoteCLIP improves zero-shot classification over CLIP across 12 remote sensing datasets, although performance remains weaker on some datasets with substantial distribution differences.

  • Zero-shot Image Classification: The zero-shot evaluation uses template-based prompts across 12 remote sensing classification datasets.The example prompt is “a satellite photo of {class name}.”
  • Zero-shot Image Classification: +2.85%, +6.39%, and +5.63% are RemoteCLIP’s average zero-shot accuracy improvements over CLIP across 12 datasets with three backbones.The ViT-Large-14 model outperforms its CLIP counterpart on 9 of 12 datasets.
  • Zero-shot Image Classification: RemoteCLIP’s zero-shot performance is consistently inferior to CLIP on some datasets.The authors associate this with domain gaps involving image resolution and image-distribution differences.

5) Full-shot Linear Probing and k-NN Classification:

RemoteCLIP provides strong image representations for linear probing and k-NN classification, while ablations identify the importance of pretrained backbones, richer captions, rotation preprocessing, and InfoNCE loss.

  • Few-shot Classification: On the 32-shot setting, RemoteCLIP outperforms all compared baselines on all 12 remote sensing classification datasets.The compared models include CLIP, self-supervised visual models, ImageNet-pretrained models, ViTAE, and SatMAE.
  • Full-shot Linear Probing and k-NN Classification: RemoteCLIP outperforms CLIP and other self-supervised models in linear probing and k-NN classification.The evaluation uses the same 12 remote sensing classification datasets as the zero-shot and few-shot experiments.
  • Ablation Study: The best backbone-ablation result uses pretrained image and text backbones, with image-backbone pretraining more significant than text-backbone pretraining.The ablation is conducted on the Ret-3 + Det-10 + Seg-4 dataset.
  • Ablation Study: Approximately 10% and 15% improvements are observed in retrieval and zero-shot tasks, respectively, for RemoteCLIP over prior pre-training techniques.These comparisons are reported in the pre-training model ablation.
  • Ablation Study: The Ret-3 + Det-10 + Seg-4 dataset, rotation preprocessing, and InfoNCE loss produce the strongest reported ablation outcomes.The dataset ablation links performance to richer textual information, while the loss ablation reports stronger semantic feature representations with InfoNCE.

D. Feature Visualization

RemoteCLIP uses image–text similarity to expose semantic information in high-resolution remote sensing imagery. The paper concludes that data-centric scaling supports strong downstream performance, while model size, data scale, and caption diversity remain constraints.

  • D. Feature Visualization: 64×64 patches from Potsdam, Vaihingen, and iSAID are compared with text prompts such as “a {target class name}” using cosine similarity.The visualization uses overlapping crops to map semantic similarity between patch-level visual and textual features.
  • Conclusion: RemoteCLIP is presented as a general-purpose vision-language foundation model for remote sensing.Its stated goal is to learn robust visual features with rich semantics and aligned text embeddings.
  • Conclusion: Data-centric scaling produced superior downstream results and a series of state-of-the-art benchmark performances.The authors connect this outcome to the importance of data-centric methodology for developing foundation models.
  • Limitations: 304M parameters and 400M training examples leave room for further scaling of the largest RemoteCLIP model.The authors note that larger remote sensing models, including a 2B-parameter example, indicate additional scaling potential.
  • Limitations: Scaling model size requires simultaneous data expansion, because the current dataset may be insufficient for much larger models.Future expansion targets weakly labeled classification data and unlabeled data generated through pseudo-labeling.
  • Limitations: Rule-based B2C and M2B conversion limits caption diversity, while RemoteCLIP currently includes limited modality diversity.The authors propose generative language models and additional sensory modalities as future directions.
Loading 2306.11029v4…