PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models in Remote Sensing: Current Progress and Future Trends

Xiang Li, Congcong Wen, Yuan Hu, Zhenghang Yuan, Xiao Xiang Zhu

arXiv:2305.05726v2cs.CVcs.AI

TL;DR

Remote-sensing VLM research is promising but remains limited by dataset scale, model architectures, and training cost. This paper reviews applications and progress across RS tasks, identifies broad advances over purely visual models, and outlines future research needs.

  • Problem

    Remote-sensing VLM research remains limited, with restricted datasets and insufficiently explored large-scale visual and language models.

  • Method

    The paper comprehensively reviews VLM applications and recent advances in remote sensing, including methods spanning multiple RS analysis tasks.

  • Results

    VLM studies have demonstrated superiority over purely visual models across RS tasks including captioning, generation, retrieval, question answering, classification, segmentation, and detection.

  • Takeaways & Limitations

    The review provides an in-depth understanding of the rapidly growing RS VLM field and encourages further investigation.

  • Takeaways & Limitations

    Current RS VLM research uses limited datasets, often relies on classical CNN and RNN encoders, and faces substantial computational requirements for training large models.

Abstract

from arXiv · show

The remarkable achievements of ChatGPT and GPT-4 have sparked a wave of interest and research in the field of large language models for Artificial General Intelligence (AGI). These models provide intelligent solutions close to human thinking, enabling us to use general artificial intelligence to solve problems in various applications. However, in remote sensing (RS), the scientific literature on the implementation of AGI remains relatively scant. Existing AI-related research in remote sensing primarily focuses on visual understanding tasks while neglecting the semantic understanding of the objects and their relationships. This is where vision-language models excel, as they enable reasoning about images and their associated textual descriptions, allowing for a deeper understanding of the underlying semantics. Vision-language models can go beyond visual recognition of RS images, model semantic relationships, and generate natural language descriptions of the image. This makes them better suited for tasks requiring visual and textual understanding, such as image captioning, and visual question answering. This paper provides a comprehensive review of the research on vision-language models in remote sensing, summarizing the latest progress, highlighting challenges, and identifying potential research opportunities.

I. INTRODUCTION

Remote-sensing research has largely emphasized visual understanding, while vision-language models add semantic reasoning over images, text, and object relationships. The paper reviews this evolution and identifies challenges and future directions for VLMs in RS.

  • Motivation: Remote-sensing deep learning primarily uses visual features for tasks such as object detection, semantic segmentation, and change detection.CNNs learn hierarchical representations and spatial patterns, while attention mechanisms focus models on specific input regions.
  • Motivation: VLMs jointly understand visual and textual information, including semantic patterns and relationships between objects.They can generate natural-language descriptions rather than only recognize image objects.
  • Motivation: VLMs support tasks requiring visual and textual understanding, including image captioning, text-based image retrieval, and visual question answering.Their capabilities also extend to image understanding, generation, segmentation, and detection.
  • Motivation: In remote sensing, VLMs combine visual understanding, world knowledge, and reasoning capabilities for more intelligent data analysis.They offer capabilities beyond earlier vision-only models, which primarily rely on supervised learning and face challenges with out-of-distribution data.
  • Scope and contribution: The paper provides a comprehensive review of VLM evolution in RS, surveys recent advances, and recommends future research directions.Its stated goal is to help researchers understand the field’s state of the art and explore VLMs for RS tasks.

A. Vision-Centric Models

Vision-centric models progressed from CNNs to transformer-based vision models, while language-model research developed large transformer architectures such as GPT and BERT. These developments provide the foundations for multimodal modeling.

  • Vision-centric models: CNNs extract high-level image features through convolution, pooling, and nonlinear activation, and are trained by backpropagation to minimize prediction error.Their architectures evolved through milestones including LeNet-5, AlexNet, VGG, GoogLeNet, and later designs.
  • Vision transformers: Vision transformers use self-attention to model global dependencies between different image regions.ViT was the first transformer-based architecture for image classification, followed by variants such as DeiT, TNT, PVT, and Swin.
  • Vision transformers: Self-attention computes weights from query-key dot products, applies scaling and softmax normalization, and multiplies the resulting weights by value features.This produces the final output features for the model.
  • Large language models: LLMs use large-scale transformer networks for language understanding and achieve strong performance in language modeling, text generation, and question answering.GPT and BERT are presented as major developments in this model family.
  • Large language models: GPT maximizes the likelihood of each token conditioned on preceding tokens within a context window.GPT-2, GPT-3, and later variants expanded capabilities across translation, summarization, question answering, text completion, zero-shot learning, and in-context learning.
  • Large language models: BERT is pretrained bidirectionally and uses masked language modeling alongside other unsupervised objectives.Its descendants improve data use, parameterization, training speed, or masking strategies.

C. Vision-Language Models

Vision-language models pretrain or encode visual and textual modalities through fusion-encoder or dual-encoder architectures. Their designs either jointly fuse image-text representations or align separately encoded modalities.

  • VLM architectures: VLMs incorporate both visual and textual modalities and are categorized into fusion-encoder and dual-encoder architectures.The two categories differ in whether image and text are jointly or separately encoded.
  • Fusion Encoder: Fusion encoders jointly encode image-text pairs with cross-modal Transformers to fuse visual and textual representations.Self-attention or cross-attention produces a fused latent representation.
  • Dual Encoder: Dual encoders encode images and text separately, then model cross-modal interactions using a dot product or multilayer perceptron.ALIGN is described as using contrastive loss to align matched image-text pairs and separate nonmatched pairs.

1) Fusion Encoder:

Fusion encoders jointly process visual features and text embeddings to model cross-modal interactions. Related architectures use self-attention or cross-attention to align visual and linguistic information.

  • Fusion Encoder:: Fusion encoders accept visual features and text embeddings, then apply fusion techniques to capture interactions between modalities.
  • Fusion Encoder:: VisualBERT combines BERT-based language processing with Faster-RCNN object proposals and jointly feeds text and image features through Transformer layers.
  • Fusion Encoder:: Dual-stream models process visual and textual inputs separately, using co-attentional Transformer layers for bidirectional information exchange and semantic alignment.
  • Fusion Encoder:: Dual encoders independently encode each modality and project image and text embeddings into a shared semantic space for similarity calculation.
  • Fusion Encoder:: CLIP uses symmetric InfoNCE losses to optimize image-to-text and text-to-image matching, averaging the two directional losses.

III. VISION-LANGUAGE MODELS IN REMOTE SENSING

Remote sensing vision-language research builds on foundation models and vision-centric architectures, with applications spanning multiple RS tasks. The review organizes representative models, datasets, and task categories while emphasizing the need for RS-tailored foundation models.

  • A. Vision-Centric Foundation Models: Foundation models are task-agnostic networks pretrained on massive datasets and adapted to downstream tasks through fine-tuning, few-shot, or zero-shot learning.
  • A. Vision-Centric Foundation Models: Representative RS models include RingMo, Wang et al.’s model, GFM, and related systems trained on datasets such as MillionAID and GeoPile.
  • A. Vision-Centric Foundation Models: The review’s model summary covers foundation models, image captioning, text-based image generation, visual question answering, grounding, classification, detection, and segmentation.
  • A. Vision-Centric Foundation Models: Designing a foundation model tailored for remote sensing data is presented as necessary for RS applications.
  • A. Vision-Centric Foundation Models: Supervised pretraining can improve downstream RS performance, but its dependence on substantial labeled data remains a hurdle for training larger models.

B. Image Captioning

Remote sensing image captioning requires understanding objects, attributes, scales, and spatial relationships before generating comprehensive natural-language descriptions. Existing methods show strong performance, but dataset scale and caption detail remain important constraints.

  • B. Image Captioning: RS image captioning generates natural-language descriptions that capture multiscale ground elements, their attributes, and their spatial relationships.
  • B. Image Captioning: Early methods decomposed captioning into key instance detection, environment analysis, and landscape analysis before integrating the results into descriptions.
  • B. Image Captioning: Later approaches used visual-text alignment, explainable word–sentence generation, truncation cross-entropy, structured attention, and multiscale encoding.
  • B. Image Captioning: RSGPT outperformed all compared methods across the reported BLEU, METEOR, ROUGE-L, and CIDEr-D evaluations.
  • B. Image Captioning: The commonly used UCM-caption, Sydney-caption, and RSICD datasets contain 2100, 613, and 10921 images, respectively.
  • B. Image Captioning: Except for RSICap, existing datasets mainly contain short captions, motivating large-scale datasets with detailed descriptions.

C. Text-based Image Generation

Text-based generation and retrieval extend remote sensing analysis by linking textual descriptions with image synthesis and search. Reviewed methods address synthetic data creation, semantic encoding, multimodal retrieval, and cross-modal matching.

  • C. Text-based Image Generation: Text-based RS image generation can support urban-planning visualization and the creation of synthetic labeled datasets to alleviate shortages of labeled samples.
  • C. Text-based Image Generation: GAN-based studies synthesized RS images from textual descriptions by transforming text representations into image pixel characteristics.
  • C. Text-based Image Generation: Txt2Img-MHN applies hierarchical prototype learning to text-image embeddings for coarseness, finesse, and richer semantic encoding.
  • C. Text-based Image Generation: Txt2Img-MHN achieved an Inception Score of 5.99 and an FID Score of 102.44 on the RSICD test dataset.
  • C. Text-based Image Generation: Text-based image retrieval targets efficient management of large RS collections by extracting images that match textual queries.
  • C. Text-based Image Generation: Retrieval methods use bidirectional triplet learning, local-global dynamic fusion, multilingual encoders, semantic alignment, and multiscale redundancy filtering.
  • C. Text-based Image Generation: One crossmodal method reported a slight retrieval improvement across datasets over the previous method using contrastive image-to-text and text-to-image losses.

E. Visual Question Answering

Remote sensing visual question answering has progressed from CNN–LSTM systems and benchmark datasets toward transformer- and language-model-based methods, including open-ended and change-detection applications. Despite strong overall results, object counting remains weaker than presence and classification.

  • Benchmarks and early methods: RSIVQA contains around 37k images, and its mutual-attention bilinear-fusion network achieved 77% overall accuracy.
  • Transformer-based methods: Vision-language transformers support open-ended VQA by encoding images and questions jointly, then using a transformer decoder with cross-attention.
  • Transformer-based methods: Chappuis et al. achieved 85% accuracy on low-resolution RSVQA and 85% and 81% on the two high-resolution test sets by converting image context into text prompts.
  • Emerging applications: Change-detection VQA extends the task to multitemporal aerial images using the CDVQA dataset, which contains 4,662 image pairs with semantic change maps.
  • Results and limitations: RSGPT outperformed compared methods on RSVQA-HR and RSVQA-LR, while existing methods show a performance disparity for object counting versus presence and classification.

F. Visual Grounding

Remote sensing visual grounding localizes queried objects in images, but remains under-explored because expressions involve complex geospatial relations and targets may be visually subtle. Transformer-based methods address scale variation and clutter, while zero-shot scene classification uses semantic relationships to recognize unseen classes.

  • Visual grounding: RSVG uses a remote sensing image and query expression to predict the bounding box of the specified object, with research still in a nascent stage.
  • Visual grounding: DIOR-RSVG contains 17,402 images and 38,320 referring expressions across 20 object categories, with average expression length 7.47.
  • Visual grounding: MGVML achieved the best mIoU of 68.04% on DIOR-RSVG by combining multi-scale visual and multi-granularity language features with noise filtering.
  • Visual grounding: Current RSVG performance still has room for improvement relative to conventional object detection methods, motivating further accuracy research.
  • Zero-shot scene classification: Zero-shot RSSC recognizes unseen scene concepts using visual features and semantic relationships between classes.
  • Zero-shot scene classification: A GAN-based zero-shot RSSC method synthesized image features from class semantics and achieved 62.66%, 55.86%, and 50.66% accuracy on UCM21, AID30, and NWPU45.

H. Few-Shot Object Detection.

Few-shot object detection in remote sensing has progressed through meta-learning and finetuning approaches, while vision-language methods use textual semantics to improve detection of novel classes. Related few-shot segmentation and open-vocabulary work points to broader opportunities for language-guided remote sensing analysis.

  • Few-Shot Object Detection: Few-shot object detection identifies objects from only a few annotated examples and commonly uses meta-learning or finetuning.Meta-learning learns task-agnostic knowledge across episodes, whereas finetuning adapts a base-class detector to novel classes.
  • Vision-Language FSOD: 0.77 and 0.49 mAp were achieved by TSF-RGR on 10 novel classes of NWPU VHR-10 and DIOR, respectively.The method uses region language descriptions encoding object attributes and relations; text-based semantic relations improved few-shot detection performance.
  • Vision-Language FSOD: 0.75 and 0.43 mAp were achieved by TEMO on 10 novel classes of NWPU VHR-10 and DIOR, respectively.TEMO adds text descriptions for all object categories to mitigate classification confusion.
  • Few-Shot Segmentation: Few-shot semantic segmentation extends prediction to novel classes with limited annotated images, including five-patch results on Vaihingen and Potsdam.The pioneering remote sensing method reported overall classification accuracies of 67.7% and 53.2% on the two datasets.
  • Emerging Directions: Language-guided cross-scene hyperspectral classification and open-vocabulary segmentation illustrate emerging routes beyond conventional few-shot detection and segmentation.Open-vocabulary segmentation methods match pixels or semantic regions with text-based class descriptions, but their remote sensing adoption remains limited.

IV. USEFUL RESOURCES

The review catalogs commonly used datasets, codebases, and supporting resources for remote sensing vision-language research. The listed materials include datasets, model repositories, vision-language architectures, and few-shot segmentation benchmarks.

  • Datasets: Table XI summarizes commonly used datasets for remote sensing vision-language tasks.The review notes that many existing works use scene-classification datasets, while the listed datasets may also support general-purpose remote sensing foundation models.
  • Datasets: The dataset list is representative rather than exhaustive and does not fully cover every related dataset for each task.The paper refers readers to a comprehensive remote sensing dataset review for broader coverage.
  • Codebases: Hugging Face provides models, datasets, applications, and Transformers utilities for downloading, running, and fine-tuning vision-language models.The platform is described as having over 50,000 organizational users.
  • Codebases: MiniGPT-4, LLaVA, Qwen-VL, and Shikra provide open-source multimodal capabilities including visual grounding, chat, multilingual interaction, and spatial coordinate handling.Their supported inputs and outputs span images, text, bounding boxes, and natural-language spatial references.
  • Benchmarks: Table X reports few-shot semantic segmentation performance on iSAID-5i under 1-shot and 5-shot settings using a ResNet-50 backbone.The table presents mIoU values and borrows comparison numbers from prior work.

C. Other resources

Additional online tools support remote sensing vision-language research through model APIs, multimodal frameworks, and text-to-image generation. These resources can assist dataset preparation, task development, and text-image pair creation.

  • Other Resources: The review identifies online resources and tools useful for building vision-language models in remote sensing.These resources complement the listed datasets and open-source codebases.
  • Other Resources: The OpenAI API offers text generation, image generation, text and image embeddings, and other multimodal interfaces for dataset preparation.The paper notes that OpenAI does not release the GPT-series code and models.
  • Other Resources: LAVIS is a Python framework supporting captioning, retrieval, visual question answering, more than 20 datasets, and over 30 pretrained VLMs.Its stated goal is to streamline multimodal model development.
  • Other Resources: Midjourney is presented as a text-to-image generation tool that can support diverse text-image pair data generation.The review links this capability to advancing vision-language research.

V. CONCLUSION AND FUTURE TRENDS

The review concludes that remote sensing vision-language research is expanding across tasks but remains constrained by limited data, model capacity, computational cost, and remote-sensing variability. It highlights data-efficient learning, efficient finetuning, domain knowledge integration, geolocated text, and climate applications as future directions.

  • Conclusion: Studies report VLM superiority over purely visual models across captioning, generation, retrieval, question answering, classification, segmentation, and detection.The paper presents this area as rapidly growing but still emerging for most remote sensing researchers.
  • Limitations: Current research is limited by scarce datasets, reliance on classical CNN and RNN encoders, costly training, and difficulty modeling remote-sensing variability.Variability arises from lighting, atmospheric interference, and sensor noise, while large spatial contexts create additional challenges.
  • Future Directions: Large-scale, diverse image-text datasets are proposed because remote sensing collections remain much smaller than web-scale computer-vision datasets.LAION-5B is cited as containing over 5 billion image-text pairs, compared with the smaller scale of existing remote sensing datasets.
  • Future Directions: Future VLMs should support data-efficient few-/zero-shot recognition, efficient finetuning, and integration of remote sensing expert knowledge.Suggested approaches include learnable prompts, adapter networks, and domain knowledge covering sensor imaging, spatial correlation, and spectral characteristics.
  • Future Applications: Geolocated text such as social media messages and newspapers can be fused with remote sensing data for building semantics, disaster response, and geo-aware social dynamics.This direction adds unconventional geodata sources that are complementary to remote sensing data.
  • Future Applications: VLMs can combine satellite imagery with forest-related text and scientific reports to support monitoring of forest cover, fire-prone areas, and vegetation transitions.The paper connects these capabilities to climate change adaptation and mitigation initiatives.
Loading 2305.05726v2…