PaperlayerView exact revision on arXiv

Source-linked AI summary

RSGPT: A Remote Sensing Vision Language Model and Benchmark

Yuan Hu, Jianlong Yuan, Congcong Wen, Xiaonan Lu, Xiang Li

arXiv:2307.15266v1cs.CV

TL;DR

Remote-sensing VLMs lack large, aligned image-text resources and strong reasoning coverage. The paper introduces human-annotated RSICap and RSIEval, then fine-tunes InstructBLIP components into RSGPT; RSGPT outperforms prior methods on image captioning and visual question answering, including higher average RSVQA accuracy than three compared models.

  • Problem

    Remote-sensing VLM research lacks general reasoning abilities and large-scale, high-quality aligned image-text datasets for training and evaluation.

  • Method

    The paper builds RSICap and RSIEval, then fine-tunes the Q-Former and linear layer of InstructBLIP to create RSGPT.

  • Results

    RSGPT outperformed prior methods on most image-captioning metrics and achieved average RSVQA accuracy 11.98%, 19.68%, and 43.42% higher than InstructBLIP, BLIP2, and MiniGPT4, respectively.

  • Takeaways & Limitations

    RSICap and RSIEval provide domain-specific resources for developing and evaluating remote-sensing vision-language models.

Abstract

from arXiv · show

The emergence of large-scale large language models, with GPT-4 as a prominent example, has significantly propelled the rapid advancement of artificial general intelligence and sparked the revolution of Artificial Intelligence 2.0. In the realm of remote sensing (RS), there is a growing interest in developing large vision language models (VLMs) specifically tailored for data analysis in this domain. However, current research predominantly revolves around visual recognition tasks, lacking comprehensive, large-scale image-text datasets that are aligned and suitable for training large VLMs, which poses significant challenges to effectively training such models for RS applications. In computer vision, recent research has demonstrated that fine-tuning large vision language models on small-scale, high-quality datasets can yield impressive performance in visual and language understanding. These results are comparable to state-of-the-art VLMs trained from scratch on massive amounts of data, such as GPT-4. Inspired by this captivating idea, in this work, we build a high-quality Remote Sensing Image Captioning dataset (RSICap) that facilitates the development of large VLMs in the RS field. Unlike previous RS datasets that either employ model-generated captions or short descriptions, RSICap comprises 2,585 human-annotated captions with rich and high-quality information. This dataset offers detailed descriptions for each image, encompassing scene descriptions (e.g., residential area, airport, or farmland) as well as object information (e.g., color, shape, quantity, absolute position, etc). To facilitate the evaluation of VLMs in the field of RS, we also provide a benchmark evaluation dataset called RSIEval. This dataset consists of human-annotated captions and visual question-answer pairs, allowing for a comprehensive assessment of VLMs in the context of RS.

I. INTRODUCTION

Remote-sensing VLM research is expanding, but the field lacks strong reasoning capabilities and large-scale, high-quality aligned image-text data. The paper addresses this gap with RSICap and an efficient RSGPT model built by fine-tuning InstructBLIP components.

  • Remote-sensing VLMs still largely prioritize visual feature extraction, leaving general reasoning abilities underdeveloped.
  • Existing remote-sensing image datasets mainly target visual recognition and provide limited language annotation, while available image-text datasets remain insufficient in scale and quality.
  • Small-scale, high-quality instruction data can support strong language-model performance, motivating a quality-focused dataset strategy.
  • RSICap contains 2,585 human-annotated captions with detailed scene, object, relational, and visual-reasoning information, and RSIEval provides captions and visual question-answer pairs for evaluation.
  • RSGPT fine-tunes only the Q-Former and linear layer of InstructBLIP to align remote-sensing visual features with language models efficiently.

II. RELATED WORK

Related work traces the progression from GPT language models to multimodal systems that connect visual features with language models. These approaches primarily target general natural images, motivating remote-sensing-specific adaptation.

  • A. GPT: GPT models scaled transformer-based language modeling and demonstrated capabilities including task transfer and in-context learning.
  • B. Vision Language Models: Multimodal systems such as Flamingo, ML-MFSL, MM-GPT, and Otter extend language models with visual adaptation or image-text training.
  • B. Vision Language Models: LLaVA uses a learnable fully connected layer to feed visual features directly into a language model, while KOSMOS models further extend language-model vision integration.
  • B. Vision Language Models: Earlier multimodal methods primarily focused on general natural images rather than remote-sensing imagery.

C. Related datasets

The paper situates RSICap among earlier remote-sensing caption datasets and describes its construction from DOTA imagery. Its design emphasizes diverse imagery and expert annotations covering detailed visual attributes.

  • Earlier datasets include UCM-Captions, Sydney-Captions, RSICD, and NWPU-Captions, with varying image-caption sizes and scene diversity.
  • High-quality data can align pretrained language models to specialized domains without requiring massive domain-specific datasets.
  • RSICap is constructed from DOTA-v1.5 because it offers diverse sensors, resolutions, image modalities, and 16 object categories.
  • The authors selected 2,585 512×512 image patches from DOTA training images, and five remote-sensing experts annotated them.
  • The annotation principles cover image attributes and object attributes such as quantity, color, shape, size, and absolute and relative spatial position.

2) Qualitative Analysis:

The qualitative analysis shows that RSICap captions are more detailed and diverse than prior datasets, while its examples and statistics characterize broad question, scene, resolution, and caption-length coverage.

  • Qualitative comparison: Prior datasets generally use five short, repetitive sentences focused on the main scene, whereas RSICap includes richer object-level descriptions.
  • Qualitative comparison: RSICap captions describe scene theme, object quantity, color, shape, absolute position, relative position, and object visibility.
  • Dataset diversity: RSICap spans aerial, color-satellite, panchromatic, and low-resolution low-contrast imagery, along with scenes such as airports, harbors, tennis courts, and residential areas.
  • RSIEval examples: RSIEval examples cover presence, quantity, color, absolute and relative position, image modality and resolution, and visual reasoning questions.
  • Dataset diversity: The captions include visual reasoning based on combined visual cues, such as inferring an airport from parking lots, railway tracks, and airplanes.
  • Quantitative analysis: RSICap caption length has an average of 60 vocabularies and a maximum of 188, while captions average four sentences and reach 13 sentences.

B. RSIEval

RSIEval is a manually constructed benchmark for remote sensing image captioning and visual question answering, combining annotated image-caption pairs with diverse question categories.

  • Dataset construction: RSIEval was built from 100 manually selected 512×512 image patches annotated by five remote sensing experts.The images were selected from DOTA-v1.5 validation patches.
  • Question coverage: Its visual question-answering component covers object, image, scene, and reasoning-related questions.Object questions include attributes and spatial relations, while reasoning questions require answers based on image content and external knowledge.
  • Dataset composition: The benchmark contains 100 image-caption pairs with one caption per image and 936 questions.

IV. METHOD

RSGPT adapts frozen visual and language backbones to remote sensing through an instruction-aware Q-Former and a trainable projection layer, using InstructBLIP initialization and RSICap fine-tuning.

  • Network structure: RSGPT connects a frozen image encoder to a frozen LLM through an instruction-aware Q-Former and a fully connected layer.The Q-Former extracts visual features, while the fully connected layer projects them into the LLM input space.
  • Network structure: The Q-Former uses learnable query embeddings and instruction tokens to extract task-relevant visual features through cross- and self-attention.
  • Training strategy: Only the Q-Former and linear layer are trained, while the image encoder and LLM remain frozen for remote sensing adaptation.
  • Training strategy: RSGPT initializes from InstructBLIP and fine-tunes its Q-Former and linear layer on the high-quality RSICap dataset.The initialization leverages InstructBLIP’s prior image captioning, visual question answering, and visual reasoning training.

V. EXPERIMENTS

The experiments define a four-level caption-quality rating system spanning detail, position, and hallucination, with levels ranging from A to D.

  • Rating criteria: Level A denotes comprehensive details covering almost all object types in the ground truth.
  • Rating criteria: Level B denotes rich details covering most object types and their attribute information.
  • Rating criteria: Level C denotes only a small amount of detail covering a few object types and their attributes.
  • Experimental details: The caption-quality evaluation scores three dimensions: detail description, position description, and hallucination description.
  • Experimental details: Each dimension uses four levels, A through D, to rate generated remote sensing image captions.

B. Benchmark evaluation

RSGPT is evaluated against BLIP2, MiniGPT4, and InstructBLIP on RSIEval using manual caption-quality ratings, with results reported across detail, position, and hallucination dimensions.

  • Benchmark setup: The benchmark compares RSGPT, BLIP2, MiniGPT4, and InstructBLIP on remote sensing captioning and visual question answering.Manual scoring is used because the model responses are open-ended.
  • Benchmark setup: Caption ratings measure detail, position, and hallucination using four levels, A through D.The three dimensions correspond to panels (a), (b), and (c) in Figure 6.
  • RSIC results: 53 A, 44 B, and 3 C detail ratings place RSGPT ahead of MiniGPT4, InstructBLIP, and BLIP2.
  • RSIC results: 18 position ratings were A for RSGPT, compared with 1 for MiniGPT4, 1 for InstructBLIP, and 0 for BLIP2.The other models rarely described object positions and relative positioning.
  • RSIC results: 98 hallucination ratings were A for RSGPT, the highest A count among the compared models except for MiniGPT4’s result pattern.MiniGPT4 produced more hallucination descriptions, while BLIP2’s brief captions rarely hallucinated.

2) RSVQA:

The RSVQA evaluation organizes questions into categories and compares RSGPT with existing methods across remote-sensing datasets. RSGPT shows strong average performance on RSIEval, while quantity and positional questions remain challenging.

  • 936 RSIEval questions are divided into 10 categories spanning object-, image-, scene-, and reasoning-level questions.
  • RSGPT outperforms other models on almost all RSIEval question types.
  • RSGPT’s average accuracy is 11.98%, 19.68%, and 43.42% higher than InstructBLIP, BLIP2, and MiniGPT4, respectively.
  • RSGPT performs well on presence, image, and scene questions but poorly on quantity, absolute-position, and relative-position questions.
  • The paper also compares RSGPT with state-of-the-art methods across three RSIC and two RSVQA datasets.

1) RSIC:

RSGPT is fine-tuned and evaluated on established remote-sensing image-captioning datasets using standard captioning metrics. It surpasses previous methods on most metrics, including BLEU-1 across all three datasets.

  • The RSIC comparison uses UCM-captions, Sydney-captions, and RSICD, each providing five descriptions per image.
  • The paper reports comparisons with state-of-the-art methods on RSVQA-HR test sets 1 and 2, represented by Tables VI and VII.
  • RSGPT is fine-tuned for 5, 5, and 15 epochs on UCM-captions, Sydney-captions, and RSICD, respectively.
  • The evaluation uses BLEU, ROUGE L, METEOR, and CIDEr, with BLEU computed for n=1,2,3,4.
  • RSGPT surpasses the previous best BLEU-1 method by 4.55%, 3.89%, and 3.42% on UCM-captions, Sydney-captions, and RSICD, respectively.

2) RSVQA:

RSGPT is evaluated on RSVQA-HR and RSVQA-LR against prior methods. It matches or exceeds state-of-the-art performance across these datasets, with especially strong results on RSVQA-LR.

  • RSVQA-HR contains 10,569 high-resolution images and 1,066,316 question-answer pairs covering presence, comparison, area, and count.
  • On RSVQA-HR test set 1, RSGPT achieves results comparable to the previous best method SHRNet.
  • On RSVQA-HR test set 2, RSGPT outperforms SHRNet on all metrics.
  • On RSVQA-LR, RSGPT outperforms previous methods on all metrics.
  • Across five RSIC and RSVQA datasets, the experiments demonstrate RSGPT’s effectiveness on multiple vision-language tasks.
Loading 2307.15266v1…