PaperlayerView exact revision on arXiv

Source-linked AI summary

EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain

Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, Xuerui Mao

arXiv:2401.16822v3cs.CV

TL;DR

MLLMs remain limited in remote sensing because natural and multi-sensor RS imagery differ substantially, while existing RS models often lack broad task and modality coverage. EarthGPT addresses this gap with unified multimodal architecture, specialized perception and alignment mechanisms, instruction tuning, and MMRS-1M. It surpasses specialist models and MLLMs across diverse RS interpretation tasks and demonstrates open-set reasoning performance in supervised and zero-shot settings.

  • Problem

    Natural-image MLLMs have limited adaptability to RS imagery, while existing RS approaches lack unified multi-sensor, multi-task, and open-set capabilities.

  • Method

    EarthGPT combines visual-enhanced perception, cross-modal mutual comprehension, unified instruction tuning, and the MMRS-1M multi-sensor instruction-following dataset.

  • Results

    EarthGPT surpasses specialist models and MLLMs across diverse RS interpretation tasks, including supervised and zero-shot open-set evaluations.

  • Takeaways & Limitations

    EarthGPT provides a unified paradigm for multi-sensor RS image comprehension and open-set reasoning across downstream tasks.

Abstract

from arXiv · show

Multi-modal large language models (MLLMs) have demonstrated remarkable success in vision and visual-language tasks within the natural image domain. Owing to the significant diversities between the natural and remote sensing (RS) images, the development of MLLMs in the RS domain is still in the infant stage. To fill the gap, a pioneer MLLM named EarthGPT integrating various multi-sensor RS interpretation tasks uniformly is proposed in this paper for universal RS image comprehension. In EarthGPT, three key techniques are developed including a visual-enhanced perception mechanism, a cross-modal mutual comprehension approach, and a unified instruction tuning method for multi-sensor multi-task in the RS domain. More importantly, a dataset named MMRS-1M featuring large-scale multi-sensor multi-modal RS instruction-following is constructed, comprising over 1M image-text pairs based on 34 existing diverse RS datasets and including multi-sensor images such as optical, synthetic aperture radar (SAR), and infrared. The MMRS-1M dataset addresses the drawback of MLLMs on RS expert knowledge and stimulates the development of MLLMs in the RS domain. Extensive experiments are conducted, demonstrating the EarthGPT's superior performance in various RS visual interpretation tasks compared with the other specialist models and MLLMs, proving the effectiveness of the proposed EarthGPT and offering a versatile paradigm for open-set reasoning tasks.

I. INTRODUCTION

EarthGPT addresses the limitations of task-specific and optical-only RS MLLMs by unifying multi-sensor image comprehension and diverse interpretation tasks. It combines three model techniques with MMRS-1M and reports strong supervised and open-set performance.

  • Current RS methods mainly use one-task-one-architecture designs, limiting multi-sensor handling, multi-task coverage, and open-set reasoning.
  • EarthGPT unifies multi-sensor RS image comprehension across optical, SAR, and infrared modalities under language-guided visual-language interaction.
  • MMRS-1M contains over 1M image-text pairs transformed from 34 diverse RS datasets, covering optical, SAR, and infrared imagery.
  • 77.37% and 74.72% accuracy are achieved on CLRS and NaSC-TG2 zero-shot scene classification, while MAR20 detection reaches 90.47% AP@40%.
  • EarthGPT combines visual-enhanced perception, cross-modal mutual comprehension, and unified instruction tuning for RS multi-task interpretation.
  • Experiments report superior performance over specialist models and MLLMs across supervised RS interpretation and open-set reasoning tasks.

A. MLLMs

Natural-image MLLMs have strong multimodal reasoning but do not adequately capture RS-specific complexities. EarthGPT and MMRS-1M are introduced to align language with diverse RS modalities and tasks.

  • Existing MLLMs are designed mainly for natural scenes and have limited adaptability to the contextual complexities of RS data.
  • Earlier RS large models emphasize visual pretraining without language alignment, restricting multimodal understanding and broad application.
  • EarthGPT adapts MLLMs to RS through visual-enhanced perception, cross-modal mutual comprehension, and unified instruction tuning.
  • Existing RS datasets commonly focus on a single modality and task, producing models with limited generalization and task-specific scope.
  • MMRS-1M contains more than 1M image-text pairs spanning classification, detection, captioning, VQA, visual grounding, optical, infrared, and SAR imagery.

B. Visual-enhanced Perception

The visual-enhanced perception mechanism combines complementary multi-layer and multi-scale features to represent both broad semantics and fine-grained RS details. Projected visual tokens are aligned with language-token dimensions for multimodal processing.

  • The mechanism uses diverse image encoders to refine multi-granularity visual information while mitigating disturbances in RS images.
  • Multi-Layer Visual Perception: ViT extracts multi-layer features whose fusion combines early spatial-aware information with later semantic information.
  • Multi-scale Visual Perception: CNN extracts multi-scale localized features such as edges and textures through spatial hierarchies and local receptive fields.
  • Multi-scale Visual Perception: Features from multiple scales are transformed to a common channel dimension and concatenated channel-wise, capturing broad semantics and intricate information.
  • A learnable projection layer aligns concatenated visual features with language tokens and produces new visual tokens Vp.
  • The integrated enhanced visual perception information improves image-interpretation accuracy and efficiency.

C. Cross-modal Mutual Comprehension

EarthGPT fuses visual and language information into a unified multimodal input for LLM alignment and interaction training. After visual-language alignment, the language-only LLM becomes an MLLM capable of generating visual interpretation responses.

  • Cross-modal mutual comprehension fuses visual and language perception information and delivers multimodal input into the LLM for alignment and interaction training.
  • Mixed visual tokens and language instruction tokens are concatenated to form the multimodal input sequence X before LLM fusion and integration.
  • Visual features are represented as mixed tokens Vp, while Nv and Nl denote the visual and language token lengths, respectively.
  • The approach uses transformer self-attention components with trainable linear projections and an unfrozen RMSNorm scale parameter γ during training.
  • The cross-entropy objective models word prediction over the multimodal sequence using EarthGPT parameters θ and target word wi.
  • After alignment, EarthGPT generates visual interpretation responses from integrated multimodal information.

D. Unified Instruction Tuning for RS

Unified instruction tuning extends EarthGPT from natural-domain multimodal capabilities to instruction-following across remote-sensing downstream tasks. The stage freezes prior weights and adds bias and shift parameters to preserve captioning while improving task compliance.

  • MMRS-1M is developed to improve EarthGPT’s instruction following for diverse remote-sensing downstream tasks and broaden its applicability beyond the natural domain.
  • All cross-modal comprehension weights are frozen while learnable bias β and shift α parameters are added to LLaMA-2 linear layers.
  • The tuning stage preserves visual captioning capability while enhancing compliance with task instructions.
  • EarthGPT gains versatile language-guided multi-sensor visual comprehension skills with potential for practical remote-sensing applications.

IV. DATASET CONSTRUCTION

MMRS-1M addresses the lack of domain-specific remote-sensing datasets by transforming existing data into a comprehensive instruction-following resource. It supports image-text conversation across multiple tasks and visual modalities.

  • Domain-specific dataset scarcity hampers MLLM application to intelligent geographic-information interpretation and open-set dialogue.
  • MMRS-1M is created by cleaning and transforming existing remote-sensing datasets into a unified instruction-following format.
  • The dataset covers classification, detection, image captioning, VQA, and visual grounding across optical, SAR, and infrared modalities.
  • EarthGPT is fine-tuned on MMRS-1M to align visual and language modalities and support coarse-grained conversation and fine-grained localization.

A. Coarse-grained Conversation Scenarios

MMRS-1M converts diverse remote-sensing datasets into instruction-following conversations for coarse-grained understanding and fine-grained localization. Its construction spans image-level tasks, object detection, and visual grounding across multiple sensors.

  • Coarse-grained Conversation Scenarios: Ten classification, five image-captioning, and four VQA datasets are collected for image-level coarse-grained question answering.
  • Coarse-grained Conversation Scenarios: Classification data use category-to-instruction prompts that ask the model to answer each image’s category with a word or phrase.
  • Coarse-grained Conversation Scenarios: Caption-to-instruction cleans duplicate captions and asks EarthGPT to provide detailed one-sentence descriptions, with caption counts determining dialogue rounds.
  • Coarse-grained Conversation Scenarios: VQA-to-instruction appends output-format guidance to original questions, and multiple question-answer pairs create multi-round image dialogues.
  • Fine-grained Conversation Scenarios: Object-detection data combine optical, SAR, and infrared datasets, while DIOR-RSVG supports visual grounding and region-level captioning.
  • Fine-grained Conversation Scenarios: Detection-to-instruction supports HBB and OBB outputs with normalized coordinates and object categories, while VG-to-instruction supports region localization and region description.
  • Fine-grained Conversation Scenarios: The construction process transforms training and validation data after image cutting and dataset splitting into MMRS-1M samples.
  • Fine-grained Conversation Scenarios: Thirty-four existing remote-sensing datasets are cleaned and converted into a uniform format spanning optical, SAR, and infrared modalities.

V. EXPERIMENTS AND ANALYSIS

The experiments evaluate EarthGPT across supervised and zero-shot remote-sensing classification settings against specialist models and other MLLMs. Results indicate strong supervised performance and notable zero-shot gains on unseen datasets.

  • Experimental scope: EarthGPT is evaluated on classification, captioning, VQA, visual grounding, and object detection against MLLMs and specialist models.The experimental section covers both conventional supervised evaluations and open-set or zero-shot comparisons.
  • Supervised classification: NWPU-RESISC45 provides 45 scene categories, with each category containing 700 images and 80% used for testing.The images are 256×256 pixels, and the test set supports comparison with specialist models.
  • Zero-shot classification: Zero-shot classification uses CLRS and NaSC-TG2, which contain 15,000 images across 25 land-use types and 20,000 images across 10 land-cover types, respectively.The datasets provide evaluation on imagery and categories outside the supervised NWPU-RESISC45 setting.
  • Supervised classification: EarthGPT exceeds CSDS and T-CNN by 0.25% and 0.79%, respectively, on supervised NWPU-RESISC45 classification.Its performance is reported as comparable with MGSNet.
  • Zero-shot classification: 25.61% top-1 accuracy on CLRS is reported for EarthGPT, exceeding Qwen-VL-Chat, LLaVA-1.5, and Sphinx at 25.61%, 21.51%, and 16.65%, respectively.On NaSC-TG2, EarthGPT improves over those MLLMs by 38.63%, 34.05%, and 24.96%, respectively.

C. Image Captioning

EarthGPT is evaluated for image captioning, visual question answering, and visual grounding using supervised and zero-shot remote-sensing benchmarks. It reports improvements over specialist models and competitive or stronger performance against other MLLMs.

  • Image captioning: NWPU-Caption contains 31,500 aerial remote-sensing images and 157,500 caption sentences for supervised evaluation.Evaluation uses BLEU1, BLEU2, BLEU3, BLEU4, METEOR, ROUGEL, and CIDErD.
  • Image captioning: On NWPU-Captions, EarthGPT improves over other methods by 10.1% to 66.2% across BLEU, METEOR, ROUGE-L, and CIDEr-D metrics.The reported gains are 10.1%, 13.8%, 17.5%, 17.7%, 10.5%, 18.2%, 66.2%, and 3.7% across the listed captioning metrics.
  • Visual question answering: 72.05% average accuracy on RSVQA-HR places EarthGPT above or commensurate with other MLLMs in zero-shot visual question answering.The reported improvements over Qwen-VL-Chat, LLaVA v1.5, MiniGPTv2, and Sphinx are 3.66%, 9.00%, 25.60%, and 2.27%, respectively.
  • Visual grounding: EarthGPT improves visual grounding by 0.11%, 1.19%, 1.30%, and 3.13% on Pr@0.8, Pr@0.9, mIoU, and cIoU, respectively.The evaluation uses 758 grounding questions from DIOR-RSVG.

F. Object Detection

EarthGPT extends language-guided interpretation to zero-shot horizontal and oriented object detection, while qualitative experiments examine multi-turn, cross-task reasoning. It improves HBB detection over open-set alternatives and remains competitive for OBB detection.

  • Object detection: Zero-shot detection on MAR20 evaluates EarthGPT with AP@40 and AP@50 for horizontal bounding boxes.MAR20 contains 3,842 images and 22,341 aircraft instances annotated with horizontal and oriented bounding boxes.
  • Object detection: EarthGPT uses clip-score as a confidence logit for noise filtering because other MLLMs do not predict confidence scores.RemoteCLIP weights are used to compute the clip score.
  • Object detection: EarthGPT improves over GroundingDINO by 1.92% on AP@40 and 1.90% on AP@50 in MAR20 HBB detection.The comparison includes other MLLMs and open-set detection models.
  • Object detection: For MAR20 OBB detection, EarthGPT achieves competitive performance with specialist models trained on DOTA.This comparison uses the oriented-bounding-box format of MAR20.
  • Qualitative analysis: Qualitative results show EarthGPT handling classification, captioning, VQA, detection, region-level captioning, and visual grounding through multi-turn dialogue.The examples emphasize image-level and region-level perception across diverse remote-sensing tasks.

B. Multi Visual Modality Inference

EarthGPT is presented as a unified model for interpreting optical, SAR, and infrared remote-sensing imagery across multiple tasks. Qualitative examples also show language-guided cross-task reasoning, including using counts or HBB results to improve OBB detection.

  • Multi Visual Modality Inference: MMRS-1M supplies multi-sensor information for EarthGPT, which is described as generalizing across optical, SAR, and infrared modalities.The dataset contains less SAR and infrared data than optical data, while language integrates the modalities.
  • Optical imagery: EarthGPT qualitatively counts objects, identifies ships, and describes dense residential areas in optical imagery.These examples illustrate image-level and task-diverse interpretation through language interaction.
  • SAR imagery: In SAR imagery, EarthGPT is shown counting storage tanks, evaluating ship presence, and producing detailed captions.The examples demonstrate that the same language-guided model is applied across visual modalities.
  • Cross-task reasoning: Chain-of-thought prompting improves visual reasoning by using object-counting results to enhance OBB detection completeness.In an airport example, prompting with the total number of planes helps recover initially missed detections.
  • Cross-task reasoning: EarthGPT uses HBB detection results as hints to rectify omissions and produce more precise OBB results.The paper frames this as incorporating insights from related tasks into visual interpretation and reasoning.

VII. CONCLUSION

EarthGPT is presented as a universal RS image-comprehension MLLM that integrates multi-sensor interpretation and diverse downstream tasks. Its accompanying MMRS-1M dataset and experiments support multi-task, supervised, and zero-shot RS reasoning.

  • VII. CONCLUSION: EarthGPT integrates coarse-scale and fine-scale visual perception for universal multi-sensor RS image comprehension.The model bridges cross-modal mutual comprehension and vision reasoning.
  • VII. CONCLUSION: EarthGPT supports scene classification, image captioning, region-level captioning, VQA, visual grounding, and object detection.The unified task scope includes both horizontal and oriented detection in the broader method description.
  • VII. CONCLUSION: MMRS-1M contains more than 1M image-text pairs in a multi-sensor, multimodal RS instruction-following dataset.The dataset is constructed to address limitations in MLLM RS expert knowledge.
  • VII. CONCLUSION: EarthGPT surpasses numerous specialist models and MLLMs across various RS visual interpretation tasks in supervised and zero-shot settings.The paper also reports open-set reasoning capability for multiple RS downstream tasks.
  • VII. CONCLUSION: Future work targets improved OBB detection performance and the incorporation of more modalities into EarthGPT.The stated goal is to extend the model toward all-purpose capabilities.
Loading 2401.16822v3…