PaperlayerView exact revision on arXiv

Source-linked AI summary

SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model

Yang Zhan, Zhitong Xiong, Yuan Yuan

arXiv:2401.09712v1cs.CV

TL;DR

Remote-sensing vision-language modeling remains limited by domain differences and task-specific systems. SkyEyeGPT addresses this with a unified model and a manually verified 968k-sample instruction dataset, achieving strong results across diverse tasks, including performance comparable to or better than GPT-4V in some tests.

  • Problem

    Remote-sensing vision-language modeling remains emerging, while existing systems often require separate task-specific models and lack multi-task conversation ability.

  • Method

    SkyEyeGPT combines visual features, an alignment layer, language instructions, and an LLM-based decoder, trained with the manually verified SkyEye-968k dataset and two-stage tuning.

  • Results

    SkyEyeGPT achieves strong performance across diverse remote-sensing vision-language tasks, with results comparable to or better than GPT-4V in some tests and about 10% above specialist models in one reported comparison.

  • Takeaways & Limitations

    A unified instruction-tuned model can handle image-level and region-level remote-sensing tasks while supporting multi-task and multi-round conversational ability.

  • Takeaways & Limitations

    The instruction dataset excludes images from validation and test sets, so its training instructions are constructed without those evaluation images.

Abstract

from arXiv · show

Large language models (LLMs) have recently been extended to the vision-language realm, obtaining impressive general multi-modal capabilities. However, the exploration of multi-modal large language models (MLLMs) for remote sensing (RS) data is still in its infancy, and the performance is not satisfactory. In this work, we introduce SkyEyeGPT, a unified multi-modal large language model specifically designed for RS vision-language understanding. To this end, we meticulously curate an RS multi-modal instruction tuning dataset, including single-task and multi-task conversation instructions. After manual verification, we obtain a high-quality RS instruction-following dataset with 968k samples. Our research demonstrates that with a simple yet effective design, SkyEyeGPT works surprisingly well on considerably different tasks without the need for extra encoding modules. Specifically, after projecting RS visual features to the language domain via an alignment layer, they are fed jointly with task-specific instructions into an LLM-based RS decoder to predict answers for RS open-ended tasks. In addition, we design a two-stage tuning method to enhance instruction-following and multi-turn dialogue ability at different granularities. Experiments on 8 datasets for RS vision-language tasks demonstrate SkyEyeGPT's superiority in image-level and region-level tasks, such as captioning and visual grounding. In particular, SkyEyeGPT exhibits encouraging results compared to GPT-4V in some qualitative tests. The online demo, code, and dataset will be released in https://github.com/ZhanYang-nwpu/SkyEyeGPT.

1 Introduction

SkyEyeGPT addresses the limited transfer of general vision-language models to remote sensing by unifying diverse RS tasks in one open-ended conversational model. It combines manually verified instruction data, a simple visual-language architecture, and two-stage tuning, with strong results across evaluations.

  • General vision-language advances have not transferred seamlessly to remote sensing because natural and remote sensing domains differ intrinsically.
  • Existing RS models often train task-specific models separately, limiting open-ended multi-task conversation.
  • SkyEyeGPT unifies open-ended RS vision-language tasks using a visual encoder, alignment layer, and LLM-based decoder without extra encoder or external plugin modules.Its task inputs and outputs are represented in natural language, including bounding box coordinates.
  • SkyEye-968k contains 968k manually verified instruction-following samples spanning single-task image-text instructions and multi-task conversation instructions.A two-stage tuning method uses these instruction types separately to develop multi-task and multi-turn dialogue capabilities.
  • Experiments on 8 remote sensing vision-language datasets demonstrate SkyEyeGPT’s superiority across image-level and region-level tasks.The authors report results comparable to or better than GPT-4V in some qualitative comparisons.
  • The work releases an online remote sensing chatbot, model checkpoint, instruction-following dataset, and codebase for real-world applications.

2 Related Work

Remote sensing vision-language research spans image-level, region-level, and spatiotemporal tasks, but most existing systems remain task- and dataset-specific. Related multimodal instruction-tuning methods likewise have limited fine-grained remote sensing coverage, motivating a unified framework.

  • Remote sensing vision-language tasks include image captioning, VQA, visual grounding, and UAV video captioning across diverse granularity levels.
  • Many state-of-the-art RS methods are trained for one specific dataset and task rather than unified multi-task understanding.
  • BLIP-2 connects a frozen language model and visual encoder through a Q-Former, while LLaVA and related methods use a linear mediator.
  • Existing instruction-tuning approaches primarily address image-level coarse-grained tasks and cannot effectively handle fine-grained perception challenges.

3 Method of SkyEyeGPT

SkyEyeGPT unifies remote-sensing vision-language tasks with a visual encoder, alignment layer, LLM-based decoder, curated instruction data, and two-stage tuning. Its inputs and outputs use natural-language task instructions, with normalized bounding-box coordinates for region-level tasks.

  • Visual Encoder: EVA-CLIP extracts patch embeddings from standardized 448×448 remote-sensing images, while UAV video features concatenate embeddings from multiple frames.The visual encoder remains frozen during training.
  • Alignment Layer: A linear alignment layer bridges remote-sensing visual features and LLM language features while reducing the contextual and resource cost of processing many image patches.The method avoids directly passing the excessive patch sequence produced by high-resolution inputs into the LLM.
  • Overall Architecture: SkyEyeGPT combines a visual encoder, alignment layer, and LLM-based decoder without extra encoder or external plugin modules.The decoder receives visual tokens and language instructions to generate task-specific answers.
  • Unified RS Vision-Language Instruction: The dataset covers captioning, VQA, and grounding, including bounding-box outputs for single or multiple grounded objects.Captioning combines five image-captioning datasets and one UAV video-captioning dataset; VQA and grounding combine multiple public and constructed datasets.
  • Input and Output Template: Task-specific identifiers such as [caption], [vqa], and [refer] unify task inputs, while region-level outputs encode normalized box coordinates as natural-language tokens.Coordinates are multiplied by 100 and rounded to integers.
  • Instruction Tuning: Two-stage tuning first aligns remote-sensing image-text representations with single-task instructions, then improves multi-task, multi-round conversation handling.In the second stage, previous-turn instructions and answers serve as references for the current response; LoRA fine-tunes the alignment layer and LLM.

4 Experiments

Experiments evaluate SkyEyeGPT across captioning, video captioning, VQA, and visual grounding, combining quantitative benchmarks with qualitative comparisons and ablations. Results are strong across tasks, while VQA remains constrained by imagery modality differences.

  • Main Results: SkyEyeGPT demonstrates competitive or superior performance across image-level and region-level remote sensing vision-language tasks.The study compares generalist and specialist models across the evaluated tasks.
  • Experimental Setup: Experiments cover four representative tasks: RS image captioning, UAV video captioning, RS visual question answering, and RS visual grounding.
  • Captioning: SkyEyeGPT achieves the best performance on most captioning metrics, except CIDEr on UCM-caption, and reaches state-of-the-art results on CapERA aerial video captioning.The authors note that caption-length differences make existing metrics, especially CIDEr, difficult to interpret reliably.
  • Qualitative Analysis: Qualitative comparisons show SkyEyeGPT providing more comprehensive descriptions than several baselines and identifying scene elements omitted by GPT-4V.The examples include sports fields, buildings, parking lots, and other surrounding objects.
  • Visual Grounding: SkyEyeGPT outperforms specialist models by about 10% on visual grounding and shows robustness and precise localization for small objects.The evaluation includes RSVG and DIOR-RSVG, while qualitative examples use NWPU and DOTA images.
  • Visual Question Answering: SkyEyeGPT's average VQA accuracy is 8% lower than RSGPT's separately fine-tuned model.The authors attribute this performance loss to modality differences between satellite imagery in RSVQA and aerial imagery in other tasks.
  • Ablation Studies: Ablation studies examine multi-task learning, two-stage instruction tuning, task identifiers, Q-Former inclusion, and LoRA rank.The variants compare separately trained, single-task models with one-stage and multi-task instruction tuning.

5 Conclusion

The paper presents SkyEyeGPT as a unified remote sensing multimodal model supported by a large instruction dataset and two-stage tuning. It reports strong performance across tasks and conversational remote sensing understanding.

  • SkyEyeGPT is a unified open multimodal large language model tailored to remote sensing.
  • The authors construct an instruction-following dataset with single-task and multi-task conversation instructions.
  • Two-stage tuning is designed to develop multi-task and multi-round conversational ability, with task identifiers supporting unified open-ended task treatment.
  • SkyEyeGPT's effectiveness is validated across tasks with different granularities, achieving new state-of-the-art accuracy on many tasks.
  • The paper reports an exceptional remote sensing multimodal chatting experience through a versatile unified framework for open-ended tasks.

Appendices

The appendices provide supplementary comparisons, task instructions, quantitative results, ablations, and qualitative examples. They also document common MLLM architectures and the instruction formats used for remote sensing tasks.

  • Supplementary Materials: The supplementary material contains structural comparisons, task instructions, additional quantitative results, ablations, and qualitative results.
  • Architecture Comparisons: MiniGPT-4, LLaVA, and Shikra use linear projection to align visual features with the language model, while RSGPT inserts an instruction-aware Q-Former before projection.
  • Architecture Comparisons: Existing MLLMs generally use a ViT visual encoder sourced from CLIP or EVA and connect visual features to an LLM with either a linear layer or Q-Former plus linear layer.
  • Instruction Formats: Appendix instructions cover RS image captioning, visual grounding, and VQA using natural-language task prompts.Captioning prompts vary the wording while preserving the same intended meaning; grounding prompts request object localization.
  • Appendix Tables: The appendix includes tables listing captioning, grounding, and VQA instructions, plus an architecture comparison of existing state-of-the-art MLLMs.

C Additional Results

Additional experiments examine SkyEyeGPT across remote-sensing VQA and captioning, while ablations assess evaluation, alignment, task identifiers, and LoRA rank. Qualitative visualizations cover multiple image-level and region-level tasks.

  • Additional Results: Sharing one visual encoder across modalities can reduce SkyEyeGPT’s VQA performance, with the separately trained specialist outperforming the open-ended model.This effect is reported for RSVQA-HR test set 1 and is linked to differences between satellite and aerial imagery.
  • Additional Results: SkyEyeGPT achieves the best results on UCM-caption, Sydney-caption, and RSICD, although CIDEr remains worse than RSGPT.The authors attribute the CIDEr pattern to different caption lengths or richer semantics.
  • Additional Results: 51.72% and 56.90% success accuracy were achieved by SkyEyeGPT in ChatGPT-based Sydney-caption evaluation, exceeding other methods.The authors argue these captions are closer to real visual objects and relations, while traditional metrics rank methods differently.
  • Ablation Studies: A single linear alignment layer is sufficient for aligning remote-sensing visual features with the LLM; alternatives using multiple layers or Q-Former perform poorly.The comparison includes removing the linear layer, replacing it with two or three linear layers, and using a Q-Former.
  • Ablation Studies: Adding task identifiers improves results on four tasks and supports more efficient multi-task learning.The ablation compares SkyEyeGPT with and without task identifiers in all instructions.
  • Ablation Studies: LoRA rank 64 produces the best model performance, with performance increasing and then decreasing as rank rises.This result comes from the rank ablation summarized in Table 16.
Loading 2401.09712v1…