PaperlayerView exact revision on arXiv

Source-linked AI summary

SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding

Junwei Luo, Zhen Pang, Yongjun Zhang, Tingzhu Wang, Linlin Wang, Bo Dang, Jiangwei Lao, Jian Wang, Jingdong Chen, Yihua Tan, Yansheng Li

arXiv:2406.10100v2cs.CVcs.AI

TL;DR

Existing datasets provide limited support for fine-grained semantic relations among objects in complex remote-sensing scenes. The paper builds FIT-RS and related benchmarks, then introduces SkySenseGPT, which is reported to excel across granular tasks and outperform existing RSLMMs on public datasets.

  • Problem

    Existing datasets provide limited support for RSLMMs’ understanding of rich semantic relations among objects in complex remote-sensing scenes.

    Source: §1 · p. 2

  • Method

    The paper constructs FIT-RS with basic and complex-comprehension tasks, establishes FIT-RSFG and FIT-RSRC, and develops SkySenseGPT using combined instruction data.

    Source: §1 · pp. 2–3

  • Results

    Across different granular tasks, SkySenseGPT excels and surpasses current RSLMMs on public datasets; on FIT-RSRC it achieves 55.5% overall accuracy, surpassing existing LMMs and RSLMMs.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The authors hope FIT-RS can contribute to building more powerful RSLMMs.

    Source: §6 · p. 10

    The authors note that challenging complex tasks still have room for improvement.

    Source: §6 · p. 10

Abstract

from arXiv · show

Remote Sensing Large Multi-Modal Models (RSLMMs) are developing rapidly and showcase significant capabilities in remote sensing imagery (RSI) comprehension. However, due to the limitations of existing datasets, RSLMMs have shortcomings in understanding the rich semantic relations among objects in complex remote sensing scenes. To unlock RSLMMs' complex comprehension ability, we propose a large-scale instruction tuning dataset FIT-RS, containing 1,800,851 instruction samples. FIT-RS covers common interpretation tasks and innovatively introduces several complex comprehension tasks of escalating difficulty, ranging from relation reasoning to image-level scene graph generation. Based on FIT-RS, we build the FIT-RSFG benchmark. Furthermore, we establish a new benchmark to evaluate the fine-grained relation comprehension capabilities of LMMs, named FIT-RSRC. Based on combined instruction data, we propose SkySenseGPT, which achieves outstanding performance on both public datasets and FIT-RSFG, surpassing existing RSLMMs. We hope the FIT-RS dataset can enhance the relation comprehension capability of RSLMMs and provide a large-scale fine-grained data source for the remote sensing community. The dataset will be available at https://github.com/Luo-Z13/SkySenseGPT

1 Introduction

Existing remote-sensing models and datasets cover basic interpretation but remain limited in fine-grained semantic relations among targets in complex scenes. The paper introduces FIT-RS and related benchmarks, and presents SkySenseGPT as a model for complex remote-sensing comprehension.

1 Introduction

  • Existing remote-sensing datasets and models cover basic tasks, but simple spatial templates do not capture fine-grained semantic relations in complex scenes.The paper distinguishes relations such as “driving in the same lane with” from rule-based relations such as “on the left of.”Source: §1 · p. 2
  • FIT-RS contains 1,800,851 instruction samples covering basic interpretation and novel complex-comprehension tasks, including relation reasoning and scene-graph generation.The dataset aims to improve RSLMMs’ understanding of object relations in complex scenes.Source: §1 · pp. 2–3
  • FIT-RS contains 1,800,851 high-quality instruction samples, outperforms existing datasets in multiple aspects, and includes detailed image-caption tasks with an average length of 619.21, alongside novel complex comprehension tasks.The comparison is presented in Table 1.Source: §1 · p. 2, Table 1 · p. 2
  • FIT-RSRC is introduced as the first benchmark for evaluating LMMs’ remote-sensing relation comprehension, with expert- and GPT-4-based distractors.It uses multiple-choice questions and the CircularEval strategy [36] [19].Source: §1 · p. 2
  • SkySenseGPT is presented as a model that performs excellently on public remote-sensing datasets and handles varied complex-comprehension tasks.The introduction reports its overall FIT-RSRC accuracy separately.Source: §1 · p. 3
  • SkySenseGPT reaches 55.5% overall accuracy on FIT-RSRC, surpassing existing LMMs and RSLMMs.The benchmark evaluates relation comprehension on remote-sensing scenes and includes high-quality distractors and unanswerable questions.Source: §1 · p. 3

2 Related Work

General LMMs have gained fine-grained capabilities through tasks that model explicitly annotated object relationships. Existing remote-sensing LMM work covers basic and region-level tasks, but simple relationship templates miss complex working or motion-state relations.

2.1 Complex Comprehension Tasks of General Large Multi-Modal Models

  • General LMMs use scene-graph datasets and tasks such as Pink’s referential comprehension [38] and All-Seeing project v2’s whole-image scene-graph generation [19] to model object relationships.CRPE evaluates object recognition and relation comprehension.Source: §2.1 · p. 3

2.2 Large Multi-Modal Models in Remote Sensing

  • Remote-sensing LMMs cover conversational and region-level tasks, but simple relationship templates miss complex relations involving targets’ working or motion states.Examples include grounded image captioning in GeoChat and phrase grounding in SkyEyeGPT [27].Source: §2.2 · p. 3

3 The FIT-RS Dataset

FIT-RS builds fine-grained remote-sensing instruction data from scene graphs, extending basic interpretation tasks with progressively harder relation, object, and scene-graph tasks. The paper also introduces FIT-RSRC to evaluate remote-sensing relation comprehension.

3.1 Data Collection and Processing

  • FIT-RS, an extension of STAR, contains more than 400,000 triplets and over 210,000 objects in 1,273 RSIs, with 48 target categories and 58 relationship subcategories.Images are cropped into 512 × 512-pixel patches with 100-pixel overlap, yielding 82,532 valid image patches.Source: §3.1 · pp. 3, 5

3.2 Data Construction for Basic Comprehension Tasks

  • Basic tasks use scene-graph relations to enrich image and region captions, VQA, and single- or multi-label classification.VQA includes presence, category comparison, counting, and relation questions, including counts constrained by a triplet.Source: §3.2 · pp. 5–6

3.3 Data Construction for Complex Comprehension Tasks

  • Complex tasks progressively probe relation understanding, subject/object comprehension, and interpretation of complete subject-relation-object triplets.They vary which scene-graph elements are supplied as input and which the model must produce.Source: §3.1 · p. 5, §3.3 · p. 6
  • Relation detection outputs relations for entities with partially specified categories and locations, while relation reasoning also predicts the subject and object categories.Object detection outputs coordinates for a specified category; object reasoning returns categories and locations satisfying a relation query, potentially within an input region.Source: §3.3 · pp. 6–7
  • Region-level scene-graph generation describes a target and its related triplets, while image-level generation produces a whole-image graph and appends isolated targets.The tasks include negative samples for background images and are mixed into multi-turn conversations.Source: §3.3 · p. 7

3.4 Evaluation of Remote Sensing Relation Comprehension

  • FIT-RSRC evaluates remote-sensing relation comprehension with four single-choice question types, CircularEval, screened distractors, and unanswerable questions.Incorrect options are drawn from a GPT-4-generated commonsense list screened by experts; distractors may include correct content, and answer lengths are matched when the correct option contains multiple relations.Source: §3.4 · pp. 7–8

4 The SkySenseGPT Model

SkySenseGPT combines a visual encoder, multimodal projector, and language model. Its training uses FIT-RS alongside transformed public datasets.

4 The SkySenseGPT Model

  • SkySenseGPT uses CLIP-ViT-L14 at 336 × 336 resolution, an MLP multimodal projector, and Vicuna-v1.5.Input images are resized to 504 × 504 pixels.Source: §4 · p. 8
  • During instruction tuning, the vision encoder is frozen, the projector is fine-tuned, and LoRA is used to fine-tune the language model.The projector is initialized following LLaVA-v1.5.Source: §4 · p. 8
  • Training uses 1,415k FIT-RS instruction samples and 365k samples transformed from public scene-classification, VQA, and object-detection datasets.The FIT-RS train and validation sets are used for instruction tuning; a portion of the test set is sampled for evaluation.Source: §4 · p. 8

5 Experiments

SkySenseGPT outperforms existing RSLMMs on the reported public datasets and shows stronger relation recognition than other evaluated models on FIT-RSRC. Fine-tuning also improves performance on complex FIT-RS tasks over the baselines.

5 Experiments

  • Experiments cover FIT-RSFG, FIT-RSRC, and generalization on existing public datasets, with test-set details reported in the Appendix.All experiments use four NVIDIA A100 GPUs with 40 GB each.Source: §5 · p. 8

5.1 Benchmarks

  • FIT-RSFG evaluates FIT-RS tasks with mainstream models, while FIT-RSRC evaluates LMMs and RSLMMs to assess the benchmark’s reliability and generalizability.For FIT-RSFG, LLaVA-v1.5 and Geochat are baselines and SkySenseGPT is evaluated after fine-tuning.Source: §5.1 · p. 8, §5.1.1 · p. 9

5.1.1 FIT-RSFG Benchmark

  • Fine-tuning yields clear improvements over LLaVA-v1.5 and Geochat baselines on complex FIT-RS tasks.Localization is evaluated with mAP at IoU 0.25, scene-graph generation with recall and mean recall, and GPT-eval scores range from 0 to 10.Source: Table 3 · p. 8, §5.1.1 · p. 9

5.1.2 FIT-RSRC Benchmark

  • Existing models generally identify whether triplets are present, but perform poorly on fine-grained relationships; SkySenseGPT recognizes relationships better than the other models, including in cross-task transfer.LLaVA-HR achieves high zero-shot accuracy, which the authors note indicates the significance of high-resolution input for fine-grained RSI understanding.Source: §5.1.2 · p. 9, Table 4 · p. 9

5.2 Generalization Experiments

  • 6.25% higher accuracy than H2RSVLM on SIRI-WHU; SkySenseGPT also surpasses current RSLMMs on scene classification and performs strongly on low- and high-resolution VQA.The VQA evaluation omits area and count questions, following prior works.Source: Table 5 · p. 9, §5.2 · p. 10

6 Conclusion

The paper introduces FIT-RS and FIT-RSFG, adds FIT-RSRC for remote-sensing relation comprehension, and presents SkySenseGPT using FIT-RS combined with other multimodal corpora. It also notes that challenging complex tasks leave room for improvement.

6 Conclusion

  • FIT-RS, FIT-RSFG, and FIT-RSRC support fine-grained remote-sensing comprehension, while SkySenseGPT trained on combined multimodal data excels across granular tasks and surpasses current RSLMMs on public datasets.FIT-RSRC evaluates remote-sensing relationship comprehension; FIT-RSFG is built from FIT-RS.Source: §6 · p. 10
  • Challenging complex tasks remain an area for improvement, even as the paper presents FIT-RS as a resource for building more powerful RSLMMs.Source: §6 · p. 10

A Overview

  • The appendix provides dataset and prompt details, training and testing dataset information, experimental results, and visualizations; the dataset, generation scripts, and model weights are slated for release.Source: A Overview · p. 15

B The FIT-RS Dataset

FIT-RS combines conventional remote-sensing interpretation tasks with detailed image and region descriptions and VQA, using scene-graph annotations to support fine-grained content. Its data preparation includes task-specific cropping thresholds and prompt construction.

B The FIT-RS Dataset

  • The dataset specifies formats for image caption, region caption, VQA, scene classification, complex comprehension, and multi-turn conversation tasks.The cropping completeness threshold is 0.3 for detailed image captioning and 0.55 for other tasks.Source: Table A1 · p. 15, B The FIT-RS Dataset · p. 15

B.1 Detailed Image Caption

  • For detailed image captions, foreground scene-graph annotations are converted to sentences, while TinyLLaVA generates concise background descriptions that are combined with foreground information for GPT-generated captions.TinyLLaVA generated descriptions for 85,000 512×512 images at approximately 1.1 seconds per image; 2,000 images were randomly reviewed.Source: §B.1 · pp. 15–16
  • Combining detailed foreground information with concise background descriptions yields thorough image descriptions, with object relationships contributing to interpretation of states and remote-sensing scenes.The paper illustrates the combined inputs and generated descriptions with examples in Tables A4 and A5.Source: §B.1 · p. 16, Table A2 · p. 16, Table A4 · p. 18

B.2 Detailed Region Caption

  • Region captions combine a target’s category, size, position, and visibility with natural-language descriptions of its scene-graph relationships.Cropping uses an iof threshold of 0.55 for target-related tasks, so targets in subsequent complex tasks are mostly partially visible.Source: §B.2 · pp. 17, 20

B.3 Visual Question Answering

  • FIT-RS VQA templates generate questions and answers from image annotations, covering object presence, category counts and comparisons, and relationship counts and consistency.When effective relationship triplets are absent, the process also creates questions verifying absent relationships.Source: §B.3 · pp. 21–22

B.4 Complex Comprehension Tasks

FIT-RS adds complex comprehension tasks beyond existing region-level tasks, including relation and object detection and reasoning, plus region- and image-level scene graph generation. The appendix illustrates using regions and relationships to help identify a target from a perceived scene graph.

B.4 Complex Comprehension Tasks

  • FIT-RS introduces relation detection, relation reasoning, object detection, object reasoning, and region- and image-level scene graph generation as complex comprehension tasks.Scene-graph generation uses the detailed region- and image-caption instructions with a [grounding] prefix.Source: §B.4 · p. 22, Table A8 · p. 23, Table A9 · p. 24
  • The appendix’s multi-turn example uses user-provided regions and relationships as aids for identifying a target from the perceived scene graph.Source: Figure A1 · p. 25

C.1 Additional Datasets for Training

The paper supplements training with converted instructions from public datasets, describes its evaluation dataset-selection practice, and reports the additional benchmark-training setup.

C.1 Additional Datasets for Training

  • About 365k instruction samples from public datasets are collected and converted; GeoChat-Instruction image-description and multi-turn portions based on DOTA, DIOR, and FAIR1M are selected to enhance overall image perception.Source: §C.1 · p. 26

C.2 Datasets for Evaluation

  • Evaluation uses a dataset’s test set when clearly defined, and otherwise typically evaluates on the entire dataset.Source: §C.2 · p. 26

D Detailed Results on Benchmarks

  • The reported additional training setup uses LoRA with rank 64, four GPUs, total batch size 128, learning rate 1 × 10^-6, cosine scheduling, 0.03 warmup ratio, and one epoch.Source: D Detailed Results on Benchmarks · p. 26

D.1 FIT-RSFG Benchmark

FIT-RSFG reports improved performance on some challenging tasks after fine-tuning, with overall accuracy distributed according to the benchmark’s progressive task difficulty. Its results use an IoU threshold of 0.25 for mAP and define four question-accuracy metrics.

D.1 FIT-RSFG Benchmark

  • After fine-tuning, performance improved significantly on some challenging tasks, and overall accuracy distribution aligned with the designed progressive task difficulty.Table A12 presents the detailed FIT-RSFG results; the mAP IoU threshold is 0.25 because triplet matching emphasizes category correctness.Source: §D.1 · p. 27
  • The mAP evaluation uses an IoU threshold of 0.25, emphasizing category correctness in scene-graph triplet matching.Source: §D.1 · p. 27
  • Table A12 reports Count-Acc, Pre-Acc, Comp-Acc, and Rel-Acc; detailed image captions use test descriptions generated by GPT-4.Source: Table A12 · p. 27

D.2 FIT-RSRC Benchmark

FIT-RSRC balances four question types and uses human-expert-crafted commonsense word lists to constrain distractors. Existing evaluated LMMs favor option D, while SkySenseGPT exhibits less answer-option bias.

D.2 FIT-RSRC Benchmark

  • Relationship, object, subject, and existence questions each make up 25% of FIT-RSRC, with negative answers distributed across types.Source: §D.2 · p. 27
  • Options have four or five choices: 87.5% of questions have four, and 12.5% have five.Each option is 4–5 words long.Source: §D.2 · p. 27
  • Existing LMMs favor option D on four-option questions, whereas SkySenseGPT exhibits less answer-option bias.Figure A2 shows the choice distribution across four models on CirularEval records.Source: §D.2 · p. 28, Figure A2 · p. 28
  • Human experts, assisted by GPT-4, crafted subject-relation and relation-object word lists that constrain distractors and prevent text-only deduction.Generated distractors are cross-checked against these lists.Source: §D.2 · p. 28, Table A13 · p. 29, Table A14 · p. 30
Loading 2406.10100v2…