GeoChat: Grounded Large Vision-Language Model for Remote Sensing

Kartik Kuckreja; Muhammad Sohail Danish; Muzammal Naseer; Abhijit Das; Salman Khan; Fahad Shahbaz Khan

Source-linked AI summary

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, Fahad Shahbaz Khan

arXiv:2311.15826v1Nov 24, 2023cs.CVcs.AI

TL;DR

General-domain VLMs struggle with remote-sensing imagery, where region-level reasoning and domain-specific instruction data are needed for reliable, open-ended interaction. GeoChat addresses this gap with a unified RS VLM, spatial representations, and a 318k-example instruction dataset, achieving robust zero-shot performance across diverse RS tasks. The paper also establishes benchmarks for multitask RS conversations.

Problem
General-domain VLMs perform poorly on RS imagery, while existing RS VQA methods limit open-ended generation and instruction-following; region-level reasoning and domain-specific data remain needed.
Method
GeoChat unifies image- and region-level RS reasoning using task tokens, spatial location representations, grounded outputs, and a 318k-example multimodal instruction-following dataset.
Results
GeoChat demonstrates robust zero-shot performance across diverse RS tasks, including scene classification, VQA, multiturn dialogue, captioning, grounded conversations, and referring detection.
Takeaways & Limitations
GeoChat provides a unified framework and benchmark for conversational remote-sensing understanding spanning image-level, region-level, and visually grounded tasks.
Takeaways & Limitations
RSGPT, a related remote-sensing VLM, cannot perform region-level reasoning or visual grounding, marking capabilities GeoChat aims to add.

Abstract

from arXiv · show

Recent advancements in Large Vision-Language Models (VLMs) have shown great promise in natural image domains, allowing users to hold a dialogue about given visual content. However, such general-domain VLMs perform poorly for Remote Sensing (RS) scenarios, leading to inaccurate or fabricated information when presented with RS domain-specific queries. Such a behavior emerges due to the unique challenges introduced by RS imagery. For example, to handle high-resolution RS imagery with diverse scale changes across categories and many small objects, region-level reasoning is necessary alongside holistic scene interpretation. Furthermore, the lack of domain-specific multimodal instruction following data as well as strong backbone models for RS make it hard for the models to align their behavior with user queries. To address these limitations, we propose GeoChat - the first versatile remote sensing VLM that offers multitask conversational capabilities with high-resolution RS images. Specifically, GeoChat can not only answer image-level queries but also accepts region inputs to hold region-specific dialogue. Furthermore, it can visually ground objects in its responses by referring to their spatial coordinates. To address the lack of domain-specific datasets, we generate a novel RS multimodal instruction-following dataset by extending image-text pairs from existing diverse RS datasets. We establish a comprehensive benchmark for RS multitask conversations and compare with a number of baseline methods. GeoChat demonstrates robust zero-shot performance on various RS tasks, e.g., image and region captioning, visual question answering, scene classification, visually grounded conversations and referring detection. Our code is available at https://github.com/mbzuai-oryx/geochat.

1. Introduction

GeoChat addresses the limited performance and open-ended instruction-following of general-domain VLMs on remote-sensing imagery by unifying image- and region-level reasoning with grounded responses. It also contributes a 318k-example instruction-following dataset and evaluation protocols for diverse RS conversations.

Motivation: General-domain VLMs perform poorly on remote-sensing imagery, producing inaccurate or hallucinated information because RS image-text content differs from web data.Earlier RS VQA methods also restrict answers to predetermined responses, limiting open-ended generation and instruction-following.
Dataset: GeoChat uses an automated pipeline and Vicuna-v1.5 to generate nearly 318k remote-sensing multimodal instruction-following examples from diverse existing RS datasets.The dataset extends image-text pairs and incorporates multiple task types.
Model: GeoChat unifies image- and region-level reasoning through task tokens, spatial inputs, and object locations in generated responses.These mechanisms support visually grounded conversations alongside image/region captioning, VQA, scene classification, natural-language conversation, and referring expression detection.
Contributions: The introduction presents GeoChat as a multitask RS conversational assistant designed to retain general conversation and instruction-following abilities while extending RS domain knowledge.Its contributions include both the instruction-following dataset and the GeoChat model.
Evaluation: The work establishes evaluation protocols and a task suite for comparing supervised and zero-shot RS conversational VLMs.Evaluations cover image captioning, VQA, and scene classification, among other capabilities.

2. Related Work

Prior VLMs support visual-language understanding in natural scenes, but remote-sensing systems have largely neglected semantic relationships, region-level reasoning, and visual grounding. GeoChat is positioned to address these gaps with broader spatial reasoning capabilities.

Large Vision-Language Models: General VLMs combine a visual backbone, language model, and cross-modal connector, while remote-sensing applications remain comparatively sparse.Existing examples demonstrate instruction-following and visual reasoning primarily for natural scenes.
Remote Sensing VLMs: Remote-sensing VLM research has often emphasized object identification while neglecting semantic understanding of items and their relationships.The related-work discussion contrasts this focus with deeper visual comprehension.
Research Gap: RSGPT cannot perform region-level reasoning or visual grounding, limitations that GeoChat aims to address.GeoChat is described as supporting visually grounded conversations and spatial reasoning across multiple granularity levels.

3. GeoChat: Grounded Remote Sensing VLM

GeoChat unifies image-level, region-level, and grounded conversation tasks for remote sensing through task prompts, spatial inputs and outputs, and a LLaVA-style vision-language architecture. It uses higher-resolution visual encoding and domain-specific instruction tuning to support diverse RS interpretation tasks.

Conversation Tasks: GeoChat handles image-level VQA, scene classification, and image captioning using holistic image context.
Conversation Tasks: Region-level tasks add spatial boxes to image and query inputs, directing attention toward region captioning, region-specific VQA, and multi-turn dialogue.
Conversation Tasks: Grounded tasks use task-specification tokens to generate object locations alongside conversational responses for grounding and referring-expression detection.
Architecture: GeoChat combines a global image encoder, a two-layer MLP adaptor, and an LLM, while adding task prompts and spatial positions in both inputs and outputs.
Architecture: Bounding boxes are represented textually by normalized [0, 100] corner coordinates and a rotation angle, enabling spatial interaction through model inputs and outputs.
Architecture: 1296 patches per image at 504×504 resolution nearly double the patch count and support higher-resolution RS understanding and visual grounding.
Architecture: GeoChat uses a frozen CLIP-ViT(L-14) backbone, an MLP projecting 1024-dimensional tokens to 4096-dimensional language inputs, and Vicuna-v1.5(7B) as its LLM.
Training: The training setup uses LoRA to fine-tune smaller matrices in the language model, accelerating training while avoiding forgetting its original knowledge.

4. RS Multimodal Instruction Dataset

GeoChat’s instruction dataset combines diverse remote-sensing sources and automated language generation to support image- and region-level conversational tasks. The pipeline adds missing classes, extracts object attributes and relationships, and converts these resources into referring and grounding examples.

Instruction and task design: Task-specific instruction formats use Vicuna-generated conversations and task tokens to support captioning, visual question answering, scene classification, grounding, and referring detection.GeoChat uses spatial representations for local-region reasoning and can generate object locations for visual grounding.
Instruction-set scale: Nearly 306k image-instruction pairs are produced for training, with 12k reserved for testing.
Constituent datasets: The instruction set combines object detection, scene classification, visual question answering, and flood-detection datasets to provide diverse remote-sensing training data.The object detection sources include DOTA, DIOR, and FAIR1M through SAMRS; other sources include NWPU-RESISC-45, LRBEN, and a flood-detection VQA dataset.
Dataset enrichment: Pseudo-labeling with ViTAE-RVSA adds missing categories such as buildings, roads, and trees while removing predictions already covered by ground truth.
Referring expressions: Object attributes and spatial relationships are extracted to create referring expressions describing category, color, size, position, and inter-object relations.Attributes use image pixels, bounding-box statistics, and a 3×3 spatial grid; relationships are assigned from object distances, labels, and containment checks.

5. Experiments

Experiments evaluate GeoChat across scene classification, visual question answering, and visual grounding using zero-shot and benchmark-specific comparisons. GeoChat performs strongly on scene classification and VQA, while grounding remains weaker for small or multiple objects despite advantages in region captioning.

Scene classification: GeoChat achieves 84.43% accuracy on UCMerced and 72.03% on AID, significantly outperforming other evaluated VLMs for zero-shot scene classification.MiniGPT-4-v2 returns unrelated classes in this setting, whereas Qwen-VL and LLaVA-1.5 follow instructions but trail GeoChat.
Visual question answering: On RSVQA-HRBEN, GeoChat exceeds other VLMs by 3.9% average accuracy and beats the LLaVA-v1.5 Comparison subset by 15.9%.On RSVQA-LRBEN, it performs close to the specialist RSGPT model and matches the urban-rural classification state of the art.
Visual grounding: The grounding benchmark contains 7,653 referring, 758 grounding, and 555 grounding-description questions evaluated with accuracy@0.5.A prediction counts as correct when its box overlaps the ground-truth box by more than 0.5 IoU.
Visual grounding: Grounding performance is low for small objects and multiple-box predictions, while GeoChat performs better than MiniGPT-4-v2 on medium-size images.
Visual grounding: GeoChat provides better grounding descriptions with slightly better box accuracy and significantly higher ROUGE and METEOR scores for region-level captioning than MiniGPT-4-v2.

6. Conclusion

GeoChat is presented as a unified remote-sensing VLM for image- and region-level conversation, spatial grounding, and multitask evaluation. Its 318k-pair instruction dataset supports robust zero-shot performance across several remote-sensing tasks.

Conclusion: GeoChat unifies image-level queries, region-specific dialogue, and spatially grounded responses for high-resolution remote-sensing imagery.
Conclusion: The 318k-pair multimodal instruction-following dataset uses a diverse multitask format for remote-sensing training.
Conclusion: GeoChat shows robust zero-shot performance across scene classification, VQA, multiturn dialogue, visual grounding, and referring object detection.