PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoNLI - A Natural Language Interpreter for Satellite Imagery

Ashutosh Gandhe, Anupam Rawat, Geet Sethi, Kabir Nasiruddin, Madhav Kotecha, Panav Shah, Rakshit Sawarn, Soumitra Nayak

arXiv:2609.28741v1cs.CV

TL;DR

Heterogeneous training data and task-dependent performance complicate building and comparing unified remote-sensing models. This paper combines EarthMind with multiple grounding pipelines and majority voting in a captioning, VQA, and grounding framework. It reports strong captioning and VQA performance for EarthMind and improved grounding performance from majority voting.

  • Problem

    Heterogeneous training data and inconsistent evaluation settings make it difficult to establish an effective unified pipeline across captioning, VQA, and grounding.

    Source: §I · p. 1

  • Method

    The unified pipeline uses EarthMind for captioning and VQA and combines seven grounding models or pipelines with majority voting.

    Source: §II · p. 1, C. Overall Architecture · p. 4

  • Results

    EarthMind performs strongly on captioning and VQA, while majority voting yields improved grounding performance.

    Source: §IX · p. 6

  • Takeaways & Limitations

    The modular system brings captioning, VQA, and grounding together for remote-sensing scene understanding.

    Source: §IX · p. 6

Abstract

from arXiv · show

Multi-modal multitasking models have shown strong performance on remote sensing datasets. However, because these models are trained on heterogeneous data and vary across tasks, designing a unified model that performs well in captioning, visual question answering (VQA), and visual grounding remains challenging. In this work, we evaluate several models on the VRS Bench and NWPU-VHR-10 datasets. The EarthMind model demonstrates strong results in both captioning and VQA. For grounding, we propose multiple pipelines - RemoteSAM-SAM-v1, RemoteSAM-SAM-v2, and DiffuSAM - and ultimately adopt a majority-voting ensemble across EarthMind, RemoteSAM, SAM3, Falcon, RemoteSAM-SAM3-v1, RemoteSAM-SAM3-v2, and DiffuSAM predictions. Our unified, modular pipeline integrates advanced SAM variants with multimodal LLMs to jointly perform captioning, VQA, and grounding. It achieves 82% accuracy on captioning and 83.32% on VQA, with 90.94%, 52.04%, and 92.06% for binary, numeric, and semantic question types respectively. For grounding, it attains 64.94% accuracy. By combining diverse VLMs with our custom RemoteSAM-SAM3 models through ensemble majority voting, the system delivers more accurate and consistent remote-sensing understanding than task-specific approaches.

I. INTRODUCTION

The paper addresses the challenge of evaluating heterogeneous vision-language models across remote-sensing tasks and proposes a unified system for captioning, VQA, and grounding. It combines EarthMind with new grounding pipelines and describes the three task inputs and outputs.

I. INTRODUCTION

  • Heterogeneous training and inconsistent evaluation complicate model comparison, and EarthMind performs well on captioning and VQA but poorly on grounding.The paper motivates a standardized evaluation and a unified framework across tasks.Source: §I · p. 1

II. INNOVATION & NOVELTY

  • The proposed pipeline uses EarthMind for captioning and VQA, adds three RemoteSAM-SAM3 and DiffuSAM grounding pipelines, and evaluates models on VRS Bench and NWPU-VHR-10.Grounding predictions are also evaluated individually and in majority-voting ensembles.Source: §II · p. 1

III. DETAILED PROBLEM DESCRIPTION

  • The framework covers image-to-text captioning, image-and-question-to-text VQA, and image-and-phrase-to-bounding-box visual grounding.VQA questions may require geometric or semantic understanding and can call for binary, numeric, or semantic answers.Source: §III · p. 2

IV. RELATED WORKS

Prior work spans remote-sensing image-text datasets, general-purpose task models, and remote-sensing-specific and multitask VLMs. The paper evaluates models on a common benchmark by task and uses a majority-voting strategy for grounding, where EarthMind struggles.

IV. RELATED WORKS

  • Remote-sensing datasets expanded from paired image-caption collections to benchmarks supporting multitask evaluation, high-resolution imagery, and diverse vision-language training.The cited examples include RSICD, UCM-Captions, Sydney-Captions, BigEarthNet, RSVG, RS5M, VRSBench, and XLRS-Bench.Source: §IV · p. 2
  • The literature includes general-purpose segmentation, grounding, and captioning models, remote-sensing-specific models, and multitask VLMs adapted for geospatial imagery.Remote-sensing models address dense objects, ultra-high resolution, multispectral variation, and fine-grained land-use semantics.Source: §IV · p. 2

V. METHODOLOGY

  • EarthMind performs strongly on captioning and VQA but struggles with grounding, so the paper evaluates tasks separately on a common benchmark and applies majority voting across grounding pipelines.The grounding set includes EarthMind, RemoteSAM, SAM3, Falcon, RemoteSAM-SAM3-v1, RemoteSAM-SAM3-v2, and DiffuSAM.Source: §V · pp. 2–3

A. RemoteSAM-SAM3

The section develops complementary RemoteSAM–SAM3 and DiffuSAM grounding pipelines, then combines model predictions through majority voting. It also adapts EarthMind for VQA while preserving original training examples.

A. RemoteSAM-SAM3

  • RemoteSAM-SAM3 combines remote-sensing-trained RemoteSAM with general-purpose SAM3, with v2 using DBSCAN clusters and IoU selection to improve spatial continuity over v1.In v1, Qwen identifies the target and SAM3 receives RemoteSAM’s output; fragmented RemoteSAM outputs can still hinder accurate grounding.Source: A. RemoteSAM-SAM3 · p. 3

B. DiffuSAM

  • For small crops SAM3 performs better, while RemoteSAM performs better with multiple candidate objects; DiffuSAM selects between them at a 10% crop-area threshold.DiffuSAM uses a diffusion-generated red box and SAM3 mask to estimate a bounding box, then removes directional prompt words because spatial information is encoded in the crop.Source: B. DiffuSAM · pp. 3–4

C. Overall Architecture

  • EarthMind is used directly for captioning, while VQA fine-tuning emphasizes numeric questions and retains original training examples to help prevent catastrophic forgetting.The final fine-tuning dataset contains 65% numeric, 20% semantic, and 15% binary questions.Source: C. Overall Architecture · p. 4
  • Grounding predictions from seven models are combined by majority voting, selecting the prediction with the highest agreement across models.The ensemble includes EarthMind, RemoteSAM, SAM3, Falcon, RemoteSAM-SAM3-v1, RemoteSAM-SAM3-v2, and DiffuSAM; an exponential term penalizes differing numbers of predicted outputs.Source: C. Overall Architecture · p. 4

D. SAR-to-Optical Image translation

The section describes a conditional diffusion model for translating SAR imagery into optical imagery and a set of image-statistic checks for classifying SAR versus optical input. Experiments use inference mode for captioning and grounding, with EarthMind specifically fine-tuned for VQA.

D. SAR-to-Optical Image translation

  • A conditional diffusion model translates SAR images into corresponding optical imagery for interpretability within the unified remote-sensing pipeline.The model details are provided in Appendix A-E.Source: D. SAR-to-Optical Image translation · p. 5
  • SAR classification checks RGB-channel difference, speckle ratio, coefficient of variation, ENL, and histogram skewness and kurtosis against thresholds.If all checks fail, the image is treated as optical; the authors state that the multiple conditions make optical-to-SAR misclassification very unlikely.Source: D. SAR-to-Optical Image translation · p. 5

A. Experimental Setup

  • Experiments use two 80 GB NVIDIA A100 GPUs; captioning and grounding run in inference mode, while EarthMind is fine-tuned specifically for VQA.The reported VQA fine-tuning followed an observed performance gain.Source: A. Experimental Setup · p. 5

B. Dataset Details •

The section describes VRS Bench and NWPU-VHR-10, then specifies task metrics and a weighted score for overall pipeline evaluation. It also outlines the application’s frontend deployment.

B. Dataset Details •

  • VRS Bench supports captioning, VQA, and visual grounding, with 29,614 images, 29,614 human-verified captions, 52,472 referring expressions, and 123,221 VQA pairs.Its 512×512 images are annotated through semi-automatic generation and human verification.Source: B. Dataset Details • · p. 5
  • NWPU-VHR-10 contains 800 images across 10 object categories, with axis-aligned bounding-box annotations for detection evaluation.It includes 650 positive images with 3,651 annotated objects and 150 negative images.Source: B. Dataset Details • · p. 5

C. Evaluation Details

  • Captioning uses BERT-BLEU4, VQA uses binary, numeric, and semantic scores, and grounding uses mIoU with a count penalty.The overall pipeline score weights captioning at 0.20, grounding at 0.30, and each VQA score—binary, numeric, and semantic—at 0.10, 0.20, and 0.20 respectively.Source: C. Evaluation Details · pp. 5–6

D. Front-End

  • The React frontend handles authentication, uploads, and analysis workflows, communicating with a FastAPI backend through REST endpoints and serving optimized static assets via Nginx.Source: D. Front-End · p. 6

VII. RESULTS ANALYSIS

EarthMind provides the reported captioning and VQA results, while qualitative grounding comparisons favor the proposed majority-voting framework. The paper also identifies numeric VQA and fixed DiffuSAM thresholds as challenges and describes the unified system’s scope.

A. Quantitative Results

  • 82% captioning accuracy is reported for pretrained EarthMind; VQA accuracy on numeric and semantic questions improves after fine-tuning, especially on RSVQA.Binary-question performance remains nearly unchanged; the passage refers to Tables I–II for VQA results but supplies no further values.Source: A. Quantitative Results · p. 6

B. Qualitative Results

  • Majority voting suppresses diffusion-induced hallucinations and produces the most reliable grounding predictions among the illustrated pipelines.SAM3 alone often misses relevant objects; RemoteSAM-SAM3-v1 and v2 improve outputs, while EarthMind performs moderately well.Source: B. Qualitative Results · p. 6

VIII. CHALLENGES AND FUTURE WORK

  • Numeric VQA remains difficult, and DiffuSAM’s fixed 99% DBSCAN and 10% cropping thresholds may not generalize across scenarios.The authors propose adaptive thresholding based on model confidence or image characteristics as future work.Source: §VIII · p. 6

IX. CONCLUSION

  • The unified modular pipeline combines EarthMind for captioning and VQA with three new grounding pipelines and majority voting, with optional SAR-to-optical translation.The conclusion also reports fine-tuning EarthMind as improving VQA, particularly for numeric questions.Source: §IX · p. 6

APIS AND LICENSING

  • The implementation uses no third-party APIs, and its open-source models use permissive licenses including MIT, SAM, and Apache-2.0.Source: APIS AND LICENSING · p. 6

APPENDIX A EXISTING MODEL ARCHITECTURES

EarthMind combines optical and SAR imagery with task text to form visual tokens for language-model processing. Its architecture uses cross-sensor attention and text-guided attention, with adaptive pooling to align modalities of differing resolutions.

A. EarthMind [22]

  • EarthMind uses a global semantic encoder and grounding encoder whose outputs are projected into language-model-aligned visual tokens for generative and dense prediction tasks.A segmentation-guidance token supports both task types; EarthMind is the captioning and VQA backbone in the illustrated unified model.Source: A. EarthMind [22] · p. 9, Figure 9 · p. 10

1) Cross-Sensor Attention:

  • Adaptive pooling aligns optical imagery to differing resolutions before bidirectional cross-modal attention relates optical and SAR tokens.The attention represents optical-to-SAR and SAR-to-optical token relationships.Source: 1) Cross-Sensor Attention · p. 9

2) Text-Guided Attention:

3) Adaptive Fusion:

EarthMind fuses cross-sensor information with text relevance through learned balance weights, producing compact multimodal tokens. The section also describes how RoI-based summarization and self-instruct question generation support captioning and VQA, alongside RemoteSAM’s mask-based processing.

3) Adaptive Fusion:

  • Adaptive fusion combines cross-modal complementarity and text relevance using a learnable balance parameter, then normalizes weights to produce compact multimodal tokens.The fused representation preserves cross-sensor information while remaining efficient for LLM processing.Source: 3) Adaptive Fusion · pp. 10–11

4) Improvement of Captioning and VQA tasks:

  • Segmentation masks highlight regions for GPT-4o caption refinement, and few-shot self-instruct prompting expands enhanced captions into diverse VQA pairs.The passage attributes strong captioning and VQA performance to these steps while noting that EarthMind can still struggle with grounding.Source: 4) Improvement of Captioning and VQA… · p. 11

B. RemoteSAM [15]

  • RemoteSAM’s visual and text features are fused into a score map, thresholded into a binary mask, then converted to boxes; EPOC refines adjacent-target handling.For counting, the number of objects is the cardinality of distinct connected components in the mask.Source: B. RemoteSAM [15] · p. 11

C. SAM3 [11]

The section describes SAM3’s prompt-conditioned segmentation architecture, Falcon’s generalized prompt-based sequence-to-sequence design, and a SAR-to-optical translation model trained with consistency and adversarial losses.

C. SAM3 [11]

  • SAM3 uses a prompt-agnostic image encoder, prompt tokens, a unified detector, and a transformer head to produce masks in one forward pass, but may perform worse than remote-sensing-specific models.Its training data is broad and concept-rich but not specific to remote sensing.Source: C. SAM3 [11] · p. 11

D. Falcon [21]

  • Falcon converts task prompts to a predefined phrasing and processes image and text features with an encoder-decoder model to generate outputs.The passage notes that compared models differ in training datasets and parameter counts.Source: D. Falcon [21] · p. 11

E. SAR-to-Optical Image Translation [32]

  • SAR-to-optical translation conditions a U-net on SAR imagery and a noised optical target; consistency loss aligns teacher and student predictions, while adversarial loss aligns student output with ground truth.The model uses consistency distillation to generate in few steps; Table V is described as summarizing model details for EarthMind, RemoteSAM, SAM3, and Falcon.Source: E. SAR-to-Optical Image Translation [32] · p. 12, Table V · p. 12

APPENDIX B DETAILS OF EVALUATION METRICS

Captioning quality combines semantic and n-gram similarity with a length penalty. VQA metrics vary by answer type: exact match for binary answers, relative error for numeric predictions, and BERT-BLEU for descriptive answers.

A. Captioning Metric •

  • Captioning similarity combines BERT semantic similarity with BLEU n-gram structure and applies a length penalty.The final caption similarity score is presented after the length penalty.Source: A. Captioning Metric • · p. 12

B. VQA Metrics •

  • VQA uses exact match for binary answers, relative error for numeric predictions, and the captioning BERT-BLEU formulation for descriptive answers.Source: B. VQA Metrics • · p. 12

C. Grounding Metrics •

Grounding metrics assess localization overlap, thresholded correctness, count deviation, and a combined localization-and-count score.

C. Grounding Metrics •

  • Grounding evaluation computes IoU between predicted and ground-truth boxes, then averages IoU across matched instances.Source: C. Grounding Metrics • · p. 12
  • A grounding prediction counts as correct when its IoU exceeds threshold τ, and count accuracy is measured by average absolute count deviation.Source: C. Grounding Metrics • · p. 12
  • mIoU with Count Penalty jointly evaluates localization quality and instance-count correctness using an exponential penalty.The penalty applies when predicted and ground-truth output counts differ.Source: C. Grounding Metrics • · p. 12

Related papers

Loading 2609.28741v1…