PaperlayerView exact revision on arXiv

Source-linked AI summary

VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

Xiang Li, Jian Ding, Mohamed Elhoseiny

arXiv:2406.12384v2cs.CV

TL;DR

Remote-sensing vision-language datasets have been constrained by narrow task coverage, limited object detail, and inadequate quality control. VRSBench addresses these gaps with a semi-automatic, human-verified benchmark spanning captions, object references, and visual question answering, and evaluates models across three tasks. Its demanding grounding scenarios remain challenging even for strong models.

  • Problem

    Remote-sensing vision-language datasets often focus on single tasks, lack detailed object information, or use automatically generated annotations without sufficient quality control.

  • Method

    VRSBench uses a semi-automatic pipeline combining object attributes, prompt engineering, GPT-4 inference, and human verification to create a unified benchmark.

  • Results

    VRSBench supports evaluation of image captioning, visual grounding, and visual question answering, with GeoChat reaching 49.8% grounding accuracy at a 0.5 threshold while still failing to achieve satisfactory performance.

  • Takeaways & Limitations

    The benchmark provides human-verified, object-rich annotations for developing and evaluating vision-language models across multiple remote-sensing understanding tasks.

  • Takeaways & Limitations

    Even the best-performing GeoChat model achieves only 49.8% grounding accuracy at a 0.5 threshold in VRSBench’s demanding multi-instance scenarios.

Abstract

from arXiv · show

We introduce a new benchmark designed to advance the development of general-purpose, large-scale vision-language models for remote sensing images. Although several vision-language datasets in remote sensing have been proposed to pursue this goal, existing datasets are typically tailored to single tasks, lack detailed object information, or suffer from inadequate quality control. Exploring these improvement opportunities, we present a Versatile vision-language Benchmark for Remote Sensing image understanding, termed VRSBench. This benchmark comprises 29,614 images, with 29,614 human-verified detailed captions, 52,472 object references, and 123,221 question-answer pairs. It facilitates the training and evaluation of vision-language models across a broad spectrum of remote sensing image understanding tasks. We further evaluated state-of-the-art models on this benchmark for three vision-language tasks: image captioning, visual grounding, and visual question answering. Our work aims to significantly contribute to the development of advanced vision-language models in the field of remote sensing. The data and code can be accessed at https://github.com/lx709/VRSBench.

1 Introduction

VRSBench addresses limitations in remote-sensing vision-language datasets by unifying detailed captions, object references, and open-ended question answering with human verification. It supports training and evaluating models across image captioning, visual grounding, and visual question answering.

  • 1 Introduction: The benchmark responds to remote-sensing-specific challenges, including very small objects and complex overhead spatial reasoning.These characteristics make datasets tailored to remote sensing important for developing effective vision-language models.
  • 1 Introduction: Existing datasets are limited by single-task focus, inconsistent multi-source integration, brief captions, and insufficient quality control.These limitations include automatically generated annotations, limited object detail, and simplistic grounding or question-answer scenarios.
  • 1 Introduction: 29,614 images are enriched with human-verified detailed captions, complex object references, and question-answer pairs.The benchmark emphasizes object-rich annotations and human verification rather than brief or automatically generated descriptions alone.
  • 1 Introduction: VRSBench unifies detailed captioning, visual grounding, and visual question answering in one benchmark for remote sensing images.The dataset is designed for both training and comprehensive evaluation across these vision-language capabilities.
  • 1 Introduction: The work benchmarks state-of-the-art vision-language models on detailed image captioning, visual grounding, and visual question answering.The benchmark suite is intended to evaluate multiple capabilities using the unified dataset.

2 Pipeline

The VRSBench pipeline extracts image and object attributes, uses engineered prompts and GPT-4 inference to generate annotations, and applies human verification. It is designed to produce detailed captions, unambiguous references, and diverse question-answer pairs at scale.

  • 2 Pipeline: The pipeline combines attribute extraction, prompt engineering, GPT-4 inference, and human verification to construct VRSBench annotations.Object information is derived from detection labels and supplied alongside images during annotation generation.
  • 2 Pipeline: Object metadata includes categories, bounding boxes, positions, sizes, and whether objects are unique within their categories.Uniqueness information supports references that distinguish among multiple similar objects.
  • 2 Pipeline: DOTA-v2 patches average 14.2 instances versus 3.3 in DIOR, creating a denser and more challenging training environment.The dataset uses 512 × 512 patches from DOTA-v2 and DIOR and focuses visual grounding on horizontal bounding boxes.
  • 2 Pipeline: Prompts request detailed captions, referring sentences for 1-5 objects, and 3-10 question-answer pairs in a structured JSON format.Referring sentences are required to identify individual objects unambiguously.
  • 2 Pipeline: GPT-4V outputs are recursively regenerated up to five times when excluding phrases appear, after which affected annotations are removed.This filtering procedure targets phrases such as “not provide,” “not specified,” and “unknown.”
  • 2 Pipeline: Human annotators validate every generated annotation, while remaining errors are attributed partly to limited remote-sensing exposure and hallucinations.The validation process uses domain experts and guidelines for removing uncertain or irrelevant content and ensuring unambiguous references.

3 VRSBench Dataset

VRSBench contains 29,614 remote sensing images with human-verified captions, referring sentences, and question-answer annotations. Its annotations combine detailed image and object descriptions with complex referring expressions and diverse questions.

  • Dataset Overview: 29,614 images are annotated with 29,614 caption sentences, 52,472 referring sentences, and 123,221 question-answer pairs.Each image is 512 × 512 pixels, and original object detection labels and attributes are also provided.
  • Detailed Caption: Captions combine general scene overviews with object-specific details such as quantity, color, shape, size, and spatial position.They also describe image source, resolution, color or panchromatic distinction, and scene type.
  • Detailed Caption: The caption statistics summarize caption length, sentence count, and aggregate dataset characteristics.Figure 3 presents probability density functions for caption length and sentence number alongside summative statistics.
  • Object Referring: Each image contains referring sentences for 1-5 distinct objects, designed to identify each object independently and unambiguously.The descriptions use attributes including color, shape, position, size, relative position, and relative size.
  • Visual Question Answering: Question-answer annotations cover object, scene, and reasoning topics, including category, existence, quantity, color, size, position, direction, and scene characteristics.The dataset also includes questions about visually obvious houses, roads, and trees beyond source detection labels.
  • Object Referring: Object-reference statistics cover frequent categories, referring-sentence length, object size, common words, and unique versus non-unique objects.Figure 4 organizes these distributions across five views of the referring-sentence annotations.
  • Visual Question Answering: Question-answer statistics report question-type distributions and the most frequent words in questions and answers.Figure 5 presents the question and answer vocabulary through separate word clouds.

4 Benchmark Evaluation

VRSBench defines three remote-sensing vision-language tasks and evaluates state-of-the-art models using standardized benchmark settings. Fine-tuning on VRSBench improves captioning, grounding, and question answering, while challenging non-unique-object grounding remains limited.

  • Benchmark Overview: VRSBench-Cap predicts detailed image descriptions, VRSBench-Ref localizes text-referred objects, and VRSBench-VQA answers questions about image content.The benchmark uses separate nonoverlapping training and evaluation splits based on official DOTA and DIOR splits.
  • Benchmark Overview: The benchmark evaluates captioning, grounding, and VQA with established caption metrics, accuracy@τ grounding thresholds of 0.5 and 0.7, and categorized VQA accuracy.Grounding uses horizontal bounding boxes, while supplementary results cover oriented bounding boxes.
  • Detailed Image Caption: LLaVA-1.5 fine-tuned on VRSBench achieves BLEU-1 48.1 and CIDEr 33.9, while GPT-4V obtains the best CHAIR score.Generated captions average 49 words, close to the ground-truth average length.
  • Visual Grounding: Fine-tuning on VRSBench improves visual grounding over the unfine-tuned GeoChat baseline, although unique-object referring remains easier than non-unique referring.MiniGPT-v2 performs better on non-unique objects despite worse overall grounding performance.
  • Benchmark Overview: GPT-4V performs worse than models trained on remote-sensing captioning and grounding, partly because its prompts omit object information.The experiment directly queries GPT-4V for captions, object locations, and visual-question answers.
  • Visual Grounding: Even the best-performing GeoChat model reaches only 49.8% grounding accuracy at threshold 0.5, reflecting the difficulty of distinguishing multiple same-category instances.The authors identify this difficulty as motivating more advanced visual grounding techniques.

5 Related Work

Remote-sensing captioning datasets range from brief descriptions to large automatically generated collections, while grounding and VQA datasets address increasingly complex spatial and question-answering scenarios. VRSBench emphasizes detailed, human-verified annotations to improve reliability and object-level coverage.

  • Remote Sensing Image Captioning Datasets: VRSBench provides 29,614 human-verified captions that are detailed and object-rich, addressing the brevity of older datasets and quality concerns in automatically generated large-scale captions.RSGPT offers human-generated detail but only 2,585 image-text pairs, whereas RS5M contains five million automatically generated captions.
  • Remote Sensing Visual Grounding Datasets: Remote-sensing visual grounding involves complex geospatial relationships and objects that may not be prominently visible in imagery.The supplied passage introduces this research direction and its early datasets.
  • Remote Sensing Visual Question Answering Datasets: Remote-sensing VQA datasets include large automatically generated question-answer collections built from imagery sourced from OpenStreetMap and other sources.RSVQA includes separate low- and high-resolution collections with different image and question-answer counts.

6 Conclusion and future work

The paper presents VRSBench as a versatile, human-verified benchmark combining detailed captions, object referring, and VQA. Its current scope is limited to RGB imagery, with future expansion planned for additional remote-sensing modalities.

  • Conclusion: VRSBench combines detailed captions, object referring, and visual question answering with rich object information and human verification.The benchmark is intended to address limited object detail and quality-control problems in earlier datasets.
  • Future Work: VRSBench currently covers RGB images, with future work targeting infrared, multispectral, hyperspectral, SAR, and temporal remote-sensing data.The planned expansion is intended to broaden utility across diverse observation conditions.

7 Broader Impact

VRSBench is a comprehensive benchmark for developing and evaluating generalist vision-language models across remote sensing tasks. Its annotations and standardized files support research, model comparison, and education.

  • VRSBench supports training and evaluation of vision-language models across image captioning, visual grounding, and visual question answering.
  • The benchmark contains 29,614 images, 29,614 captions, 52,472 object refers, and 123,221 visual question-answer pairs.
  • Standardized training and evaluation files organize images and annotations for captioning, visual grounding, and VQA experiments.
  • VRSBench is intended for academic research, model evaluation, and educational activities involving complex visual and textual tasks.
  • The dataset is positioned as a resource for advancing remote sensing image analysis by supporting multiple tasks and standardized model comparisons.

A.5 Limitations

VRSBench has geographic and interpretive limitations despite human verification and planned community feedback. Its coverage is bounded by the source datasets and annotations may reflect subjective judgments.

  • Geographic diversity is limited to the regions represented in the DOTA-v2 and DIOR source datasets.
  • Human verification does not eliminate potential annotation bias arising from subjective interpretations of visual data.
  • The dataset uses publicly available non-sensitive satellite imagery, with no identifiable individuals or private property.
  • Users are encouraged to report issues or biases so the dataset and annotations can be refined over time.

B Dataset Collection Details

VRSBench is constructed from DOTA-v2 and DIOR image patches filtered for object annotations. The source-dataset statistics are summarized in Table 6.

  • The dataset source comprises over 20,310 DOTA-v2 patches and 9,304 DIOR patches after filtering out patches without object annotations.
  • Each source image is divided into 512 × 512 pixel patches before image-level and object-level information is extracted.
  • Table 6 reports statistics for the source object detection datasets.

C URL to Data and Metadata

VRSBench is distributed through a dedicated platform and Hugging Face repository. Dataset metadata is documented using the Croissant framework and MLCommons standards.

  • The dataset and its annotations can be accessed and downloaded through a dedicated platform.
  • The Hugging Face repository provides practical examples, dataset downloads, and detailed metadata.
  • Metadata is documented with the Croissant framework to support comprehensive coverage and MLCommons Croissant compliance.

D Author Statement and Data License

The dataset is released under CC-BY-4.0 with organized JSON annotations, while GPT-4V prompts guide caption, referring-expression, and VQA generation and verification.

  • D Author Statement and Data License: CC-BY-4.0 permits unrestricted use, distribution, and reproduction when the original work is properly cited.
  • D Author Statement and Data License: Annotations are organized in standard JSON format, and the dataset is archived on Hugging Face for long-term accessibility.
  • D Author Statement and Data License: GPT-4V prompts require detailed captions, unambiguous object references, and diverse VQA pairs returned in a prescribed JSON structure.
  • D Author Statement and Data License: Verification instructions require correct caption information, distinguishable referring sentences, and clear answers limited to a single word or phrase of at most three words.

G.1 Training details

Comparative models use standardized training settings, while the visual-grounding evaluation compares horizontal and oriented bounding boxes and reports qualitative outputs for three tasks.

  • G.1 Training details: Comparative methods train for five epochs with batch size 32 on four Nvidia A100 GPUs, using a 2e-4 learning rate with cosine decay and warm-up.
  • G.1 Training details: The evaluation uses horizontal boxes in the main setting and extends GeoChat testing to oriented boxes defined by center, dimensions, and rotation angle.
  • G.1 Training details: 24.3% grounding accuracy at threshold 0.5 with oriented bounding boxes is below 49.8% with horizontal bounding boxes.
  • G.1 Training details: Qualitative examples cover detailed captioning, visual grounding, and visual question answering, with correct information or answers highlighted in green.
Loading 2406.12384v2…