PaperlayerView exact revision on arXiv

Source-linked AI summary

XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?

Fengxiang Wang, Hongzhen Wang, Mingshuo Chen, Di Wang, Yulin Wang, Zonghao Guo, Qiang Ma, Long Lan, Wenjing Yang, Jing Zhang, Zhiyuan Liu, Maosong Sun

arXiv:2503.23771v1cs.CV

TL;DR

Existing RS benchmarks use smaller images, face annotation-quality concerns, and insufficiently cover complex perception and reasoning. XLRS-Bench addresses these gaps with large human-verified imagery, broad task coverage, and evaluation of general and RS-focused MLLMs, which still struggle on ultra-high-resolution RS understanding.

  • Problem

    Existing RS benchmarks use small images, may contain annotation errors or biases, and insufficiently assess interactive perception and complex reasoning needed for real-world RS scenarios.

  • Method

    XLRS-Bench combines 1,400 large real-world RS images, expert annotation and verification, a semi-automated captioning pipeline, and 16 sub-tasks covering perception and reasoning.

  • Results

    Current general-purpose and RS-specific MLLMs still struggle to understand ultra-high-resolution RS imagery, with GPT-4o below 25% accuracy on complex spatiotemporal reasoning tasks.

  • Takeaways & Limitations

    XLRS-Bench provides a multidimensional test bed emphasizing real-world decision-making and spatiotemporal change detection for further MLLM development in RS.

  • Takeaways & Limitations

    The benchmark addresses limitations of prior RS evaluations, but its reported scope remains evaluation of MLLMs in ultra-high-resolution RS scenarios.

Abstract

from arXiv · show

The astonishing breakthrough of multimodal large language models (MLLMs) has necessitated new benchmarks to quantitatively assess their capabilities, reveal their limitations, and indicate future research directions. However, this is challenging in the context of remote sensing (RS), since the imagery features ultra-high resolution that incorporates extremely complex semantic relationships. Existing benchmarks usually adopt notably smaller image sizes than real-world RS scenarios, suffer from limited annotation quality, and consider insufficient dimensions of evaluation. To address these issues, we present XLRS-Bench: a comprehensive benchmark for evaluating the perception and reasoning capabilities of MLLMs in ultra-high-resolution RS scenarios. XLRS-Bench boasts the largest average image size (8500$\times$8500) observed thus far, with all evaluation samples meticulously annotated manually, assisted by a novel semi-automatic captioner on ultra-high-resolution RS images. On top of the XLRS-Bench, 16 sub-tasks are defined to evaluate MLLMs' 10 kinds of perceptual capabilities and 6 kinds of reasoning capabilities, with a primary emphasis on advanced cognitive processes that facilitate real-world decision-making and the capture of spatiotemporal changes. The results of both general and RS-focused MLLMs on XLRS-Bench indicate that further efforts are needed for real-world RS applications. We have open-sourced XLRS-Bench to support further research in developing more powerful MLLMs for remote sensing.

1. Introduction

Existing RS MLLM benchmarks are limited by small imagery, annotation-quality concerns, and narrow evaluation dimensions. XLRS-Bench addresses these gaps with large real-world images, human-involved verification, and broad perception and reasoning coverage.

  • Real-world RS images can reach 10,000×10,000 pixels, while existing benchmarks often use 512×512 slices that miss long-range spatial-semantic relationships.
  • Automated annotation can introduce hallucinations, errors, and linguistic biases that compromise benchmark quality and may favor similar models.
  • Existing RS benchmarks primarily assess basic perception, leaving interactive perception and complex reasoning insufficiently evaluated.
  • XLRS-Bench contains 1,400 ultra-high-resolution RS images averaging 8,500×8,500 pixels, with 45 experts annotating and cross-verifying 16 sub-tasks.The benchmark includes VQA, captioning, and visual grounding formats.
  • XLRS-Bench covers 10 perception indicators and 6 reasoning dimensions across 16 sub-tasks and 45,942 questions.Its reasoning tasks include planning and change detection in long spatial-temporal scenarios.
  • The benchmark evaluates prominent general and RS-focused MLLMs to provide insights into understanding real RS scenarios.

2. Related Work

General and RS multimodal benchmarks have expanded, but prior evaluations typically cover limited tasks, small datasets, or small images. XLRS-Bench is positioned as a larger test bed for perception and reasoning in real-world RS imagery.

  • General multimodal benchmarks often assess large-scale VLMs on one or a few tasks, motivating more comprehensive evaluation datasets.
  • RSIEval and LHRS-Bench provide relatively small human-annotated RS benchmark collections focused mainly on captioning and VQA.RSIEval contains 100 captions and 936 visual question-answer pairs, while LHRS-Bench provides 108 images and 690 questions.
  • General and RS-specific MLLMs still face challenges because they are insufficiently tested on large real-world RS images, with RS models often evaluated below 512×512 pixels.
  • XLRS-Bench provides a rigorous test bed for large-size real RS images and evaluates both perception and reasoning capabilities.

3. XLRS-Bench

XLRS-Bench evaluates MLLMs across three levels and 16 sub-tasks, combining broad perception and reasoning dimensions with ultra-high-resolution RS imagery. Its construction uses diverse large images, structured vision-language tasks, and semi-automated detailed captioning.

  • XLRS-Bench evaluates perception and reasoning through three levels and 16 sub-tasks, including complex reasoning and spatiotemporal change detection.
  • Data Source: The benchmark collects 1,400 large-size RS images from detection, segmentation, and change-detection datasets, including 840 images at 10,000×10,000 pixels.
  • Data Construction: Its visual-language data include multiple-choice VQA, visual grounding references, and detailed captions covering global context, regional details, object attributes, counts, anomalies, and reasoning.
  • Image Caption: The captioning pipeline divides each image into nine sub-images, combines them with a compressed full image, and prompts GPT-4o to produce general-to-specific descriptions.
  • Resolution: With an average image size of 8,500×8,500 pixels, XLRS-Bench provides extensive views of real RS scenes and includes more than 10,000 question-answer pairs from 840 10,000×10,000 images.

4. Experiment

The experiments compare open-source, proprietary, and RS-specialized MLLMs under zero-shot evaluation with uniform prompts. Results show task-dependent strengths, including Qwen2-VL’s VQA performance, GPT-4o’s long-caption generation, and persistent difficulty with fine-detail and spatiotemporal RS understanding.

  • Evaluation Setup: The evaluation compares open-source VLMs, closed-source VLMs, and GeoChat using zero-shot inference with uniform prompts.
  • VQA Results: Qwen2-VL performs reliably on spatiotemporal VQA sub-tasks, while GPT-4o remains below 25% accuracy on these complex change-detection tasks.
  • VQA Results: Models allowing higher-resolution inputs, such as Qwen2-VL, outperform models constrained by CLIP-based visual encoders on detailed RS tasks.
  • Captioning Results: GPT-4o and GPT-4o-mini outperform other models on lengthy detailed captions, although open-source and RS-specific models perform poorly.
  • Limitations: Current high-resolution MLLMs are limited to 4K images, so compressing larger RS imagery can reduce 5–10-pixel objects to a single pixel.

5. Conclusion

XLRS-Bench is a comprehensive benchmark for evaluating MLLMs on ultra-high-resolution RS scenarios, combining large images, human-verified annotations, and multidimensional tasks. Results show that current general-purpose and RS-specific MLLMs still struggle with these images.

  • XLRS-Bench evaluates MLLM perception and reasoning in ultra-high-resolution RS scenarios using multidimensional tasks and human-verified annotations.It emphasizes real-world decision-making and spatiotemporal change detection.
  • Current general-purpose and RS-specific MLLMs still struggle to understand ultra-high-resolution RS imagery.

A.1. Overview of the Appendix

The appendix supplements the main paper with details about XLRS-Bench, human evaluations, capability analyses, task results, visualizations, datasheets, and limitations.

  • The appendix provides details excluded from the main paper because of space constraints.
  • Its sections cover benchmark details, human evaluations, L-2 and L-3 analyses, sample visualizations, dataset datasheets, and limitations with societal impact.

A.2. More Details of XLRS-Bench

Additional dataset details clarify how VQA, visual grounding, and image captioning relate to XLRS-Bench’s L3 sub-tasks. Visual grounding spans both perception and reasoning categories.

  • Table 5 presents statistics for VQA, visual grounding, and image captioning and relates them to L3 sub-tasks.
  • Fine-grained Visual Grounding is classified under perception, whereas Condition-based Visual Grounding is classified under reasoning.

A.3. Human Evaluations on XLRS-Bench

Human evaluation of XLRS-Bench used randomly selected VQA questions answered simultaneously by two groups, with final accuracy averaged across groups. Humans exceeded 90% accuracy, while existing MLLMs performed significantly worse, although the human task itself was demanding.

  • 30 questions from each VQA sub-task were answered simultaneously by two groups, and final accuracy was averaged across both groups.
  • Human accuracy consistently exceeded 90%, supporting the reliability of XLRS-Bench.
  • Analyzing ultra-high-resolution RS images required intense focus and frequent zooming, making human evaluation challenging, especially for global counting.
  • Existing MLLMs, including GPT-4o, performed significantly worse than humans, possibly because of insufficient training on real ultra-high-resolution RS data.

A.4. More Analysis of Results on XLRS-Bench

XLRS-Bench exposes broad weaknesses across ultra-high-resolution remote-sensing tasks, with especially large gaps in spatiotemporal reasoning and visual grounding. Results also suggest that model scale alone does not resolve these challenges.

  • Overall performance: Most MLLMs score below 50% across all 16 evaluation dimensions, compared with the 80%–90% accuracy typically observed on common benchmarks.The authors attribute this pattern to the distinct difficulty of ultra-high-resolution remote sensing and limited pretraining on annotated data.
  • Reasoning gaps: About 70% accuracy on Anomaly Reasoning falls to 15.2% on Spatiotemporal Reasoning, revealing a sharp performance gap.Anomaly tasks rely on global patterns, whereas spatiotemporal tasks require intricate local temporal modeling.
  • Model scale: LLaVA-Next (Llama3-8B) offers little advantage over smaller Qwen2-VL (Qwen2-7B) on Counting and Scene Classification subtasks.The analysis points instead to diverse pretraining data, task alignment, and visual-module capability as important factors.
  • Visual grounding: Most models achieve less than 1.0% on both Acc@0.5 and Acc@0.7 for visual grounding in the Chinese and English benchmarks.The reported issues include inadequate local feature extraction, weak cross-modal alignment, poor real-world generalization, and limited reasoning.

A.5. Sub-tasks (L-3 capability) Results on XLRS-Bench

The L-3 analysis shows that XLRS-Bench reverses the usual perception–reasoning pattern: models struggle more with perception on its much larger images, especially when tasks depend on local details.

  • Evaluation structure: XLRS-Bench evaluates L-3 VQA capabilities across perception and reasoning dimensions, with visual-grounding results reported separately.Tables 6 and 7 cover perception and reasoning VQA results, while Table 8 summarizes visual grounding.
  • Cross-capability pattern: MLLMs generally perform worse in perception than reasoning on XLRS-Bench, reversing the trend reported on most other benchmarks.The benchmark’s images average 8,500×8,500 pixels, 24 times the resolution of MME-Realworld, while current models commonly face a 4K input limit.
  • Perception challenges: Object Spatial Relationship and Object Color tasks achieve significantly lower accuracy because sparse local details and subtle relationships are difficult to capture.For example, LLaVA1.5 achieves only 17.45% on Object Color in the Chinese benchmark.
  • Perception challenges: Downsampling caused by resolution and computational constraints degrades local features, while current visual encoders emphasize global semantics over local-global interactions.The analysis identifies stronger local feature modeling as necessary for detailed perception tasks.

A.6. Samples and Hard Cases of XLRS-Bench

XLRS-Bench’s examples span visual question answering, captioning, grounding, counting, land-use classification, route planning, anomaly analysis, and dataset construction. Case studies organize model failures into perceptual, reasoning, knowledge, capability, and instruction-following errors, while the benchmark’s scope remains bounded by its sensor coverage and deployment risks.

  • Case-study scope: The case studies present VQA, image captioning, visual grounding, and error analyses for L-3 subtasks, including five error categories.The categories are perception error, reasoning error, lack of knowledge, lack of capability, and failure to follow instructions.
  • Error categories: Perception errors are especially prevalent because visual encoders struggle to detect objects with minimal pixel representation in ultra-high-resolution images.The paper illustrates these errors with examples from multiple figures.
  • Representative tasks: The examples include anomaly detection, environmental reasoning, route planning, change-aware counting, complex counting, and overall or regional land-use classification.These samples use questions about industrial clustering, shortest routes, house-count changes, vehicle counts, and selecting all land-use types present.
  • Dataset composition: XLRS-Bench contains 1,400 ultra-high-resolution images, 934 detailed captions, 32,389 VQA pairs, and 12,619 visual-grounding instances.The images come from existing detection and segmentation datasets, while textual annotations were independently created by the authors.
  • Scope and deployment: The benchmark currently focuses on visible-light imagery and may not generalize to sensor data such as multispectral imagery.The authors also warn that excessive reliance on evaluation datasets can create overconfidence in autonomous systems, requiring safety measures and human supervision.
Loading 2503.23771v1…