PaperlayerView exact revision on arXiv

Source-linked AI summary

DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response

Junjue Wang, Weihao Xuan, Heli Qi, Zhihao Liu, Kunyi Liu, Yuhan Wu, Hongruixuan Chen, Jian Song, Junshi Xia, Zhuo Zheng, Naoto Yokoya

arXiv:2505.21089v2cs.CV

TL;DR

Disaster scenes across hazards, regions, and sensors pose challenges for vision-language models, while existing remote-sensing datasets leave disaster-specific coverage limited. DisasterM3 assembles bi-temporal optical and SAR imagery with disaster-focused instruction tasks and benchmarks 14 VLMs. The authors report that fine-tuning four models produces stable improvements across tasks and sensors.

  • Problem

    Disaster scenes spanning varied hazards, regions, and sensors challenge VLMs, while remote-sensing datasets have limited coverage of specialized disaster assessment.

    Source: §1 · pp. 1–2

  • Method

    DisasterM3 combines 26,988 bi-temporal optical and SAR images with 123,010 instruction pairs, and the authors benchmark 14 VLMs and fine-tune four models.

    Source: §1 · pp. 2–3

  • Results

    Fine-tuning four VLMs on DisasterM3 yields stable improvements across evaluation tasks and sensor modalities.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The fine-tuned models are presented as solid baselines for disaster-related vision-language tasks.

    Source: §1 · p. 3

    Standardizing imagery to 0.8 m limits evaluation of robustness to the diverse spatial resolutions found in operational settings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large vision-language models (VLMs) have made great achievements in Earth vision. However, complex disaster scenes with diverse disaster types, geographic regions, and satellite sensors have posed new challenges for VLM applications. To fill this gap, we curate a remote sensing vision-language dataset (DisasterM3) for global-scale disaster assessment and response. DisasterM3 includes 26,988 bi-temporal satellite images and 123k instruction pairs across 5 continents, with three characteristics: 1) Multi-hazard: DisasterM3 involves 36 historical disaster events with significant impacts, which are categorized into 10 common natural and man-made disasters. 2)Multi-sensor: Extreme weather during disasters often hinders optical sensor imaging, making it necessary to combine Synthetic Aperture Radar (SAR) imagery for post-disaster scenes. 3) Multi-task: Based on real-world scenarios, DisasterM3 includes 9 disaster-related visual perception and reasoning tasks, harnessing the full potential of VLM's reasoning ability with progressing from disaster-bearing body recognition to structural damage assessment and object relational reasoning, culminating in the generation of long-form disaster reports. We extensively evaluated 14 generic and remote sensing VLMs on our benchmark, revealing that state-of-the-art models struggle with the disaster tasks, largely due to the lack of a disaster-specific corpus, cross-sensor gap, and damage object counting insensitivity. Focusing on these issues, we fine-tune four VLMs using our dataset and achieve stable improvements across all tasks, with robust cross-sensor and cross-disaster generalization capabilities. The code and data are available at: https://github.com/Junjue-Wang/DisasterM3.

1 Introduction

DisasterM3 is introduced as a multi-hazard, multi-sensor, multi-task dataset for disaster assessment and response. The paper benchmarks existing VLMs and reports gains after fine-tuning four models on the dataset.

1 Introduction

  • The motivation is timely disaster relief: remote sensing is used for damage assessment and mapping, while disaster impacts create urgent demands for response.The paper frames VLMs as potentially useful to rescue teams and planning personnel.Source: §1 · pp. 1–2
  • Its nine tasks organize five capabilities—recognition, counting, localization, reasoning, and report generation—around paired pre- and post-disaster scenes.Post-disaster imagery may be optical or SAR.Source: Figure 1 · p. 2, §1 · p. 2
  • Fine-tuning Qwen2.5-VL, InternVL3, LISA, and PSALM yields consistent improvements across evaluation tasks and sensor modalities, with prompt-variation stability.The authors present the fine-tuned models as strong baselines.Source: §1 · p. 3
  • The dataset contains 26,988 bi-temporal optical and SAR images and 123,010 instruction pairs spanning disaster recognition, damage assessment, segmentation, relational reasoning, description, and restoration advice.It covers 36 disaster events across five continents.Source: §1 · pp. 2–3
  • The authors benchmark 14 advanced VLMs and report that cutting-edge models struggle with disaster tasks.The paper identifies disaster-focused vision-language understanding as an area for future improvement.Source: §1 · pp. 2–3

2 Related Work

Related work covers general VLMs and remote-sensing vision-language datasets, identifying limits in fine-grained damage assessment and specialized disaster coverage.

2 Related Work

  • General VLMs that produce holistic textual descriptions are limited for damage assessment requiring pixel-level detail.Grounding and mask-decoder approaches extend fine-grained visual understanding, but the passage notes generic VLM limitations in disaster scenarios.Source: §2 · p. 3
  • The table-comparison passage defines MT pairs as the number of pre- and post-disaster image pairs.Source: §2 · p. 3
  • Remote-sensing datasets cover VQA, captioning, grounding, ultra-high-resolution imagery, and instruction following, but many do not target specialized geoscience challenges.FloodNet focuses on Hurricane Harvey and simple tasks, limiting its breadth for VLM evaluation.Source: §2 · p. 3
  • DisasterM3 is framed as addressing this gap with multi-sensor imagery, bi-temporal inputs, refined damage masks, and diverse disaster-focused visual tasks.Table 1 is described as comparing DisasterM3 with existing remote-sensing vision-language datasets; its caption does not provide cell-level findings.Source: §2 · p. 3, Table 1 · p. 3

3 DisasterM3 Dataset

DisasterM3 combines globally distributed, multi-hazard optical and SAR imagery with nine tasks for disaster assessment and response. Its task design spans scene recognition, damage analysis, object reasoning, and comprehensive reports, supported by an expert-reviewed construction pipeline.

3.1 Perception and Reasoning Tasks in the Context of Disaster

  • Nine tasks assess disaster response, including disaster and land-use recognition, building and road damage assessment, segmentation, relational reasoning, and comprehensive reporting.Recognition covers 13 land-use types and 12 disaster-bearing-body types; building damage is labeled intact, damaged, or destroyed, while road damage is classified as intact, flooded, or debris covered.Source: §3 · p. 4, §3.1 · pp. 4–5

3 DisasterM3 Dataset

  • DisasterM3 contains 36 significant natural and man-made disasters spanning 10 types across five continents, pairing spatially aligned pre- and post-disaster optical imagery with post-disaster SAR imagery.It combines 26 xBD [9] and BRIGHT [5] events with 10 new Maxar Open Data events; optical images were preprocessed to 0.8 m.Source: Figure 2 · p. 4, §3 · p. 4

3.1 Perception and Reasoning Tasks in the Context of Disaster

  • Referring segmentation maps disaster-specific objects and risk-relevant places, while relational reasoning describes spatial dependencies among damaged objects and key facilities.Examples include identifying rescue locations and available vehicles, mapping lava and intact buildings within 100 meters, and analyzing relationships relevant to disaster response and spread prevention.Source: §3.1 · pp. 4–5
  • Comprehensive reports combine disaster-scene descriptions with immediate response advice and longer-term recovery strategies.Examples address earthquake shelters and rebuilding, and flood-related infrastructure repair, temporary housing, disease prevention, and drainage.Source: §3.1 · p. 5, Figure 5 · p. 6

3.2 Dataset Construction Pipeline

  • The construction pipeline combines GPT-4o prompt variation and option generation with expert answer annotation, polygon-based counts, expert relational labels, and multi-round verification.Experts draft disaster reports from bi-temporal imagery and task information, after which GPT-4o polishes them; SAR answers for uninterpretable areas use co-registered optical imagery.Source: §3.2 · pp. 6–7, Figure 6 · p. 6

4 Benchmark Experiments

The benchmark evaluates vision-language models across optical–optical and optical–SAR settings, QA, and referring segmentation. Results expose disaster-domain and cross-sensor gaps, while fine-tuning generally helps but leaves challenges in counting and prompt robustness.

4 Benchmark Experiments

  • The benchmark covers optical–optical and optical–SAR QA and referring segmentation, evaluating six QA task types with accuracy, open-ended tasks with GPT-4.1 scores, and segmentation with cIoU and mIoU.The benchmark includes generic, commercial, and remote sensing VLMs, and fine-tunes Qwen2.5-VL-7B, InternVL3-8B, LISA, and PSALM.Source: §4 · p. 7

4.1 Comparative Results

  • Existing VLMs show a disaster-domain gap, with near-random object relational reasoning; larger models generally perform better, while remote sensing VLMs also struggle on disaster tasks.The Bench set identifies performance gaps, and the Instruct set provides complementary training data.Source: §4.1 · pp. 7–8
  • Fine-tuning Qwen2.5-VL and InternVL3 improves performance across tasks, but InternVL3 degrades on building damage counting because of overfitting.Disaster-specific terminology also improves report generation quality.Source: §4.1 · p. 8
  • Optical–SAR performance is substantially lower across tasks than optical–optical performance, and scaling alone yields no stable gains; fine-tuning improves cross-sensor results but leaves a large gap.The authors point to multi-modal pretraining and alignment strategies as directions for further improvement.Source: §4.1 · p. 8, Figure 7 · p. 8
  • Fine-tuned LISA and PSALM gain on referring segmentation in both sensor settings, and PSALM surpasses smaller-parameter LISA with its multiple-mask-token approach.Performance stabilizes above 50 mask tokens; the experiments use 100.Source: §4.1 · pp. 8–9, Table 3 · p. 8

4.2 Detailed Analysis

  • Across disaster categories, methods perform better on landslides than on earthquake, tornado, and explosion scenarios, which the authors associate with simpler rural scenes versus complex urban ones.The multi-disaster dataset supports evaluation across categories using unified metrics for multiple vision-language tasks.Source: §4.2 · p. 9, Table 4 · p. 9
  • InternVL models’ counting accuracy falls and then rises with building density, whereas GPT-4 accuracy decreases with density; fine-tuned InternVL3-8B improves below 50 buildings but degrades in other ranges.The authors suggest pixel-level semantics as a possible way to reduce overfitting risk.Source: §4.2 · p. 9, Figure 8 · p. 9
  • Across five prompts, LLaVA-OV, TeoChat, EarthDial, and Kimi models are more prompt-sensitive, while larger InternVL3 and Qwen2.5-VL models show enhanced stability.Fine-tuned Qwen2.5-VL-7B and InternVL3-8B are stable across prompt variations; GPT-4.1 is more consistent than GPT-4o.Source: §4.2 · pp. 9–10, Figure 9 · p. 9

5 Limitations and Future Directions

The authors identify limits in spatial-resolution coverage, SAR polarization, cross-sensor performance, and damage counting, alongside a commitment to expand the benchmark.

5 Limitations and Future Directions

  • Standardizing imagery to 0.8 m supports controlled experiments but limits evaluation of robustness across operational spatial resolutions.The authors propose adding multi-resolution imagery from Sentinel-2 and Landsat using geo-registration with existing annotations.Source: §5 · p. 10
  • The SAR data use single polarization, limiting the sensor diversity available for damage assessment.The authors propose integrating multi-polarization data such as Sentinel-1 VV+VH.Source: §5 · p. 10
  • Performance degrades significantly on SAR imagery, motivating further work on multi-modal pretraining and cross-modal alignment.Source: §5 · p. 10
  • Overfitting in damage object counting remains a concern; proposed directions include object-sensitive encoders, numerical difference loss, synthetic high-density data, and knowledge distillation.Source: §5 · p. 10
  • The authors plan to regularly add new Maxar Open Data Program disaster events to extend the benchmark’s geographic and temporal coverage.Source: §5 · p. 10

6 Conclusion

DisasterM3 combines bi-temporal imagery and instruction pairs from disasters across continents. Benchmarking and fine-tuning experiments show broad performance gains, and the authors expect the dataset and baselines to help bridge gaps in disaster applications of Earth-vision VLMs.

6 Conclusion

  • DisasterM3 contains 26,988 bi-temporal images, 123k instruction pairs, and 36 disaster events across five continents.Source: §6 · p. 10
  • Benchmarking 14 advanced VLMs evaluates their capabilities and limitations in disaster contexts.Source: §6 · p. 10
  • Fine-tuning four VLMs with DisasterM3 data produces substantial performance enhancements across all evaluation tasks.Source: §6 · p. 10

NeurIPS Paper Checklist

The checklist points readers to the paper’s contribution and experimental scope, while asserting that the settings and materials needed for reproduction are provided. It also notes the absence of theoretical results and identifies a counting-related overfitting tendency.

NeurIPS Paper Checklist

  • The checklist justification presents DisasterM3 and the main experimental results as the paper’s contributions.Source: NeurIPS Paper Checklist · p. 14
  • The checklist states that the paper has no theoretical results and reports potential InternVL3-8B overfitting on damage counting after fine-tuning.Source: NeurIPS Paper Checklist · p. 14
  • The authors state that experimental settings and evaluation metrics are detailed, with training and test settings in Section 4 and further details in Appendix B.Source: NeurIPS Paper Checklist · p. 14
  • The authors report that the dataset and code are available with prompt-design and fine-tuning instructions in Appendix B.Source: NeurIPS Paper Checklist · p. 14

10. Broader impacts

The broader-impacts checklist raises questions about societal effects, safeguards, asset licensing, human-subject research, and LLM use. The paper says GPT-4o helped generate incorrect options and polish manual answers, and points to Appendix §I for broader impacts.

10. Broader impacts

  • The checklist asks whether the paper discusses positive and negative societal impacts and safeguards for assets that may carry misuse risks.Source: §10 · p. 15
  • It also asks about human-subject risks and approvals, crowdsourcing instructions and compensation, and documentation of new assets.Source: §10 · p. 15
  • The paper reports using GPT-4o to generate incorrect options and polish manual answers; it says broader impacts are clarified in Appendix §I and that used datasets and code are open-source and properly cited.Source: §10 · p. 15

A Dataset Quality Control

DisasterM3 uses a staged annotation and checking process, with damage definitions, task-specific prompts, and evaluation rubrics. The section also specifies benchmark and fine-tuning configurations.

A Dataset Quality Control

  • Annotators train on disaster-specific guidance, annotate separate subsets, cross-review other teams’ work, and revise flagged samples; leaders verify a random 10–20% and repeat validation until standards and agreement are met.The process also checks statistical outliers and uses GPT-4.1 to assess semantic consistency between multi-level questions.Source: A Dataset Quality Control · p. 16

A.1 Building damage-level definitions.

  • FEMA-guided, standardized building-damage definitions train annotators to recognize visual features for damage-level classification.Source: §A.1 · p. 16, Table 5 · p. 16

B.1 Benchmark model settings.

  • Benchmark models use vLLM and default input configurations; referring-segmentation comparisons use the stated default configurations for LISA, PSALM, HyperSeg, and GeoPixel.The passages specify their distinct vision-language and segmentation architectures.Source: §B.1 · p. 17

B.3 InternVL3 and Qwen2.5-VL

  • InternVL3 and Qwen2.5-VL fine-tuning freezes the vision encoder and applies LoRA to the LLM using DisasterM3 question-answer samples.Training uses rank 64, alpha 16, dropout 0.05, four H100 GPUs, batch size 256, and one epoch.Source: §B.3 · p. 17

B.4 LISA and PSALM

  • LISA uses LoRA fine-tuning on segmented instruction data for 1,000 steps, while PSALM is directly fine-tuned with its LLM unfrozen for 10 epochs.Both configurations use four H100 GPUs; LISA’s step limit is stated as intended to prevent overfitting.Source: §B.4 · p. 17

C.1 Instruction prompts

  • Prompts cover paired-image classification and analysis, single-image relational reasoning, six-domain disaster captions, and immediate and long-term recovery advice.The prompt designs impose structured response formats for automated evaluation.Source: §C.1 · pp. 17–18, Table 6 · p. 18

C.2 GPT-based Evaluation Rubric and Prompts

  • GPT-4.1 evaluates disaster captions on precision, detail recall, and factual correctness, and restoration advice on four dimensions using integer scores from 0 to 5.The four restoration dimensions are necessity recognition, action-priority precision, strategic completeness, and implementation feasibility.Source: §C.2 · p. 18, Table 7 · p. 19

D Experimental results on Optical-SAR setting

On the Optical-SAR benchmark, performance varies across model groups and tasks, and open-ended results show marked differences between base and fine-tuned models. Recognition and generation remain uneven, with damage counting and detailed assessment particularly challenging.

D Experimental results on Optical-SAR setting

  • 35.2% overall accuracy for GPT-4.1 leads the Optical-SAR benchmark; GPT-4o reaches 32.1%, InternVL3-78B 31.8%, and fine-tuned InternVL3-8B 34.1%.Source: D Experimental results on Optical-SAR… · p. 19, Table 8 · p. 20
  • 22.1% accuracy for the best Bearing-Body Damage recognition model contrasts with over 70% for top Disaster Type Recognition models; GPT-4.1 reaches 49.4% on Object Relational Reasoning.The passage attributes the bearing-body challenge to SAR’s limited information for recognizing natural objects.Source: D Experimental results on Optical-SAR… · p. 19
  • 3.65 average score for fine-tuned Qwen2.5-VL-7B on disaster captioning compares with 0.98 for its base model, a 3.7× improvement.Factual correctness scores higher than damage precision and detail recall; recovery-necessity scores exceed action-priority precision and strategic completeness across models.Source: D Experimental results on Optical-SAR… · p. 19

E Experimental results on numerical tasks

The section compares numerical-task evaluation formats, PSALM scaling, and geographic performance. It reports consistent model rankings across numerical formats, scaling gains with a persistent sensor gap, and balanced results across US and non-US disasters.

E Experimental results on numerical tasks

  • Models maintain consistent relative rankings between open-ended RMSE and multiple-choice overall accuracy, which the authors say validates the robustness of their multiple-choice design.Lower RMSE indicates better counting accuracy; the comparison covers Building Damage Counting and Damage Road Estimation.Source: E Experimental results on numerical… · p. 20, Table 9 · p. 20

F Scaling up LLMs on PSALM

  • Fine-tuning PSALM raises its near-single-digit cIoU baseline, and scaling from 1.3B to 3B and 7B improves Opt.-Opt. by roughly ten points and Opt.-SAR by around five to seven points.Opt.-SAR remains notably below Opt.-Opt.; the authors attribute scaling gains primarily to linguistic disambiguation and phrase-to-region grounding, and say LLM scaling alone is insufficient across the visual-statistics shift.Source: F Scaling up LLMs on PSALM · pp. 20–21, Figure 11 · p. 21

G Potential Geographic Bias

  • Performance is well balanced between US and non-US disasters across VLMs, whether or not they are fine-tuned on DisasterM3.The authors attribute this robustness to regional dataset coverage and disaster-related visual cues they describe as consistent across national boundaries.Source: G Potential Geographic Bias · p. 21, Table 10 · p. 21

H Visualizations on different disasters

Across flooding and earthquake visualizations, DisasterM3-fine-tuned models outperform baselines on several disaster tasks, while optical-SAR reasoning and segmentation remain difficult. The authors note potential disaster-response benefits but recommend assistive rather than autonomous deployment.

H Visualizations on different disasters

I Broader impacts

  • The authors say DisasterM3-trained models could support more timely damage assessment and responder resource prioritization, but recommend use alongside human experts rather than as autonomous decision-makers.They identify incorrect assessments and misallocation of vital resources as risks of over-reliance.Source: I Broader impacts · p. 23
Loading 2505.21089v2…