PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards reliable multimodal disaster severity assessment through preference optimization and explainable vision-language reasoning

Yuanjun Zhang, Fuzel Ahamed Shaik, Suvojit Acharjee, Fahad Khalid, Mourad Oussalah

arXiv:2609.00879v1cs.AI

TL;DR

Reliable disaster damage assessment is limited by scarce annotated rationales and inadequate evaluation of explanation quality. The paper introduces a unified HITL pipeline that derives SFT and DPO datasets for jointly training and aligning vision-language models. The framework improves classification and explanation quality, with stronger recognition of mild damage, fewer high-risk misclassifications, and better alignment with human judgment.

  • Problem

    Existing multimodal disaster-assessment approaches lack sufficient annotated rationales and systematic evaluation of reasoning quality for reliable, auditable decision support.

  • Method

    A single HITL annotation workflow produces ReasoningSet for SFT and paired PreferenceSet examples for DPO, followed by joint prediction-and-explanation training and multi-tier evaluation.

  • Results

    SFT improves classification and explanation quality, while subsequent DPO alignment further enhances interpretability and human-preference alignment across evaluated models.

  • Takeaways & Limitations

    The framework provides a reproducible route toward auditable disaster insights, improving mild-damage detection and reducing high-risk misclassifications for emergency decision support.

  • Takeaways & Limitations

    The dataset covers one social-media platform and limited disaster types, while cross-event and temporal generalization were not explicitly evaluated.

Abstract

from arXiv · show

Reliable disaster damage assessment requires models that provide both accurate predictions and transparent explanations. However, existing multimodal approaches are limited by scarce annotated data and insufficient evaluation of reasoning quality. This study proposes a two-stage training framework that integrates Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) within a unified data construction pipeline. From a single Human-in-the-Loop (HITL) annotation workflow, two complementary datasets are derived, namely ReasoningSet, which contains validated rationales for SFT, and PreferenceSet, which comprises paired rationales for DPO-based alignment. The framework evaluates both classification performance and explanation quality using automatic metrics, model-based scoring, and human ranking. Experimental results show that SFT improves accuracy from 73.64% to 78.29% and increases Macro-F1 by 29% compared to the baseline, while explanation quality improves by approximately 25%. Subsequent DPO alignment further enhances interpretability on the PreferenceSet. Cross-model validation on InternVL-3-8B and LLaVA-1.5-7B demonstrates the robustness and generalizability of the approach. The proposed framework improves detection of underrepresented mild damage cases, reduces high-risk misclassifications, and strengthens alignment between model reasoning and human judgment. Overall, it provides a reproducible pathway to develop reliable multimodal systems that deliver auditable, actionable disaster insights for emergency management.

1. Introduction

The paper frames reliable multimodal disaster assessment as requiring accurate, explainable, and auditable outputs, while addressing limited rationale data and evaluation protocols through a human-in-the-loop, preference-aligned framework.

  • Motivation: Social media offers timely, broad disaster information but introduces noise, unverifiable content, misinformation, and reliability risks.These risks can hinder high-stakes emergency decision-making.
  • Research gap: Human-validated rationales are needed because existing crisis-informatics work rarely evaluates explanation quality systematically or aligns outputs with practitioner expectations.The paper identifies task-aligned, human-centered, and auditable evaluation as an unresolved requirement.
  • Contributions: CrisisMMD-R extends CrisisMMD with paired human-validated explanations linking textual and visual cues to disaster-severity labels.The dataset is intended to support explainable damage-assessment models and preference-based alignment.
  • Contributions: A scalable HITL protocol combines contributor scoring, expert review, audit trails, and quality-improvement mechanisms for disaster-management annotation.The workflow produces validated rationales and preferred–rejected rationale pairs.
  • Contributions: The framework fine-tunes vision-language models with supervised and preference-based methods to jointly predict disaster severity and generate explanations.It evaluates classification and explanation quality together and tests generalizability on InternVL3-8B and LLaVA1.5-7B.

2. Related work

Related work establishes the promise of social-media and vision-language systems for crisis response while emphasizing credibility, accountability, and explanation-quality gaps. The paper addresses these gaps with human-validated rationales, preference alignment, and joint evaluation.

  • Crisis informatics: Crisis informatics and DRR research emphasize timely, reliable, accountable information for situational awareness and emergency resource allocation.These requirements frame crisis analytics as practitioner-centered decision support.
  • Crisis data: Social-media analytics support event detection, spatiotemporal tracking, and disaster-impact assessment, but operational uptake is limited by credibility, reliability, representativeness, and annotation concerns.Multimodal evidence adds visual information while also raising authenticity and annotation-reliability issues.
  • Explainability gap: Prior crisis datasets commonly provide labels without explanations, whereas the proposed approach adds rationales linking textual and visual evidence to assigned damage levels.These rationales make classification decisions inspectable and address practitioner information needs.
  • Vision-language models: Vision-language models have shown broad multimodal generalization, but disaster response requires outputs that are trustworthy as well as accurate.The study therefore targets disaster-specific rationale generation and preference optimization.
  • Explainable AI and alignment: Explainable-AI and preference-optimization research supplies precedents for rationale generation and human-guided alignment in language and vision-language tasks.Examples include rationale datasets for inference and visual question answering, alongside RLHF and DPO methods.

3. Methodology

The methodology builds an explainable disaster-assessment pipeline by generating and human-validating rationales, then separating them into supervised and preference-alignment data. It trains models with LoRA-SFT followed by DPO and evaluates annotation reliability through reviewer consistency checks.

  • Pipeline overview: The pipeline combines prompt-based rationale generation, human-in-the-loop validation, two-stage LoRA-SFT → DPO training, and multimodal evaluation.The framework spans data construction, model training, and evaluation of both predictions and explanations.
  • Data construction: 3526 tweet-image pairs from seven natural disasters provide the original CrisisMMD damage-severity data, which lacks explanatory text.This makes the corpus a test-bed for augmenting categorical damage labels with rationales.
  • Data construction: 3526 draft explanations are generated with Qwen-VL-Max using a prompt that restates official criteria, treats tweets as secondary evidence, and constrains rationale length.The prompt produces one draft explanation for each CrisisMMD item.
  • Human-in-the-loop annotation: Human validation rates drafts on Evidence Recognition, Reasoning Chain, and Text Naturalness using 1–5 scores, with approval requiring Total score ≥12 and no dimension below 3.Contributors inspect the image, tweet, and model explanation before professional review.
  • Reliability assessment: Reviewer outcomes were statistically independent of reviewer identity, supporting consistency of the professional verification procedure.The reported test failed to reject the null hypothesis, and the authors interpret this as evidence against evaluator-specific decision bias.
  • Final datasets: 1284 validated rationales form the ReasoningSet, while 287 corrected rejected drafts are paired with preferred revisions to form the PreferenceSet for DPO.The remaining corrected entries are excluded from DPO construction to preserve separation from supervised data.

3.2. Problem definition

The task is to generate a damage label and a free-form rationale from an image and its accompanying tweet. Both outputs are produced together in one autoregressive decoding pass to jointly support correctness and interpretability.

  • The model maps an image I and accompanying tweet T to a predicted damage label and textual rationale.
  • The damage-label space comprises LITTLE OR NO, MILD, and SEVERE.
  • The rationale is usually written in English as free-form text.
  • Prediction and explanation are generated in a single autoregressive decoding pass rather than through post-hoc explanation.

3.3. Base architecture

The framework primarily uses Qwen2.5-VL-7B and applies the same adaptation and alignment procedures to InternVL3-8B and LLaVA1.5-7B for generalizability. Qwen2.5-VL-7B combines a ViT vision encoder, MLP merger, and language decoder with multimodal positional encoding.

  • Qwen2.5-VL-7B is the primary model, while InternVL3-8B and LLaVA1.5-7B provide cross-model validation of the training pipeline.
  • Qwen2.5-VL-7B uses a Vision Transformer vision encoder, an MLP merger, and an LLM decoder.
  • The vision encoder uses 2D-RoPE, window attention, and periodic full-attention layers for global feature interaction.
  • The decoder replaces one-dimensional rotary embeddings with Multimodal RoPE to align textual and visual tokens positionally.

3.4. Prompt design and data serialization

The study serializes multimodal data in ShareGPT format under a fixed system prompt and uses distinct record structures for SFT and DPO. Training incorporates human-refined rationales while excluding ground-truth labels from inputs to prevent leakage.

  • 3.4. Prompt design and data serialization: ShareGPT is consistently adopted for multimodal dialogue serialization in both SFT and DPO.The corpus is formatted for compatibility with the LLaMAFactory toolkit.
  • 3.4. Prompt design and data serialization: The fixed system prompt defines damage-assessment criteria, prioritizes the image as evidence, and constrains generated responses.
  • 3.4. Prompt design and data serialization: SFT records contain an image, tweet, system prompt, and human-refined gold rationale, with the rationale stored in final_response.
  • 3.4. Prompt design and data serialization: DPO records add chosen and rejected rationales to the image, tweet, and system-prompt structure.
  • 3.4. Prompt design and data serialization: Ground-truth damage labels are excluded from ShareGPT inputs during SFT, DPO training, and evaluation to prevent label leakage.
  • 3.5. Stage I: Supervised fine-tuning (SFT): ReasoningSet contains 1027 training instances and 128 validation instances for SFT.
  • 3.5. Stage I: Supervised fine-tuning (SFT): SFT jointly learns the damage label and rationale through a single autoregressive output sequence.
  • 3.5. Stage I: Supervised fine-tuning (SFT): The SFT objective treats s = [Y, R] as the concatenated output sequence and selects the checkpoint with the lowest validation loss.Training uses three epochs, batch size eight, AdamW, and a 7 × 10^-5 learning rate with cosine decay.

3.6. Stage II: Direct preference optimization (DPO)

DPO aligns model outputs with human-preferred reasoning using paired chosen and rejected responses from PreferenceSet. The resulting models are evaluated for deployment and presented through a prototype that exposes both predictions and rationales for inspection.

  • PreferenceSet supplies 229 instances for DPO, with samples containing paired chosen and rejected structured responses.
  • Each structured response jointly contains a predicted damage label and its corresponding rationale, while preserving the original ground-truth label.
  • LoRA targets all linear layers with rank r = 16 and scaling factor α = 32 while freezing the vision encoder and multimodal projector.
  • DPO increases the likelihood of preferred responses relative to rejected responses for the same image and tweet.
  • The temperature parameter τ controls preference-weighting sharpness, with chosen and rejected outputs denoted ŝ_c and ŝ_r.
  • DPO training runs for four epochs with batch size eight, AdamW, and a 5 × 10^-6 learning rate under cosine decay.
  • The prototype accepts tweet-image pairs and displays both predicted damage levels and natural-language rationales for real-time inspection.
  • The DPO-aligned models are intended to improve interpretability, trustworthiness, and practical utility in reliability-oriented decision support.

4. Results and discussions

SFT substantially improves classification, explanation quality, and practical reliability across evaluated backbones, especially for Mild damage. DPO adds more modest gains by aligning explanations with human preferences and improving interpretability.

  • Classification performance: 78.29% accuracy and 57.42% Macro F1 are achieved by Qwen2.5-VL-7B after SFT, versus 73.64% and 44.46% at baseline.Macro F1 increases 29.1% relatively, while Mild-damage F1 rises from 15.38% to 48.78%.
  • Classification performance: 45.5% of Qwen2.5-VL-7B Mild cases are correctly classified after SFT, while Severe misclassifications fall to 40.9%.The baseline correctly classifies 2 Mild cases and labels 77.3% as Severe; after SFT, 10 are correct and 9 are labeled Severe.
  • Classification performance: 13.3%, 18.0%, and 21.4% reductions in Qwen2.5-VL-7B expected cost occur with overestimation penalties of 2, 3, and 4.Risk-aware gains are also reported for InternVL3-8B and, under stricter penalties, LLaVA1.5-7B.
  • Generation quality: SFT improves BLEU-4 and ROUGE across all backbones, with ROUGE-2 gains indicating more detailed and contextually linked explanations.Qwen2.5-VL-7B provides the strongest overall balance across n-gram metrics, while InternVL3-8B nearly doubles ROUGE-2.
  • Explanation quality: 20.7% higher Diagnostic Helpfulness, 39.5% higher Evidence Integration, 20.3% higher Contextual Awareness, and 22.7% higher Decision Support Value follow Qwen2.5-VL-7B SFT.Comparable scorer improvements are reported for InternVL3-8B and LLaVA1.5-7B.
  • Practical implications: SFT reduces refusals and high-risk misclassifications while producing richer, evidence-based explanations that integrate visual and textual cues.Examples include correct severe-damage assessment for a wildfire and a flooded museum that the baseline classified as little or no damage.
  • DPO results: DPO primarily improves interpretability, preference alignment, and decision-support relevance, while its classification gains over SFT remain modest.The PreferenceSet contains 229 training and 29 validation instances, limiting the size of the additional alignment stage.
  • Human evaluation: ρ=0.386 with 0.783 pairwise agreement indicates moderate alignment between automated scorer ratings and human judgments for Baseline versus SFT.For SFT versus DPO, the smaller n=29 evaluation and differing supervision signals make winner-level agreement noisier despite positive correlation.

5. Conclusion

The framework combines human-in-the-loop data construction, SFT, DPO alignment, and multi-tier evaluation to improve reliable multimodal disaster damage assessment. The authors also identify risk-analysis and generalization boundaries that motivate future work.

  • Framework: The human-in-the-loop pipeline produces ReasoningSet for SFT and PreferenceSet for DPO within one annotation workflow.This design provides complementary supervision while reducing annotation redundancy, cost, and time.
  • Training and evaluation: SFT jointly trains disaster classification and rationale generation, improving accuracy, robustness, interpretability, Mild-damage recognition, and reducing high-risk misclassifications and refusals.DPO then incorporates human preferences to refine contextual and ethical rationale quality and create an auditable reasoning trail.
  • Training and evaluation: The three-tier evaluation framework combines automatic metrics, model-based scoring, and human ranking to assess predictive performance and explanation quality.The evaluation is presented as evidence of reliability and operational utility.
  • Limitations: The study does not perform quantitative severity-weighted error-risk analysis, including formal penalties for operationally consequential misclassification types.Future evaluation is intended to quantify severity-level disagreements and weight them by operational risk.
  • Limitations: CrisisMMD-R has limited scale and diversity, experiments use only 7B/8B models, and DPO requires comparison with alternative strategies under dynamic real-world conditions.These constraints may limit generalization and leave larger-model performance unexplored.

CRediT authorship contribution statement

The authors’ contributions span writing, visualization, methodology, data curation, conceptualization, software, project administration, supervision, and review.

  • Yuanjun Zhang contributed writing, visualization, methodology, data curation, and conceptualization.
  • Fuzel Ahamed Shaik contributed writing, visualization, project administration, methodology, and conceptualization.
  • Suvojit Acharjee contributed software, while Fahad Khalid contributed writing and review.
  • Mourad Oussalah contributed writing, review, and supervision.

Data availability

Derived text datasets are available for research use under restrictions, while original CrisisMMD media and Platform X data are not redistributed.

  • Access conditions: Access to derived text datasets requires a research-only license and an institutional-email request.Applicants must agree not to use the data for re-identification or commercial purposes.
  • Release scope: The release excludes raw CrisisMMD media and Platform X data, providing only an incremental layer aligned to CrisisMMD samples.Researchers must obtain CrisisMMD through its original channel and follow applicable licensing and Platform X policies.
  • Release scope: The incremental release includes ReasoningSet, PreferenceSet, and the train/dev/test split indices used in the study.

Appendix A. Evaluation Stability under an Alternative Judge Model

The appendix tests explanation-scoring stability by replacing Qwen-VL-Max with GPT-4o-mini while holding the evaluation protocol constant. Both judges preserve the main ranking across backbones, with one minor dimension-specific deviation.

  • Evaluation setup: Table A.14 compares Qwen-VL-Max and GPT-4o-mini using the same test sets, four dimensions, judging prompt, and evaluation template.
  • SFT-Testset: Across all three backbones, both judges rank SFT above Baseline on every SFT-Testset explanation dimension.
  • DPO-Testset: On the DPO-Testset, both judges identify SFT+DPO as best or tied-best on most dimensions, with consistent gains for Qwen2.5-VL-7B and InternVL3-8B.
  • DPO-Testset: For LLaVA1.5-7B on EI, Qwen-VL-Max favors SFT at 4.41 versus 4.34, whereas GPT-4o-mini favors SFT+DPO at 4.17 versus 4.34.This deviation does not reverse the overall conclusion.

Appendix B. Reproducibility Details

Appendix B documents the implementation details needed to reproduce the training and inference pipeline, including framework settings, prompts, data formats, and SFT/DPO configurations.

  • Implementation Environment: Training and inference use LLaMA-Factory version 0.9.4 or later, with unspecified parameters set to framework defaults.Experiments run on an NVIDIA V100 with 32GB of memory.
  • Prompts: The reproducibility materials include the visual-tweet reasoning draft prompt and the fixed system prompt used across experiments.These prompts are provided in Figures B.11 and B.12.
  • Data and Training Configurations: Listings 3–6 document the ShareGPT formats and complete LLaMA-Factory training configurations for SFT and DPO.The appendix identifies Listings 5 and 6 as the SFT and DPO YAML configurations.
Loading 2609.00879v1…