Source-linked AI summary
Verifiable Disaster Storylines and Causal Knowledge Graphs: A Citation-Grounded Pipeline from Heterogeneous Humanitarian Sources
Ivan Decostanzi, Michele Ronco, Sergio Consoli, Christina Corbane, Lorenzo Bertolini, Indaco Biazzo, Daria Mihaila, Manuel Garcia-Herranz, Felix Schwebel, Yelena Mejova, Kyriaki Kalimeri
TL;DR
Humanitarian response requires rapid synthesis of heterogeneous information that can exceed human analytical capacity and leave conventional databases incomplete. The paper combines EM-DAT, ReliefWeb, and EMM with RAG to generate 17-field storylines and citation-grounded causal knowledge graphs. Across three crisis use cases, the evaluation reports high retrieval precision, strong citation support, and preference for citation-grounded components, while storyline citations remain less reliable than KG citations.
Problem
High-volume unstructured disaster information exceeds manual analytical capacity, while existing systems and databases lack complete, traceable coverage of impacts and causal evidence.
Method
The pipeline combines EM-DAT with ReliefWeb and EMM, uses RAG to extract 17-field storylines, and validates every KG node and edge with citation-grounded narratives.
Results
85.8% retrieval precision, 86.7% of causal triples grounded in source material, and an overall expert trust rating of 6.56 out of 10 were reported across three crisis use cases.
Takeaways & Limitations
Citation-grounded components received the highest utility ratings, supporting source anchoring as a practical design choice for verifiable humanitarian knowledge graphs.
Takeaways & Limitations
Storyline citations were markedly less reliable than KG citations, with the authors identifying unsupported answering and explicit abstention as a possible mitigation.
Abstract
from arXiv · showhide
Effective humanitarian response depends on the rapid synthesis of heterogeneous, high-volume information sources - a task that routinely exceeds human analytical capacity in the critical early hours of a crisis. We present a pipeline that combines structured disaster records from EM-DAT with unstructured documents from ReliefWeb and the European Media Monitor (EMM) to produce source-grounded disaster storylines and causal knowledge graphs supporting situational awareness for responders and analysts. Using Retrieval-Augmented Generation, the pipeline extracts structured storylines - tabular event profiles covering 17 fields, from severity and key drivers to child-sensitive impact indicators - and constructs causal knowledge graphs where each node and edge is enriched with citation-grounded explanatory narratives, enabling full traceability back to primary sources. We evaluate the system on three diverse crisis use cases through a human evaluation involving 9 domain expert and 9 non-expert evaluators. Results confirm high retrieval precision, strong faithfulness of extracted causal relations, and a clear expert preference for citation-grounded components over ungrounded alternatives. The pipeline is designed to scale to the full EM-DAT catalogue, with the goal of publicly releasing a narrative-enriched version of the database.
1 Introduction
Humanitarian responders face more information than they can manually synthesize quickly, while standard disaster databases omit qualitative, cascading, and child-sensitive impacts. The paper addresses these gaps by combining heterogeneous sources with RAG to produce traceable storylines and causal graphs.
- Information gap: High-volume, fast-moving unstructured information routinely exceeds human analytical capacity during disaster response.Manual review of thousands of reports to identify causal links is often infeasible within operational timeframes.
- Information gap: EM-DAT records aggregate impacts but exclude smaller events below reporting thresholds, leaving situation-awareness gaps.
- Complementary evidence: News, field, and humanitarian documents add details on impacts, vulnerabilities, exposed populations, and cascading effects absent from conventional catalogues.
- Traceability gap: Existing LLM-generated knowledge graphs commonly lack explicit links between graph elements, narratives, and their source evidence.This weakens traceability in settings where claims must be verifiable and can limit user trust and practitioner adoption.
- Paper response: The paper extends prior EM-DAT–EMM work with ReliefWeb integration, field-level Multi-Shot RAG, citation-grounded KG validation, child-sensitive indicators, and an exploration dashboard.The expanded storyline covers indicators including displacement, casualties, and access to education and health services.
2 Methods
The pipeline merges ReliefWeb and EMM evidence with EM-DAT-linked events, extracts 17-field storylines, and builds causal graphs. Multi-Shot retrieval and per-element validation attach source citations and explanatory narratives to outputs.
- Data integration: ReliefWeb situation reports, assessments, and coordination documents are aligned with EM-DAT events through standardized GLIDE identifiers.The merged ReliefWeb and EMM documents form the evidence base for downstream tasks.
- Storyline extraction: Storylines are structured 17-field event profiles covering standard impact dimensions and child-specific casualties, displacement, and service-access indicators.
- Storyline strategies: One-shot jointly extracts all fields from one context block, whereas Multi-Shot retrieves focused evidence independently for each field and records supporting sources.One-shot is efficient but lacks a mechanism for tracing individual outputs to passages.
- Knowledge graph construction: The causal KG represents storyline-derived entities and relations as nodes and subject–predicate–object triples.
- KG validation: A secondary RAG process dynamically queries evidence for every KG node and triplet, then generates a concise explanatory narrative with explicit citations.This creates an audit trail for checking whether graph elements are supported by official sources.
3 Evaluation
Evaluation covers three humanitarian crisis typologies selected for documentation richness and diversity in geography, temporal dynamics, and reporting density. The cases comprise Haiti cholera, Hurricane Melissa in the Dominican Republic, and Syria conflict escalation.
- Use cases: The evaluation spans a public health emergency, a sudden-onset natural disaster, and a protracted conflict.
- Haiti Cholera Outbreak: Haiti’s 2022 cholera resurgence tests extraction of multicausal chains linking public health, governance failure, and conflict.
- Hurricane Melissa: Hurricane Melissa in the Dominican Republic tests KG faithfulness using well-documented disruption to roads, schools, and health facilities.
- Syria Conflict Escalation: The late-2024 Syria conflict escalation tests handling of rapidly evolving, information-dense, structurally complex conflict settings.
3.2 Human Evaluation
The human evaluation uses independent expert and non-expert annotators assigned to tasks according to their domain-knowledge requirements. Agreement is reported with multiple measures, including prevalence-adjusted statistics.
- Participants: 18 independent annotators participate: nine domain experts and nine non-experts, with nine annotators judging each item.
- Agreement measures: Agreement is summarized using Krippendorff’s α, mean pairwise percentage agreement, and mean pairwise PABAK.PABAK is included to address prevalence and response-bias artifacts.
- Evaluator assignments: Non-experts assess retrieval, KG text, and citation quality, while experts assess storylines, KG faithfulness, and the overall interactive system.
- Retrieval assessment: Retrieval quality is assessed by judging whether retrieved passages are relevant to the target disaster and informative rather than boilerplate.
Retrieval Quality.
The evaluation compares One-shot and Multi-shot storyline generation across 17 fields and assesses explanatory texts for relevance and informativeness. It also distinguishes whether generated text targets the correct disaster concept or relationship.
- Storyline comparison: Evaluators compare One-shot and Multi-shot approaches field by field across 17 storyline categories.Each field is judged as favoring One-shot, Multi-shot, or neither.
- Storyline comparison: Overall storyline quality is rated on a 1–5 Likert scale.
- Narrative assessment: Explanatory texts are evaluated for relevance and informativeness by non-expert annotators.Relevance concerns the correct node concept or link relationship; informativeness concerns concrete, useful disaster information.
Knowledge Graph Text Quality.
Domain experts assess whether extracted causal triples are supported by the storyline from which they were derived. Support is categorized from explicit to absent or indeterminate.
- Support categories: Each causal triple is classified as Fully supported when explicitly stated in the source storyline.
- Support categories: Partially supported triples contain one entity or an implicit relation in the storyline.
- Support categories: Triples may also be classified as Not present or Cannot determine.
Knowledge Graph Faithfulness.
The evaluation measures whether citations support generated statements collectively and whether individual citations are relevant. It applies these assessments to storyline and knowledge-graph narratives.
- Citation evaluation: Citation recall tests whether each statement is supported by its citation set as a whole.Citation precision instead evaluates each individual citation’s relevance to the statement.
- Citation evaluation: Citation precision evaluates the relevance of each individual citation to the statement it accompanies.Recall and precision coincide when a statement has a single citation.
- System evaluation: Experts evaluate the full dashboard, including graph visualisation, source-grounded storylines, and the natural-language database interface.The assessment covers component usefulness, workflow utility, and trust in generated outputs.
Expert System Assessment.
The system assessment includes structured usability questions about component usefulness, humanitarian workflow utility, and trust in generated outputs. Trust uses a distinct 0–10 rating scale.
- Holistic assessment: Experts assess perceived usefulness of individual system components.
- Holistic assessment: The evaluation measures the system’s utility for humanitarian workflows.
- Holistic assessment: Overall trust in generated outputs is rated on a 0–10 scale.Other structured responses use a five-point Likert scale.
4 Evaluation Results
Human evaluation found strong retrieval precision, broadly faithful causal relations, and high citation relevance, while storyline comparisons were more subjective and expert trust remained adequate rather than high.
- Retrieval Quality: 85.8 ± 4.7% of retrieved paragraphs were judged relevant to the target event, indicating high retrieval precision.Relevance agreement was 83.1%, with PABAK = 0.662 and Krippendorff’s α = 0.305.
- Storyline Quality: 62.1 ± 20.0% of storyline fields favored Multi-shot over 26.1 ± 19.5% favoring One-shot, although agreement was low at 51.6%.Event-level preferences varied considerably, and Event 2 showed sharp disagreement among annotators.
- Storyline Quality: Multi-shot received a mean overall rating of 3.67 ± 0.71 out of 5 versus 2.78 ± 0.83 for One-shot, with seven of nine annotators preferring Multi-shot.Annotators valued its source citations for verifiability, while some fields favored One-shot’s concision.
- Knowledge Graph Faithfulness: 86.7% of extracted causal triples were judged supported by the storyline, with 56.1 ± 26.5% fully supported and 30.6 ± 20.7% partially supported.Only 9.2 ± 9.4% were rated not present and 4.2 ± 12.5% could not be determined.
- Citation Quality: 93.5% of claims were at least partially supported by their citation sets, while 91.7% of citations were at least partially relevant to their statements.KG node and link citations were more reliable than storyline citations, whose lower performance appeared related to over-citation.
- Expert System Assessment: Experts rated KG Citations highest at 4.33, followed by Storyline & Fields at 4.00, whereas Causal Graphs received 2.78 on the 1–5 utility scale.Operational impact means were 3.67 for both Efficiency and Situational Awareness, while overall trust averaged 6.56 on a 0–10 scale.
5 Discussion and Conclusion
Across three crisis use cases, the pipeline achieved strong retrieval and causal-grounding results, while citation-grounded components were preferred by experts. The evaluation also identified weaker storyline citation reliability and concerns that causal graphs may oversimplify without human validation.
- Evaluation: 85.8% retrieval precision and 86.7% of causal triples grounded in source material demonstrate strong evaluation performance across three crisis use cases.The evaluation involved 18 independent annotators.
- Evaluation: Citation-grounded components received the highest utility rating (M=4.33), supporting the design choice of anchoring graph elements to retrievable evidence.Experts also reported potential for rapid situational overview, crossagency comparison, and preliminary impact assessment.
- Limitations: Causal graphs received the lowest utility rating (M=2.78) because experts perceived them as oversimplified and potentially misleading without human validation.The authors identify refined graph complexity and continued validation as future-work needs.
- Future work: Further planned extensions include sector-aligned structuring and ingestion of user-supplied or restricted-access documents.
- Limitations: Storyline citation reliability was 71.7%, compared with above 92% for KG citations, because the model continued citing when evidence was insufficient.Explicit abstention is identified as a possible mitigation.
- Future work: The planned full-catalogue deployment would produce a publicly released, narrative-enriched EM-DAT version using EMM and ReliefWeb sources.The intended dataset would add contextual detail beyond current aggregate event statistics.