PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Emergency Decision-making with Knowledge Graphs and Large Language Models

Minze Chen, Zhenxiang Tao, Weitong Tang, Tingxin Qin, Rui Yang, Chunli Zhu

arXiv:2311.08732v1cs.CL

TL;DR

Emergency decision support must contend with LLM hallucinations and limited reasoning, as well as fragmented information in emergency documents. E-KELL structures emergency knowledge in a knowledge graph and guides LLM reasoning over it. The authors report superior decision-support capabilities across emergency scenarios, while noting limited data sources and the need for further prompt testing.

  • Problem

    LLMs can produce hallucinations and have constrained reasoning, while emergency standards often contain fragmented information that is difficult for models to interpret.

    Source: §1 · p. 3

  • Method

    E-KELL structures emergency-related text in a standardized knowledge base and guides LLM reasoning over a knowledge graph through an interpretable framework.

    Source: §1 · p. 3, §7 · p. 13

  • Results

    E-KELL demonstrates superior decision-support capabilities across various emergency scenarios in the authors’ case study and evaluation.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    The system can assist emergency managers with decision support within standards and regulations.

    Source: §7 · p. 13

    The study is limited by its data sources, which affect E-KELL’s reasoning quality, and its prompt design needs further testing.

    Source: §7 · p. 13

Abstract

from arXiv · show

Emergency management urgently requires comprehensive knowledge while having a high possibility to go beyond individuals' cognitive scope. Therefore, artificial intelligence(AI) supported decision-making under that circumstance is of vital importance. Recent emerging large language models (LLM) provide a new direction for enhancing targeted machine intelligence. However, the utilization of LLM directly would inevitably introduce unreliable output for its inherent issue of hallucination and poor reasoning skills. In this work, we develop a system called Enhancing Emergency decision-making with Knowledge Graph and LLM (E-KELL), which provides evidence-based decision-making in various emergency stages. The study constructs a structured emergency knowledge graph and guides LLMs to reason over it via a prompt chain. In real-world evaluations, E-KELL receives scores of 9.06, 9.09, 9.03, and 9.09 in comprehensibility, accuracy, conciseness, and instructiveness from a group of emergency commanders and firefighters, demonstrating a significant improvement across various situations compared to baseline models. This work introduces a novel approach to providing reliable emergency decision support.

1. Introduction

Emergency decision support must address the limitations of direct LLM use and fragmented emergency documents. E-KELL combines a knowledge graph with LLM reasoning to provide decision support, and is reported to perform well across emergency conditions.

1. Introduction

  • Emergency decision support systems can save response time and mitigate misjudgment, but typically leave managers to interpret incident-related information; LLMs offer direct guidance and multi-turn conversation.Source: §1 · p. 2
  • LLM hallucinations and constrained reasoning are critical in emergencies, while fragmented standards documents make it difficult for models to understand relationships between components.Emergency decision support also needs interpretability and must consider both standard procedures and real-world conditions.Source: §1 · p. 3
  • The paper develops an emergency decision-support framework based on knowledge graphs over an LLM application framework, structuring Chinese emergency documents and guiding graph reasoning through a prompt chain.The framework is intended to provide interpretable guidance and prompt feedback.Source: §1 · p. 3
  • E-KELL is reported to have superior decision support capabilities in various emergency conditions.Source: §1 · p. 3

2. Related work

Related work discusses fine-tuned and prompt-based industry LLMs, then considers integrating knowledge graphs with LLMs. E-KELL is presented as an emergency-specific KG-enhanced LLM framework.

2.1. Industry LLMs

  • Fine-tuning adapts LLMs to industry tasks, while prompt-based frameworks provide queries and related text; neither approach resolves all concerns around hallucinations and emergency use.The review notes ethical and legal concerns and emphasizes transparency and interpretability.Source: §2.1 · pp. 3–4
  • Prompt-based emergency systems use multi-round dialogue, but providing adequate context remains difficult when real-world knowledge is unstructured.This motivates consideration of knowledge graphs as a way to represent factual knowledge and relationships for LLM use.Source: §2.1 · p. 4, §2.2 · p. 4

2.2. Integration of KG and LLM

  • Prior KG–LLM work includes using LLMs to construct or complete graphs and using LLMs to analyze graph content and reason over it; E-KELL is presented as an emergency-specific framework.The review notes that some KG reasoning approaches still require automated processes to be completed.Source: §2.2 · p. 4

3. Methodology

E-KELL builds an emergency knowledge graph from decision requirements and authoritative documents, then guides an LLM through stepwise reasoning over relevant graph content. Its schema-first design and semi-automatic construction support structured emergency knowledge and query reasoning.

3.1. Knowledge structuring

  • The decision-demand schema covers eight primary classes and 22 subclasses.The classes are identified from common emergency tasks and standard documents.Source: Figure 2 · p. 6, §3.1 · p. 6
  • The KG uses a schema-first structure, defining entity, relation, and attribute patterns before populating the data layer.This top-down construction is described as suitable for domain-specific KGs.Source: §3.1 · p. 6
  • A semi-automatic process uses authoritative emergency standards and policies to extract concepts, relations, and subject-predicate-object triples for the KG.An LLM-based approach deconstructs triples, while the schema specifies entity and relation types.Source: §3.1 · p. 6
  • Knowledge fusion and updates are primarily manual because entity meanings can be difficult to distinguish and automatic clustering or embedding was insufficiently accurate in testing.Revised standards require corresponding triples to be replaced, and abolished standards require their triples to be removed.Source: §3.1 · p. 6

3.2. Logical reasoning

  • The KG integrates knowledge from multiple standards and policies, enabling inference across documents, such as applying actions associated with a power outage to a flood that causes one.The example connects a flood-to-power-outage relation with response actions documented for power outages.Source: §3.2 · pp. 6–7
  • E-KELL automatically decomposes queries into logical expressions using projection, intersection, union, and negation, then guides the LLM through stepwise prompts over relevant KG segments.The system retrieves relevant graph portions rather than requiring access to the entire KG.Source: §3.2 · p. 7
  • E-KELL adds automatic query decomposition to a prior prompt-based KG reasoning approach whose query logic had to be specified in advance.The added module uses examples to guide the LLM in breaking down queries.Source: §3.2 · p. 7
  • The prompt chain converts a query into a logical form and then into sequential prompts that guide the LLM through KG triples to generate a response.The authors report that step-by-step sub-prompts were more effective in testing than presenting them together in one paragraph.Source: §3.2 · p. 7

4. System implementation

The prototype focuses on hazardous chemical leaks, constructing a knowledge graph from selected emergency documents and using vector retrieval to provide relevant knowledge for prompt-guided LLM reasoning. It offers separate PC and mixed-reality interfaces for decision-center personnel and frontline operators.

4. System implementation

  • The emergency standards and interfaces were compiled in Simplified Chinese to align with regional laws, regulations, and local-language users.The authors translated the documents, interfaces, and prompts into English for accessibility, stating that this did not affect development or practical application.Source: §4 · p. 8
  • The prototype targets hazardous chemical leaks, using timely, relevant, representative materials from official Chinese government and legal sources to construct its KG.Selected documents are mined with OCR, structured into entities, relations, and attributes, and converted into subject-predicate-object triples.Source: §4 · p. 8
  • 2,264 triples make up the final KG after volunteers with hazardous-chemical expertise revise the preliminary graph and add cross-text triples.The KG draws on two legal regulations, ten standard specifications, and other relevant materials.Source: §4 · pp. 8–9
  • Vector retrieval returns relevant KG portions and their source texts, which are supplied with predefined prompts to guide the LLM's reasoning from the provided knowledge.The system is intended to have the LLM reason from that knowledge rather than its own database, reducing the possibility of hallucinations.Source: §4 · p. 9
  • The system uses ChatGLM-6B, an open bilingual LLM, and is deployed locally on an NVIDIA A100 graphics card.The paper states that this choice considers emergency-knowledge data security and common interactive language.Source: §4 · p. 9
  • PC interfaces serve decision-center personnel, while mixed-reality interfaces provide frontline operators with voice-command interaction and hands-free access to answers.The MR interface can be moved to different spatial locations to avoid obstructing the operator's work.Source: §4 · p. 9, Figure 5 · p. 10

5. Case study

The case study tests E-KELL on ten hazardous-chemical-leak decision queries against ChatGLM-6b and GPT-3.5. Objective checks and expert ratings assess response quality.

5.1. Use case

  • For Q1, ChatGLM-6b's response conflicted with the standard, while GPT-3.5 was more detailed but redundant, making critical information harder to obtain.The authors emphasize that concise, direct, quantifiable support is crucial in emergencies.Source: §5.1 · p. 10
  • For the PPE query, E-KELL intersects chemical and hazard-zone factors, expands the zone to a protection level, then retrieves standards for an operational guide.The example concerns sulfur dioxide, acetylene, and a moderate-toxicity, low-hazard zone.Source: §5.1 · p. 10, Figure 6 · p. 11

5.2. Evaluation

  • Across the provided examples, E-KELL had no grammatical or factual errors and did not violate standards, while the other models showed factual errors and non-compliance.The other models gave incorrect propylene formulas; chatGLM-6b's Q8 isolation response would violate standards.Source: Table 1 · p. 11, §5.2 · p. 11
  • Nineteen firefighters and emergency commanders blindly rated randomly selected responses on four attributes using a 1–10 scale.The attributes were comprehensibility, accuracy, conciseness, and instructional value.Source: §5.2 · p. 12
  • E-KELL's mean scores across four expert-rated metrics exceeded both ChatGLM-6b and GPT-3.5.The ratings were averaged across the queries.Source: Table 2 · p. 12, §5.2 · p. 12

6. Discussion

The discussion identifies implementation needs for E-KELL, including prompt and knowledge-base development, and outlines self-checking and multimodal or agent-based extensions.

6. Discussion

  • Prompt templates require adjustment and ongoing optimization; structured knowledge-base construction still relies heavily on manual labeling, raising costs and limiting logical reasoning.The discussion also calls for adding common-sense knowledge and real-time data interfaces.Source: §6 · p. 12
  • The authors call for continuous prompt-based self-checking to mitigate hallucinations and identify AI agents with access to a local KG, web search, and other tools as a future direction.They describe the agent framework as promising for further improving reliability and decision-making capabilities.Source: §6 · p. 12
  • A proposed multimodal extension would convert scene images into language descriptions for the LLM and visualize guidance through real-time holographic animations.The authors describe the image-understanding module as a feasible input approach.Source: §6 · pp. 12–13

7. Conclusion

E-KELL uses knowledge graphs and LLMs to support emergency decision-making, with the study reporting superiority across emergency scenarios. The authors identify limited data as a constraint on reasoning quality.

7. Conclusion

  • E-KELL combines knowledge graphs and LLMs to enhance emergency decision support, and the study reports superiority across various emergency scenarios.The system structures emergency-related text in a standardized knowledge base and uses user queries as the basis for reasoning within it.Source: §7 · p. 13
  • The study’s main limitation is its limited data source, which significantly affects E-KELL’s reasoning quality.The authors also say the prompt design needs further testing.Source: §7 · p. 13
  • The authors plan to consult real-world emergency decision-making organizations and develop an AI-agent collaboration framework.Their proposed workflow includes data retrieval, data review, and comprehensive problem-solving.Source: §7 · p. 13

A. Prompt Templates

Table 3 contains prompt templates for queries.

A. Prompt Templates

  • The prompt templates for queries are provided in Table 3.The passage does not specify the template contents.Source: A. Prompt Templates · p. 13

B. Prompts and responses for evaluation

This section presents sample emergency-management evaluation queries and E-KELL responses, spanning equipment provision, hazardous-chemical response, investigation, hazard identification, firefighting, personal protection, first aid, and report compilation.

Q1: Material Reserves

  • For Class II hazardous-chemical units, E-KELL specifies quantities of detectors, hoses, rescue gear, protective equipment, medical supplies, and other supporting equipment.Examples include two toxic-gas detectors, 1,200 meters of fire hose, ten escape masks, and one first-aid kit.Source: B. Prompts and responses for evaluation · p. 13

Q2: On-site Command

  • For a hazardous-chemical leak, E-KELL recommends hazard-appropriate protective gear, rescuing trapped people, and evacuating others along designated upwind or side-upwind routes.It also calls for isolating the leakage zone and handing rescued people over to medical institutions after first aid and registration.Source: B. Prompts and responses for evaluation · p. 15

Q3: Accident Investigation

  • For the pipeline leak with natural disasters and human factors ruled out, E-KELL lists corrosion, design, construction, installation, protection, stress-prevention, and safety-management defects as possible causes.It also identifies improper gas storage or leakage as a possible contributor to pipeline corrosion.Source: B. Prompts and responses for evaluation · pp. 16–17

Q4: Hazard Source Identification

  • For sulfur dioxide and propylene, E-KELL reports sulfur dioxide as nonflammable but potentially explosive with potassium chlorate, and identifies explosive hazards for both gases.It gives sulfur dioxide’s boiling point as -10°C and gaseous relative density as 2.25, and propylene’s gaseous relative density as 1.5 and explosive limit as 1.0% to 15.0%.Source: B. Prompts and responses for evaluation · p. 18

Q5: Fire Extinguishment

  • For a warehouse leak and fire, E-KELL advises removing containers, cooling them with ample water without letting water enter them, and having skilled professionals manage the toxic-gas hazards.The prompt names sulfur dioxide and acetylene, but the response warns about sulfur dioxide and propylene.Source: B. Prompts and responses for evaluation · p. 19

Q6: Personal Protective Equipment

  • For the specified moderate-toxicity, low-hazard zone, E-KELL recommends protective suits and respiratory protection, chemical-resistant gloves and boots, and promptly notifying authorities.The response also calls for cutting off the gas source and closing ventilation ports.Source: B. Prompts and responses for evaluation · p. 20

Q7: Rescuing

  • For people who inhaled the leaked gases, E-KELL advises moving them to fresh air, keeping the airway clear, and seeking immediate medical attention for propylene inhalation.For sulfur dioxide inhalation, it advises removing contaminated clothing and rinsing skin and eyes with clean water; rescued people should be transferred to medical institutions.Source: B. Prompts and responses for evaluation · p. 21

Prompts and responses for evaluation

  • E-KELL recommends initial isolation of at least 500 meters and downwind evacuation of at least 1500 meters when contamination is unclear.Gas-concentration tests should guide later distance adjustments; storage tanks, tank trucks, or tank cars call for 1600 meters of isolation, and evacuation routes should be upwind or side upwind.Source: B. Prompts and responses for evaluation · p. 22
  • E-KELL advises safely closing valves and sealing leaks, preventing gas from entering sewers or confined spaces, and using water spray to control the gas.It also calls for eliminating ignition sources and using explosion-proof communication tools; its response warns that sulfur dioxide is toxic and says not to extinguish flames at the leakage site.Source: B. Prompts and responses for evaluation · pp. 23–24
  • E-KELL’s firefighting report checklist covers actions taken, water sources, fire-control and extinguishing times, spread area, people evacuated or rescued, and protected-property value.Source: B. Prompts and responses for evaluation · p. 25

CRediT authorship contribution statement

The authors report contributions spanning conceptualization, methodology, resources, software, data curation, writing, and project oversight.

CRediT authorship contribution statement

Related papers

Loading 2311.08732v1…