Source-linked AI summary
GeoRisk-RAG: A Hierarchy-Aware Risk Framework for Improving RAG Reliability through Selective Answering
Meenu Ravi, Shailik Sarkar, Lulwah AlKulaib, Yordanos Tessema, Chang-Tien Lu
TL;DR
Existing RAG and reliability methods often distinguish semantic relevance from geographic validity poorly, a consequential gap in natural-hazard QA. GeoRisk-RAG uses a Wikidata-derived geographic DAG for hierarchy-aware retrieval and selective answering. On a 449-sample wildfire QA dataset, it reduced false confidence on location-specific queries and received higher human preference alignment, while its current evaluation remains U.S.-based and dependent on Wikidata coverage.
Problem
Reliability methods often struggle to distinguish semantic similarity from geographic validity, although geographic granularity is significant for natural-hazard decision-making.
Method
GeoRisk-RAG uses an external-knowledge-graph geographic DAG to estimate applicability during retrieval and guide selective answering under geographic uncertainty.
Results
GeoRisk-RAG significantly reduces false confidence on location-specific queries while maintaining high factual correctness and higher human preference than baseline responses.
Takeaways & Limitations
Geographic validity and selective-answering behavior provide an explainable basis for safer, more transparent location-aware RAG in natural-hazard scenarios.
Takeaways & Limitations
The benchmark is modest and U.S.-based, and the framework depends heavily on the coverage and correctness of external knowledge graphs such as Wikidata.
Abstract
from arXiv · showhide
Current work on improving reliability in large language model (LLM)- generated answers has primarily leveraged Retrieval-Augmented Generation (RAG), knowledge-graph augmentation, and reinforcement learning. While these methods are adept at enhancing and measuring reliability through semantic similarity and faithfulness, they often struggle to distinguish semantic similarity from geographic validity. This is especially critical in natural hazard management domains where geographic granularity (i.e., town vs. city vs. state) is significant for decision-making, as responses valid in one municipality may not transfer to another. In such domains, a confidently wrong answer carries greater risk than abstaining. We present GeoRisk-RAG, a novel hierarchy-aware framework that addresses this geographic-validity gap through selective answering. This framework explicitly estimates geographic applicability using a Directed Acyclic Graph (DAG)-based distance for context retrieval before response generation. Experiments on a novel held-out wildfire-related question-answering (QA) dataset show that GeoRisk-RAG significantly reduces false confidence rates for location-dependent questions, lowering the rate to 0.009 compared with ~0.090 for standard semantic similarity and reranking baselines, while consistently achieving higher human preference alignment. This work provides a more comprehensive assessment of end-to-end RAG pipelines by integrating geographic validity and selective-answering behavior for safer decision-making in geospatial domains.
1 Introduction
LLM reliability methods often overlook whether retrieved information is geographically applicable. GeoRisk-RAG addresses this gap by combining hierarchy-aware retrieval with selective answering for natural-hazard QA.
- Motivation: Semantic similarity can retrieve geographically inapplicable passages, while spatial proximity alone may not ensure that a response transfers across jurisdictions.Geographic granularity matters because guidance valid in one region may not apply to another.
- Framework: GeoRisk-RAG uses a Wikidata-derived geographic DAG to measure hierarchical distance between a query location and retrieved passages.The framework classifies context as exact, broader, more-specific, or different-place context.
- Framework: The framework jointly optimizes semantic relevance and geographic alignment for natural-hazard question answering.This extends conventional retrieval beyond semantic similarity.
- Selective answering: GeoRisk-RAG restricts answers to exact geographic matches, warns about broader or narrower context, and abstains for different jurisdictions.These behaviors implement geographic grounding under imperfect retrieval.
- Benchmark and evaluation: The benchmark contains 449 diverse wildfire QA pairs representing realistic public information needs across geographic granularities.The dataset supports evaluation of location-aware QA.
- Benchmark and evaluation: Evaluation incorporates geographic validity alongside correctness, answering behavior, context quality, and user preference.This broadens reliability assessment beyond semantic similarity.
2 Related Works
Prior work improves factuality and retrieval through RAG, knowledge graphs, and other reliability techniques, but domain-specific geographic applicability remains under-addressed. GeoRisk-RAG is motivated by this gap and evaluated with a public-centered wildfire QA benchmark.
- Hallucination mitigation: Existing reliability methods mitigate hallucinations through RAG, self-consistency checks, activation-based prediction, and human-in-the-loop reinforcement learning.These approaches remain largely general-purpose.
- Research gap: Existing approaches generally do not explicitly model domain-specific dependencies that determine whether retrieved context applies to a query.Geographic granularity is identified as a significant applicability dependency in disaster scenarios.
- Graph-augmented RAG: GraphRAG, KG-RAG, GeoRAG, and HippoRAG use graph structure or spatial relationships to enrich retrieval and reasoning.Their graph uses range from document relationships to spatial proximity and entity-text connections.
- Natural-hazard QA: Natural-hazard systems such as WildfireGPT, DisasterResponseGPT, SafeMate, and ClimateGPT ground responses in data, modular retrieval, domain training, or knowledge graphs.These tools are described as primarily adapted for domain-expert decision-making.
- Benchmark gap: The proposed benchmark addresses a gap in public-centered natural-hazard QA datasets by labeling geographic granularity and location dependence.It is constructed from government wildfire-related FAQs and targets geographically appropriate grounding.
3 Methodology
GeoRisk-RAG constructs a wildfire QA benchmark and a hierarchy-aware retrieval framework that combines semantic relevance with geographic structure. Its DAG-based representation supports geographic distance measurement and rank optimization for retrieving location-appropriate context.
- 3.1 Benchmark Dataset Creation: The benchmark contains 449 wildfire-related QA pairs manually annotated for geographic granularity, topic, location dependence, and public-information attributes.It combines event-centric reporting with institutionally grounded procedural guidance.
- 3.2 Knowledge Base Construction: The document base combines approximately 2,000 wildfire news articles with 125 official agency guidelines from 30 U.S. states.Sources were filtered for credibility, and duplicate articles were removed before processing guideline PDFs.
- 3.2.2 Knowledge-base Chunking Strategy: Semantic segmentation introduces document boundaries when adjacent-sentence cosine distance exceeds the document-specific 95th-percentile threshold.This restricts segmentation to the top 5% of major contextual shifts; the resulting corpus contains 12,975 nodes.
- 3.3 Location Named Entity Recognition: Flair was selected for location NER because it achieved the highest extraction quality, with F1 = 0.9290 while mitigating latency from larger transformer models.The models were evaluated on the out-of-sample GeoVirus dataset (n=229 texts).
- 3.4.1 Defining the DAG Model: The geographic knowledge space is modeled as a Wikidata DAG whose vertices are geographic entities and whose directed edges encode containment relationships.Edges use Wikidata relations including p131 and p361; entity hierarchies are defined through reachable ancestors.
- 3.4.2–3.4.3 Geographic Retrieval and Similarity: GeoRisk-RAG ranks candidate passages first by minimizing query-to-passage geographic distance and second by maximizing semantic cosine similarity.The retrieval objective formalizes this ordering as an optimization over permutations of the candidate set.
4 Experimental Study
The experimental study compares retrieval and model configurations while evaluating geographic applicability, selective-answering decisions, and reliability under geographic uncertainty.
- 4.1 Baselines and Models: Four retrieval methods span semantic similarity, lexical reranking, geographic granularity, and GeoRisk-RAG assumptions.The study also evaluates four embedding-and-generative-model combinations.
- 4.2 Model Configuration Stability Analysis: Stability analysis measures each configuration’s mean cross-configuration pairwise cosine similarity using an independent embedding model.Bootstrap resampling over 1,000 query-set samples provides 95% confidence intervals for evaluation metrics.
- 4.2 Model Configuration Stability Analysis: Consistent similarity scores indicate that performance differences are driven by retrieval method rather than model choice, motivating use of bge-small-en-v1.5 + gpt-oss-120b for efficiency.The selected configuration shows no meaningful stability drop relative to bge-base-en-v1.5.
- 4.3 Grounded Reliability and Correctness Behavior: The evaluation uses six metrics balancing retrieval and generation quality with geographic alignment, including GDR, faithfulness, and appropriate warning rate.An LLM judge receives the query, target geography, ground truth, retrieved context, and generated response.
- 4.4 Geographic Applicability Experiments: The experiments test whether retrieval methods preserve geographic applicability beyond topic relevance through decision-consistency and error-sensitivity analyses.Semantically grouped wildfire questions are evaluated across substituted locations to expose location-dependent answering behavior.
- 4.4.1 Geographic errors.: GeoRisk-RAG evaluates geographic relationships using exact matches, missing context, mismatches, and broader or finer geographic granularities.These categories support analysis of how geographic evidence failures affect selective-answering decisions and correctness.
5 Results
GeoRisk-RAG achieves strong reliability on both location-dependent and non-location-dependent wildfire queries while reducing false confidence and maintaining consistent selective-answering behavior across locations.
- Overall reliability: 0.946 GDR on location-dependent queries and 0.974 on non-location-dependent queries give GeoRisk-RAG the highest overall reliability.The framework remains competitive on general wildfire queries with 0.963 GDR.
- False confidence: 0.009 false confidence rate for GeoRisk-RAG compares with 0.090 for standard semantic similarity and reranking baselines on location-dependent questions.The framework also maintains strong factual correctness at 0.761.
- Decision consistency: GeoRisk-RAG maintains high GDR across seven out of eight question types when the same questions are posed across different locations.Wildfire-season timeframe questions require retrieval at specific geographic granularities because answers can vary within one state.
- Error sensitivity: Cases without geographic evidence have the lowest correctness, averaging 0.404 across five configurations.Broader or narrower evidence can still support correct responses, but selective answering is needed when geographic scope does not align.
6 Case Study
The Albuquerque case study shows GeoRisk-RAG using broader state-level context when local evidence is unavailable, while warning the user about the geographic scope.
- Case setup: The case study asks whether wildfire-sensing improvements exist for Albuquerque, New Mexico.It examines location-specific question answering in Figure 6.
- Evidence availability: No knowledge-base passages address wildfire-sensing technologies specifically in Albuquerque.The system therefore uses broader geographic context rather than local evidence.
- Selective answering: GeoRisk-RAG escalates to New Mexico-level information, explicitly warns about the state-level scope, and provides an answer aligned with ground truth.The response was manually verified against the retrieved sources.
7 Discussion and Conclusion
The paper presents GeoRisk-RAG as a hierarchy-aware approach for transparent, location-aware wildfire QA, reports reduced false confidence and higher human preference, and identifies scope boundaries for future work.
- Discussion and Conclusion: GeoRisk-RAG distinguishes semantic similarity from geographic applicability through a Wikidata-derived DAG and explainable selective answering.The framework is designed for location-aware question answering in natural-hazard scenarios.
- Discussion and Conclusion: The 449-sample wildfire QA evaluation finds reduced false confidence, high factual correctness, and greater human preference than baseline responses.Blind human evaluations selected GeoRisk-RAG responses as more preferable.
- Limitations and Future Work: The benchmark is modest in size and U.S.-based, limiting current coverage across countries, languages, and natural hazards.The framework also depends heavily on the coverage and correctness of external knowledge graphs such as Wikidata.
9 GenAI Usage Disclosure
The paper discloses that LLMs were used in the experimental methodology, while ChatGPT-5 was limited to manuscript grammar and typographical review.
- LLMs were used as part of the experimental methodology.
- ChatGPT-5 was used only to review grammatical and typographical errors, not for technical or text generation.
A Wildfire QA Dataset
The section presents a table of wildfire question categories used in the dataset.
- Table 8 lists the wildfire question categories.
B News Article Retrieval
The news-retrieval process used reputable domains and a set of wildfire-related search terms, with annotators evaluating answer preference for location-aware quality.
- Google News articles were retrieved using wildfire-related query terms.The terms included “wildfire”, “forest fire”, “bushfire”, “brushfire”, “fire season”, “wildland fire”, and “wildland-Urban Interface”.
- Table 9 lists reputable news domains used for filtering Google News articles.
- Annotators selected the answer they would most prefer as a user, considering accuracy, usefulness, warnings, abstention, and location.
D Top-K Retrieval
The section identifies the elbow-point method for selecting top-k retrieval and documents the LLM configuration and evaluator prompt used in the wildfire RAG experiments.
- Figure 7 shows optimal top-k selection using the elbow-point method on the wildfire benchmark dataset.
- Table 10 lists the LLM models’ configuration settings.
- Figure 8 presents the prompt given to nemotron-3-super to evaluate a location-aware wildfire RAG system.