Source-linked AI summary
GeoOutageBench: Benchmarking Ambiguity-aware, Ontology-grounded Geospatiotemporal KGQA for Multimodal Power Outage and Resilience Analysis
Ethan D. Frakes, Amy Kvien, Rishabh Kundu, Redad Mehdi, Van D. Tran, Vibha S. Mandayam, Kristopher O. Davis, Erika I. Barcelos, Roger H. French, Yinghui Wu, Mengjie Li
TL;DR
Existing benchmarks offer limited support for jointly evaluating ambiguity, ontology utility, and multimodal evidence in spatiotemporal KGQA for outage and storm resilience. GeoOutageBench provides a multimodal KGQA benchmark and configurable evaluation of those tasks. Its experiments report Claude Opus 4.7 as the strongest overall performer for NL spatiotemporal understanding.
Problem
Existing KGQA benchmarks provide limited support for assessing how ontologies, geospatial KGs, and multimodal evidence jointly affect spatiotemporal outage and storm-resilience KGQA.
Source: §1 · p. 2Method
Results
Claude Opus 4.7 provides the strongest overall Task 1 performance for NL spatiotemporal understanding, combining the best alignment, schema-linking, and spatiotemporal-constraint scores.
Source: §4 · p. 8, §5 · p. 10Takeaways & Limitations
GeoOutageBench provides evaluation infrastructure and assessment tools for LLM-based spatiotemporal analysis of power-system resilience.
Source: §1 · p. 3Black Marble and EAGLE-I layers are sampled within Florida, and future work will extend them beyond the sampled Florida counties.
Source: §5 · p. 10
Abstract
from arXiv · showhide
We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis. Unlike existing KGQA benchmarks for Web knowledge, GeoOutageBench considers a spatiotemporal KG that integrates visual, textual, and structured data from outage records, remote sensing, weather observations, storm and power events, geographic entities, and domain ontologies. It provides a competency query taxonomy at different difficulty levels from spatiotemporal containment and proximity, spatiotemporal co-occurrence analysis, multimodal evidence, to hypothetical evaluation. Over multimodal KG and query classes, GeoOutageBench provides user-configurable evaluation of three important, highly coherent yet less studied tasks: (1) LLMs' understanding for ambiguous geospatiotemporal questions in terms of NL to SPARQL interpretation, (2) query-driven assessment of ontology utility, and (3) answer accuracy of multimodal KGQA retrieval. GeoOutageBench provides a design principle and foundation for assessing LLM-KG systems that support real-world infrastructure resilience analysis. Our benchmark, source code, data, results, and other documentation are available at https://github.com/UCF-SAGE/GeoOutageBench.
1 Introduction
Existing KGQA benchmarks provide limited support for jointly assessing ontology use, geospatial KGs, and multimodal evidence in outage and storm-resilience analysis. GeoOutageBench introduces a benchmark for evaluating these capabilities alongside ambiguity-aware interpretation of geospatiotemporal questions.
1 Introduction
- Existing benchmarks offer limited support for assessing how ontologies, geospatial KGs, and multimodal evidence jointly affect spatiotemporal KGQA for outage and storm-resilience analysis.The paper identifies domain-specific spatiotemporal understanding, ontology and KG utility, and multimodality as undermeasured areas.Source: §1 · p. 2
- A single outage-and-storm question can license different bindings for state, year, SVI threshold, and event-count semantics, yielding different executable queries.The example presents two ontology-grounded readings that preserve county-year co-occurrence while taking different query paths.Source: §1 · p. 2, Figure 1 · p. 2
- GeoOutageBench contributes a multimodal geospatiotemporal KG, seven query categories, three benchmark tasks, and a configurable sandbox for outage and storm-resilience KGQA.The tasks assess NL-to-SPARQL interpretation, multimodal KGQA, and query-driven ontology utility.Source: §1 · p. 2
- The paper positions GeoOutageBench as evaluation infrastructure for LLM-based spatiotemporal power-system resilience analysis, not as a new KGQA system.Its motivation is to support accurate location- and time-sensitive power-event analysis and help users assess LLMs, ontologies, and multimodal data together.Source: §1 · pp. 2–3
2 GeoOutageBench Overview
GeoOutageBench organizes evaluation around configurable questions, reference interpretations, ontologies, KGs, models, and metrics, with operators for interpretation and retrieval. Its ontology and KG integrate multimodal evidence through shared geographic grounding, while its competency queries span varied reasoning types and are paired with validated reference interpretations.
2 GeoOutageBench Overview
- A user-specified profile combines NL questions, reference SPARQL interpretations, ontology, KG, models, and metrics; operators support interpretation, query evaluation, ontology-aware retrieval, and multimodal answers.The operator library also includes user-defined tools such as geodesic buffers, spatiotemporal joins, kNN search, and raster-tile access.Source: §2 · pp. 3–4
2.1 GeoOutageBench Ontologies and KGs
- GeoOutageOnto supplies a semantic schema for heterogeneous geospatial and temporal resources, while GeoOutageKG grounds entities in location, acquisition time, and record-time metadata.The ontology is designed to integrate satellite imagery, meteorological products, outage records, and storm reports; it is curated to align conceptually with MDS-Onto.Source: §2.1 · p. 4
- The KG integrates county-level outage observations, storm events, hurricane metadata, and social vulnerability measures; Black Marble and EAGLE-I-derived assets are currently sampled within Florida.Storm Events, SVI, HURDAT, and TIGER geometries are ingested for all counties, while broader raster and outage-record coverage requires rerunning ingestion over additional TIGER geometries.Source: §2.1 · p. 5
- County geography links county-partitioned nighttime-light imagery and outage-severity maps with spatially grounded tabular records, forming evidence for multimodal geospatiotemporal questions.The raster layer compares daily radiance with the preceding three-month average to encode anomalous radiance loss as outage severity.Source: Figure 4 · p. 5, §2.1 · p. 5
2.2 Competency Query Classes
- Competency questions cover seven categories and one-hop through multi-hop complexity, spanning spatial and temporal filtering to links among geographies, hazards, outages, imagery, and vulnerability.Questions are manually authored or generated from parameterized templates, with human experts adding ambiguity and query-generation difficulty.Source: §2.2 · p. 6
- Computer-science experts validate gold SPARQL interpretation sets, and domain experts manually validate answers generated by executing the reference queries on supporting data engines.The built-in question set combines expert-designed questions with LLM-assisted generation.Source: §2.2 · p. 6
3 Benchmark Tasks and Evaluation Metrics
GeoOutageBench evaluates geospatiotemporal KGQA through three tasks: interpreting ambiguous questions, assessing ontology utility, and measuring multimodal answer quality. Its metrics cover query interpretation, ontology-term coverage and closeness, and answer-level correctness.
3 Benchmark Tasks and Evaluation Metrics
- The benchmark evaluates LLM interpretation of geospatiotemporal questions, ontology utility, and multimodal KGQA answer quality through three configurable tasks.The task configuration uses questions, an ontology, reference interpretations, a KG, models, and evaluation metrics.Source: §3 · p. 6, §3.1 · p. 7, §3.2 · p. 7
3.1 Task 1: Assessing LLM’s ability in Geospatiotemporal Understanding
- Task 1 scores interpretation ambiguity and query accuracy, including syntactic validity, endpoint compatibility, and alignment with gold ontology grounding and spatiotemporal constraints.Alignment uses class, property, spatial-constraint, and temporal-constraint features; the best generated–gold interpretation pair is selected for each question.Source: §3.1 · p. 7
- Task 1 also reports schema-level F1 for classes, properties, and schema-linking, alongside spatial, temporal, and spatiotemporal constraint F1 measures.Schema-linking F1 is the harmonic mean of class and property F1; question-level spatiotemporal scoring requires both constraint types in the same interpretation.Source: §3.1 · p. 7
3.2 Task 2: Ontology Utility Evaluation
- Task 2 assesses ontology utility with query coverage and query closeness, measuring workload-term overlap and penalizing ontology concepts not seen in the questions.The scoped ontology term set is built from ontology terms associated with nodes within h hops of KG instances typed by workload class labels.Source: §3.2 · pp. 7–8
3.3 Task 3: Multimodal Spatiotemporal KGQA
- Task 3 compares structured and multimodal KGQA workflows using answer-quality metrics, including spatial and temporal correctness and answer-level Spatiotemporal Relevance Score.The workflows execute reference or generated interpretations and compare their answers with reference answers.Source: §3.3 · p. 8
4 Experimental Evaluation
GeoOutageBench evaluates ambiguous NL-to-SPARQL interpretation, ontology utility, and multimodal KGQA using GeoOutageOnto, GeoOutageKG, and a workload spanning competency-query categories. Across these evaluations, model performance varies by task and query complexity, while multimodal retrieval adds date-aligned visual evidence to county-level answers.
Evaluation-1: Assessing LLM for Power Outage KGQA
- Task 1 evaluates 48 questions with up to three valid SPARQL interpretations each; all three models generate syntactically valid queries, while exact query-text match is near zero.GPT-5.5 and Claude Opus 4.7 have full endpoint compatibility; Gemini 3.1 Pro reaches 0.979.Source: §4 · p. 8
- The three models generate different numbers of interpretations for the 48-question set, treated as an ambiguity diagnostic balancing execution cost against coverage of alternative readings.Gemini 3.1 Pro generated 51 interpretations, GPT-5.5 generated 60, and Claude Opus 4.7 generated 61.Source: §4 · p. 8
- Claude Opus 4.7 provides the strongest Task 1 overall performance, leading on interpretation alignment, schema-linking, and spatiotemporal-constraint scores.It leads corpus-level spatial, temporal, and spatiotemporal constraint F1 at 0.886, 0.754, and 0.711, respectively; GPT-5.5 is the closest alternative on spatial F1.Source: §4 · p. 8
Evaluation-2: Ontology Utility Evaluation
- GeoOutageOnto covers every extracted domain term in the 72-interpretation workload, while query closeness varies from 0.0196 to 0.1373 with a 0.0673 mean.The authors attribute the range to workload heterogeneity: more complex queries activate broader schema coverage.Source: §4 · pp. 8–9
Case study on competency queries
- Alternative interpretations of competency questions vary in state, year, vulnerability threshold, event count, Hurricane Ian time window, and geographic radius.The examples include three different readings of “around Hurricane Ian” and three readings of “within the general area of Cleveland County.”Source: §4 · p. 9
Evaluation-3: Multimodal KGQA
- Claude Opus 4.7 obtains the highest answer F1 and SRS, while GPT is comparable in answer F1 and slightly higher on Hits@1; Gemini lags in answer F1 despite strong Task 1 syntax and class-recovery scores.Answer-level errors can result from property choice, temporal filtering, aggregation, or administrative joins even when query form remains plausible.Source: §4 · p. 9, Table 6 · p. 10
- Multi-hop questions are substantially harder than two-hop questions for GPT and Claude, and Gemini drops most sharply on multi-hop questions.Hurricane-landfall ranking and event-sequence questions remain difficult because they require coordinated event ranking, spatial aggregation, storm semantics, and vulnerability-threshold setting.Source: §4 · p. 9
Single Modal vs. Multimodal Querying: A Case Study
- For the outage-and-storm case, tabular answers omit remote-sensing evidence, whereas multimodal queries return date-aligned nighttime-light images and outage maps linked to the same counties and time constraints.The benchmark can assess entity correctness alongside whether the retrieved multimodal bundle supports a spatiotemporal explanation; Figure 5 presents the single- versus multimodal comparison.Source: §4 · p. 10, Figure 5 · p. 10
5 Conclusions
GeoOutageBench evaluates LLM-based spatiotemporal KGQA for outage, social-vulnerability, and storm-resilience analysis using a multimodal knowledge graph, ontology, and competency queries. Its evaluation reports model differences across query understanding, executability, spatial grounding, and class recovery, and finds GeoOutageOnto and multimodal GeoOutageKG effective for answer accuracy.
5 Conclusions
- GeoOutageBench combines a multimodal geospatiotemporal knowledge graph and ontology with seven query templates and metrics for spatiotemporal understanding, ontology utility, and multimodal KGQA.The benchmark targets outage, social-vulnerability, and storm-resilience analysis.Source: §5 · p. 10
- Claude Opus 4.7 performs best overall in NL spatiotemporal understanding; GPT-5.5 is competitive on executability and spatial grounding, while Gemini 3.1 Pro leads class recovery.Query-level and finer-grained assessment also verifies the effectiveness of GeoOutageOnto and multimodal GeoOutageKG for improving answer accuracy in power outage analysis.Source: §5 · p. 10
- Future work will expand query semantics and disruption causes, and extend Black Marble and EAGLE-I beyond the sampled Florida counties.Planned query extensions include cross-modal and cross-model retrieval and counterfactual spatiotemporal reasoning; additional disruption causes include cyberattacks and grid overload.Source: §5 · p. 10
A.1 Query-Interpretation: Prompts
The NL2STQ prompt provides an ontology-grounded template for generating default SPARQL interpretations without access to gold queries. It accommodates distinct readings of ambiguity while imposing vocabulary and formatting constraints.
A.1 Query-Interpretation: Prompts
- NL2STQ gives the LLM benchmark metadata, the GeoOutageOnto ontology, a compact vocabulary file, and a JSON prediction template, but no gold SPARQL queries.The interpreter fills the template’s sparql_query field with its best default interpretation.Source: §A.1 · p. 12
- The prompt requires ontology-vocabulary alignment when available and distinct interpretations for different spatial, temporal, semantic, or spatiotemporal readings.It discourages variants that only rename variables or reorder triples.Source: §A.1 · p. 12
- The prompt standardizes SPARQL prefixes, projected-variable binding, ISO 8601 dateTime filters, half-open day ranges, aggregation, ranking, and namespaces.It internally prompts consideration of answer type, ontology class, properties, filters, multimodal evidence, and ambiguity, but returns only the completed JSON prediction file.Source: §A.1 · p. 12
A.2 Evaluation Metrics
The evaluation combines formal query-generation checks, ontology-utility measures, and answer-level KGQA assessment. It scores interpretation structure and ambiguity, measures ontology vocabulary coverage and activation, and evaluates execution and returned answers.
A.2 Evaluation Metrics
- GeoOutageBench evaluates formal query generation, ontology utility, and answer-level spatiotemporal KGQA.Source: §A.2 · p. 12
- An empty generated and gold feature set scores 1, while an empty prediction against a non-empty gold set scores 0.Source: §A.2 · p. 12
- Task 1 also reports normalized exact-query match, interpretation alignment, syntax validity, and endpoint compatibility.Syntax validity checks generated interpretations for a coarse SELECT/ASK and WHERE/braces structure; compatibility credits at least one interpretation that executes over the configured graph or endpoint.Source: §A.2 · p. 12
- Ambiguity-aware Task 1 aligns generated and reference interpretation sets, selecting the pair with the highest average class, property, spatial, and temporal F1.Schema-linking and spatiotemporal-constraint F1 use harmonic means; axis-level reporting groups questions by their spatiotemporal annotations.Source: §A.2 · pp. 12–13
- Task 2 measures how much workload vocabulary the ontology declares through Coverage and how densely queries activate its scoped vocabulary through Closeness.All 72 ground-truth queries had Coverage of 1.00; Table 7 lists their Closeness scores.Source: §A.2 · p. 13, Table 7 · p. 13
- Task 3 executes generated SPARQL interpretations against GeoOutageKG and compares normalized returned bindings with gold answers.Gold answers come from executing ground-truth interpretations or, when all interpretations return empty results, from stored human-validated answers.Source: §A.2 · p. 13
- For ranked outputs, MRR is reported as the macro-average of RR_i; row relevance is represented by a binary indicator at rank ℓ, but the relevance criterion is not specified in these passages.Source: §A.2 · p. 13
- Spatial and temporal correctness use explicit annotations when available, otherwise deriving from generated–gold constraint overlap; dimensions not required by a question receive full credit.When a required dimension has no query-level signal, it falls back to answer F1; answer-level alignment selects the generated–gold pair maximizing the interpretation-selection score.Source: §A.2 · pp. 12–13