PaperlayerView exact revision on arXiv

Source-linked AI summary

KGVoyager: Knowledge Graph Agnostic Question Answering via Agentic Navigation

Essam Wisam, Chengkai Li

arXiv:2609.01780v1cs.IR

TL;DR

Domain-specific KGQA often lacks ontologies and curated text-SPARQL examples. KGVoyager uses an endpoint-only agentic exploration process to generate and refine SPARQL queries, improving F1 by ∼8 points while reducing cost and runtime by ∼22% each across four benchmarks.

  • Problem

    Domain-specific KGQA is challenged by unavailable or opaque ontologies and scarce curated text-SPARQL examples.

  • Method

    KGVoyager uses search, exploration, and execution tools in a think-act-observe loop to discover graph structure and generate validated SPARQL queries from natural language.

  • Results

    Across four domain-specific benchmarks, KGVoyager improves F1 by ∼8 points while reducing cost and runtime by ∼22% each.

  • Takeaways & Limitations

    Endpoint-only KGQA is viable and efficient for domain-specific knowledge graphs where curated resources are often unavailable.

  • Takeaways & Limitations

    The current tool suite assumes standard RDFS/RDF typing patterns, making KGVoyager best suited to RDFS-based or RDFS-compatible graphs.

Abstract

from arXiv · show

Knowledge Graph Question Answering (KGQA) over RDF graphs remains challenging in domain-specific settings, where formal ontologies and curated text-SPARQL pairs are often unavailable. We present KGVoyager, a KG-agnostic agentic architecture that generates SPARQL queries from natural language questions by dynamically discovering graph structure and semantics, requiring only a query endpoint of the underlying graph. Using a think-act-observe loop with search, exploration, and execution tools, KGVoyager maps terms to graph IRIs, uncovers structure, and refines queries through execution feedback - all without pre-existing ontologies or examples. Unlike the prior state of the art, KGVoyager requires only a lightweight class index which renders it applicable for far more real-world endpoints. Across four benchmarks, KGVoyager improves F1 by ~8 points while cutting cost and runtime by ~22% each.

1 Introduction

KGVoyager addresses domain-specific KGQA settings where ontologies, documentation, and text-SPARQL examples may be unavailable by discovering graph structure through endpoint access. Its agentic tool suite generates and validates SPARQL queries, improving benchmark performance while reducing resource use.

  • Motivation: Domain-specific KGQA is hindered by missing formal ontologies, opaque graph structure, and scarce text-SPARQL examples.These resources commonly ground natural-language-to-SPARQL systems but are rarely available for arbitrary domain-specific graphs.
  • Approach: KGVoyager assumes only a SPARQL query endpoint and dynamically discovers graph structure, semantics, and instance data.Its searching, sampling, and navigation tools support exploration resembling a human expert’s first encounter with an unfamiliar graph.
  • Approach: KGVoyager uses a think-act-observe loop to map question terms to graph IRIs, explore their relationships, and refine queries through execution feedback.The agent validates query drafts until it derives an accurate answer query.
  • Results: Across four domain-specific benchmarks, KGVoyager improves F1 by ∼8 points while reducing cost and runtime by ∼22% each.The comparison is against the prior endpoint-only, KG-agnostic state of the art, GRASP.
  • Availability: KGVoyager is released as open source for deployment against SPARQL endpoints with minimal configuration.The release includes the system needed for practitioner use.

2 Related Work

KGVoyager extends endpoint-only, KG-agnostic KGQA beyond approaches tied to exhaustive entity indexes, local graph deployment, initial entities, or domain-specific engineering. Its distinctions include lighter indexing, data-driven property semantics, and unified hybrid retrieval.

  • GRASP: GRASP is the closest prior endpoint-only system, but its complete entity index succeeds on only ∼21% of Jena-served and 0% of QLever-served okn.us endpoints.KGVoyager instead uses a lightweight class-level index.
  • KGVoyager: KGVoyager infers property semantics directly from graph data through dedicated exploration tools.This differs from approaches relying on available schema annotations or other external semantic resources.
  • KGVoyager: KGVoyager combines semantic and keyword search through hybrid retrieval, whereas GRASP uses them as separate signals across tools.The comparison identifies retrieval mechanism as one of the systems’ fundamental differences.
  • ToG: ToG traverses instance-level graph data hop by hop, while KGVoyager explores structure and semantics to construct executable SPARQL queries.ToG also requires substantial adaptation effort and is evaluated only on open-domain graphs in the described comparison.
  • ToG: ToG requires local graph deployment, a graph-specific traversal interface, and initial entities, whereas KGVoyager uses unified SPARQL tools and performs grounding within the endpoint-only task.KGVoyager’s SPARQL queries can express arbitrarily many hops in a single execution.
  • Other systems: Spinach and Expasy depend on domain-specific tools, schemas, or curated text-query pairs, limiting their generalization to arbitrary knowledge graphs.Their engineering assumptions are tied respectively to Wikidata or bioinformatics settings.

3 Methodology

KGVoyager formulates natural-language-to-SPARQL generation as iterative graph exploration using search, exploration, and execution tools. The agent resolves graph elements, inspects their connections, and tests query drafts against the endpoint.

  • Architecture: KGVoyager assumes RDFS-compatible class definitions so KG-agnostic tools can operate through standard SPARQL queries.The framework’s applicability depends on this structural convention.
  • Architecture: The think-act-observe loop organizes search, exploration, and execution into an iterative query-construction process.The agent reasons about the next action, invokes a tool, observes its result, and continues refining the query.
  • Search Tools: Search tools retrieve candidate classes, instances, and properties from natural-language cues and map them to graph IRIs.Class and property searches use locally cached indexes with ranked retrieval and relevance scores.
  • Search Tools: Adaptive filtering prunes low-relevance search candidates using a score-ratio threshold while ensuring a minimum number of results.The mechanism is intended to limit low-quality candidates entering the agent’s context.
  • Search Tools: Class-scoped instance lookup verifies candidate classes or resolves named entities without requiring an exhaustive entity index.Exact lexical matching can miss abbreviations, alternative spellings, or overly specific names.
  • Exploration Tools: Exploration tools inspect property usage and entity neighborhoods to reveal graph connections that search results do not expose.Property usage aggregates subject and object classes with representative examples, while entity exploration retrieves incoming and outgoing relations.
  • Execution: The execution tool runs final and intermediate SPARQL queries, using results and errors to test hypotheses, validate answers, and self-correct drafts.Execution feedback is central to the agent’s iterative reasoning process.
  • Running Example: In the DBLP example, ten tool calls resolve the author, conference, and publication relation before a COUNT query returns 1.The agent discovers the needed publishedInStream relation by inspecting the conference entity and then confirms the complete triple pattern through execution.

4 Experiment Results

Experiments assess indexing feasibility, benchmark performance, tool-use dynamics, and ablations across domain-specific knowledge graphs and models. KGVoyager generally improves answer quality while reducing resource use, with model-dependent and benchmark-specific exceptions.

  • Index Feasibility: 96.6% of QLever endpoints supported KGVoyager’s class indexing, while entity indexing succeeded on 0% of endpoints.Under Jena, class indexing succeeded on 69.0% of endpoints versus approximately 21% for entity indexing.
  • Evaluation Setup: The study compares KGVoyager with GRASP using three-run averages of F1, monetary cost, agent iterations, and wall-clock time.Figure 3 reports run-to-run standard deviations as error bars, while Table 2 reports performance by benchmark averaged across four backbone models.
  • QA Performance: KGVoyager improved F1 on all four benchmarks, with gains ranging from 3.9 percentage points on DBLP to 14.0 points on Climate Models KG.It also reduced cost and agent iterations on every benchmark; runtime was lower on three benchmarks, with DBLP as the exception.
  • QA Performance: Separate class, instance, and property tools, neighborhood exploration, and adaptive search filtering were recurring patterns in cases where KGVoyager outperformed GRASP.The analysis links these patterns to fewer fragile iterations and less context rot, while noting that the explanation for context rot is qualified as likely.
  • Tool-Use Dynamics: Gemini showed cleaner search–exploration–execution phase separation than Devstral, whose F1 was 46.8% versus 70.1% and iterations were 22.30 versus 12.39.Both models nevertheless followed the intended ordering of search early, exploration in the middle, and execution later.
  • Ablation Study: Removing entity-neighborhood exploration reduced F1 by 1.8 to 3.4 points, while hybrid retrieval improved F1 by 2.7 to 6.1 points over keyword-only search.Property-usage exploration improved F1 by up to 8.7 points on SOCKG and 4.8 points on DREAMKG with Gemini 3 Flash, but added time, cost, and iterations.

5 Conclusion

KGVoyager enables natural-language question answering over arbitrary RDF knowledge graphs using only SPARQL endpoint access. Its lightweight class index is substantially more feasible than GRASP’s entity index, while benchmark experiments show improved F1 with generally lower runtime, cost, and agent iterations.

  • Conclusion: KGVoyager answers natural-language questions over arbitrary RDF knowledge graphs using only SPARQL endpoint access.The framework is designed for settings where curated resources are often unavailable.
  • Conclusion: 96.6% of QLever okn.us endpoints supported KGVoyager’s lightweight class index, compared with 0% for GRASP’s required entity index.This feasibility result establishes the practical difference between the two indexing requirements.
  • Conclusion: Across four domain-specific benchmarks and four backbone models, KGVoyager improved F1 over GRASP while generally reducing runtime, cost, and agent iterations.The conclusion characterizes endpoint-only KGQA as viable and efficient for domain-specific knowledge graphs where curated resources are often unavailable.

Limitations

KGVoyager’s applicability is bounded by structural assumptions in its tool suite, while domain-specific evaluation data remain scarce.

  • Structural scope: KGVoyager assumes endpoints expose classes and instance membership through standard RDFS/RDF typing patterns.This makes it best suited to RDFS-based or RDFS-compatible knowledge graphs.
  • Structural scope: Graphs with substantially different modeling conventions, notably Wikidata, are outside the current implementation’s strongest applicability scope.Supporting them would likely require adapter tools or graph-specific normalization layers.
  • Data availability: Domain-specific text-SPARQL benchmarks remain scarce, and the authors could not gather larger datasets despite contacting many KG authors and maintainers.The paper identifies broader public benchmark availability as a need.
  • Evaluation scope: Table 3 reports ablation results as mean ± standard deviation over three independent runs.The table covers ablations rather than introducing a separate limitation claim.

Ethical Considerations

The paper reports responsible use of public research artifacts and highlights verification and environmental considerations for LLM-based KGQA.

  • Data and artifacts: All endpoints and benchmarks used were publicly available, and no private or personal data were collected or processed.Released artifacts are intended for research, evaluation, and reproducibility under applicable third-party terms.
  • Reliability: KGVoyager may generate syntactically valid SPARQL queries that are semantically incorrect.The paper advises verification when query results inform scientific or policy decisions.
  • Environmental considerations: LLM inference carries an environmental cost, which the paper notes is partially mitigated by approximately 22% lower monetary cost and wall-clock runtime than the prior state of the art.The reduction is reported as a relative comparison.

A Implementation Details

The implementation section identifies the parameters used across experiments, with Table 4 serving as their reference.

  • Implementation parameters: Table 4 lists the key implementation parameters used across all experiments.The passage does not enumerate the individual parameter values.
  • Experimental scope: The implementation parameters are intended to cover all experiments rather than a single benchmark or model.This scope is stated in the surrounding description of Table 4.
  • Implementation parameters: Table 4 is presented as an implementation-parameters table.Its caption identifies the table’s scope without specifying individual settings.

B Query Complexity Analysis

The query-complexity analysis measures structural complexity from gold SPARQL queries and compares paired KGVoyager-minus-GRASP F1 differences across complexity levels. KGVoyager’s average improvement appears larger for queries with at least five triple patterns.

  • Complexity measurement: Query complexity is measured using triple-pattern counts and distinct special SPARQL constructs parsed from gold-query algebra.Constructs include aggregation, FILTER, OPTIONAL, UNION, MINUS, EXISTS, subqueries, and property paths.
  • Figure 5: Figure 5 reports mean paired F1 difference, defined as KGVoyager minus GRASP, by query complexity.Rows encode the number of distinct special SPARQL constructs, while columns encode triple-pattern counts.
  • Observed pattern: KGVoyager’s average improvement appears larger for queries with at least five triple patterns.The analysis pools the four benchmarks and uses paired F1 differences.

C Failure Analysis

KGVoyager’s complete failures are concentrated in concept resolution, especially property selection, while some errors arise from query construction and defective gold queries. Tool-use distributions also indicate that the system often solves questions with relatively few interactions.

  • Failure-analysis setup: 148 complete failures were identified across four models and benchmarks, with 50 randomly sampled for manual root-cause analysis.The analysis used triples scoring F1 of 0 across all three independent runs.
  • Error distribution: 48% of errors involved concept resolution, led by wrong property choice at 24%, followed by wrong entity grounding at 18% and wrong class grounding at 6%.These categories were derived from the manually inspected complete failures.
  • Error distribution: 22% of errors were query-construction failures, mostly involving missing constraints rather than incorrect join paths.
  • Evaluation caveat: 12% of cases reflected over-constrained gold queries that unfairly penalized valid answers, suggesting reported F1 slightly understates true performance.
  • Interaction efficiency: KGVoyager often required fewer tool invocations or explicit SPARQL executions per question across the evaluated benchmarks, models, and runs.Figure 6 pools executions across four benchmarks, four backbone models, and three runs.

E KGVoyager System Prompt and Tool Specification

KGVoyager uses a think-act-observe workflow in which search, exploration, and execution tools progressively ground natural-language questions in a knowledge graph. Its tools combine retrieval, data-based property analysis, and query execution with iterative feedback.

  • System prompt: KGVoyager generates SPARQL queries through a think-act-observe loop that reasons about a question, invokes tools, and interprets their results before acting again.
  • Search tools: Search tools identify candidate classes, properties, and entities from natural-language terms, with class retrieval combining semantic and keyword matching.Low-relevance class results are automatically pruned.
  • Exploration tools: Property exploration samples real triples to infer domain and range from graph usage rather than relying on ontology annotations.This targets settings where ontology annotations are missing or inaccurate.
  • Execution tools: The execution tool runs final queries and provides iterative feedback for testing intermediate drafts and diagnosing partial-query errors.Verification queries can use LIMIT to avoid fetching excessive data.
  • Execution tools: After verification, the system returns the final SPARQL query and answer based on the query results, ending the generation process.
Loading 2609.01780v1…