PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window

Blake G. Fitch

arXiv:2609.14652v1cs.DBcs.AI

TL;DR

LLMs need a graph’s domain model to generate queries, but schema concepts are often conveyed informally. NLKGQ places a complete OWL ontology and query instructions in context and generates SPARQL in one zero-shot call, using wrappers when native terms are opaque. It reports strong results across DBLP, SemOpenAlex, and neuroimaging, and proposes a revised DBLP-QuAD benchmark for more deterministic evaluation.

  • Problem

    Knowledge-graph query generation requires the model to know graph classes, relationships, and property types, while prior work often transfers this schema information through informal descriptions or examples.

    Source: Introduction · p. 1

  • Method

    NLKGQ generates SPARQL zero-shot in one LLM call with the complete domain OWL ontology and instructions, using wrapper ontologies and deterministic rewriting for opaque native vocabularies.

    Source: Introduction · p. 1, Formal Ontologies as Concept Transfer · p. 2, Wrapper Ontologies · p. 3

  • Results

    NLKGQ scores 98/100 on SemOpenAlex against the published baseline’s 86/100 on the identical test set, and reports results on DBLP and neuroimaging as well.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The paper concludes that controlled semantics can deliver high accuracy without fine-tuning, agentic exploration, or very large models.

    Source: Introduction · pp. 1–2

    Every call includes the complete domain ontology, and the paper does not address selection or reduction for vocabularies that outgrow the context window.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Large Language Model (LLM) applications often transfer domain concepts into the model's context informally, through prompt prose, schema dumps, and examples. We show that for database queries, data model concepts pass to LLMs more effectively through representations whose vocabulary terms carry declared, machine-readable semantics (controlled semantics). NLKGQ is a working system and reusable framework that does this for data modeled in a knowledge graph. A formal OWL ontology serves as the transfer mechanism, concentrating the meaning of the data into semantically precise tokens the model can use directly. In a single LLM call, NLKGQ places in the context a system prompt instructing on SPARQL, the complete domain OWL ontology, and a domain-specific prompt addition, together with the user's natural language query. The model then generates the SPARQL query directly, zero-shot. Where the native vocabulary of an existing database or federation of databases is opaque, a wrapper ontology substitutes clean terms and a runtime rewriter restores the native forms. Evaluating on DBLP-QuAD 2.0 showed that its scores depend on the graph snapshot, the endpoint used, and the wording of its machine-generated questions, so we propose DBLP-QuAD 3.1, which maintains the intent of 2.0 while making reference results deterministic, revising reference SPARQL where needed, and rewriting the natural language questions, with a frontier model, to state each reference query's intent clearly and completely. We evaluate on the DBLP-QuAD 2.0 benchmark (57.6% Match under deterministic re-scoring), DBLP-QuAD 3.1 (89.9% Match on 1,000 questions), SemOpenAlex (98% Match against a published baseline's 86% on the identical test set), and neuroimaging metadata (100%).

Introduction

The paper argues that LLMs query knowledge graphs more accurately when schema terms carry declared, machine-readable semantics. It extends NLKGQ with wrapper ontologies and revises DBLP-QuAD to support more reliable evaluation.

  • Controlled semantics means that schema tokens in context carry declared, machine-readable meaning, rather than requiring the model to infer domain concepts from informal descriptions.The paper tests this approach on public knowledge graphs with vocabularies the authors do not control.Source: Introduction · p. 1
  • NLKGQ generates SPARQL zero-shot in one LLM call using the complete domain OWL ontology, SPARQL instructions, a domain rider, and the user’s query.For opaque native vocabularies, a wrapper ontology supplies clean terms and a deterministic rewriter restores native forms before execution.Source: Introduction · p. 1
  • On the original neuroimaging domain, the purpose-built ontology reached 100% accuracy on 21 competency questions, versus 43% with auto-generated SQL and 57% with ontology annotations in column comments.Source: Introduction · p. 2
  • On SemOpenAlex, NLKGQ scores 98/100 against the published baseline’s 86/100; it also reaches 89.9% Match on DBLP and 100% on neuroimaging.Source: Introduction · p. 2
  • DBLP-QuAD 3.1 revises reference queries and clarifies question intent while making evaluation deterministic.The revision responds to systematic quality issues identified in DBLP-QuAD 2.0.Source: Introduction · p. 2

Related Work

Prior work shows that schema information in context matters for knowledge-graph query generation, but leaves open which formal schema properties best serve LLMs. Related systems use runtime exploration, fine-tuning, or retrieval, while benchmark quality and vocabulary representation remain central concerns.

  • Schema context substantially improves SPARQL generation in prior studies, but which formal schema properties matter and how to optimize them for LLMs remain largely unexamined.Source: Related Work · p. 2
  • Prior systems use runtime exploration, fine-tuning, or retrieval and revision, whereas NLKGQ relies on formal ontology vocabulary in a single zero-shot call.GRASP reports 51.0% F1 on 50 original DBLP-QuAD samples; FIRESPARQL reports 85% on SciQA but 0% zero-shot.Source: Related Work · p. 2
  • QuAD 2.0’s arbitrary 10-row reference slices penalize systems that return full answer sets, while SemOpenAlex provides a 100-question test set with reference queries.Source: Related Work · p. 2
  • Prior comparisons associate readable vocabulary and richer RDF/OWL semantics with stronger query generation, while wrapper ontologies map clean conceptual terms onto native schemas.The paper distinguishes its wrappers as designed for LLM consumption and applied through deterministic SPARQL-to-SPARQL rewriting.Source: Related Work · p. 2

Formal Ontologies as Concept Transfer

NLKGQ treats an ontology in the context window as a transfer mechanism for a graph’s domain vocabulary. The section explains which OWL semantics guide query generation, how wrappers adapt opaque vocabularies, and how ontology content and prompt instructions divide their roles.

Formal Ontologies as Concept Transfer

What OWL Provides That SQL Does Not

  • OWL provides explicit domain and range, class hierarchies, property typing, human-readable annotations, and semantic names that informal schema descriptions may lack.These features distinguish URI joins from literal filters and can support generalizing a query across subclasses.Source: What OWL Provides That SQL Does Not · p. 3

Prior Results on a Controlled Domain

  • On neuroimaging data, full OWL context yielded 100% SPARQL accuracy; removing comments reduced it to 81%, and bare graph structure to 5%.Three system-prompt variants tied once the vocabulary was right, while 35B MoE variants peaked at 57% against the dense 27B’s 100%.Source: Prior Results on a Controlled Domain · p. 3

Wrapper Ontologies

  • Wrapper ontologies replace problematic native terms with readable, unambiguous terms and deterministic mappings; dblpx covers DBLP and OpenCitations, while oax flattens intermediary relationships.The rewriter translates generated queries to native predicates and patterns before execution.Source: Wrapper Ontologies · p. 3

A Worked Example

  • In the worked DBLP example, the model queries dblpx terms and the rewriter restores native vocabulary before endpoint execution.The example asks for author signatures of publications in a specified venue and year, sorted by publication.Source: A Worked Example · p. 3

The Role of the Domain Rider

  • The ontology carries domain knowledge expressible as vocabulary; the domain rider is reserved for a small, stable set of query conventions.The experiments found the rider ineffective for incremental concept repair.Source: The Role of the Domain Rider · p. 3

Benchmark Critique: DBLP-QuAD 2.0

The 1,000-question QuAD 2.0 test set has systematic defects that make reported scores depend on evaluation mechanics and question/reference quality. The paper identifies unstable references, ambiguous questions and entities, defective queries, data dependencies, and reproducibility gaps.

Non-deterministic evaluation

  • 52.1–52.9% Match across five runs rose to a stable 57.6% with fixed reference results, while SPINACH F1 stayed near 17.5%.On QuAD 3.1, the same system scores 79.0% SPINACH F1; the authors attribute most of the 61.5-point F1 gap to nondeterministic references.Source: Benchmark Critique: DBLP-QuAD 2.0 · p. 4

LLM-generated questions

Ambiguous entity references

  • Questions often leave the intended URI unrecoverable when DBLP authors share names, limiting reliable reference reproduction with or without entity linking.Source: Benchmark Critique: DBLP-QuAD 2.0 · p. 4

Defective reference queries

Misdocumented data dependencies

  • 211 cases require OpenCitations data absent from the referenced plain DBLP dump, and citation questions depend on identifier coverage of only 71% of publications.Source: Benchmark Critique: DBLP-QuAD 2.0 · p. 4

Reproducibility

DBLP-QuAD 3.1

DBLP-QuAD 3.1 updates QuAD 2.0’s 1,000 test cases while preserving their provenance. It makes evaluation deterministic, repairs references, clarifies questions, and documents data and entity-reference choices.

DBLP-QuAD 3.1

  • 954 nondeterministic LIMIT clauses were removed, 305 under-constrained queries were scoped, and LIMIT remains only in 44 ranked queries with deterministic ordering.Reference results are unique for semantically equivalent queries, except for ties on the ordering key in ranked cases.Source: DBLP-QuAD 3.1 · p. 4
  • Reference queries were repaired for incorrect predicates, missing GROUP BY clauses, broken aggregations, and 122 VALUES-only echo cases.Source: DBLP-QuAD 3.1 · p. 4
  • Questions were rewritten for natural phrasing and unambiguous intent, and 547 cases were amended to restore omitted columns, ordering, or scope.Source: DBLP-QuAD 3.1 · p. 4
  • 266 questions use bracketed entity URIs to pin intended referents; systems with entity linking can ignore the brackets.The NLKGQ web interface also accepts the convention so users can pin an intended entity.Source: DBLP-QuAD 3.1 · pp. 4–5
  • The benchmark documents its combined DBLP+OpenCitations data requirement and 71% citation-coverage bound, while retaining 28 cases dependent on live public endpoints.Source: DBLP-QuAD 3.1 · p. 5
  • QuAD 3.1 is planned for release as a self-contained benchmark with cases, snapshot-recreation scripts, and a QLever instance.Source: DBLP-QuAD 3.1 · p. 5

Evaluation

The evaluation traces how reference repairs change DBLP scores, compares results across domains and models, and analyzes where query generation still fails. Results depend materially on benchmark construction: QuAD 3.1’s SPINACH F1 is 61.5 points above QuAD 2.0’s, while the tested system reaches 89.9% Match on DBLP-QuAD 3.1.

Evaluation

  • Evaluation uses single calls at temperature 0.0 without fine-tuning, few-shot examples, or entity linking, and repeated deterministic runs reproduce scores within 0.1 points.Match and Exact are strict outcomes; F1 measures award partial overlap and can penalize correct answers with extra rows or columns.Source: Evaluation · p. 5, Table 1 · p. 5

From QuAD 2.0 to QuAD 3.1

  • The 2.0-wrapper row raises Match and Exact but lowers both F1 measures; consistent 2.1 re-scoring gives 57.6% Match without improving either F1 measure.The 2.1 protocol keeps questions and reference queries unchanged and computes reference results once against the evaluation snapshot.Source: From QuAD 2.0 to QuAD 3.1 · p. 5, Table 2 · p. 6
  • SPINACH F1 rises from 17.5 on QuAD 2.0 to 79.0 on QuAD 3.1, a 61.5-point gap attributed mainly to benchmark defects.QuAD 3.1 makes references deterministic, repairs defective queries, and rewrites questions for clear intent; the repair stages were not separately ablated.Source: From QuAD 2.0 to QuAD 3.1 · p. 5, Table 2 · p. 6

Cross-Domain Generality

  • DBLP-QuAD 3.1 reaches 89.9% Match, while SemOpenAlex scores 98/100 against 86/100 on the identical test set; the smaller datasets are exact or near-exact.The DBLP comparison with GRASP is not same-benchmark: GRASP used original DBLP-QuAD, whereas this result uses QuAD 3.1.Source: Cross-Domain Generality · pp. 5–6, Table 3 · p. 6
  • Two SemOpenAlex misses reflect ambiguous questions whose natural reading is co-authorship, while the UNION references mean papers by either author.The duplicate question is retained, and disambiguated variants pass.Source: Cross-Domain Generality · p. 6

Model Comparison: The Concept Layer Outweighs Scale

Error Analysis

  • Among 101 non-matching cases, 24 involve citation queries and 22 require federated SERVICE calls outside the dblpx wrapper’s coverage.Other errors include aggregation structure or execution, ontology introspection, and heterogeneous cases; 30 cases fail to produce an executable query or encounter a query error.Source: Error Analysis · pp. 6–7

Discussion

The discussion argues that formal concept transfer matters more than prompt refinement, and that the principle may extend beyond SPARQL. Benchmark defects substantially affect measured performance, while the evaluation remains bounded by context size and other stated limitations.

Discussion

  • The authors argue that typed schemas in SQL, API, and code generation face the same natural-language-to-formal-construct mapping challenge as SPARQL.In neuroimaging SQL experiments, carrying ontology annotations into column comments lifted accuracy from 43% to 57%.Source: Discussion · p. 7
  • At temperature 0.0, rider additions can fix targeted queries while breaking unrelated ones, so mature riders approach zero-sum refinement.The authors interpret natural-language tokens as semantically wide and formal vocabulary tokens as narrow, reserving the rider for stable conventions.Source: Discussion · p. 7
  • QuAD 2.0 and 3.1 differ by 61.5 SPINACH F1 points, from 17.5% to 79.0%, demonstrating that benchmark defects drive measured performance.The authors place formal concept transfer above model architecture, model scale, and prompt engineering in their proposed hierarchy.Source: Discussion · p. 7
  • The wrapper-versus-native ablation uses one model, the ten-model sweep uses only the wrapper, proprietary models are untested, and the complete ontology must fit in context.The authors do not address selection or reduction for vocabularies that outgrow the context window; wrapper design is manual and entity resolution remains out of scope.Source: Discussion · p. 7

Conclusion

OWL ontologies transfer data-model concepts as semantically precise tokens, while wrapper ontologies and runtime rewriting support existing databases and federations. Across three domains and four model families, the paper reports high accuracy and says controlled semantics contributes more than model scaling or prompt revision.

Conclusion

  • The framework reports 89.9% Match on 1,000 DBLP questions, 98% on SemOpenAlex, and 100% on neuroimaging metadata.The conclusion says these results span three domains and four model families.Source: Conclusion · p. 7
  • OWL ontologies transfer schema concepts as semantically precise tokens that map directly to correct query patterns.The paper concludes that concepts pass to LLMs more effectively through formal mechanisms than informal descriptions.Source: Conclusion · p. 7
  • For suboptimal native vocabularies, wrapper ontologies substitute precise terms and a runtime rewriter preserves compatibility with existing databases and federations.Source: Conclusion · p. 7
  • The controlled-semantics layer contributes more to accuracy than model scaling or prompt revision; the system prompt tunes query generation, while the ontology and rider tune concepts.Source: Conclusion · p. 7

GenAI Usage Disclosure

The disclosure describes generative AI assistance in editing, formatting, coding, and data analysis, and distinguishes that assistance from the work's scientific content and system under study.

GenAI Usage Disclosure

Loading 2609.14652v1…