PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing

Remo Grillo, Lukas Klic, Giovanni Colavizza

arXiv:2610.08095v1cs.AIcs.CL

TL;DR

LLMs can produce plausible SPARQL that misrepresents unfamiliar RDF graphs, where populated data may diverge from declared ontologies. QRAKEN distills empirical graph patterns into TTQL and validates generated queries deterministically; on CK25 it outperformed the strongest matched recomputed participant on strict F1. The authors present this as an initial signal, with evidence limited to one relatively small benchmark.

  • Problem

    On unfamiliar RDF graphs, LLMs can generate valid-looking queries that misrepresent populated data, a challenge distinct from general code generation and Text-to-SQL.

    Source: §1 · p. 2

  • Method

    QRAKEN prepares empirical graph context offline as TTQL and combines LLM query generation with deterministic syntax, vocabulary, and data-model checks.

    Source: §1 · p. 2, §4.2 · p. 5, §4.4 · p. 7

  • Results

    30% with GPT-4.1 mini and 32% with GPT-5.4: QRAKEN’s relative strict F1 gains over the best matched recomputed baseline, ARUQULA in the same base model family.

    Source: §5.3 · p. 8

  • Takeaways & Limitations

    The CK25 results indicate that graph distillation plus deterministic validation can be more decisive than heavier online agent orchestration in domain-specific KGQA.

    Source: §6 · p. 12

    All quantitative claims rest on CK25’s 50 questions; validation on a larger, more heterogeneous graph remains the most important missing experiment.

    Source: §5.8 · p. 11

Abstract

from arXiv · show

Natural-language access to RDF knowledge graphs is a core Semantic Web ambition. Large language models (LLMs) have advanced Text-to-SPARQL, yet on unfamiliar graphs they often generate valid queries that misrepresent the populated data model. QRAKEN is a training-free, ontology-agnostic neurosymbolic pipeline grounding generation in empirical graph evidence rather than schema expectations. An offline distiller produces TTQL, a compact description of populated multi-hop patterns, conditional frequencies and path-conditioned literal examples, plus a class-property co-occurrence matrix. Online, TTQL guides the LLM, while deterministic syntax, vocabulary and data-model checks provide diagnostics for iterative refinement. On CK25 (First International Text2SPARQL Challenge), under matched-condition recomputation on a QLever snapshot, QRAKEN achieves strict F1 of 0.643 $\pm$ 0.026 with GPT-4.1 mini and 0.652 $\pm$ 0.012 with GPT-5.4: relative gains of 30% and 32% over the strongest recomputed participant, outperforming systems using the same base model family. Ablations identify TTQL patterns as the dominant driver (+0.31 strict F1 over a shape-only baseline); the refinement loop provides a cheap safety net, rejecting triple patterns unsupported by the co-occurrence matrix. Compared with auto-derived SHACL, TTQL yields 64% higher strict F1, supporting the value of empirical patterns beyond schema exposure. With two local 35B 4-bit open-weight models at zero marginal cost, the same pipeline matches the strongest recomputed participant, and TTQL advantages over shape-only and SHACL baselines persist. Results on a single, relatively small benchmark provide an initial empirical signal; monolithic TTQL injection on very open cross-domain graphs remains the main limitation.

1. Introduction

LLMs make Text-to-SPARQL promising, but unfamiliar RDF graphs and mismatches between declared ontologies and populated data make graph-grounded correctness a persistent challenge. QRAKEN addresses this with training-free graph distillation and deterministic validation, evaluated on CK25.

1. Introduction

  • LLM-generated queries can be syntactically plausible yet misrepresent populated graph usage through wrong predicates, directions, shortcuts, or projections.The introduction identifies graph-grounded correctness, rather than language understanding, as the dominant difficulty.Source: §1 · p. 2
  • Bare prompting is under-grounded on unfamiliar graphs, while fine-tuning needs graph-specific question-query pairs and does not adapt to schema drift.Private or evolving graphs may lack the required training pairs.Source: §1 · p. 2
  • QRAKEN distills a graph into a compact LLM-readable representation and combines in-context generation with deterministic checks, without requiring a new-graph question-query training corpus.The graph-side TTQL context is prepared offline, then used with a single supervised prompt and deterministic validation.Source: §1 · p. 2
  • On CK25, matched-condition recomputation found QRAKEN outperformed the challenge participant systems on strict F1; the authors frame this as a single-benchmark signal.The paper also distinguishes syntactic, lexical, and data-model correctness, identifying the last as where most LLM failures occur.Source: §1 · p. 2
  • The authors argue that graph grounding can matter more than model scale in domain-specific KGQA, while calling for validation on additional benchmarks.Their claim contrasts faithful representation of populated graph usage with larger models or elaborate agent choreography.Source: §1 · p. 2

2. Related Work

Related work spans online graph exploration, stored traces or fine-tuning, and prompt-injected structural context. The paper positions QRAKEN in an under-explored design space: training-free, cheap at query time, and grounded in empirical graph usage.

2. Related Work

  • The Text2SPARQL Challenge standardized the endpoint protocol but left model, retrieval, indexing, and repair choices open, while cost reporting remains limited.The paper groups prior systems into three design families and treats cost as an additional evaluation dimension.Source: §2 · p. 2, §2.4 · p. 3

2.1. Online agentic graph exploration

  • Online exploration systems re-derive graph understanding per question; ARUQULA uses about one minute per question, while GvD uses a compact three-action design.ARUQULA is the closest same-model comparator; GvD reports that sub-10B models are insufficient for reliable agent loops.Source: §2.1 · pp. 2–3

2.2. Memory- and training-based approaches

  • Memory and training approaches store successful traces or adapt model weights, but annotated-data requirements make fine-tuning less attractive for private or evolving graphs.The paper places QRAKEN alongside context-injection approaches, replacing per-question retrieval with one-off graph-level distillation.Source: §2.2 · p. 3

2.3. Shape- and metadata-informed prompting

  • Shape- and metadata-informed prompting injects graph context, but QRAKEN instead uses empirical multi-hop usage patterns, literal examples, and co-occurrence statistics from populated data.The comparison includes schema-derived ontology path patterns and retrieved metadata and example queries [17].Source: §2.3 · p. 3

2.4. Cost and the open region in the design space

  • The design space remains under-explored for systems that are training-free, cheap at query time, and grounded in empirical graph usage.The paper contrasts per-question agentic costs, preparation and data needs for memory or training, and retrieval overhead for shape or metadata prompting.Source: §2.4 · pp. 3–4

3. What Does It Mean for a Query to Be Correct?

The paper treats query correctness as layered: syntax and vocabulary checks are insufficient when a query misrepresents how the populated graph connects entities. Such data-model errors can execute yet return nothing.

3. What Does It Mean for a Query to Be Correct?

  • Syntactic correctness means conforming to SPARQL grammar, but only establishes that a query is executable in principle.Lexical correctness concerns whether classes, properties, and namespaces exist in the populated graph; data-model correctness concerns their semantic use against actual data.Source: §3 · p. 4
  • A real vocabulary term can still connect classes through an impossible direction or shortcut, yielding an executing query with no results.The CK25 example’s ProductCategory-to-ProductManager shortcut fails because the populated path runs through Hardware; the data-model check rejects its zero-support edge.Source: §3 · p. 4

4. QRAKEN: Method and Architecture

QRAKEN separates graph distillation from question answering: TTQL provides an LLM with empirical graph patterns, while a separate co-occurrence matrix supports deterministic checks. Candidate queries are refined using diagnostics before execution.

4.1. Design overview

  • The offline phase produces a compact TTQL description of populated classes, multi-hop paths, conditional frequencies, and example literals, plus a separate machine-side co-occurrence matrix.TTQL is reused across questions; the matrix is not injected into the prompt.Source: §4.1 · p. 4, Figure 1 · p. 5

4.2. Graph distillation as empirical graph summarization

  • The distiller enumerates class-level co-occurrences exhaustively, ranks classes by structural signals, samples instances while preserving rare properties, and harvests patterns up to a chosen depth.It completes uncovered class-level edges before serializing patterns as TTQL and retaining the matrix for later checks.Source: §4.2 · p. 5
  • TTQL frequencies are conditional on the root class or already-followed parent path, exposing populated multi-hop combinations useful for join planning.Unlike prescriptive shape languages, TTQL describes observed patterns, including path-conditioned literals that reveal graph-specific lexical conventions.Source: §4.2 · p. 6, §4.3 · p. 6

4.3. TTQL: a descriptive language for populated graph patterns

  • TTQL describes observed graph patterns rather than serializing instance data, and records multi-hop paths and literals conditioned on root class.It complements rather than replaces SHACL or ShEx, which specify allowed structures and conformity.Source: §4.3 · p. 6

4.4. NTQ and correctness-guided self-healing

  • The NTQ module prompts an LLM with the question and relevant TTQL content, then checks candidate queries for syntax, vocabulary, and class-property-target constraints.Exhaustive co-occurrence evidence makes a zero matrix entry a certificate that the required class-level edge does not exist.Source: §4.4 · p. 7
  • Failed checks produce targeted diagnostics for a refinement prompt, keeping repair grounded rather than asking the model to introspect freely.The stated design aims to avoid context drift from unfiltered execution traces.Source: §4.4 · p. 7

5. Evaluation

The evaluation uses CK25, a small but structurally dense benchmark unfamiliar to the models, to test QRAKEN on recurring multi-hop query patterns. Published participant answers are recomputed on a shared QLever snapshot and assessed with strict and lenient F1.

5.1. Text2SPARQL and CK25

  • CK25 comprises 50 questions over a curated graph with 13 classes, 30 properties, about 26.5K statements, and about 2.6K resources.Its bounded vocabulary and recurring multi-hop patterns make it a test of query composition rather than graph volume.Source: §5.1 · p. 7

5.2. Experimental protocol

  • The headline run distills TTQL from a local QLever endpoint with per-class sample size N=100 and maximum hop depth H=5, producing a 122 KB file from a 970 KB graph.The protocol uses up to five refinement iterations and temperature 0.1; results use six replicates for gpt-4.1-mini and three for other models.Source: §5.2 · pp. 7–8
  • Strict set-based F1 is the main metric; lenient F1 canonicalizes URI-label equivalences for entity answers and is diagnostic, not directly comparable to official challenge results.For this dataset, the official combined F1+NDCG collapses to the same set-based F1 in practice.Source: §5.2 · p. 8
  • Published participant answer files are re-executed against the same local QLever snapshot and rescored, preserving relative team ordering rather than re-ranking the official leaderboard.The recomputation also audits and deduplicates answer files and applies identical timeouts.Source: §5.2 · p. 8

5.3. Results

On CK25, QRAKEN exceeds the strongest matched recomputed participant baseline on strict F1 with both hosted models. Ablations attribute the largest performance contribution to TTQL pattern content, with a smaller, bounded role for self-healing.

5.3. Results

  • QRAKEN beats the best matched recomputed baseline, ARUQULA using the same base model family, by 30% on strict F1 with gpt-4.1-mini and 32% with gpt-5.4.The figure compares strict and lenient F1 for the recomputed baselines and QRAKEN configurations; open-weight configurations are shown separately.Source: §5.3 · p. 8, Figure 2 · p. 9
  • The closest comparator, ARUQULA, uses online graph exploration, whereas QRAKEN combines offline graph distillation with online deterministic validation.On this bounded, curated graph, TTQL describes empirical graph usage rather than only schema constraints.Source: §5.3 · p. 8

5.4. Ablation: what drives the result?

  • Replacing full TTQL with the co-occurrence matrix costs +0.309 strict F1 on gpt-4.1-mini and +0.153 on gpt-5.4, making TTQL patterns the dominant performance driver.The matrix-only prompt is about 11 times more compact; the passage attributes the gain to descriptive empirical pattern information, not compactness.Source: §5.4 · p. 9, Table 1 · p. 9
  • Auto-derived SHACL reaches 0.376±0.018, while TTQL patterns outperform it by −0.241 absolute (+64% relative in A2’s favor) at identical iteration budget.The comparison supports pattern-level empirical content over shape-level graph context on CK25.Source: Table 1 · p. 9, §5.4 · p. 10
  • Self-healing rejects only queries with a zero-support triple pattern, and its contribution ranges from −0.009 to +0.067 across four models at below $0.01 per question.The loop intervenes when a first-shot query is provably empty against the target graph; it is not a general-purpose quality optimizer.Source: §5.4 · p. 10

5.5. Open-weight models

QRAKEN’s full ablation grid was run with two local, quantized 35B open-weight models at zero marginal cost. TTQL’s advantages persist, and Qwen3.6 reaches the strongest recomputed participant’s strict F1 level.

5.5. Open-weight models

  • The full ablation grid uses Ornith-1.5 and Qwen3.6, two open-weight 35B-A3B models quantized to 4 bits and served locally, with 30 runs across 1,500 questions at zero marginal cost.The pipeline remains training-free and unchanged; only the NTQ model differs.Source: §5.5 · p. 10
  • Qwen3.6 reaches 0.491 ± 0.036 strict F1, on par with ARUQULA’s 0.493; TTQL also beats the SHACL baseline by +0.165 and +0.221 on the two open-weight models.TTQL’s gains over the co-occurrence-only condition are +0.136 and +0.115 strict F1, respectively.Source: §5.5 · p. 10
  • The remaining open-weight performance gap is attributed to SPARQL validity and answer-shape errors, including filtering on absent literals and projecting entity URIs where gold answers are literals.The passage characterizes these as pipeline-side losses rather than a model-capability ceiling.Source: §5.5 · p. 10

5.6. Cost analysis

The cost analysis accounts for the online per-query path separately from one-off offline distillation. QRAKEN’s headline configuration is low-cost relative to the reported comparators, while using fewer calls and less time than ARUQULA on the same questions.

5.6. Cost analysis

  • The accounting covers online per-query calls, tokens, cost, and median latency; offline TTQL distillation takes about 15 minutes for the graph and is amortized across questions.The validator consults the pre-distilled matrix machine-side, while the runner executes the validated query once against the endpoint.Source: §5.6 · p. 10, Table 2 · p. 11
  • QRAKEN’s headline A2 costs $1.18/100q, comparable to SPARQL-LLM at about $1/100q and 2.6× cheaper than mKGQAgent/GPT-4o at $3.06/100q, while reaching 0.617 versus 0.333 strict F1.Full TTQL uses five times more tokens yet stays under $0.02 per question and 5 seconds latency; the distillation step is a one-off offline cost.Source: §5.6 · p. 10
  • Against ARUQULA on the corporate dataset, QRAKEN’s A1 uses 1.43 LLM calls and one endpoint call at 3.5 seconds median latency, versus ARUQULA’s 9.92 agent steps and 59.6 seconds per question.QRAKEN reports 0.643 versus 0.493 strict F1; ARUQULA reports neither tokens nor monetary cost, so the comparison is limited to reported dimensions.Source: §5.6 · pp. 10–11

5.7. Error analysis

The error analysis identifies recurring CK25 failure modes, while the validity of the findings is bounded by one benchmark and one unvaried distillation setting.

5.7. Error analysis

  • At the GPT-4.1 mini headline, 16 zero-score answers cluster around multi-constraint composition, literal canonicalisation, and aggregation-scope errors.Path shortening is rarer and usually caught by the data-model validator.Source: §5.7 · p. 11

5.8. Threats to validity

  • All quantitative claims rely on CK25's 50 questions; four open-weight configurations broaden model coverage, not graph coverage.The authors identify validation on a larger, more heterogeneous graph as the most important missing experiment.Source: §5.8 · p. 11
  • The shape baseline strips TTQL to its co-occurrence matrix, while a full replica of a shape-prompting system would be a stronger comparator.Hosted-model API snapshot dates are recorded; determinism applies to generated queries, not model internals.Source: §5.8 · p. 11
  • Only one distillation setting (N=100, H=5) is evaluated, so the value of changing pattern-block and literal-example richness remains unknown.The deterministic checks are insensitive to these parameters because the matrix is exhaustively enumerated and TTQL covers every class-level edge.Source: §5.8 · pp. 11–12

6. Conclusion, Applicability, and Future Work

QRAKEN addresses graph-specific query correctness through empirical graph distillation and deterministic validation. On the single CK25 benchmark, it leads recomputed participant systems using the same base-model family, motivating validation on additional benchmarks.

6. Conclusion, Applicability, and Future Work

  • On CK25's 50 questions, QRAKEN attains the highest strict F1 among recomputed participant systems at the same base-model family.The authors state that this finding warrants validation on additional benchmarks.Source: §6 · p. 12
  • QRAKEN is strongest on curated, semantically rich graphs with bounded vocabularies and recurring multi-hop patterns; monolithic TTQL injection can saturate on graphs with thousands of heterogeneous classes.The stated scaling ceiling is context budget rather than distillation.Source: §6 · p. 12
  • Retrieval over TTQL and a dedicated multi-constraint reasoning layer are identified as next steps; broader evaluation on private and scholarly graphs is ongoing.Source: §6 · p. 12
  • The reproducibility package releases the CK25 TTQL output and raw run artifacts, while the evolving offline Graph Distiller is withheld.The authors state that the output and Sect. 4.2 support re-implementation and that empirical claims remain independently verifiable.Source: §6 · p. 12

Declaration on Generative AI

The authors disclose using Claude for language and code assistance, state that they reviewed the work and take responsibility, and clarify that LLMs are also studied as pipeline components.

Declaration on Generative AI

Loading 2610.08095v1…