Source-linked AI summary
GraphRAGProv: GraphRAG Meets How-Provenance
Wadie El Amrani, Luis Galárraga
TL;DR
GraphRAG systems can use graphs to retrieve context while leaving the LLM to produce an answer, making the computation behind a returned value difficult to inspect. GraphRAGProv executes SPARQL over source-grounded RDF graphs and links provenance derivations to their text. In a 400-case demonstration, this trace supports inspection of extraction and query errors, including cases with correct final answers.
Problem
In many GraphRAG architectures, a graph determines what evidence reaches the LLM, but the LLM still produces the final answer, leaving unclear how a returned value was computed in settings where that computation may need verification.
Source: §1 · p. 1Method
GraphRAGProv executes SPARQL over RDF statements linked to source text and computes answer bindings with how-provenance identifying the contributing statements and their combinations.
Source: §1 · pp. 1–2Results
The demonstration examines 400 cases from 2WikiMultiHopQA and MuSiQue, with trace inspection revealing extraction, query, and entity-representation errors.
Source: §1 · p. 2, §4 · pp. 3–4, §5 · p. 4Takeaways & Limitations
Linking query results, provenance, and source text lets users inspect both the interpretations made during graph construction and the computation specified by the query.
Source: §1 · p. 2, §5 · p. 4The supervised evaluation does not establish generalization to unseen questions.
Source: §4 · p. 3
Abstract
from arXiv · showhide
GraphRAG systems often use knowledge graphs to retrieve relevant context for an LLM that produces the final answer. That makes answers more inscrutable and therefore less appealing for domains where transparency is paramount. We present GraphRAGProv, an architecture that executes SPARQL queries over source-grounded RDF graphs to compute answers and their how-provenance polynomials. These polynomials record which statement occurrences were used and how they were combined. Because each occurrence is linked to its source text, users can follow these links to check whether the extracted statements reflect the text and whether the query combines them to answer the question. Querying thereby becomes a way to inspect and evaluate the text-to-graph transformation. Our interactive demo examines 400 cases from two multi-hop QA benchmarks, revealing extraction and query errors even when the final answer is correct.
1 Introduction
GraphRAGProv answers questions by querying an RDF graph built from text, rather than asking an LLM to produce an answer from retrieved context. It pairs answer bindings with provenance and source links so users can inspect how statements and queries contribute to answers.
1 Introduction
- GraphRAGProv uses SPARQL query results as answers and how-provenance to record the contributing statements and how they were combined.The database computes answers and derivations without an LLM interpreting retrieved context.Source: §1 · pp. 1–2
- Each extracted statement occurrence links to its source passage and evidence spans, letting users check the text behind provenance factors.The extractor supplies quotations for statement components, and a deterministic localizer resolves them to exact character spans.Source: §1 · p. 2
- Query execution, how-provenance, and textual grounding let users inspect whether extracted statements reflect the source and whether the query combines them as intended.The chain can expose extraction errors, inconsistent entity identifiers, or SPARQL that fails to capture the question.Source: §1 · p. 2
- The demonstration examines trace availability and diagnostic value on 400 cases from 2WikiMultiHopQA and MuSiQue.Users can move through the question, SPARQL, answer binding, provenance polynomial, RDF occurrence, and source spans, and modify the question or query.Source: §1 · p. 2
2 GraphRAGProv
GraphRAGProv turns text into source-grounded RDF statement occurrences, then uses generated SPARQL queries and provenance-aware execution to return answers with derivations.
2 GraphRAGProv
- An LLM extracts RDF statements and supporting quotations, which a deterministic localizer resolves to exact source spans.Quotations that cannot be located trigger a correction request; unresolved quotations cannot be accepted as exact evidence.Source: §2 · p. 2
- Each source-grounded statement occurrence, or symbolic chunk, gets an identifier linking its RDF triple to its source paragraph and evidence spans.Identical triples from different passages retain separate occurrence identifiers and textual evidence, materialized using RDF-star reification.Source: §2 · p. 2
- An LLM generates SPARQL from the question and graph; GraphDB returns answer bindings, while NPCS rewriting computes their how-provenance.Multiplication in a provenance polynomial represents joint use of statement occurrences, and addition represents alternative derivations.Source: §2 · p. 2
3 Demonstration
The workbench lets users explore benchmark cases and inspect how questions, queries, answers, provenance, and source text relate.
3 Demonstration
- Users can inspect SPARQL queries, answer bindings, and provenance polynomials, then follow a provenance factor to its statement and textual evidence.Selecting a graph statement highlights its supporting text, and selecting text highlights the associated statement.Source: §3 · p. 3, Figure 3 · p. 3
- Users can edit natural-language questions or SPARQL queries to observe changes in answers and derivations.The workbench contains recorded cases from 2WikiMultiHopQA and MuSiQue; its source code is available under the MIT license.Source: §3 · pp. 2–3
4 Evaluation
The evaluation examines answer-to-source trace availability and cases showing how inspecting the graph and query can reveal different errors.
4 Evaluation
- The evaluation uses 200 compositional 2WikiMultiHopQA questions and 200 answerable MuSiQue questions, with graphs built from designated supporting passages and no document retrieval.Graph construction and SPARQL generation use DeepSeek-V4-Flash with access to the question, expected answer, and reasoning evidence; the supervised setting does not establish generalization to unseen questions.Source: §4 · p. 3
- Complete traces were available for 193 of 200 2Wiki cases and 118 of 200 MuSiQue cases.All 311 decoded traces passed occurrence and source-span checks, and their answer bindings matched ordinary query execution; 16 MuSiQue cases returned an answer without decoded provenance.Source: §4 · p. 3, Table 1 · p. 4
- In the Aldobrandino case, inconsistent IRIs for one person split the father relation and birth date, causing the required join to fail despite both facts being in the graph.Inspecting the graph and query identifies this failure, which has no result provenance to follow.Source: §4 · p. 4
5 Conclusion
GraphRAGProv makes the path from a question through its query and provenance to source-grounded statements inspectable. This lets users examine query and graph interpretations and locate possible failures.
5 Conclusion
- SPARQL specifies the computation, while how-provenance identifies contributing statement occurrences linked to their textual origins.Together, these layers let users inspect the interpretations made during graph construction.Source: §5 · p. 4
- A correct answer may rely on unsupported interpretations, while supported statements may participate in an incorrect query.Missing answers may also result from inconsistent entity representations rather than absent knowledge.Source: §5 · p. 4
- Provenance exposes the chain so users can locate and diagnose failures.Source: §5 · p. 4