Source-linked AI summary
The Answer Path and the Grounding Instruction in LLM Question Answering over Knowledge Graphs
Arquimedes Canedo
TL;DR
GraphRAG pipelines vary which triples they retrieve, how they serialize and order them, and how they instruct models to use them. This paper tests those choices across models and benchmarks, finding that answer-path presence and grounding instructions matter while several intermediate choices are flat. However, depth is confounded with benchmark and format-sensitive scoring prevents measuring the grounding cost when correct context is present.
Problem
The central question is which GraphRAG choices materially affect knowledge-graph question answering: retrieved content, serialization, ordering, or grounding instruction.
Method
Across 16 experiments and 30,841 trials, the study varies these four choices across six models and two benchmarks while using controlled comparisons.
Results
The answer-path choice is consequential: replacing surrounding triples changes answer F1 by +0.003 at constant volume, whereas removing the chain causes a large accuracy loss.
Takeaways & Limitations
The results support prioritizing answer-path recall over surrounding-triple precision, while treating the grounding-instruction contrast with correct context as unresolved.
Takeaways & Limitations
Depth is perfectly aliased with benchmark, and the format-sensitive scorer was not validated across response formats, limiting interpretation of depth and prompt contrasts.
Abstract
from arXiv · showhide
A graph retrieval-augmented generation pipeline chooses which triples to put in the prompt, a syntax to write them in, an order to write them in, and a sentence telling the model what to do with them. We vary all four over six large language models and two knowledge-graph question answering benchmarks. Two of the four choices move the answer and the other two are flat. The first is whether the answer path, the triples needed to reach the answer, is in the prompt at all. Holding the number of triples fixed and replacing every triple that is not on the chain with material from an unrelated entity changes answer accuracy by +0.003 F1, while removing the chain costs most of what the graph was worth. Retrieval budget belongs on recall, and precision in the range we can test buys nothing. There is no retriever here: subgraphs come from gold SPARQL, so precision describes the context we build, not a system setting. The second is the grounding instruction. With no facts in the prompt, telling a model to answer using only the provided facts drops F1 from 0.299 to 0.035, a factor of 8.63. That figure describes an evaluation with an empty context arm rather than a working pipeline, and an experiment that applies the instruction to its context arm but not to its no-context baseline manufactures a spurious finding that graph context hurts at depth. We found one in our own results and retract it. Syntax, triple order and subgraph size produce no effect we can measure at multi-hop depth. The comparison that would price the grounding instruction against correct context is not measurable with a format-sensitive scorer, because the instruction determines the response format; we report it as an open contrast rather than a number.
I. INTRODUCTION
The study evaluates four GraphRAG design choices across models and benchmarks, finding that answer-path presence and grounding instructions matter, while syntax, ordering, and context composition generally do not. It also identifies evaluation designs that confound graph context with prompt regime or response format.
- Retrieval choices: The answer path contains the gold-SPARQL chain triples needed to reach the answer, whereas distractors are genuine neighboring triples outside that chain.Answer accuracy is measured by token-level F1 against gold labels.
- Grounding instruction: With no facts available, the strict grounding prompt scores F1 = 0.035 versus 0.299 for the permissive prompt, an 8.63-fold suppression.This is an empty-context evaluation and not a functioning retrieval pipeline.
- Other design choices: Serialization syntax, triple order, and subgraph size produce no measurable effects once reasoning is required, although syntax can matter for one-hop parsing.The observed one-hop difference tracks whether the serializer provides a human-readable predicate name.
- Measurement limits: The comparison that would measure grounding-instruction cost with correct context is unmeasurable because prompt regime determines response format and the scorer is format-sensitive.The paper reports this contrast as open rather than assigning it a numerical effect.
- Study scope: Across 16 experiments and 30,841 trials, the paper varies which triples appear, their serialization, their order, and the grounding instruction.The design spans six models from three vendors and treats these choices as distinct pipeline variables.
- Retrieval choices: With 50 triples fixed, reducing context precision from 1.00 to 0.51 changes answer F1 by +0.003, while removing the answer chain lowers F1 to 0.231 or 0.005.The first value is under the permissive prompt; the second is under the strict prompt.
II. RELATED WORK
The paper situates its controlled precision and serialization experiments among work on irrelevant triples, graph encodings, adversarial context, and reliance on retrieved text. Its main related-work contribution is methodological: response-format-sensitive scoring prevents a clean estimate of grounding-instruction effects.
- Retrieval and graph encoding: The precision sweep controls triple count while changing only provenance, separating relevance from context volume in response to prior findings on irrelevant triples.This design tests whether unrelated material itself reduces answer accuracy.
- Retrieval and graph encoding: Prior work reports accuracy changes from graph encoding choices and format-dependent single-hop comprehension, whereas this study tests RDF serialization in a broader question-answering setting.The cited comparisons cover synthetic structural tasks, tabular formats, and RDF comprehension.
- Retrieval and graph encoding: The study extends graph-encoding and format-comparison work to real knowledge-graph question answering with seven RDF-native formats and depth-stratified analysis.Naming and syntax vary together, so their individual effects remain aliased.
- Reliance on retrieved text: The study’s suppression result is not directly comparable to prior override-rate measures because it reports F1 change, with a measured no-context-relative effect of −0.068 F1.Two of six models show no suppression, so the evidence does not establish suppression as intrinsic to structured context.
- Reliance on retrieved text: The paper’s methodological contribution is negative: endpoint prompts produce different response formats, preventing an end-task scorer from estimating the effect of moving between them.This measurement failure is documented as a limitation rather than resolved by the related-work comparison.
- Reliance on retrieved text: A clinical grounding result may predict a small context benefit here, but the present comparison also changes the prompt by forbidding parametric knowledge.The paper therefore attributes the larger observed benefit partly to prompt differences rather than context alone.
E. Multi-Hop Knowledge Graph Question Answering
The study evaluates GraphRAG over two benchmarks using oracle subgraphs and six models, while warning that benchmark and reasoning depth are confounded. Its related work situates iterative graph exploration and serialization-format effects, but the paper’s depth comparisons cannot isolate depth itself.
- Related work: Prior work shows that iterative graph exploration and bounded path history can improve multi-hop KGQA, while the paper reports format effects analogous to earlier code-generation findings only for parsing.The reported format spread is 2.6× larger for mid-tier than frontier models, and disappears at multi-hop depths.
- Dataset and pipeline: The pipeline uses 125 Wikidata-grounded questions from LC-QuAD 2.0 and QALD, with gold SPARQL queries identifying reasoning chains.LC-QuAD supplies one- and two-hop questions, while QALD supplies three- and four-hop questions.
- Scope qualification: Depth is perfectly aliased with benchmark, so depth-stratified contrasts cannot distinguish reasoning-depth effects from dataset differences.The paper retains depth-stratified reporting but does not support recommending a depth threshold.
- Experimental design: The experiments send oracle subgraphs through seven formats and five triple orders to six LLMs, scoring answer F1 and evidence faithfulness.Figure 1 summarizes this experimental pipeline.
- Corpus differences: LC-QuAD and QALD differ in answer-set size, answer coverage, and graph effects, with QALD scoring 0.427 without a graph versus 0.270 with correct context.These properties covary with depth and benchmark under the study design.
- Scope qualification: The paper cautions that gold-answer coverage declines with depth and hop count measures predicate count rather than a verified sequential chain.These properties further qualify depth-stratified analyses.
C. Serialization Formats and Triple Ordering
The study varies graph serialization, triple ordering, prompt wording, and evaluation parsing as distinct design choices. It describes seven representations and five orders, while documenting that prompt format affects parsing and scorer behavior.
- C. Serialization Formats and Triple Ordering: The study tests seven graph representations, ranging from human-readable prose to machine-oriented formats including Turtle, N-Triples, Cypher, JSON-LD, and RDF/XML.The formats reflect practitioner-available options rather than a systematic sampling of serialization design space.
- C. Serialization Formats and Triple Ordering: Five triple orders are tested: entity-centric, path-centric, breadth-first, seeded random, and alphabetical lexicographic.Some serializers preserve supplied order, while Turtle, JSON-LD, and RDF/XML regroup triples by subject.
- D. Prompt Templates: The strict prompt requires answers using only supplied facts and an evidence block in a specific format.Its template includes an Answer field and cited Evidence lines.
- D. Prompt Templates: The permissive prompt allows the model to combine graph information with its own knowledge.The two prompt regimes differ in both grounding restriction and response-format requirements.
- E. Two-Axis Evaluation: Responses receive separate answer-quality and evidence-faithfulness scores, allowing correct answers and valid citations to vary independently.Faithfulness measures the fraction of cited triples matching triples in the supplied subgraph.
- E. Two-Axis Evaluation: Answer F1 uses token-level matching with normalization and alias resolution, while multi-answer questions use optimal bipartite matching.Answer order therefore does not affect the score.
- E. Two-Axis Evaluation: Prompt wording determines response format, which selects the parsing path and creates a measurement problem for comparing strict and permissive regimes.The primary parser expects an “Answer:” line, while the fallback reads free-form responses.
IV. RETRIEVAL PRECISION DOES NOT MATTER
At fixed context volume, replacing distractors with unrelated triples barely changes answer accuracy when the answer chain remains present. The experiment instead shows a small decline in evidence faithfulness and supports separating chain recall from surrounding-triple precision.
- Experimental setup: Retrieval quality has two distinct failure modes: omitting the answer chain or returning it with additional irrelevant material.The experiment isolates these by keeping the chain present while varying surrounding triples.
- Methodological qualification: The precision sweep’s cross-run agreement is reassurance rather than validation because it compares unpaired means from four versus six models.The sweep’s internal contrast remains unaffected because both arms use the same runner.
- Result: +0.003 answer F1 results from replacing every distractor slot with unrelated triples at fixed 50-triple volume.The 95% CI is [−0.019, +0.031], with permutation p = 0.812 across 125 questions.
- Restricted analyses: +0.004 answer F1 is observed among 92 fully covered questions, confirming the pooled null is not caused by missing gold answers alone.The fully covered subset scores 0.579 under full context precision.
- Restricted analyses: +0.015 answer F1 is observed on 48 questions with zero unaided performance, where models had to use supplied information.This restricted analysis also shows no measurable precision effect.
- Result: 0.914 to 0.888 is the decline in evidence faithfulness as context precision decreases, despite stable answer accuracy.Models cite irrelevant triples somewhat more often while continuing to derive correct answers from retained chain triples.
- Result: Table IV holds triple count constant while varying imported-neighbourhood fraction, showing flat answer accuracy and slightly declining evidence faithfulness.The table summarizes the precision experiment’s central pattern.
C. Where the null stops
Matched prompts show that graph context remains beneficial through three hops and is non-negative but inconclusive at four hops; the earlier claim that context hurt at depth was retracted as an artifact of mismatched prompts. The grounding instruction itself strongly suppresses no-context performance, but its cost with correct context remains unmeasurable.
- Grounding instruction: F1 falls from 0.299 to 0.035 when the strict grounding instruction is applied with no facts, an 8.63-fold suppression.This is an empty-context evaluation arm, not a measurement of instruction cost when correct evidence is supplied.
- Open contrast: The unresolved comparison is the strict-versus-permissive instruction cost when context is correct, because prompt regime also changes response format and scorer behavior.The current apparatus therefore reports this contrast as unmeasurable rather than assigning it a number.
- The retracted artifact: The original depth result compared strict context with permissive no-context, confounding graph context with permission to use parametric knowledge.That mismatch produced apparent harm at three and four hops and does not establish worse reasoning over the graph.
- Matched context effects: At four hops, matched strict prompts show a non-negative but inconclusive context effect, replacing the earlier claim that graph context hurt.The benefit declines from +0.809 F1 at one hop to +0.147 at four hops, with p = 0.109 at four hops.
C. Why the design is easy to build
The design is easy to build because removing facts often also removes a grounding instruction, creating coherent but unmatched arms. Once prompt regime and context are separated, wrong-context effects and suppression cannot be cleanly isolated with the available comparisons.
- Why the artifact arises: Removing facts while also removing the grounding instruction varies prompt regime and context presence simultaneously.The resulting arms are individually coherent but do not isolate which change produced the performance difference.
- Unmatched comparisons: The four-hop permissive comparison is not dispositive because its baseline uses a third prompt and only 12 questions.It reports −0.120 F1 at four hops, but the arms remain unmatched.
- Wrong context: Wrong-context F1 is 0.005 under the strict prompt versus 0.579 for correct context, while the permissive comparison is 0.231 versus 0.627.These contrasts combine loss of useful facts with any suppression of parametric knowledge.
- Unmeasurable suppression: The suppression-isolating contrast is unavailable because wrong-context and no-context arms use different prompts, yielding −0.068 F1 that may be entirely prompt-driven.The scorer fallback does not change the sign or significance, addressing parsing rather than the prompt mismatch.
B. What this does and does not say about a poisoned graph
Wrong triples substantially reduce performance relative to correct context, but the experiment does not measure adversarial poisoning or cleanly separate missing evidence from added irrelevant evidence. At multi-hop depth, serialization and format effects largely disappear, while several apparent differences are bounded by evaluation artifacts.
- What wrong context measures: Under the permissive prompt, wrong-context F1 is 0.551 when models could answer unaided but 0.006 when they could not.The surviving performance reflects parametric recall despite substituted context, which may be unavailable for unfamiliar graphs.
- Scope of the experiment: The substitution experiment is not a poisoning test because it removes correct evidence, adds non-adversarial unrelated evidence, and changes reliance under the strict prompt.It establishes behavior under substitution, not vulnerability severity against semantically conflicting or adversarial triples.
- Format effects: The five human-readable formats cluster between F1 = 0.580 and 0.589, while JSON-LD and RDF/XML trail at 0.503 and 0.495.Format sensitivity is larger for mid-tier models and converges at three hops; predicate naming, rather than verbosity, aligns with the tier split.
- Multi-hop nulls: At three hops, format F1 spread is 0.036, and serialization, ordering, and subgraph size show no measurable multi-hop effect.The seven-format spread is concentrated at one hop and collapses once multi-hop reasoning is required.
- Answer versus citation quality: Correct answers and citation faithfulness can diverge: Turtle has the highest answer F1 at 0.589 but the lowest faithfulness at 0.796.Cypher’s initially low faithfulness was partly a scoring defect corrected by resolving variables before matching.
B. Triple ordering
Triple ordering has no measurable effect in the tested comparisons, while the apparent subgraph-size effect disappears after matching question sets. The initial size pattern was therefore a selection artifact rather than evidence that larger contexts improve performance.
- Triple ordering: Triple ordering changes overall F1 by only +0.006 and prose ordering by −0.003, indicating no measurable ordering effect.The follow-up with 200 triples and cross-question distractors also finds no interaction with scale.
- Selection artifact: The initial subgraph-size ordering was confounded because the three arms ran on different question sets with different difficulty distributions.The conditions covered 125, 77, and 105 questions, and subgraph size correlated with reasoning depth.
- Matched size comparison: Among the 77 shared questions, every subgraph-size contrast crosses zero, including a −0.016 F1 change from chain-only to twenty-five triples.The matched analysis removes the apparent benefit of larger subgraphs on both answer F1 and faithfulness.
D. Four further levers
The four further levers show measurable effects for evidence-first prompting and several parser-dependent prompt comparisons, while other interventions remain difficult to interpret or flat. The central unresolved contrast is the grounding instruction with correct context, because response format and parsing change together.
- Evidence-first prompting: +0.028 faithfulness and −0.021 F1 resulted from requiring an evidence block before the answer.Models identified relevant triples more often, but better evidence selection did not produce better answers.
- Chain-of-thought prompting: −0.020, −0.043, and −0.031 F1 occurred at 2-hop, 3-hop, and 4-hop after adding step-by-step traversal instructions.The six-model result did not replicate a two-model pilot’s 2-hop gain.
- Dynamic oracle scoping: 0.520 F1 for dynamic oracle scoping was below 0.578 for full context and 0.604 for chain-only context.Per-hop decomposition supplied oracle-correct triples plus the preceding answer but removed the holistic view available to the model.
- Sampling: +0.726 context benefit at 1-hop and wrong-context F1 = 0.015 persisted under temperature 0.7 sampling.Five samples per trial preserved the condition rankings observed under greedy decoding.
- Parser-dependent contrast: +0.045 with fallback enabled and −0.039 with fallback disabled are opposite significant estimates from the same correct-context trials.The contrast changes only in response parsing, so the current apparatus cannot identify the grounding instruction’s effect.
- Parser-dependent contrast: The strict prompt combines a grounding restriction with a structured response format, so format-sensitive scoring cannot isolate the sentence’s cost.A valid comparison requires matched output structure and a scorer validated across both formats.
C. Two problems that are not the parser
Several apparent findings are constrained by coverage, corpus composition, parser design, and incomplete evidence about recall. The supported practical guidance is to preserve answer paths, match prompts across arms, and avoid treating tested nulls as universal pipeline claims.
- Coverage and corpus split: Among 11 covered questions, 3 failed the stricter reachability test, while the remaining 8 yielded +0.114 with a confidence interval spanning zero.The coverage filter detected answer strings anywhere in the subgraph rather than verifying reachability along the gold chain.
- Parser and prompt controls: The strict prompt bundles grounding and response-format interventions, so even a format-insensitive scorer would estimate their combined effect.The paper calls for matched structured output, scorer validation, and a 2 × 2 comparison.
- Coverage and corpus split: Within QALD, strict-permissive gaps range from +0.091 to +0.306, whereas LC-QuAD gaps are −0.017, −0.011, and −0.007 with intervals spanning zero.The paper does not assign a mechanism to this corpus split, and LC-QuAD’s higher context score may indicate a ceiling effect.
- Retrieval implications: Replacing surrounding triples at constant volume changes F1 by +0.003, while removing the answer chain lowers F1 to 0.231 under permissive prompting and 0.005 under strict prompting.The evidence is stronger for context composition than for intermediate recall levels.
- Retrieval implications: Relative to no context, wrong-context suppression averages −0.068 F1, but wrong-context F1 is 0.006 on an unmemorised graph.The aggregate effect is smaller and less uniform than the raw correct-versus-wrong comparison suggests.
- Parser and prompt controls: The paper recommends matching prompts across evaluation arms and validating that the scorer reads both response formats equally well.An unmatched grounding instruction produced an apparently publishable but artifactual context finding.
IX. LIMITATIONS
The study’s conclusions are bounded by constructed rather than retrieved subgraphs, incomplete coverage, corpus-depth aliasing, format-sensitive scoring, and limited transfer beyond static plain-text systems. These constraints are especially important for interpreting depth-stratified and retrieval claims.
- Coverage and depth: Full answer coverage declines from 46 of 46 one-hop questions to 4 of 11 four-hop questions.Coverage was tested by answer-label presence in the subgraph, not necessarily reachability from the question entity.
- Coverage and depth: Depth cannot be separated from benchmark because LC-QuAD supplies all one- and two-hop questions while QALD supplies all three- and four-hop questions.Every depth-stratified result is therefore also a between-corpus result.
- Measurement: The scorer is format-sensitive and was not validated across formats, so comparisons that change response format may measure parser validity rather than end-task accuracy.The paper requires validation on both arms before comparing such conditions.
- Training-data contamination: All questions concern Wikidata entities likely to have appeared in pretraining data, and unaided zero scores remain only a proxy for missing training knowledge.The paper notes that a private graph or post-cutoff facts would be needed for a genuine contamination test.
- Retrieval setting: The study has no retriever: chain triples come from gold SPARQL, so precision describes constructed context rather than an implemented retrieval setting.Real retrievers may omit chain triples in ways this precision sweep does not represent.
- Scope of transfer: The results use plain-text fixed subgraphs and do not establish transfer to iterative retrieval, learned graph tokens, embeddings, or mutable model aliases.The precision sweep also covers only four models, and donor triples may be more coherent than real retrieval errors.
X. CONCLUSION
Across 16 experiments and 30,841 trials, only two of four GraphRAG choices changed answers: preserving the answer chain and choosing the grounding instruction. The latter is measurable in empty-context evaluation but remains unresolved when correct context is present.
- Conclusion: Across 16 experiments and 30,841 trials, two of four subgraph-related choices changed the answer, at the beginning and end of the pipeline.The conclusion identifies answer-chain inclusion and the grounding instruction as the consequential choices.
- Conclusion: +0.003 F1 was the effect of replacing surrounding triples while retaining the answer chain.That null held across tested depths and among questions the model could not answer unaided; the intervening choices showed no measurable multi-hop benefit.
- Conclusion: 8.63-fold suppression occurred when the grounding instruction was evaluated with no facts in the prompt.This largest measured effect characterizes an empty-context arm, not a functioning retrieval pipeline.
- Conclusion: Matched prompts show context helping at one-, two-, and three-hop while four-hop remains inconclusive.The earlier depth finding that graph context hurt was artifactual because the prompt differed across arms.
- Conclusion: The cost of the grounding instruction with correct context cannot be measured because prompt regime changes response format and parser choice reverses the estimated effect.Resolution requires a cross-format scorer and identical output structure across permissive and strict prompts.