KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering

1 Introduction

• Semantic-parsing KGQA can provide precise answers but often has low coverage because it depends on rigorous schemas and is sensitive to parsing errors.Generating only a few retrieval paths with LLMs does not fully address retrieval recall.Source: §1 · p. 2
• KGQA must handle entity neighborhoods too large for context, multi-hop retrieval boundaries, and questions requiring aggregation or reasoning.A head entity can have 2 million triples.Source: §1 · p. 2
• KERAG retrieves a broader subgraph around the question's topic entity, filters irrelevant information, and summarizes the remainder to generate an answer.Unlike retrieving only entities and triples deemed strictly necessary, its broader retrieval aims to include information likely to be relevant.Source: §1 · p. 2
• KERAG combines entity-level retrieval with predicate-level filtering and fine-tuned Chain-of-Thought reasoning over neighborhood subgraphs.The approach is presented as applicable to both SPARQL-supporting graph databases and API-based knowledge systems.Source: §1 · p. 2
• 7% in quality over state-of-the-art solutions and 10-21% over GPT-4o (Tool) across two KG benchmarks.The experiments cover API-based and SPARQL-based benchmarks.Source: §1 · p. 2

2 Related Work

• Semantic-parsing approaches convert questions into logical forms or structured queries, using multi-step search or seq2seq generation.WikiSP is a few-shot seq2seq parser fine-tuned on modified SPARQL queries.Source: §2 · pp. 2–3
• Information-retrieval KGQA retrieves query-related subgraphs, while KERAG uses a more relaxed neighborhood to give its summarizer broader context.A cited trainable subgraph retriever prioritizes smaller, more relevant subgraphs.Source: §2 · p. 3
• LLM tool-calling approaches can propagate planning errors into retrieval, leaving insufficient KG content for answering.ToG explores KG reasoning paths with beam search, while ToG-2 supplements KG content with external Wikipedia pages.Source: §2 · p. 3
• KERAG's neighborhood retrieval is reported to improve recall by about 11% over ToG and accuracy by up to 20% over ToG and 4% over ToG-2.It also outperforms StructGPT by 7% in truthfulness and supports both KGs and APIs.Source: §2 · p. 3

3 Problem Statement

• A knowledge graph K is defined as K = (E, R, D), where E is entities, R is relations, and D optionally represents covered domains.Each entity can have properties, and relations are binary relations between entities.Source: §3 · p. 3
• A knowledge graph is normally accompanied by an ontology describing entity types and relationships between types.Source: §3 · p. 3
• KG-based RAG answers a natural-language question Q by leveraging knowledge in graph K.Source: §3 · p. 3

4.1 Solution overview

• Planning selects a topic entity, hop count, and relations to filter; retrieval executes the plan, and summarization reasons over retrieved knowledge to produce the answer.The retrieval plan defines a neighborhood scope, excluding only relations deemed absolutely irrelevant rather than retrieving only nodes strictly necessary to answer.Source: §4.1 · p. 4, §4.2 · p. 4

4.2 Planning

• Planning identifies the domain and topic entity, explores neighborhood schema, filters irrelevant relations and checks information sufficiency, then continues or terminates retrieval.Ontology-guided hopping and filtering are intended to reduce the cost of full data retrieval.Source: §4.2 · pp. 4–5
• 5.5%: On CRAG, multi-hop retrieval improved truthfulness, while filtering improved truthfulness by 3.9%.The passage reports both effects in Table 4.Source: §4.2 · p. 5

4.3 CoT Summarization

• CoT prompting supports complex questions that require cross-referencing KG facts, and the summarizer is fine-tuned using generated reasoning and answers checked against ground truth.Correct generated answers and reasoning are retained as training targets; otherwise, the original answer is retained and the prompt is replaced with a non-CoT prompt.Source: §4.3 · pp. 5–6
• On the CRAG dataset, plain CoT reasoning refrained from generating hallucinated answers for 18% of questions, while fine-tuning further increased accuracy by 10%.Source: §4.3 · p. 6

5.1 Datasets

• CRAG uses API-accessed KG questions; Head2Tail uses DBPedia via SPARQL, with 375 test questions sampled from each popularity category.The experiments also include QALD-10-en, WebQSP, AdvHotpotQA, and CWQ.Source: §5.1 · p. 6

5.2 Baselines

• Comparisons include LLMs and tool variants, CRAG KDD Cup winners, and SPARQL KGQA systems; ToG-2 is omitted on Head2Tail because its external-page setup is incompatible.KERAG (8B) is included for a fair comparison with KDD Cup solutions that may use 8B models.Source: §5.2 · p. 6

5.3 Evaluation

• CRAG and Head2Tail answers are classified as accurate, missing, or hallucinated; truthfulness scores accuracy positively and hallucinations negatively.For QALD-10-en, WebQSP, AdvHotpotQA, and CWQ, the experiments use each dataset's own evaluation methods and metrics.Source: §5.3 · p. 6

5.4 Experimental Results

• 7.1% higher truthfulness on CRAG; Table 2 reports a 7% improvement on Head2Tail, while KERAG wins on three SPARQL datasets and is comparable on CWQ.The CRAG and Head2Tail comparisons are reported in Tables 1 and 2, respectively.Source: Table 1 · p. 7, §5.4 · p. 7, Table 2 · p. 7

5.5 Ablation Study

• Removing multi-hop retrieval, filtering, fine-tuning, or CoT lowers truthfulness by 7.4%, 10.4%, 14.4%, and 43.1%, respectively.On CRAG, CoT prompting and fine-tuning most improved quality, raising accuracy by 9% and reducing hallucinations by 15%; exhaustive retrieval also raised accuracy 15% and reduced missing rate from 30% to 7%, while two-hop retrieval raised hallucinations by 9%.Source: §5.5 · pp. 7–8

5.6 Robustness

• KERAG's performance is relatively stable across head, torso, and tail entities; multi-entity adaptation raises CRAG comparison-question truthfulness from 0.7 to 0.75 and accuracy from 0.817 to 0.85.The adapted setting targets comparison questions, the only CRAG type involving multiple entities.Source: §5.6 · p. 8
• DeepSeek-V3 raises KERAG's CRAG accuracy by 2.4% and truthfulness by 11.8% over the 70B backbone; error analysis identifies summarization as the main CRAG error source.For Head2Tail, reported errors occur mainly in entity linking and summarization.Source: §5.6 · p. 8

5.7 Latency

• KERAG has slightly higher latency than apex on CRAG but lower latency than StructGPT and ToG on Head2Tail.The authors characterize its overall latency as acceptable.Source: §5.7 · pp. 8–9

• KERAG leverages entity-level information to reduce reliance on triple-level semantic parsing.Source: §6 · p. 9
• A data-generation scheme supports fine-tuning Chain-of-Thought summarizers to handle complex queries.Source: §6 · p. 9
• The authors report that KERAG outperforms existing state-of-the-art methods and applies across knowledge graphs accessed through SPARQL or APIs.Source: §6 · p. 9

Limitations

• Evaluation covered only CRAG, Head2Tail, QALD-10-en, WebQSP, AdvHotpotQA, and CWQ, so performance on other KGs and domains is unknown.The authors also note potential error propagation in the multi-stage pipeline and identify entity linking as the main error source.Source: Limitations · p. 9
• Existing state-of-the-art entity-linking approaches could be used as a prestep to integrate with KERAG and improve overall performance.The current method does not focus on deploying a specific entity-linking mechanism.Source: Limitations · p. 9

Ethics Statement

• The experiments use real-world named entities and publicly available datasets, which cannot be anonymized, and public availability does not guarantee the absence of privacy violations.The cited datasets have been extensively used in prior research.Source: Ethics Statement · p. 9
• Training and inference of the models consume energy and produce carbon emissions.Source: Ethics Statement · p. 9

A Dataset Details

• Table 9 presents training- and testing-set statistics for the CRAG KG subset.Source: Table 9 · p. 12, A Dataset Details · p. 12
• CRAG covers open, movie, finance, sports, and music domains, with questions spanning eight types and four dynamism categories.Its question types include simple, conditional, set, comparison, aggregation, multi-hop, post-processing, and false-premise questions; dynamism categories are real-time, fast-changing, slow-changing, and static.Source: A Dataset Details · p. 12
• CRAG questions require one hop in 43% of cases, two hops in 49%, and three hops in 8%.Source: A Dataset Details · p. 12
• Head2Tail's open-domain set contains 9,219 questions, with 375 questions sampled from each head, torso, and tail split for testing.The sampling guarantees a margin of error below 5%.Source: A Dataset Details · p. 12
• CRAG and Head2Tail use Attribution-NonCommercial 4.0 International licenses; QALD-10-en, WebQSP, AdvHotpotQA, and CWQ use MIT, CC-BY-SA-4.0, CC-BY-SA-4.0, and Apache-2.0, respectively.Source: A Dataset Details · pp. 12–13
• Table 10 presents statistics for each question type across different question splits.Source: Table 10 · p. 13

B Advantages of KERAG over State-of-the-arts.

• KERAG gathers relevant neighborhoods rather than prompting for retrieval paths, makes filtering and expansion decisions at schema level, and applies to both SPARQL- and API-based KGs.For D hops, KERAG invokes the LLM D+2 times, compared with ND+D+2 calls for ToG and ToG-2, where N is the number of paths investigated.Source: B Advantages of KERAG over… · p. 13

C Baseline Details

• GPT-4o and GPT-4o (tool) serve as closed-source LLM baselines, while Llama-3.1-70B-Instruct and its tool variant represent open-source LLM solutions.The tool variants can write API calls or SPARQL queries for KG access.Source: C Baseline Details · p. 13
• db3 and apex are CRAG benchmark-winning solutions adapted to KGQA by supplying empty web content; apex uses a router-based adaptive RAG pipeline.Source: C Baseline Details · p. 13
• StructGPT linearizes retrieved structured data for LLM reasoning, while ToG explores KG reasoning paths through beam search.KERAG compares its approach with ToG and ToG-2 in the stated advantages discussion.Source: B Advantages of KERAG over… · p. 13, C Baseline Details · p. 14
• The baseline set includes semantic-parsing methods sparql-qa and WikiSP, plus Decaf, which jointly generates answers and logical forms.WikiSP is used as a state-of-the-art semantic-parsing representative, with its QA module replaced by GPT-4o in the experiments.Source: C Baseline Details · pp. 13–14
• CBR retrieves similar question cases and revises reused logical-form components, while CoK adapts heterogeneous knowledge through staged reasoning.Source: C Baseline Details · p. 14

D Evaluation Details

• The evaluation adopts the CRAG scheme to assess answer correctness on both CRAG and Head2Tail.Responses are categorized as correct, missing, or incorrect.Source: D Evaluation Details · p. 14
• Llama-3.1-70B-Instruct serves as the automatic evaluation critic for all responses.The evaluation prompts are in Appendix I.5.Source: D Evaluation Details · p. 14

E Configuration Details

• For CRAG, planning uses Llama-3.1-70B-Instruct, while the summarizer is fine-tuned from Llama-3.1-8B-Instruct.FSDP and LoRA are used for fine-tuning.Source: E Configuration Details · p. 14
• For Head2Tail, Llama-3-8B-Instruct handles planning and summarization without CoT reasoning or summarizer fine-tuning.DPR-based filtering retains up to 30 predicates for summarization.Source: E Configuration Details · p. 15
• For QALD-10-en, WebQSP, AdvHotpotQA, and CWQ, planning uses Llama-3-8B-Instruct and summarization uses GPT-4o, without CoT reasoning or summarizer fine-tuning.DPR filtering is applied.Source: E Configuration Details · p. 15

F Head-Torso-Tail Experiment Results

• Tables 12 and 13 provide head/torso/tail results for CRAG and Head2Tail, respectively.The supplied passages do not report their cell values.Source: F Head-Torso-Tail Experiment Results · p. 15, Table 12 · p. 15, Table 13 · p. 15
• Figure 5 compares one-hop, two-hop, and exhaustive retrieval on CRAG.The supplied caption identifies the comparison but does not state its outcome.Source: Figure 5 · p. 15

G Full Ablation Study Results

• Table 14 presents the full ablation study on CRAG.The supplied passage does not report ablation values or outcomes.Source: G Full Ablation Study Results · p. 15, Table 14 · p. 16

H Templates

• The planning template asks for a topic entity and domain, then uses KG predicates to extend a retrieval plan and assess its sufficiency for answering.Examples and optional instructions may also be supplied.Source: H Templates · pp. 15–16
• The summarization template requests a brief answer based on the question and retrieved triples, with step-by-step reasoning and an “I don’t know” response when confidence is insufficient.Source: H Templates · p. 16

I.1 Planning Prompts

• Planning prompts classify queries by domain and extract domain-specific structured fields, while requiring flat JSON output.Fields include entities or metrics, such as finance identifiers and metrics or sports type, tournament, and team.Source: §I.1 · p. 16
• A separate time-extraction prompt uses the query time when no date is explicit and represents periods with a time range.Examples cover relative dates such as “last day,” explicitly stated dates, and ranges such as the past week.Source: §I.1 · pp. 16–17
• Head2Tail planning prompts ask a semantic-parsing model to select the best candidate (entity, predicate) pair or return (No, No).The prompt explicitly instructs the model not to answer the question directly.Source: §I.1 · p. 17

I.2 Filtering Prompts

• Filtering prompts provide finance-KG functions for price histories, dividends, market capitalization, EPS, P/E ratio, and ticker metadata, with constrained function-call formatting.The appendix presents CRAG finance filtering as an example and says other-KG prompts can be adapted by changing function descriptions and examples.Source: §I.2 · p. 17
• Head2Tail’s multi-hop retrieval-boundary prompt labels retrieved DBpedia triples insufficient, wrongly linked, or sufficient using <NO>, <NA>, or <YES>.The model is instructed to return only one of those labels.Source: §I.2 · p. 18

I.3 Summarization Prompts

• CRAG summarization prompts request a brief, step-by-step answer grounded in refined knowledge-base content and say to answer “I don’t know” when uncertain.The prompt includes the query time; the appendix also gives a base-prompt version with the same answer and uncertainty instructions.Source: §I.3 · p. 18
• Head2Tail’s summarization prompt requests a brief answer based on DBpedia subject-predicate-object triples and permits “I don’t know” when uncertain.The prompt supplies triples as repeated inputs alongside the question.Source: §I.3 · p. 18

I.4 Baselines Tool-use Prompts

• Baseline tool-use prompts ask GPT-4o and Llama to select APIs for relevant content, spanning open-domain, movie, and finance functions.The supplied API list also includes music artist and song search functions.Source: §I.4 · pp. 18–19
• The baseline tool prompts specify functions with descriptions, parameter schemas, and required arguments for queries across domains such as music and sports.Examples include song and artist queries, soccer games by date, and NBA games or play-by-play data.Source: §I.4 · p. 20
• Tool-use instructions require a prescribed function-call format, use <CONTINUE> for chained calls, separate multiple calls with semicolons, and say “I don’t know” when no function applies.Examples map questions to tools and provide the current query-time and question placeholders.Source: §I.4 · p. 21
• The Head2Tail prompt directs the model to select a question’s main entity and predicate from a supplied list, then construct a DBPedia SPARQL query, returning only the entity and predicate.Source: §I.4 · p. 21

I.5 Auto-evaluation Prompts

• The custom evaluator assigns a binary score: uncertainty, contradictions, irrelevant answers, or mismatches score 0, while correct exact answers, concise summaries, and sufficiently close numbers can score 1.Set-valued answers must contain exactly the ground-truth items; numeric answers must almost exactly match.Source: §I.5 · p. 21
• Examples show that incomplete or extra set items and incorrect values receive 0, while equivalent names, rounded numbers, and concise correct answers may receive 1.The prompt also rejects answers that omit a needed qualifier, such as stating a value without indicating it is “over” the ground-truth amount.Source: §I.5 · p. 22
• Head2Tail’s evaluator instead asks whether an answer is correct and returns a brief Yes or No, with examples illustrating accepted equivalents and incorrect responses.Source: §I.5 · pp. 22–23