Source-linked AI summary
KG-RAG: Bridging the Gap Between Knowledge and Creativity
Diego Sanmartin
TL;DR
Knowledge-intensive language model agents face hallucination, forgetting, long-context, and dense-retrieval challenges. KG-RAG extracts triples into a graph and uses retrieved graph paths to generate answers; on ComplexWebQuestions, it reports a lower hallucination rate than Embedding-RAG, alongside lower reported answer metrics.
Problem
Language model agents face hallucinations, forgetting of learned knowledge, long-context difficulties, and dense retrieval limitations on complex queries.
Source: §1 · pp. 1–2Method
KG-RAG extracts triples from text into a knowledge graph and uses retrieved paths to condition answer generation.
Source: §1 · p. 3, §3.3 · p. 5Results
KG-RAG’s hallucination rate was 15%, versus 30% for Embedding-RAG, while its EM was 19% versus 28% and accuracy was 32% versus 46%.
Source: §5.4 · p. 11Takeaways & Limitations
The preliminary experiments suggest KG-RAG as a possible approach for knowledge-intensive tasks, with reduced hallucinated content reported in the paper’s evaluation.
Source: §1 · p. 3, §8 · p. 13Low-quality CWQ web snippets and errors or omissions in extracted triples could affect result reliability and propagate into CoE performance.
Source: §6 · p. 12
Abstract
from arXiv · showhide
Ensuring factual accuracy while maintaining the creative capabilities of Large Language Model Agents (LMAs) poses significant challenges in the development of intelligent agent systems. LMAs face prevalent issues such as information hallucinations, catastrophic forgetting, and limitations in processing long contexts when dealing with knowledge-intensive tasks. This paper introduces a KG-RAG (Knowledge Graph-Retrieval Augmented Generation) pipeline, a novel framework designed to enhance the knowledge capabilities of LMAs by integrating structured Knowledge Graphs (KGs) with the functionalities of LLMs, thereby significantly reducing the reliance on the latent knowledge of LLMs. The KG-RAG pipeline constructs a KG from unstructured text and then performs information retrieval over the newly created graph to perform KGQA (Knowledge Graph Question Answering). The retrieval methodology leverages a novel algorithm called Chain of Explorations (CoE) which benefits from LLMs reasoning to explore nodes and relationships within the KG sequentially. Preliminary experiments on the ComplexWebQuestions dataset demonstrate notable improvements in the reduction of hallucinated content and suggest a promising path toward developing intelligent systems adept at handling knowledge-intensive tasks.
1 Introduction
The introduction frames hallucination, forgetting, long-context handling, and dense retrieval granularity as challenges for knowledge-intensive LMAs. KG-RAG uses structured graph knowledge to augment agents and retrieve information for complex queries.
1 Introduction
- LLMs can produce factually incorrect answers, lose relevant information in extended contexts, and forget previously learned knowledge during training.The paper identifies these issues as challenges for deploying LMAs in production.Source: §1 · p. 1
- Dense vector retrieval can struggle with complex queries because it has difficulty finding the necessary granularity among similar text chunks.Metadata filtering and hybrid search remain constrained by predefined metadata scope.Source: §1 · p. 2
- Knowledge graphs represent entities and relationships explicitly, supporting retrieval of relevant information rather than relying only on vector similarity.The introduction describes triples as accurate, updatable, and interpretable knowledge facts.Source: §1 · p. 3
- KG-RAG extracts triples from raw text, stores them in a knowledge graph, and searches the graph to augment LMAs with external knowledge.The authors present it as addressing hallucination, catastrophic forgetting, and granularity in dense retrieval.Source: §1 · p. 3
2 Related Work
The related work situates KG-RAG among approaches to external memory, graph construction, and KG question answering. It presents a domain-agnostic homogeneous graph and Chain of Explorations as components of its approach.
External Databases and Agent Memory
- Prior agent-memory approaches include SQL-based reasoning over stored memories and triple generation from past dialogues for conversational memory.The paper places KG-RAG among systems that integrate external databases with language model agents.Source: §2 · p. 3
Knowledge Graph Construction
- The proposed homogeneous knowledge graph is domain-agnostic, unlike approaches that use rigid, domain-specific ontologies.The authors describe this flexibility as accommodating dynamic knowledge.Source: §2 · p. 3
Knowledge Graph Question Answering (KGQA)
- KG-RAG introduces Chain of Explorations for more precise, contextually relevant lookups within graph-structured knowledge.This work builds on LLM integration with KGs for question answering.Source: §2 · p. 3
Our Contribution
- KG-RAG uses dynamically updated knowledge graphs and granular, context-sensitive retrieval to address hallucination in language model agents.The paper positions its pipeline as an external knowledge module for LMAs.Source: §2 · p. 3
3 Preliminaries
The preliminaries define language-model generation, graph representations, and multi-hop KG question answering. They then formalize triple extraction, graph-path retrieval, and answer generation as stages of the pipeline.
3 Preliminaries
- A language model generates an output token sequence conditioned on its prompt and prior output tokens.The preliminaries describe this as a conditional probability distribution over output sequences.Source: §3 · p. 4
- A homogeneous knowledge graph has entity and relationship sets, while a triple represents a subject, predicate, and object.Complex KGQA can require retrieving information from multiple triples along a multi-hop path.Source: §3 · p. 4
3.1 Storage
- In storage, a few-shot language model extracts entity–relationship–entity triples from unstructured text chunks for the graph.The passage describes using typically 5–10 examples for few-shot learning.Source: §3.1 · pp. 4–5
3.2 Retrieval
- Retrieval searches the graph for paths connecting relevant entities through relationships suited to the natural-language query.The retrieval objective is to find paths likely to answer the query.Source: §3.2 · p. 5
3.3 Answer Generation
- Answer generation synthesizes retrieved graph paths into a response conditioned on the query and path context.The answer is generated as a sequence whose next token depends on prior answer tokens, the query, and retrieved paths.Source: §3.3 · p. 5
4 Methodology
The methodology builds a KG from unstructured text, retrieves query-relevant information through Chain of Explorations (CoE), and has an LLM answer using only the retrieved KG knowledge.
4.1 Storage
- A 6-shot LLM extracts triples; recursive triple hypernodes encode nested relations and let their components connect onward, while embeddings support later retrieval.The hypernodes can represent contextual structures such as triples linked to dates.Source: §4.1 · pp. 6–7
4.2 Retrieval
- CoE plans query-guided graph exploration, combines vector-ranked starting nodes with Cypher lookups and LLM selection, then evaluates whether to continue, refine, or answer.Its initial plan uses the query; evaluation checks the traversed path against that plan and may revise the exploration steps.Source: §4.2 · pp. 7–9
4.3 Answer Generation
- The LLM answers the user’s question using only knowledge found in the KG during the preceding retrieval stage.Figures 4 and 5 illustrate step-by-step CoE executions for complex queries.Source: §4.3 · p. 9, Figure 4 · p. 9
5 Experiments
The experiments evaluate KG-RAG on ComplexWebQuestions (CWQ) using sampled development questions and compare its answer metrics with vector-RAG and leaderboard models. KG-RAG has lower reported EM, F1, and accuracy than Embedding-RAG, but a lower hallucination rate.
5.1 Experimental Setup
- The setup used NebulaGraph for the KG, SentenceTransformer embeddings with Redis vector search, and GPT-4 Turbo 1106-Preview hosted on Azure.The experiments ran locally on an M2 Ultra with 128 GB unified memory.Source: §5.1 · pp. 9–10
5.2 ComplexWebQuestions Dataset
- CWQ tests complex queries involving multi-hop reasoning, temporal constraints, and aggregations, using questions paired with web snippets and Freebase SPARQL queries.The dataset contains 34,689 questions, with an average of 367 web snippets per question.Source: §5.2 · p. 10
- Experiments sampled 100 CWQ development questions, excluded 11 without answers in the snippets, and built a KG with 9,604 connected nodes and 3,175 unique relationship names.Answer-containing snippets were filtered to reduce construction costs.Source: §5.2 · p. 10
5.3 Evaluation Metrics
- Evaluation used Exact Match, F1 Score, conventional accuracy, and a modified precision metric for assessing hallucinations.Hallucination flags used imperfect precision and heuristic uncertainty indicators, followed by manual review.Source: §5.3 · pp. 10–11
5.4 Results and Analysis
- KG-RAG recorded EM 19% and F1 Score 25% (Embedding-RAG: EM 28% and F1 Score 37%), accuracy 32% vs 46%, and hallucination rate 15% vs 30%.The comparison uses the CWQ web-snippets dataset.Source: Table 1 · p. 11, §5.4 · pp. 11–12
6 Limitations and Future Work
The experiments were constrained by costs and by the quality of the CWQ web snippets and extracted triples. Future work could test other datasets and develop a specialized dataset for KG construction from raw text.
6 Limitations and Future Work
- LLM costs limited the use of web snippets for each question and prevented testing the full CWQ development split.The authors suggest expanding the experiments as budgets increase or LLM computing costs decrease.Source: §6 · p. 12
- Low-quality CWQ snippets and incorrect or missed triple extractions could affect result reliability, with extraction errors potentially propagating to CoE.The authors suggest testing other question-answer datasets and refining KG-construction models, including entity resolution and linking.Source: §6 · p. 12
- A specialized dataset for constructing KGs with triple hypernodes from raw text could support open-source model fine-tuning and affordable local KG construction.The authors say this could mitigate cost issues and improve data control and quality.Source: §6 · pp. 12–13
7 Discussion
The discussion presents KGs as a way to give LMAs explicit, updated information and discusses retrieval speed, costs, and further research directions. It also identifies hardware advances as a possible way to reduce CoE response time.
7 Discussion
- Structured KGs can reduce hallucinations by grounding responses in explicit information and provide accurate, updated information without resource-intensive fine-tuning.The discussion contrasts structured knowledge with information stored implicitly in model weights.Source: §7 · p. 13
- CoE's average response time is 30 seconds; the authors project Groq hardware could reduce it to 3 seconds and lower operational costs.The current response time is attributed to multiple Azure API calls.Source: §7 · p. 13
- The authors call for further research on KG construction, knowledge representation, and LLM capabilities in analytical thinking, reasoning, and complex problem-solving.They also discuss using external tools to extend agents' access to information resources.Source: §7 · p. 13
8 Conclusion
The paper introduces KG-RAG, integrating structured KGs with LLM capabilities, and reports preliminary results as promising for knowledge-intensive tasks. It presents CoE as a sequential graph-exploration method within this pipeline.
8 Conclusion
- KG-RAG integrates structured KGs with LLM reasoning and creativity; its homogeneous triple-hypernode KGs represent knowledge extracted from unstructured text.CoE sequentially explores KG nodes and relationships to find answer information, aiming to improve the relevance and precision of explored paths.Source: §8 · p. 13
- Preliminary experiments show promising results for building LMAs that handle knowledge-intensive tasks, though the authors describe the experiments as limited.The authors frame the work as a foundation for further research and development.Source: §8 · p. 13