Source-linked AI summary
Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey
Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, Huan Liu
TL;DR
LLM hallucinations arise amid knowledge gaps and probabilistic behavior, motivating methods that supply precise external knowledge. This survey reviews KG-based augmentation, categorizes it into inference, learning, and validation, evaluates empirical efficacy, and discusses challenges and future directions. It concludes that KGs can enhance LLM performance across inference, training, and output verification stages, while continued innovation remains necessary.
Problem
LLM hallucinations undermine reliability, and adding random information does not adequately improve interpretation or reasoning.
Method
The survey critically reviews KG-based LLM augmentation methods, categorizes them into inference, learning, and validation, and evaluates their empirical efficacy.
Results
KGs enhance LLM performance across inference, model training, and output verification stages while mitigating hallucinations and improving reasoning accuracy.
Takeaways & Limitations
The survey identifies continued innovation and proposed future directions as necessary for developing more advanced KG-augmented LLMs.
Takeaways & Limitations
The analysis depends on current experiments, datasets, benchmarks, and baseline models, which may change rapidly and alter evaluations.
Abstract
from arXiv · showhide
The contemporary LLMs are prone to producing hallucinations, stemming mainly from the knowledge gaps within the models. To address this critical limitation, researchers employ diverse strategies to augment the LLMs by incorporating external knowledge, aiming to reduce hallucinations and enhance reasoning accuracy. Among these strategies, leveraging knowledge graphs as a source of external information has demonstrated promising results. In this survey, we comprehensively review these knowledge-graph-based augmentation techniques in LLMs, focusing on their efficacy in mitigating hallucinations. We systematically categorize these methods into three overarching groups, offering methodological comparisons and performance evaluations. Lastly, this survey explores the current trends and challenges associated with these techniques and outlines potential avenues for future research in this emerging field.
1 Introduction
LLM hallucinations undermine reliability, while contextually relevant external knowledge from knowledge graphs offers a promising augmentation strategy. This survey reviews KG-based methods, classifies them by augmentation stage, evaluates their efficacy, and identifies research directions.
- LLM hallucinations can produce irrelevant or incorrect outputs, undermining model reliability.
- Contextually relevant and precise external knowledge can support models’ information recall, interpretation, and reasoning.
- Knowledge-graph augmentation strategies target inference, learning, and validation stages.
- The survey critically reviews KG augmentation methods for reducing hallucinations and improving LLM performance and reliability.
- The survey focuses exclusively on structured knowledge from knowledge graphs and its use in addressing LLM hallucinations.
2 Preliminaries
The preliminaries describe how language models estimate text-sequence likelihoods and how transformer-based advances produced modern LLMs. They also introduce knowledge graphs as structured representations of entities and relationships.
- Language modeling estimates the likelihood of a text sequence by computing each token’s probability conditioned on preceding tokens.
- Transformers advanced neural probabilistic language models through efficient parallel processing and recognition of long-range dependencies.
- GPT-3, GPT-4, and PaLM are presented as advanced LLMs with notable language capabilities.
- Knowledge graphs organize information as triples, using nodes for entities and edges for their relationships.
3 Knowledge Graph-Enhanced LLMs
LLMs primarily fail through inadequate context comprehension, insufficient knowledge, or failure to recall facts. The survey organizes KG-based mitigation methods into inference, learning, and validation categories.
- LLMs have three primary failure points: insufficient context comprehension, inadequate knowledge, and inability to recall specific facts.
- The survey reviews KG augmentation techniques intended to mitigate hallucinations and improve LLM reasoning capabilities.
- The taxonomy comprises Knowledge-Aware Inference, Knowledge-Aware Learning, and Knowledge-Aware Validation.
3.1 Knowledge-Aware Inference
Knowledge-aware inference integrates knowledge graphs into LLM inputs or generation to improve contextual understanding, factual answering, and multi-step reasoning. The survey distinguishes retrieval, reasoning, and controlled-generation strategies while noting that the foundations of neural reasoning remain uncertain.
- Inference challenges: LLM inference can fail because of ambiguous inputs, missing context or knowledge, training biases, and difficulty generalizing to unseen scenarios.The survey frames KG augmentation as a response to these inference and reasoning challenges.
- KG-Augmented Retrieval: KG-augmented retrieval supplies structured, curated knowledge alongside inputs, supporting factual answers without changing the LLM architecture.This extends document-based retrieval with knowledge organized to align more closely with factual accuracy.
- KG-Augmented Reasoning: KG-augmented reasoning decomposes complex multi-step questions into sub-queries and interleaves retrieval with reasoning to guide faithful explanations.Approaches include iterative graph retrieval, graph-of-thoughts prompting, and explicit reasoning paths.
- KG-Augmented Reasoning: Whether neural networks genuinely reason remains unanswered, and correct reasoning paths do not necessarily guarantee accurate answers.These unresolved questions limit how confidently prompting-based reasoning improvements can be interpreted.
- KG-Controlled Generation: Knowledge-controlled generation uses graph ontologies as domain constraints that help define output boundaries and align generated content with facts.This positions KGs as operational guardrails against misinformation.
3.2 Knowledge-Aware Training
Knowledge-aware training incorporates graph knowledge during pre-training or fine-tuning to improve language understanding and adapt models to specific domains. The survey covers knowledge-enhanced models, entity or sentiment masking, knowledge fusion, probing, and KG-based fine-tuning, while highlighting open questions about updated knowledge.
- Training overview: Knowledge-aware training improves LLM learning by enhancing pre-training data or adapting pre-trained models to specific tasks and domains.The survey divides these approaches into Knowledge-Aware Pre-Training and Knowledge-Aware Fine-Tuning.
- Knowledge-Aware Pre-Training: Knowledge-enhanced models enrich large text corpora with KG facts, combining contextual, lexical, and syntactic information during pre-training.ERNIE and ERNIE 3.0 exemplify architectures that integrate knowledge with language modeling objectives.
- Knowledge-Aware Pre-Training: Knowledge-guided masking hides key entities or sentiment information using graph-linked signals to strengthen relational or sentiment representations.Reported applications include question answering, knowledge-base completion, and sentiment analysis.
- Knowledge-Aware Pre-Training: Knowledge fusion uses graph query encoders and joint reasoning to select relevant KG nodes and synchronize text-graph updates.JointLK applies this design to commonsense question answering, while LKPNR applies KG-augmented encoding to personalized news recommendation.
- Knowledge-Aware Fine-Tuning: KG-based fine-tuning updates or expands model knowledge for domain-specific tasks using synthetic KG-derived sentences, triples, or logical datasets.Examples span named-entity recognition, summarization, link prediction, and cross-lingual reasoning.
- Knowledge-Aware Fine-Tuning: Updated models can recall facts about new entities, but inferring from those facts is harder, and effects on existing entities remain open questions.Fine-tuning is described as more efficient than training from scratch, although integration of updated knowledge is not fully determined.
3.3 Knowledge-Aware Validation
Knowledge-aware validation uses structured graph information and auxiliary verification mechanisms to check generated claims, provide explanations, and improve factual consistency. The surveyed methods range from KG-conditioned generation to explicit logical claim verification.
- Validation overview: Knowledge graphs can serve as fact-checking references that support comprehensive explanations, decision justification, and consistency across generated facts.These properties can reduce reliance on laborious human-annotated data while enhancing generated-content reliability.
- KG-Based Validation: KGLM consults a knowledge graph to generate entities and facts relevant to the surrounding context.This couples factual generation with graph-based information selection.
- KG-Based Validation: SURGE retrieves high-similarity, context-relevant triples as a subgraph, while a text critic assesses the match between input data and generated text.These methods apply graph retrieval or classification to evaluate generation against contextual evidence.
- KG-Based Validation: FOLK uses first-order-logic predicates to verify claims and generates explicit explanations to assist human fact-checkers.Its scope includes online misinformation and interpretable verification outputs.
4 Discussion, Challenges and Future
The survey evaluates KG-enhanced LLM methods for reducing hallucinations, compares their effectiveness and trade-offs, and identifies research trends and future directions. Recent work increasingly favors retrieval, reasoning, generation, and validation methods that avoid additional training costs.
- Evaluation Metrics: KG-enhanced LLM methods are assessed using accuracy, retrieval ranking, factuality, and helpful-versus-harmful triple criteria.These criteria compare augmented and unaugmented performance, evaluate Top-K and MRR retrieval, and examine factual and fabricated information.
- Performance Analysis: Over 80% higher answer correctness was reported for smaller models augmented with retrieved KG facts on question-answering tasks.The benefit addresses smaller models’ limited ability to incorporate extensive knowledge during pretraining.
- Performance Analysis: ChatGPT accuracy increased from 66.8% to 85.7% with RoG knowledge-graph augmentation in reasoning tasks.Mindmap also reached 88.2% accuracy for disease diagnosis and drug recommendation using a clinical reasoning graph.
- Challenges: Pre-training and fine-tuning improve domain-specific performance but require substantial computation, while fine-tuning remains task-specific and less transferable.Fact-checking can reduce hallucinations by checking generated data against a KG, but adds computational load and may miss inaccuracies.
- Trend Analysis: From 2019 to 2023, research shifted from KG-augmented pre-training toward task-specific fine-tuning and methods that avoid additional training costs.The survey attributes the shift partly to the impracticality and resource intensity of retraining models with billions of parameters.
- Future Directions: Future directions include context-aware, bias-aware, cross-domain, and multimodal KGs, alongside MoE integration, symbolic-subsymbolic unification, bidirectional LLM-KG reasoning, and causality awareness.These directions target richer knowledge, adaptive expert use, mutual enhancement, and understanding causation rather than correlation.
5 Conclusion
The survey systematically investigates KG integration across inference, training, and output verification to mitigate hallucinations and improve reasoning accuracy. It reports substantial progress while calling for continued innovation in KG-augmented LLMs.
- The survey investigates KG integration across inference, model training, and output verification to mitigate hallucinations and improve reasoning accuracy.
- The paper emphasizes continued innovation and proposes future directions for developing more advanced KG-augmented LLMs.
6 Limitations
The survey acknowledges limitations in coverage and technical detail, while its taxonomy and comparisons depend on current experiments, datasets, benchmarks, and baselines.
- Page limitations may exclude relevant references and detailed technical information.
- The review primarily covers state-of-the-art methods from 2019 to 2023, sourced mainly from selected conferences and arXiv.
- Hybrid studies using multiple augmentation approaches may be categorized differently according to the survey’s criteria.
- Performance comparisons reflect existing experiments and datasets, while changing benchmarks and baselines may alter evaluations.