Source-linked AI summary
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting
Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, Le Sun
TL;DR
Existing KG-based methods mainly query using the user’s input, leaving factual hallucinations that arise during LLM reasoning insufficiently addressed. KGR retrofits initial draft responses by autonomously extracting, selecting, validating, and refining claims against KG knowledge, and experiments report significant factual-QA improvements, especially for complex reasoning.
Problem
Existing KG-based methods primarily retrieve facts from entities in the user’s query, leaving factual hallucinations generated during the LLM’s reasoning process insufficiently addressed.
Method
KGR uses LLMs to extract factual claims from draft responses, retrieve and select relevant KG facts, verify claims, and retrofit the responses without manual effort.
Results
KGR significantly improves LLM performance on factual QA benchmarks, especially when complex reasoning processes are involved.
Takeaways & Limitations
The experiments demonstrate the necessity and effectiveness of KGR for mitigating hallucination and enhancing LLM reliability.
Takeaways & Limitations
The framework faces errors in claim extraction, entity-granularity detection, fact selection, claim verification, and response retrofitting.
Abstract
from arXiv · showhide
Incorporating factual knowledge in knowledge graph is regarded as a promising approach for mitigating the hallucination of large language models (LLMs). Existing methods usually only use the user's input to query the knowledge graph, thus failing to address the factual hallucination generated by LLMs during its reasoning process. To address this problem, this paper proposes Knowledge Graph-based Retrofitting (KGR), a new framework that incorporates LLMs with KGs to mitigate factual hallucination during the reasoning process by retrofitting the initial draft responses of LLMs based on the factual knowledge stored in KGs. Specifically, KGR leverages LLMs to extract, select, validate, and retrofit factual statements within the model-generated responses, which enables an autonomous knowledge verifying and refining procedure without any additional manual efforts. Experiments show that KGR can significantly improve the performance of LLMs on factual QA benchmarks especially when involving complex reasoning processes, which demonstrates the necessity and effectiveness of KGR in mitigating hallucination and enhancing the reliability of LLMs.
Introduction
LLMs can hallucinate factual statements during multi-step reasoning, while query-only KG retrieval misses intermediate facts. KGR autonomously extracts, verifies, and retrofits draft-response claims using KG knowledge, improving factual QA performance especially for complex reasoning.
- Problem: Factual hallucination is a severe challenge because LLMs frequently generate unsupported false statements about factual information.The introduction illustrates this issue with an incorrect claim about Nicolas Chopin’s birth date.
- Problem: Existing KG-augmented methods retrieve facts mainly for entities explicitly mentioned in the query, leaving reasoning-stage hallucinations insufficiently addressed.Intermediate facts may concern entities absent from the original question.
- Method: KGR retrofits initial LLM draft responses throughout reasoning by extracting claims, retrieving and selecting KG facts, verifying claims, and refining responses.The framework uses LLMs to autonomously extract, validate, and refine factual statements without manual effort.
- Contribution: The paper presents KGR as an LLM-and-KG framework for extracting, verifying, and refining factual knowledge across the entire reasoning process.Its stated contribution is to integrate KG-based factual checking into LLM reasoning rather than only the initial query.
- Results: KGR improves LLM performance on factual QA benchmarks, especially when questions involve complex reasoning processes.Experiments cover three representative LLMs and three benchmarks with different reasoning difficulties.
Related Work
Prior retrieval-augmentation methods either supply query-related information before generation or edit generations using external evidence afterward. KGR instead uses knowledge graphs to retrofit model responses while targeting factual errors beyond the question’s explicit entities.
- Retrieval Augmentation: Retrieve-before-generation methods provide additional query information, whereas retrieve-after-generation methods automatically edit model outputs using evidence.The related-work discussion distinguishes these two broad retrieval-augmentation categories.
- Retrieval Augmentation: Query-focused retrieval cannot guarantee correctness for question-unrelated portions of generated responses.This limitation motivates checking facts that arise during generation and reasoning.
- Retrieval Augmentation: KGR uses knowledge graphs as the knowledge base for retrofitting model-generated responses and reducing hallucination risk.It belongs to the retrieve-after-generation line while replacing web evidence with KG knowledge.
- KG-Enhanced LLM: Traditional KG-enhanced approaches often require dedicated architectures and model-specific training, which is costly for contemporary LLMs.Recent methods instead inject knowledge during inference.
Retrofitting
KGR retrofits an LLM’s draft response through a chain that extracts claims, retrieves and selects relevant KG facts, verifies claims, and revises the response. The process uses LLMs throughout and can be iterated to align generated facts with KG knowledge.
- KGR applies an Extraction–Detection–Selection–Verification–Retrofitting chain to automatically mitigate factual hallucinations in generated responses.The framework takes a query and draft response, then processes claims and KG evidence before producing a refined response.
- Claim Extraction: Claim extraction decomposes a draft response into atomic factual claims that can be verified separately.The framework uses an LLM to extract factual claims requiring validation from the generated draft.
- Entity Detection and Knowledge Graph Retrieval: Entity detection identifies critical entities in extracted claims, enabling retrieval of their local KG subgraphs as triples.KGR uses LLM prompting for entity detection, while prior supervised approaches require KG-specific training and generalize poorly across scenarios.
- Fact Selection: Fact selection partitions retrieved triples into chunks and uses an LLM to retain critical facts, limiting irrelevant context during claim verification.This design addresses long-context and context-window constraints while avoiding text-to-SQL requirements that can limit triple recall.
- Claim Verification: Claim verification compares each generated claim with selected KG facts and produces revision suggestions for response retrofitting.The final retrofitting step incorporates suggestions across claims, and the combined procedure can be repeated to improve factual alignment.
Experiments
KGR is evaluated across factual QA datasets spanning simple, complex, and open-domain multi-hop reasoning, with comparisons against prompting, web-based critique, and question-relevant KG retrieval. Results show consistent gains, while error analysis identifies retrieval-stage tradeoffs and entity detection and fact selection as key weaknesses.
- Overall Results: KGR improves F1 over QKR by at least 6.2 on Mintaka and 1.1 on HotpotQA, both requiring complex reasoning.The comparison attributes these gains to verifying facts used during reasoning through a chain-of-verification.
- Overall Results: KGR generalizes across compact, aligned, and misaligned language models and remains robust in HotpotQA’s open-domain setting.On HotpotQA, KGR leverages searched triples when available and parametric knowledge when no evidence is returned; with Vicuna 13B, it outperforms CoT and QKR.
- Overall Results: KGR consistently outperforms competing methods across three factual QA datasets and various conditions.The evaluation reports EM and F1 on Simple Question, Mintaka, and HotpotQA, with Table 1 covering ChatGPT and text-davinci-003 and Table 2 covering Vicuna 13B.
- Case Study: A multi-round case study shows KGR iteratively correcting successive factual errors in a multi-hop response.It first corrects Alex Shevelev’s death location from Moscow to Rome, then corrects the claim about Rome and the Central Federal District.
- Error Analysis: Error analysis identifies entity detection and fact selection as the main sources of inaccuracies, while claim extraction, verification, and retrofitting are more reliable.Entity detection may retrieve overly broad entities, and fact selection must isolate critical triples from noisy retrieved information.
- Error Analysis: Retrieval settings expose a precision–recall tradeoff: more retrieved triples raise recall but reduce precision, while smaller chunks also lower precision and raise recall.Chunk size has minimal impact except at 100, which may worsen long-distance dependency modeling; random triple ordering has little effect.
Conclusion
KGR is presented as a knowledge graph-based retrofitting framework for mitigating factual hallucination during LLM reasoning. Experiments show improved factual QA performance, especially for complex reasoning, supporting its effectiveness and potential to enhance reliability.
- KGR mitigates factual hallucination during LLM reasoning using factual knowledge stored in knowledge graphs.
- Experiments show that KGR significantly improves LLM performance on factual QA benchmarks, particularly for complex reasoning.
- The results demonstrate the necessity and effectiveness of KGR for mitigating hallucination and enhancing LLM reliability.
Impact of Multi-Turn Retrofitting
Multi-turn retrofitting exposes a contrast between KGR and search engine-based retrofitting. KGR corrects most factual errors in the first turn and relies on a more reliable knowledge source, whereas inaccurate search results can cause erroneous revisions.
- Inaccurate search-engine retrieval can cause erroneous revisions of factually correct claims, especially during multi-turn retrofitting.
- Most factual errors are retrofitted in the first KGR turn, while its effectiveness tends to remain consistent across subsequent turns.
- Figure 8 compares KGR and CRITIC behavior under multi-turn retrofitting.
- KGR uses a knowledge graph as its foundational source, providing reliable and accurate factual information for retrofitting.
Effectiveness with Entity Detection.
The paper compares entity-detection methods for identifying entities needed in claim verification. LLM-based entity extraction remains insufficient relative to alternative techniques, motivating further research.
- LLMs remain insufficient for extracting entities needed for claim verification compared with alternative techniques.
- The paper identifies entity extraction for claim verification as an opportunity for future research.
- Table 4 compares KGR with Wikifier and SpaCy for entity detection, including a hybrid Mix approach combining Wikifier, LLM, and SpaCy.