GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework

1. Introduction

• Context-based hallucinations are especially limiting where certainty matters, such as medical diagnosis; detecting them and pinpointing suspect response content are important goals.The work excludes hallucinations that deviate from the LLM’s original training data.Source: §1 · p. 1
• Evaluation approaches have progressed from N-gram metrics to LLM-based metrics, while RAG and KG reasoning have also been proposed to ground generated responses.These grounding approaches do not eliminate the need to evaluate LLM outputs.Source: §1 · p. 1
• GraphEval represents LLM outputs as knowledge graphs to identify specific information that may be hallucinated.The framework is presented as the first KG-based LLM hallucination evaluation approach, to the authors’ knowledge.Source: §1 · p. 1
• GraphEval combined with state-of-the-art hallucination detectors improves classification accuracy on benchmarks, while GraphCorrect can rectify a significant proportion of hallucinations.The introduction identifies hallucination correction as a further contribution.Source: §1 · pp. 1–2

2. Problem statement

• The target is closed-domain detection: checking an LLM output against the grounding context included in its prompt.Open-domain factuality against all knowledge in the world is not explored.Source: §2 · p. 2
• Hallucination evaluation is binary: an output is inconsistent if it contains at least one factual inconsistency with the provided context.The paper uses factual, consistent, grounded, and faithful interchangeably to mean no hallucinations with respect to context.Source: §2 · p. 2
• Correction is explored as an extension, using detection frameworks to select outputs for correction and then evaluate them, rather than a directly labeled dataset.The authors state that correction is not the study’s primary focus.Source: §2 · p. 2

3. Related work

• N-gram and embedding-based metrics score similarity to reference text, an objective that often fails to capture hallucination-detection intricacies.N-gram metrics have also been reported to perform poorly at factual inconsistency detection.Source: §3 · p. 2
• NLI methods use entailment signals for consistency prediction, while sampled-output methods such as SelfCheckGPT and ChatProtect can be prohibitively computationally expensive.SelfCheckGPT and ChatProtect require generating many responses from the LLM.Source: §3 · p. 2
• G-Eval and GPTScore use a separate LLM for evaluation, but their scores are harder to interpret as probabilities and often need extra inconsistency-classification steps.G-Eval uses user-defined criteria; GPTScore treats evaluation as conditional generation.Source: §3 · p. 2
• FactScore and SAFE evaluate atomic facts against external knowledge sources, enabling precise localization of hallucinations in responses.FactScore uses sources such as Wikipedia or scientific literature, whereas SAFE uses a search engine.Source: §3 · p. 2
• FactGraph decomposes summaries and supporting documents into structured meaning representations, and is focused on summarization.Its representations encode core semantic concepts and relations.Source: §3 · p. 2
• GraphEval’s depicted pipeline constructs a KG from the LLM output, checks each triple against context, and returns flagged triples with an overall decision.The figure describes use of an out-of-the-box detection method such as an NLI model.Source: Figure 1 · p. 3

4. GraphEval: Our evaluation method

• GraphEval is a preprocessing step that can feed its KG-based representation into existing hallucination detection methods.The paper uses NLI models to compare raw-model performance with performance supplemented by GraphEval.Source: §4 · p. 3
• The method has two stages: construct a KG from the output, then assess whether each triple is consistent with the provided context.An output is inconsistent if any triple is identified as not grounded.Source: §4 · p. 3
• GraphEval uses a single LLM-based KG-construction stage and then feeds each triple with its grounding context to an out-of-the-box NLI model for a hallucination probability.An example is classified as inconsistent if any triple’s probability exceeds 0.5; the NLI models are selected as computationally cheap while achieving state-of-the-art results.Source: §4 · p. 3
• Representing entity relationships in a KG addresses the difficulty of retaining entity references across sentences, such as a person named once and later referred to by a pronoun.The paper contrasts this with FactGraph, which represents both the output and context, whereas GraphEval represents only the output and aims to preserve its information.Source: §4 · p. 4
• GraphEval is designed for domains beyond summarization, unlike FactGraph, and uses LLMs while representing only the output as a KG.FactGraph decomposes both output and context into semantic concepts and relations; GraphEval aims to preserve information from the raw output.Source: §4 · p. 4
• GraphEval checks each piece of information in the LLM output and returns specific ungrounded triples, making the decision explainable and identifying response sections that may be untrustworthy.The method is presented as less computationally expensive than other LLM-based metrics because it needs one LLM call for KG construction and does not require the large context documents as input to that LLM.Source: §4 · p. 4

• KG construction identifies entities, resolves mentions that refer to the same entity, and extracts semantic relationships between entities.These are the three stages used to structure information from unstructured text.Source: §5 · p. 4
• The authors use chain-of-thought prompting and in-context demonstrations to construct KGs without fine-tuning.Source: §5 · p. 4
• GraphEval’s contribution is applying KG construction to hallucination detection, rather than introducing a new KG construction method.Source: §5 · p. 5
• The authors describe their prompting approach as easy to implement with any LLM and less computationally intensive than PiVE’s iterative KG improvements.Users may also choose another KG construction method.Source: §5 · p. 5

• GraphCorrect flags triples with a probability above 0.5, if any, as likely to contain hallucinations.Source: §6 · p. 5
• An LLM first corrects each flagged triple using the context; the corrected triple then guides selective replacement in the original output.Source: §6 · p. 5
• The method addresses identified hallucinations separately and leaves other text untouched, while its intermediate steps keep the context and original output from being processed together by an LLM.The authors say this safeguards against adding extra information or losing information from the output.Source: §6 · p. 5

7.1. Benchmarks

• The benchmarks cover human-annotated summary consistency against grounding context: SummEval uses consistency ratings, while QAGS records sentence-level judgments by three annotators.For QAGS, a summary is considered consistent only if all its sentences are judged consistent.Source: §7.1 · p. 5

7.2. NLI models in GraphEval

• The second-stage NLI models are HHEM, TRUE, and TrueTeacher; HHEM uses 738 MB and has shorter runtime, while TRUE requires 45.5 GB and TrueTeacher is the same size as TRUE.Source: §7.2 · pp. 5–6

7.3. Experimental settings

• Experiments requiring an LLM used Claude 2 through Amazon Bedrock with default temperature 1, top_p 1, and top_k 250.Source: §7.3 · p. 6

7.4.1. Hallucination detection with GraphEval

• 6.2 (SE=1.3) is the average balanced-accuracy improvement from adding GraphEval preprocessing to NLI models, weighted by dataset sample counts.The comparison covers HHEM, TRUE, and TrueTeacher on SummEval, QAGS-C, and QAGS-X.Source: §7.4.1 · p. 6, Table 2 · p. 6
• GraphEval detection is hypothesized to add more value for longer outputs; the authors also note that it identifies specific triples inconsistent with the context.For very short outputs, they hypothesize there may be fewer facts to check and that KG construction may lose sentence intricacies.Source: §7.4.1 · p. 6

7.4.2. Hallucination correction with GraphCorrect

• GraphCorrect outperforms direct prompting on believed corrected hallucinations in every task except two QAGS-X cases.The authors hypothesize that it brings most value to longer texts with more complex structure.Source: §7.4.2 · p. 7, Table 4 · p. 7
• GraphCorrect produces corrected summaries closer to the original LLM outputs than Direct Prompt across the examined scenarios, measured with ROUGE-1, ROUGE-2, and ROUGE-L.Source: §7.4.2 · p. 7, Table 3 · p. 7

• GraphEval addresses closed-domain hallucinations by checking consistency with provided context; open-domain detection would require retrieving external sources as grounding information.Source: §8 · p. 7
• The authors expect improved KG construction from unstructured text to improve GraphEval’s first stage and evaluation performance.Source: §8 · p. 7
• KG construction may lose information, but the method rarely reduces balanced accuracy and can identify inconsistent triples when comparable to baselines.Source: §8 · p. 7
• GraphCorrect’s effectiveness should be assessed manually rather than through evaluation frameworks that measure only believed corrections.Source: §8 · pp. 7–8

9. Conclusion

• GraphEval uses LLMs to extract information from text into KG triples that out-of-the-box hallucination detection methods can evaluate, while identifying inconsistent triples.Source: §9 · p. 8
• 6.2 (SE=1.3) average improvement in balanced accuracy with GraphEval and state-of-the-art NLI models across three popular hallucination benchmarks.Source: §9 · p. 8
• GraphCorrect addressed the majority of hallucinations in LLM outputs while maintaining extremely high similarity to the original texts.Source: §9 · p. 8

A. KG Construction Prompt

• The KG-construction prompt requests entity detection, coreference resolution, and relation extraction, with output as Python-formatted triples.It also instructs the model to include all information and discard triples that do not form a sensible sentence.Source: A. KG Construction Prompt · pp. 10–11

B. Hallucination correction (step 1)

• The first correction step gives the potentially incorrect triple and context to an LLM to produce a corrected triple.Source: B. Hallucination correction (step 1) · p. 11
• The first-step prompt requires the model to return only the corrected triple.Source: B. Hallucination correction (step 1) · p. 11

C. Hallucination correction (step 2)

• The second step replaces the old triple’s information in the initial output with information from the corrected triple, leaving other context unchanged.Source: C. Hallucination correction (step 2) · p. 12

D. Hallucination correction without a KG

• The prompt directs the model to correct factual errors using the context without changing other parts of the summary.Source: D. Hallucination correction without a KG · p. 12
• It requests minimal edits, keeping the summary no longer than the original and preserving as much wording as possible.Source: D. Hallucination correction without a KG · p. 12
• The requested output is only the corrected summary.Source: D. Hallucination correction without a KG · p. 12