PaperlayerView exact revision on arXiv

Source-linked AI summary

RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Models

Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, Zheng Zhang

arXiv:2405.14486v1cs.CL

TL;DR

LLMs can generate hallucinated content, motivating finer-grained and scalable detection across varied tasks and reference conditions. RefChecker extracts claim-triplets from responses and checks them against references, achieving stronger benchmark performance and human alignment than prior approaches. Its scope is currently centered on plain text in general domains, with advanced reasoning-related hallucinations and other formats or domains left for future work.

  • Problem

    Hallucination detection requires an appropriate comparison unit, a comprehensive benchmark, and an automated framework that scales across diverse tasks.

  • Method

    RefChecker extracts knowledge triplets from LLM responses and evaluates each claim-triplet against a reference using configurable proprietary or open-source extractors and checkers.

  • Results

    6.8 to 26.1 points of improvement over the best alternative were achieved on the benchmark, while claim-triplet checking gained 4 to 9 points over other granularities.

  • Takeaways & Limitations

    Claim-triplets provide a fine-grained basis for automated hallucination detection that scales across tasks and produces results strongly aligned with human evaluation.

  • Takeaways & Limitations

    RefChecker primarily targets plain text in general domains, while reasoning- and limited-context hallucinations may be difficult for triplets and extensions to other formats or domains remain future work.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown impressive capabilities but also a concerning tendency to hallucinate. This paper presents RefChecker, a framework that introduces claim-triplets to represent claims in LLM responses, aiming to detect fine-grained hallucinations. In RefChecker, an extractor generates claim-triplets from a response, which are then evaluated by a checker against a reference. We delineate three task settings: Zero, Noisy and Accurate Context, to reflect various real-world use cases. We curated a benchmark spanning various NLP tasks and annotated 11k claim-triplets from 2.1k responses by seven LLMs. RefChecker supports both proprietary and open-source models as the extractor and checker. Experiments demonstrate that claim-triplets enable superior hallucination detection, compared to other granularities such as response, sentence and sub-sentence level claims. RefChecker outperforms prior methods by 6.8 to 26.1 points on our benchmark and the checking results of RefChecker are strongly aligned with human judgments. This work is open sourced at https://github.com/amazon-science/RefChecker

1 Introduction

RefChecker addresses the need for fine-grained, scalable hallucination detection by representing response claims as knowledge triplets and checking them against references. Its benchmark and experiments show improved alignment with human judgments and stronger performance than prior granularities and methods.

  • 4 to 9 points of improvement show that claim-triplet checking outperforms response-, sentence-, and sub-sentence-level granularities.Triplets provide a finer-grained unit for detecting local hallucinations in complex responses.
  • The benchmark covers three context settings and includes 11,000 annotated claim-triplets from 2,100 responses generated by seven LLMs.It was designed to evaluate hallucination across diverse tasks and varying reference quality or availability.
  • Hallucination was most pronounced with Zero Context, followed by Noisy Context and Accurate Context.Human evaluation found contradiction rates decreasing from 25% without contextual cues to 13% with noisy context and 6% with accurate context.
  • RefChecker decomposes an LLM response into claim-triplets, then evaluates each triplet against a reference to assign a hallucination label.The framework consists of an extractor and a checker and can use proprietary or open-source models.

2 Related Work

Prior hallucination-detection work varies in claim granularity, checking resources, and benchmark scope. RefChecker addresses these gaps with triplet-based claims, broader settings and tasks, expanded human evaluation, and support for open-source models.

  • RefChecker uses knowledge triplets rather than sentences or sub-sentences, while supporting both proprietary and open-source extractors and checkers.The comparison also highlights broader context settings, more diverse tasks, and evaluation across more LLMs and responses.
  • Response-level checking can miss local hallucinations in long responses, motivating finer-grained claim representations.Sentence- and sub-sentence-level methods address granularity but remain limited, leading RefChecker to employ knowledge triplets.
  • Hallucination checking research includes resource-free methods based on self-contradiction, uncertainty, or self-consistency, whose effectiveness depends on checker capabilities and sampling.RefChecker instead compares extracted claims with references and distinguishes factuality and faithfulness settings.
  • Existing benchmarks often focus on response-level detection, narrow domains or tasks, or factuality alone, whereas RefChecker covers diverse tasks and domains.Its human evaluation also examines more LLMs and responses than prior benchmarks.

3 REFCHECKER: Definition and Benchmark

RefChecker defines hallucinations as claims unsupported by references and benchmarks detection across Zero, Noisy, and Accurate Context settings. Its triplet annotations distinguish entailment, contradiction, and neutral claims, and human evaluation shows contradiction rates decline as contextual information improves.

  • 3 REFCHECKER: Definition and Benchmark: Hallucination detection compares LLM claims against factual references, with contextual setting, checking granularity, and label categorization determining the evaluation design.The benchmark and annotation process operationalize this comparison across the three context settings.
  • 3.1 Context Settings and Benchmarks: Zero Context uses NaturalQuestions with Wikipedia paragraphs as reference proxies for stable internal knowledge, while Noisy Context uses MS MARCO retrieval documents.Accurate Context uses a databricks-dolly-15k subset covering summarization, closed-QA, and information extraction.
  • 3.1 Context Settings and Benchmarks: The benchmark separates Zero Context, Noisy Context, and Accurate Context to cover internal knowledge, retrieved potentially noisy information, and prompt-provided noise-free references.It contains 300 examples, with 100 examples for each setting.
  • 3.2 Claim-Triplets and Definition of Hallucination: RefChecker represents claims as (head_entity, relation, tail_entity) triplets and labels them Entailment, Contradiction, or Neutral against a reference.Neutral applies when the reference is insufficient to verify a claim; unmentioned reference content is not evaluated.
  • 3.3 Human Evaluation: 25% to 13% to 6%: average Contradiction rates decrease from Zero Context to Noisy Context to Accurate Context.The human evaluation covered 2.1k responses from seven LLMs and annotated 11k claim-triplets, with 95.0% agreement on double-annotated items.

4 REFCHECKER Framework

RefChecker decomposes responses into claim-triplets, checks each triplet against a reference, and optionally aggregates the resulting labels. Its implementation combines LLM and smaller NLI-based checker families, including open-source variants.

  • 4 REFCHECKER Framework: RefChecker’s two-stage pipeline extracts triplets from an LLM response, verifies each with a checker, and optionally aggregates the triplet categories.This design is intended to support finer-grained detection and more accurate evaluation than checking the original response directly.
  • 4.2 Checker: The checker options include GPT-4 and Claude 2, smaller NLI models such as AlignScore and RoBERTa-NLI, and representation-based classifiers built on open models.Long references are split for small-model context windows and aggregated afterward.
  • 4 REFCHECKER Framework: Triplet labels can be aggregated into category ratios and macro-averaged across responses, or mapped to scalar values such as −1, 0, and 1.For response-level comparison, RefChecker flags a response as contradictory if any triplet is contradictory.

5 Experiments

The experiments compare claim granularities and evaluate RefChecker against prior hallucination-detection methods and human judgments across multiple context settings. Claim-triplet checking performs best among granularities, while the Claude 2 + GPT-4 configuration strongly outperforms prior methods and correlates well with human rankings.

  • 5.1 Comparing with Other Granularity: The granularity comparison breaks responses into response-, sentence-, sub-sentence-, and triplet-level claims, then aggregates predictions to response level using zero-tolerance max-pooling.The ordering is Entailment < Neutral < Contradiction, so the highest-severity claim determines the response-level outcome.
  • 5.1 Comparing with Other Granularity: 10 points: Triplet-level checking exceeds response-level checking in average macro-F1, while sentence-level gains 5 points and sub-sentence checking drops 3.5 points.The sub-sentence decline is attributed partly to overlapping claims and poorer extraction quality from flexible sub-sentences.
  • 5.2 Comparing with Other Methods: 6.8 to 26.1 points: Claude 2 + GPT-4 outperforms the best alternative, FacTool, across all three context settings.RefChecker significantly outperforms previous methods with both proprietary and open-source models; it also outperforms SelfCheckGPT in 11 of 15 combinations, or 73%.
  • 5.3 Evaluation on REFCHECKER Framework: Claude 2 + GPT-4 shows the strongest correlations with human evaluation across settings, while Mistral + NLI/AlignScore is the strongest nonproprietary combination.Replacing the extractor with Mistral causes only a marginal decline, and Mistral-RepC remains robust across extractors.

6 Analysis and Discussion

The analysis evaluates extraction and checker components, identifies configuration strengths and context-dependent weaknesses, and discusses limitations affecting claim coverage, source control, attribution, and deployment.

  • 6.1 Evaluation on Extractors: 93.7% precision agreement and 91.9% recall agreement show strong alignment between GPT-4 Turbo’s automatic extraction evaluation and human evaluation.The pipeline labels extracted claims for faithfulness and completes missing claims to compute precision, recall, and F1.
  • 6.2 Evaluation on Checkers: Claude 2’s neutral F1 is below 20%, reflecting a tendency to label neutral claims as contradiction through reliance on internal knowledge.This tendency is especially visible for the MS MARCO and Dolly datasets.
  • 6.2 Evaluation on Checkers: Mistral-based checkers often achieve the best performance, but no single variant wins across all settings and their main weakness is Noisy Context.The authors attribute this weakness partly to a mismatch between short training paragraphs and references spanning thousands of tokens, requiring splitting and later aggregation.
  • 6.2 Evaluation on Checkers: Checker performance has clear gaps between Noisy or Accurate Context and Zero Context, leaving substantial room for improvement.The context-dependent differences indicate that checker reliability varies with reference availability and quality.
  • 6.3 Future Work: The triplet format may omit important semantics, including facts whose truth changes with time, while source attribution remains rudimentary and model checkers may rely on internal knowledge.The authors also identify needs for stronger checkers, custom reference retrieval, and faster deployment.

7 Conclusion

RefChecker is a unified framework for fine-grained hallucination detection that represents response claims as knowledge triplets and evaluates them against references. Its automated extractor-checker pipeline is calibrated to human annotations and achieves superior performance to prior methods.

  • 7 Conclusion: RefChecker extracts knowledge triplets from LLM responses and evaluates them against references to determine hallucination categories at fine granularity.The framework pairs an extractor with a checker in an automated pipeline.
  • 7 Conclusion: The calibrated extractor-checker pipeline achieves superior performance compared with prior hallucination-detection methods.The conclusion presents this as the framework’s overall empirical outcome.

Limitations

The authors identify limitations involving hallucinations caused by reasoning or limited context windows, which claim-triplets may handle poorly, and scope limited to plain text in general domains.

  • Reasoning- or context-window-driven hallucinations can be difficult for claim-triplets because they bias toward local contexts.
  • RefChecker primarily covers plain text in general domains, leaving tables, code, mathematics, and specialized domains for future extension.

Ethics Statement

The paper reports no negative ethical implications and argues that RefChecker can support responsible AI by identifying non-factual LLM content. Its benchmark uses public English datasets and trained expert annotators.

  • The authors report no negative ethical implications and suggest RefChecker can support responsible AI by identifying non-factual LLM responses.
  • The benchmark draws on publicly accessible English-language corpora, with human annotations conducted by six NLP experts.
  • The released fine-tuned models are based on Mistral-family models and are planned for public release under the Apache-2.0 License.
  • Benchmark curation filters inappropriate, ambiguous, low-quality, tabular, or context-deficient examples to improve annotation reliability.

A.4 Observations from Human Evaluation

Human evaluation shows that context quality, model family, copying behavior, and checker design shape hallucination outcomes across the benchmark.

  • Context Information is Critical: In Zero Context, LLMs produce more contradictions and unverifiable claims, while contextual settings reduce hallucinations but introduce sensitivity to noise.
  • GPT Family Steadily Improved Factuality: Across all three settings, factuality consistently improves from InstructGPT to ChatGPT to GPT-4, using lower contradiction rates as better.
  • Open Source LLM is Catching Up: Llama 2 70B Chat outperforms ChatGPT and surpasses Claude 2 across the evaluated settings, although proprietary models perform better on average.
  • Copy from Context is Safer: Higher claim-triplet copy rates from context correspond to increased likelihood of entailment in Noisy and Accurate Context settings.
  • RepC-LS performs best with a middle layer rather than the last layer, while RepC-LE performs better but RepC-LS uses less model size and data.
  • RepC-LS-svm outperforms RepC-LS-nn with fewer training examples.
  • The paper’s source-attribution approach matches response and reference spans through SimCSE embeddings, but remains limited by efficiency, boundary, and shallow-semantic matching issues.

C.1 Comparison on the REFCHECKER Benchmark

RefChecker is evaluated against prior hallucination-detection frameworks across Zero, Noisy, and Accurate Context settings using response-level classification and hallucination-rate correlations.

  • RefChecker aggregates claim-level labels into strict response-level binary classifications, treating any hallucinated claim as making the response non-factual.
  • The framework also compares response-level hallucination-rate correlations with human evaluation using Pearson and Spearman correlations.
  • Most RefChecker combinations outperform baselines by large margins across all five evaluation metrics on the benchmark.
  • The best results use GPT-4 as checker, while smaller AlignScore checkers still allow RefChecker to outperform baselines; fully open-source combinations can also outperform proprietary baselines.
  • On the SelfCheckGPT dataset, 11 of 15 RefChecker combinations, or 73%, outperform SelfCheckGPT.

D.1 Internal Knowledge Bias of Evaluated Model

The analysis separates evaluated models’ internal knowledge from their use of provided context, using interrogative knowledge checks and information masking. Results show distinct internal-versus-context reliance patterns across accurate and noisy settings, while checker behavior also varies in its reliance on internal knowledge.

  • Knowledge checking: Knowledge checking converts each extracted claim-triplet into an interrogative query and asks the evaluated model whether it possesses the corresponding knowledge.GPT-4-Turbo generates the queries using a designed prompt template.
  • Interpretation of labels: (No, Entailment) and (Unsure, Entailment) indicate reliance on contextual information, while (Yes, Contradiction) and (Yes, Neutral) indicate use of internal information.(No, Contradiction) suggests weaker processing of context and less reliable claim-triplets.
  • Knowledge checking: GPT-4-Turbo most often generates responses aligned with the reference in the accurate-context setting, whereas GPT-3.5-Turbo relies more on internal knowledge, producing contradictions or neutral relations.Claude 2 sometimes produces unsure but reference-neutral information, while Alpaca 7B resembles GPT-3.5-Turbo but seldom contradicts its internal knowledge.
  • Knowledge checking: Noisy context increases Neutral labels for all evaluated models compared with the accurate-context setting.The comparison is reported using the knowledge-checking results in Figures 21 and 22.
  • Checker internal knowledge bias: Information masking tests checker reliance on internal knowledge by replacing triplet information in the context with ‘####’; non-Neutral predictions indicate internal inference.In accurate-context masking, RoBERTa-NLI produces the most Neutral labels for GPT-3.5-Turbo and GPT-4-Turbo, at 62.64% and 53.70%, respectively.
Loading 2405.14486v1…