PaperlayerView exact revision on arXiv

Source-linked AI summary

Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification

Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov

arXiv:2403.04696v2cs.CLcs.AIcs.LG

TL;DR

LLM hallucinations can conceal erroneous claims within coherent text, motivating fact-checking that does not depend on external knowledge sources. The paper proposes token-level and claim-level uncertainty quantification, including CCP, and reports improved fact-checking across seven LLMs and four languages, with human evaluation competitive with FactScore.

  • Problem

    LLM hallucinations can produce coherent, persuasive false claims that are difficult for users to spot, while existing fact-checking commonly relies on external knowledge sources.

  • Method

    The paper uses token-level uncertainty from white-box LLM outputs, aggregates it over atomic claims, and introduces Claim Conditioned Probability to reduce uncertainty from claim choice and surface form.

  • Results

    CCP outperforms other methods for seven LLMs and four languages, with computational overhead potentially as low as 3% of LLM inference runtime.

  • Takeaways & Limitations

    Human evaluation finds the uncertainty-based pipeline competitive with FactScore while using only LLM outputs.

  • Takeaways & Limitations

    The implementation uses OpenAI GPT models for text segmentation and atomic-fact extraction, which may not be practical in real applications.

Abstract

from arXiv · show

Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. Such hallucinations can be dangerous, as occasional factual inaccuracies in the generated text might be obscured by the rest of the output being generally factually correct, making it extremely hard for the users to spot them. Current services that leverage LLMs usually do not provide any means for detecting unreliable generations. Here, we aim to bridge this gap. In particular, we propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification. Uncertainty scores leverage information encapsulated in the output of a neural network or its layers to detect unreliable predictions, and we show that they can be used to fact-check the atomic claims in the LLM output. Moreover, we present a novel token-level uncertainty quantification method that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. Our method Claim Conditioned Probability (CCP) measures only the uncertainty of a particular claim value expressed by the model. Experiments on the task of biography generation demonstrate strong improvements for CCP compared to the baselines for seven LLMs and four languages. Human evaluation reveals that the fact-checking pipeline based on uncertainty quantification is competitive with a fact-checking tool that leverages external knowledge.

1 Introduction

LLM hallucinations can hide false claims within coherent, persuasive text, creating a need for reliable claim-level fact-checking. The paper addresses this gap with token-level uncertainty quantification and the CCP method, evaluated across models and languages.

  • Motivation: LLM hallucinations are factually incorrect generations that can be coherent and persuasive, making occasional false claims difficult to detect.The paper highlights fictitious biographies and unfounded claims as examples.
  • Research gap: Existing fact-checking systems often rely on external knowledge sources, introducing source incompleteness and storage overhead.The paper instead argues that hallucination information can be extracted from the model output using uncertainty quantification.
  • Contributions: Claim-Conditioned Probability (CCP) is introduced as a token-level uncertainty score that improves over baselines for seven LLMs and four languages.The evaluation covers English, Chinese, Arabic, and Russian.
  • Evaluation: The paper constructs a biography fact-checking benchmark to evaluate claim-level uncertainty methods against FactScore.The benchmark addresses variability in LLM-generated outputs.
  • Contributions: The paper proposes fact-checking LLM generations by estimating token-level uncertainty and aggregating it for atomic claims.The framework maps potentially deceptive fragments back to the original response.

2 Related Work

Prior hallucination-detection and uncertainty methods commonly use external models, multiple samples, latent representations, or semantic diversity. This work instead studies token-level uncertainty from white-box LLM outputs for atomic-claim fact-checking.

  • Hallucination detection: Prior hallucination-detection methods use external models, sampled-output agreement, or extracted claims and verification.These approaches include Factool and methods based on sampling multiple black-box outputs.
  • Positioning: This work is the first to investigate token-level UQ for fact-checking using only a white-box LLM and its outputs.Its method post-processes token probabilities to quantify uncertainty for atomic claims and individual words.
  • UQ taxonomy: Information-based UQ methods use generated-token probability distributions, while latent-density methods require training-data access to fit external density models.The paper also distinguishes white-box methods from black-box methods that use generated texts only.
  • UQ taxonomy: Ensembling and Monte Carlo dropout require many predictions and are computationally intensive, while divergent samples complicate uncertainty estimation for claims.These methods are therefore difficult to apply to text fragments rather than complete generations.
  • UQ taxonomy: Self-evaluation can outperform original-prediction probability analysis but requires a second inference pass using the generated output.Semantic entropy and output-similarity methods instead measure diversity of generated meanings.

3 Fact-Checking Pipeline

The pipeline decomposes generated text into atomic claims, aligns each claim with token probabilities, and aggregates token-level uncertainty into claim-level scores. Claims exceeding a validation-set threshold are highlighted as potentially unreliable.

  • Claim extraction: The pipeline first splits generated text into atomic claims using a model specialized or fine-tuned for claim segmentation.The experiments implement this step through the OpenAI Chat API following FactScore.
  • Uncertainty aggregation: Each atomic claim is aligned with its corresponding token sequence and probability distributions in the original text.Token-level uncertainty scores are then computed and aggregated into a claim-level uncertainty score.
  • User-facing decision: Claims whose uncertainty exceeds a validation-set threshold are highlighted for users as unreliable.Tokens belonging simultaneously to reliable and unreliable claims are not highlighted.

4 Uncertainty Quantification

This section introduces uncertainty quantification methods for claim-level fact-checking, emphasizing that CCP removes uncertainty from claim type and surface form while retaining uncertainty about the factual value expressed.

  • Claim-level uncertainty: Claim-level uncertainty aggregates token-level scores over the words corresponding to an atomic claim.The paper defines an atomic claim as a set of word positions and later obtains claim-level uncertainty by multiplying word-level CCP values.
  • Claim-level UQ baselines: Existing baselines include Maximum Probability, Perplexity, Maximum Entropy, and P(True), which estimate uncertainty from token probabilities, entropy, or a model’s truthfulness judgment.P(True) asks the LLM whether the generated claim is true, while Maximum Entropy uses the highest token entropy within the claim.
  • Motivation and theoretical background: Two uncertainty types are irrelevant to fact-checking: which claim type to generate and which surface form to use; factual uncertainty concerns the specific information relayed.Different claim orders, synonyms, and hypernyms can preserve factual content, whereas uncertainty among values such as painting, acting, and sculpture may signal factual error.
  • Claim Conditioned Probability: Claim Conditioned Probability (CCP) ignores claim-type and surface-form uncertainty and focuses on uncertainty about the particular claim value expressed by the model.The method represents meaning and claim type explicitly, then conditions the meaning probability on the claim type.
  • CCP implementation: CCP compares top-K alternative word realizations with the original prediction using natural-language inference labels: entailment, contradiction, or neutrality.Whole-word distributions may require beam search because transformer LLMs often generate sub-word tokens; claim-level CCP is then obtained from the product of word-level CCPs.

5 Benchmark for Evaluation of Claim-Level UQ Methods

The benchmark evaluates claim-level uncertainty methods on unrestricted multilingual biography generations, using automatic FactScore labels alongside manually annotated claims. It supports scalable evaluation while documenting the prevalence of hallucinated claims.

  • Benchmark design: The benchmark uses unrestricted biography generation to evaluate whether uncertainty scores can identify hallucinated atomic claims.This design aims to approximate real-world LLM outputs rather than relying on a proxy model’s generations.
  • Benchmark design: Responses were generated for 100 biography prompts in English, Chinese, Arabic, and Russian, with a maximum length of 256 tokens.The evaluated models included Vicuna 13b, Mistral 7b, and Jais 13b for English, with additional multilingual generations described in the benchmark setup.
  • Annotation and scoring: Claims were decomposed into atomic statements and matched back to generated tokens to obtain the logits needed for uncertainty scoring.Only successfully matched claims were retained for analysis.
  • Annotation and scoring: English claims were automatically labeled as supported or unsupported by FactScore using retrieval plus ChatGPT with Wikipedia as an external knowledge source.Manual annotations were also collected across English, Arabic, Russian, and Chinese datasets, with two annotators checking each statement.
  • Dataset characteristics: 6–29% of claims were hallucinations across the resulting datasets, while most model-output claims were correct.FactScore automatically annotated English claims, and the evaluation pipeline is illustrated in Figure 6.

6 Experiments

Experiments compare claim-level uncertainty methods across English and multilingual biography generations, then examine sentence position, ablations, qualitative behavior, and runtime. CCP generally outperforms baselines and remains computationally practical with a smaller NLI model.

  • English FactScore results: CCP outperforms all other uncertainty techniques for each considered LLM, except PR-AUC for GPT-3.5-turbo, where P(True) performs best.The GPT-3.5-turbo exception may reflect the limited token alternatives and logits available through the OpenAI API.
  • Sentence-position analysis: CCP generally outperforms other methods as more sentences are considered, while ROC-AUC quality decreases with increasing generation length.The reported exceptions occur for GPT-3.5-turbo when only the first two or first five sentences are evaluated.
  • Manual-annotation comparison: 77.2% accuracy and 0.72 ROC-AUC describe FactScore’s automatic annotation against manual English labels.The paper reports that FactScore’s mistakes are analyzed separately in Appendix C.3.
  • Manual-annotation comparison: 0.06 ROC-AUC is CCP’s improvement over FactScore itself under human annotation, while uncertainty methods perform slightly better with human labels than FactScore labels.The authors characterize UQ techniques as a strong alternative to external-knowledge fact-checking tools for hallucination detection.
  • Multilingual results: CCP outperforms baselines for Chinese, Arabic, and Russian, including a 0.05 ROC-AUC advantage on Arabic and Jais and on Russian Vikhr generations.Metrics are low for Arabic GPT-4 generations, which the paper attributes to a small ratio of nonfactual claims.
  • Ablation studies: The product aggregation performs best, a 22M-parameter NLI model retains strong performance, and standard preceding-claim context outperforms shorter or longer alternatives.No context lowers ROC-AUC by 0.02, while longer contexts lower it by more than 0.07.
  • Ablation studies: Excluding functional words improves ROC-AUC by 0.03, while reducing CCP alternatives from K = 10 to K = 5 lowers ROC-AUC by 0.02 and performance plateaus at K = 8.Further decreasing K reduces performance even more.

7 Conclusion

The paper presents a token-level uncertainty-based pipeline for fact-checking and hallucination detection in LLM generations. Human evaluation finds it competitive with FactScore while requiring only LLM outputs.

  • The proposed approach performs fact-checking and hallucination detection using token-level uncertainty quantification.
  • Human evaluation finds similar or better results than FactScore, despite using only LLM outputs rather than an external knowledge source.
  • CCP is a computationally efficient claim-level and token-level uncertainty method that outperforms baselines across seven LLMs and four languages.Its computational overhead can be as low as 3% of LLM inference runtime.

Limitations

The paper identifies limitations involving the entailment classifier, implementation cost, human evaluation, token-level analysis, calibration, and generation control. These constraints define important boundaries for applying the method.

  • The entailment classifier was pretrained for a slightly different use case, so its performance across diverse domains and genres needs further analysis.
  • The implementation uses OpenAI GPT models for text segmentation and atomic-fact extraction, which may not be practical in real applications.
  • Part of the experimental evidence relies on subjective human evaluation, although larger-scale studies with greater annotator overlap could strengthen the results.
  • The method treats tokens as analysis units, while larger units such as noun or verb phrases may be more linguistically motivated.
  • The work does not consider calibration of CCP scores, although CCP could be post-calibrated like other probabilities or probability surrogates.
  • The approach detects potentially spurious generations but does not modify generation to exclude them, and simply removing uncertain claims may produce incoherent sentences.

Ethical Considerations

The paper frames the method as an aid for human awareness rather than a complete hallucination solution. It also cautions that internal uncertainty can miss or mischaracterize claims and may be misused.

  • The method reflects the LLM’s internal model state, so it has limited utility for claims beyond the model’s time cutoff.
  • Factually false training data and prompt variations can undermine the reliability of uncertainty-based fact-checking.
  • The solution highlights risky text spans for human consideration rather than eliminating hallucinations, supporting human-machine collaboration.
  • The approach could be misused to unfairly moderate content, so researchers and users are urged to exercise caution.

C.2 Datasets and Statistics

The experiments construct biography datasets across languages and evaluate CCP through automatic and manual annotation, aggregation, NLI-context, functional-word, and beam-count comparisons. CCP’s performance is robust across NLI models and generally superior to alternatives.

  • Datasets and statistics: The benchmark includes automatically annotated English biographies and manually annotated datasets, with additional Arabic, Chinese, and Russian biography data.Arabic, Chinese, and Russian datasets were created with language-specific generation and claim-extraction procedures.
  • Datasets and statistics: Jais 13b biographies are approximately half the length of GPT-4 biographies, average nine claims, and include seven empty biographies among 100 names.
  • Datasets and statistics: FactScore annotations can disagree with manual labels because of hallucination, missing context, and difficulties interpreting differently formulated information.
  • Ablation studies: CCPprod performs best among the four aggregation methods tested on Vicuna 13b and is selected for the final CCP formula.
  • Ablation studies: CCP performance depends minimally on the selected NLI model, with a 22M-parameter CrossEncoder achieving the highest performance.
  • Ablation studies: CCPclaim pref provides better results than sentence-prefix and no-context alternatives, while CCPconfident improves functional-word handling and Maximum Probability performs worse when functional words are specially handled.

E Additional Experimental Results

Additional experiments examine how claim support and uncertainty-quality measurements vary with generation length, token options, language, and computational cost. CCP maintains an advantage over alternatives on Chinese data and correctly reflects declining factual support in longer generations.

  • Generation length: As generation length increases, Vicuna 13b produces a lower percentage of supported claims, possibly because it generates additional facts.The passage reports this relationship for Figure 8.
  • Cross-language performance: CCP outperforms alternative methods over the entire generation length on Chinese Yi 6b data in ROC-AUC.This result is described as consistent with results for other English models.
  • Evaluation settings: Figure 7 plots ROC-AUC between FactScore classes and claim-level CCP against n, the number of token options considered.The caption defines n as the number of token options in the LLM probability distribution.
  • Evaluation settings: Figure 8 plots the percentage of supported claims against the number of sentences used to restrict generation for English Vicuna 13b.The figure varies the sentence restriction during generation.
  • Evaluation settings: Figure 9 compares token-level uncertainty methods by ROC-AUC on Chinese data, separately for facts from the first 2, 5, and all sentences.The results are divided into bins based on how many sentences are considered.
  • Computational cost: A single generation and uncertainty-method evaluation required 12 days of Nvidia A100 GPU compute time.English-model API and biography-generation costs were additionally reported as $40 and $13, respectively.

G Examples of Results Obtained with CCP and the Maximal Probability Baseline

Examples compare Claim-Conditioned Probability with Maximum Probability across three figures. CCP correctly detects wrong facts in all reported examples, while Maximum Probability highlights correct facts instead.

  • Example comparisons: Across the examples, CCP correctly detects wrong facts, whereas Maximum Probability mistakenly highlights correct facts.The comparison is reported for Figures 10, 11, and 12.
  • Example comparisons: Figure 10 compares the Maximum Probability baseline with Claim-Conditioned Probability.The caption identifies CCP as the authors’ method.
  • Example comparisons: Figure 11 compares the Maximum Probability baseline with Claim-Conditioned Probability.The caption presents the same baseline-versus-method comparison.
  • Example comparisons: Figure 12 compares the Maximum Probability baseline with Claim-Conditioned Probability.The caption presents another example of the same comparison.
Loading 2403.04696v2…