PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, Luoyi Fu

arXiv:2311.13230v1cs.CLcs.AI

TL;DR

Hallucinations can undermine LLM reliability, while probability-based estimates may misrepresent factuality. This paper proposes reference-free uncertainty-based detection that imitates human factuality checking; experiments report strong benchmark performance, including for summaries from small models.

  • Problem

    Hallucinated text can be untruthful or nonsensical, undermining LLM reliability and trustworthiness, while model probabilities may not reflect factuality confidence.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    The method uses a proxy language model’s token probabilities and three focus mechanisms to score hallucinations without external knowledge, sampled responses, or further training.

    Source: §1 · p. 1, §3.2 · p. 4, §5 · p. 8

  • Results

    The method achieves state-of-the-art performance on WikiBio GPT-3 across models of different scales and shows effectiveness on XSumFaith and FRANK for small-model summaries.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The approach detects hallucinations at sentence and passage level without external knowledge or training data, and does not require additional sampled responses.

    Source: §1 · p. 2, §5 · p. 8

    Proxy LLMs may lack recently emerged facts because they are not continuously updated after training, potentially reducing detection effectiveness.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have gained significant popularity for their impressive performance across diverse fields. However, LLMs are prone to hallucinate untruthful or nonsensical outputs that fail to meet user expectations in many real-world applications. Existing works for detecting hallucinations in LLMs either rely on external knowledge for reference retrieval or require sampling multiple responses from the LLM for consistency verification, making these methods costly and inefficient. In this paper, we propose a novel reference-free, uncertainty-based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the most informative and important keywords in the given text; 2) focus on the unreliable tokens in historical context which may lead to a cascade of hallucinations; and 3) focus on the token properties such as token type and token frequency. Experimental results on relevant datasets demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance across all the evaluation metrics and eliminates the need for additional information.

1 Introduction

Hallucination detection methods can be costly or depend on external resources, while proxy-model probabilities may not reflect factuality. The paper proposes a reference-free method that adjusts its focus to informative keywords, unreliable context, and token properties.

1 Introduction

  • Retrieval-based detection depends on accessible external knowledge, while sampling-based detection requires multiple LLM responses, making both approaches costly or inefficient.These methods assess veracity against knowledge bases or consistency across sampled responses.Source: §1 · p. 1
  • The base method uses a proxy language model’s token probabilities to calculate uncertainty-based hallucination scores and identify candidate hallucinated tokens and sentences.It assumes a sufficiently powerful LLM assigns low probability to tokens that deviate from world knowledge learned during training.Source: §1 · pp. 1–2
  • A naive proxy can add noise by treating tokens as equally informative and can be overconfident about hallucinated content or underconfident about factual content.Surface correlations in context can raise probabilities for hallucinated tokens, while plausible alternatives can lower probabilities for factual tokens.Source: §1 · p. 2
  • The proposed human-inspired focus targets informative keywords, propagates uncertainty from preceding words through attention, and conditions token probabilities on entity type and inverse document frequency.Keyword-based hallucination scores are calculated at sentence and passage level; the token-property adjustment is intended to mitigate underconfidence.Source: §1 · p. 2
  • The method achieves state-of-the-art performance on WikiBio GPT-3 across models of different scales and detects hallucinations in summaries generated by small models.It requires neither additional sampled responses nor external knowledge bases.Source: §1 · p. 2

2 Related Work

The related work covers hallucinations across text-generation tasks and reviews detection approaches developed for both smaller task-specific models and LLMs.

2.1 Hallucinations in Text Generation

  • Hallucinations occur in tasks including summarization, dialogue, and question answering, and can produce inaccurate or misleading results in user-oriented applications.The cited work describes them as prevalent in deep learning-based text generation.Source: §2.1 · pp. 2–3

2.2 Hallucination Detection

  • Earlier detection work often targeted small task-specific models, and differing hallucination patterns in LLMs challenge the generalization of those methods.The paper states that detecting hallucinations in small models is outside its primary scope.Source: §2.2 · p. 3
  • LLM detection methods include knowledge-source fact checking and consistency checks across sampled outputs, but these approaches require external knowledge or multiple responses.Examples include FACTSCORE (Min et al., 2023) and SelfCheckGPT (Manakul et al., 2023).Source: §2.2 · p. 3

3 Methods

The method addresses proxy-model uncertainty that may misalign with human factuality judgments by mimicking human checking through keyword focus, propagation of prior uncertainty, and token-property adjustments.

3 Methods

  • The approach targets proxy-model uncertainty metrics that may not align with human evaluation by imitating human factuality checking.A proxy model is used when token-level probabilities are inaccessible, as with GPT-3.Source: §3 · p. 3

3.1 Keywords selection

  • SpaCy identifies named entities and other nouns as keywords; their scores combine negative log probability and entropy, then keyword scores are averaged at sentence and passage levels.The sentence score weights tokens by keyword membership.Source: §3.1 · pp. 3–4

3.2 Hallucination propagation

  • To counter overconfidence from hallucinated context and exposure bias, prior keyword hallucination scores propagate via attention weights, with γ damping multi-hop penalties geometrically.Exposure bias is the training–inference discrepancy caused by teacher forcing.Source: §3.2 · p. 4

3.3 Probability correction

  • Entity-type prompts approximate context-matched candidate sets to raise probabilities for matching tokens, while IDF reweights rare tokens to address underconfidence.IDF is calculated using one million documents sampled from RedPajama.Source: §3.3 · p. 5

3.4 Putting things together

  • The combined method substitutes corrected probabilities into the uncertainty score, applies propagation, then computes sentence and passage scores; inserted entity labels are excluded from scoring.Source: §3.4 · pp. 5–6

4 Experiments and Results

The experiments compare the proposed method with baselines on WikiBio GPT-3 and examine its focus mechanisms, including their effects in specific hallucination cases. The reported results also identify limits associated with entity-type information and parameter settings.

4.1 Experiment setting

  • Experiments centered on WikiBio GPT-3, with XSumFaith and FRANK as supplementary datasets, using sentence-level AUC-PR and passage-level Pearson and Spearman correlations.WikiBio GPT-3 contains 1,908 annotated sentences from 238 text-davinci-003 passages; 22 proxy models were evaluated.Source: §4.1 · p. 6

4.2 Main results

  • LLaMA-30b outperformed SelfCheckGPT-Combination4 and other baselines across all five metrics, while LLaMA-7b was comparable or superior to GPT-3 uncertainties.The LLaMA-30b comparison used no sampled responses or further training.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Analysis

  • Keyword focus improved nearly all metrics, raising Pearson and Spearman correlations by 5.04% and 7.48%, respectively.The authors report stronger correlation between keyword uncertainties and passage-level human judgments.Source: §4.3 · p. 7
  • Adding propagation after keyword selection improved all metrics, including a 3.67% increase in non-factual AUC-PR on LLaMA-30b.The authors attribute this improvement to remediation of overconfidence.Source: §4.3 · p. 7
  • Generally, ρ = 0.01 delivers optimal results; a large ρ could omit crucial information because of a restricted candidate set, while a small ρ might add noise by including irrelevant tokens.Source: §4.3 · p. 8

4.4.1 Non-factual cases detected by hallucination propagation

  • In the reported examples, a preceding hallucination led to a false singer-songwriter attribution, while a penalty raised the score for duplicated text approximately fifty-fold.Table 3 presents hallucination scores for highlighted sentences with and without the penalty.Source: Table 3 · p. 8, §4.4.1 · p. 8

4.3 Analysis

  • Adding entity type and token IDF increased factual-class AUC-PR by 10.76% and both correlation coefficients by approximately 18%.The authors link these gains to probability correction that mitigates underconfidence; entity type alone yielded little non-factual* improvement across multiple proxy models.Source: §4.3 · p. 8

4.4.2 Failure cases after entity type provision

  • Providing entity types lowered average scores across classes, especially for major-inaccurate sentences, making major- and minor-inaccurate cases harder to distinguish and reducing non-factual* AUC-PR.The non-factual* class contains only sentences labeled major inaccurate.Source: §4.4.2 · p. 8, Table 4 · p. 8

5 Conclusion

The paper presents a reference-free, uncertainty-based hallucination detector built around three aspects of human factuality checking. Experiments report effectiveness at sentence and passage level and potential for detecting hallucinations in small-model summaries.

5 Conclusion

  • The detector focuses on informative keywords, preceding words, and token properties to assess hallucinations.The reported experiments cover sentence- and passage-level detection without external knowledge or training data.Source: §5 · p. 8
  • Experiments show potential for detecting hallucinations in summaries generated by small models.The authors also analyze how the three focus mechanisms perform with different proxy-model backbones.Source: §5 · p. 9

Limitations

The approach relies on SpaCy for keyword identification and named-entity recognition, and assumes proxy LLMs have current factual knowledge. Both conditions can constrain detection.

Limitations

  • SpaCy errors in keyword identification or named-entity recognition can make calculated probabilities unreliable and reduce performance.The authors note that real-world entity categories, including food and vehicles, are more diverse than those identifiable by SpaCy.Source: Limitations · p. 9
  • The method assumes proxy LLMs are current with factual knowledge, although post-training models may lack recent facts and thereby affect detection effectiveness.The authors state that this can influence assigned probabilities.Source: Limitations · p. 9

Ethics Statement

The authors state that their method uses no user data and that experiments use publicly available datasets. They condemn potential misuse to improve misinformation or deepfakes.

Ethics Statement

  • The method does not require user data, and experiments use publicly available datasets to uphold privacy and anonymity standards.The authors frame these choices as maintaining privacy.Source: Ethics Statement · p. 9
  • The authors condemn potential misuse of the work to enhance AI-generated misinformation or deepfake content.They state a commitment to ethical AI research and balancing technological progress with ethical responsibility.Source: Ethics Statement · p. 9

A Results for detecting hallucinations generated by small models

The small-model evaluation uses XSumFaith and FRANK, with dataset-specific exclusions and scoring choices. The method performs well across the reported XSumFaith metrics, while FRANK favors negative log probability alone and all-token scoring over keyword focus.

A Results for detecting hallucinations generated by small models

  • The evaluation tests summaries from small models, including TransS2S, TCONVS2S, and BART, on XSumFaith and FRANK.The datasets are drawn from the test split of the SummaC benchmark.Source: A Results for detecting hallucinations… · p. 12
  • The authors exclude FRANK OutE instances and human-written XSumFaith summaries, and note that some benchmark datasets are unsuitable for this evaluation.They explain that OutE labels can include factual extrinsic content; they also report that about 5% of such cases in XSumFaith were reported in its original paper.Source: A Results for detecting hallucinations… · p. 12
  • On XSumFaith, the method performs well across non-factual and factual AUC-PR and balanced accuracy.Results are reported in Table 6 for LLaMA-30b-SFT.Source: A Results for detecting hallucinations… · p. 12, Table 6 · p. 12
  • For FRANK, negative log probability alone outperforms its sum with entropy, and considering all passage tokens outperforms keyword focus.The authors use negative log probability as the token hallucination score and disregard keyword selection for FRANK.Source: A Results for detecting hallucinations… · p. 12, Table 7 · p. 13

B Implementation Details

The appendix documents implementation and prompt handling, illustrates attention maps and entity-typed examples, and compares LLaMA model sizes. LLaMA-30b outperforms the twice-as-large LLaMA-65b on four of five evaluated metrics.

B Implementation Details

  • The experiments used an AWS p3dn.24xlarge instance, and excluded 16 XSumFaith cases exceeding LLaMA’s 2,048-token context limit.For SFT LLaMA, prompts followed the Alpaca format; the probability of token “<” was set to zero, and entity types were included in prompts as few-shot examples.Source: B Implementation Details · p. 13

C More attention heat map cases

  • Figure 4 and Figure 5 visualize attention heat maps for two cases, marking attention erroneously directed toward preceding unreliable tokens in blue boxes.The figures omit some sentences due to space limitations.Source: C More attention heat map cases · p. 13

D Examples of passages with entity types provided

  • The examples provide entity types before each SpaCy-recognized named entity in text-davinci-003-generated Wikipedia passages.The passages illustrate this setup with examples about Michael Savage, Michael Replogle, and Tommy Nutter.Source: D Examples of passages with entity… · p. 13

E Details of the proxy models

  • The evaluation used 22 proxy models spanning LLaMA, LLaMA-2, OPT, GPT-J, GPT-NeoX, Falcon, Vicuna, RedPajama, and SFT LLaMA variants.The listed models range from OPT-125m to LLaMA-2-70b.Source: E Details of the proxy models · p. 13

F Performance comparison of LLaMA family

  • LLaMA-65b underperforms LLaMA-30b on four of five evaluated metrics despite being twice its size; larger LLaMA models generally perform better.The comparison is on the WikiBio GPT-3 dataset.Source: F Performance comparison of LLaMA family · p. 13

G Hyper parameters analysis

The analysis varies γ and ρ for LLaMA-30b. Performance decreases when ρ is set too high or too low.

G Hyper parameters analysis

H Additional Results

Additional results report consistent gains over uncertainty-only scoring across model ablations and note model-dependent optimal settings. Figures also show model-size and hyperparameter comparisons, attention maps, and generated passage examples.

H Additional Results

Loading 2311.13230v1…