Source-linked AI summary
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Potsawee Manakul, Adian Liusie, Mark J. F. Gales
TL;DR
LLMs can generate fluent but factually incorrect text, while existing checks may require unavailable token probabilities or external databases. SelfCheckGPT compares multiple stochastic responses from a black-box LLM without external resources, and it outperforms considered baselines in sentence- and passage-level factuality assessment.
Problem
LLMs can confidently hallucinate facts, while existing detection approaches may require inaccessible token probabilities or external databases.
Method
SelfCheckGPT samples multiple responses and measures their consistency to detect hallucinated or factual content in black-box LLM outputs without external resources.
Results
SelfCheckGPT outperforms considered grey-box and black-box baselines at both sentence and passage levels.
Takeaways & Limitations
Sampling-based consistency provides a zero-resource hallucination-detection approach applicable to black-box LLMs.
Takeaways & Limitations
The evaluation mainly uses 238 GPT-3-generated passages about individuals, and SelfCheckGPT-Prompt is computationally heavy.
Abstract
from arXiv · showhide
Generative Large Language Models (LLMs) such as GPT-3 are capable of generating highly fluent responses to a wide variety of user prompts. However, LLMs are known to hallucinate facts and make non-factual statements which can undermine trust in their output. Existing fact-checking approaches either require access to the output probability distribution (which may not be available for systems such as ChatGPT) or external databases that are interfaced via separate, often complex, modules. In this work, we propose "SelfCheckGPT", a simple sampling-based approach that can be used to fact-check the responses of black-box models in a zero-resource fashion, i.e. without an external database. SelfCheckGPT leverages the simple idea that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another. We investigate this approach by using GPT-3 to generate passages about individuals from the WikiBio dataset, and manually annotate the factuality of the generated passages. We demonstrate that SelfCheckGPT can: i) detect non-factual and factual sentences; and ii) rank passages in terms of factuality. We compare our approach to several baselines and show that our approach has considerably higher AUC-PR scores in sentence-level hallucination detection and higher correlation scores in passage-level factuality assessment compared to grey-box methods.
1 Introduction
LLMs produce fluent responses but can confidently hallucinate facts, while existing detection methods may require unavailable token probabilities or external resources. SelfCheckGPT addresses this gap by comparing multiple stochastic responses without an external database.
- LLMs generate fluent responses across many applications but can confidently produce fictitious information.
- Existing approaches may require token-level probability distributions that are unavailable through limited external APIs.
- Hallucinations occur across tasks beyond pure fact verification, motivating broader detection approaches.
- SelfCheckGPT samples multiple responses and measures their consistency to distinguish factual from hallucinated statements without an external database.
- SelfCheckGPT considers five consistency variants: BERTScore, question-answering, n-gram, NLI, and LLM prompting.
2 Background and Related Work
Related work includes self-consistency, hidden-state classifiers, self-evaluation, uncertainty estimation, and evidence-retrieval pipelines. These approaches differ in their access requirements, supervision, and reliance on external databases.
- Self-consistency decoding has improved chain-of-thought prompting on complex reasoning tasks, while hallucination datasets may rely on perturbed factual texts.
- Hidden-state classifiers predict sentence truthfulness but require internal LLM representations and labelled training data.
- Self-evaluation prompts an LLM to assess the truth probability of its earlier prediction.
- Token probabilities, entropy, and conditional language-model scores provide uncertainty-based signals for generated text.
- Fact-verification systems use claim detection, evidence retrieval, and verdict prediction, requiring external databases and potentially costly inference.
3 Grey-Box Factuality Assessment
Grey-box factuality assessment uses output-distribution information to connect token uncertainty with factuality. It is therefore constrained when systems expose only text responses or limited API access.
- In a zero-resource setting, factual statements are defined as grounded in valid information, with no external database used.
- Pretraining gives LLMs language understanding, contextual reasoning, and world knowledge that can support factual generation.
- Factual sentences tend to contain higher-likelihood, lower-entropy tokens, whereas hallucinations tend to arise from high-uncertainty positions.
- Probability metrics assess sentence likelihood using token probabilities, including the least likely token under Max(−log p).
- Grey-box methods require token-level probabilities, which may be unavailable through limited APIs for large models.
- Entropy-based metrics characterize uncertainty from the output distribution and are used alongside probability-based metrics.
4 Black-Box Factuality Assessment
Black-box factuality assessment avoids dependence on inaccessible output distributions. Proxy LLMs can approximate those probabilities, while SelfCheckGPT instead compares sampled responses directly.
- Text-only black-box approaches remain applicable when systems provide responses without token-level probability information.
- A proxy LLM can approximate the output token-level probabilities of the black-box model that generated the text.
5 SelfCheckGPT
SelfCheckGPT detects hallucinations in black-box LLM responses by comparing multiple stochastic samples for consistency. It includes BERTScore, QA, n-gram, NLI, and prompting-based variants, with prompting limited by the capability of the assessing model.
- Core approach: SelfCheckGPT compares a response with multiple stochastic samples from the same query to assign sentence-level hallucination scores.Scores approach 0 for grounded information and 1 for hallucinated information.
- BERTScore variant: BERTScore estimates factuality from the average similarity between each response sentence and its most similar sentence in every sampled response.Information recurring across samples is treated as more likely factual, while unsupported statements are more likely hallucinated.
- Question-answering variant: MQAG measures inconsistency by generating questions about response sentences and comparing answers produced using the sampled responses as context.Its score aggregates match and mismatch behavior across generated questions, with soft-counting available to account for question answerability.
- n-gram variant: The n-gram variant trains a language model on sampled responses and the assessed response to approximate token probabilities under limited sampling.Including the assessed response increases each token count by 1 as a smoothing method.
6 Data and Annotation
The evaluation uses GPT-3-generated WikiBio passages whose sentences are manually labeled for factuality. The dataset includes 1,908 annotated sentences and passage-level scores derived from sentence-level labels.
- Data and evaluation: The evaluation generates synthetic Wikipedia articles with GPT-3 for WikiBio concepts, manually annotates sentence factuality, and tests hallucination detection.The procedure addresses the lack of standard hallucination-detection datasets described by the authors.
- Dataset construction: The sampled dataset contains 238 WikiBio articles selected from the longest 20% of the test set to avoid very obscure concepts.GPT-3 generates each article from the prompt “This is a Wikipedia passage about {concept}:”.
- Annotation: Annotators classify sentences as major inaccurate, minor inaccurate, or accurate according to whether their information is unrelated, partly non-factual, or accurate.Major inaccurate receives label 1, while minor inaccurate and accurate receive labels 0.5 and 0, respectively.
- Annotation: Of 1,908 annotated sentences, 761 were major-inaccurate, 631 minor-inaccurate, and 516 accurate.These correspond to 39.9%, 33.1%, and 27.0% of the annotated sentences.
- Passage-level scoring: Passage-level factuality scores average sentence-level labels, with a histogram peak at +1.0 identified as total hallucination.Total hallucination means the response information is unrelated to the real concept and entirely fabricated by the LLM.
7 Experiments
Experiments evaluate SelfCheckGPT for sentence-level hallucination detection and passage-level factuality ranking, comparing variants with black-box, grey-box, proxy-LLM, and external-knowledge settings.
- Sentence-level Hallucination Detection: GPT-3 generated WikiBio passages were evaluated using N=20 stochastic samples, with factuality labels grouped into factual, non-factual, and non-factual∗ tasks.The non-factual∗ task targets major inaccuracies in passages that are not entirely hallucinated.
- Sentence-level Hallucination Detection: GPT-3 probability measures provide strong factuality baselines, with factual-sentence AUC-PR of 53.97 versus a random baseline of 27.04 and hallucination-detection AUC-PR rising from 72.96 to 83.21.Probability measures outperform top-5-token entropy in the reported comparison.
- Sentence-level Hallucination Detection: SelfCheckGPT variants outperform grey-box approaches in most sentence-level setups, while SelfCheckGPT-Prompt performs close to the strongest Prompt method.The variants include BERTScore, QA, n-gram, and NLI-based methods; NLI offers a practical performance–computation trade-off.
- Passage-level Factuality Ranking: Passage-level scores average sentence-level scores, and SelfCheckGPT methods correlate better with human judgements than probability, entropy, and proxy-LLM baselines.SelfCheckGPT-Prompt achieves the highest reported Pearson correlation of 78.32.
- Ablation Studies: With external WikiBio passages, BERTScore and QA remain comparable or better using self-samples, whereas n-gram performance drops and NLI/Prompt benefit considerably.Using external knowledge is no longer zero-resource and requires retrieving relevant information.
- Ablation Studies: SelfCheckGPT performance increases smoothly with more samples but shows diminishing gains, while n-gram requires the most samples to reach a plateau.GPT-3 can self-check its own text and outperforms the unigram method with only 4 samples.
8 Conclusions
SelfCheckGPT is presented as a zero-resource approach for hallucination detection in general LLM responses, applicable to black-box systems without external resources. It outperforms considered grey-box and black-box baselines at sentence and passage levels and includes an annotated GPT-3 hallucination-detection dataset.
- SelfCheckGPT detects hallucinations in general LLM responses without requiring external resources or access to a non-black-box system.
- SelfCheckGPT outperforms considered grey-box and black-box baseline detection methods at both sentence and passage levels.
- The authors release an annotated dataset of GPT-3 hallucination detection examples with sentence-level factuality labels.
Limitations
The study’s evaluation is limited in concept coverage and factuality granularity, while its strongest prompting variant is computationally heavy. These boundaries motivate broader concepts, finer-grained factuality evaluation, and efficiency improvements.
- 238 GPT-3-generated texts were predominantly passages about individuals from WikiBio, limiting the study’s evaluated concept range.
- The study evaluates factuality at sentence level, although a single sentence can contain both factual and non-factual information.
- SelfCheckGPT-Prompt is the best self-check method but is computationally heavy, potentially causing impractical computational costs.
Ethics Statement
The supplied passages describe implementation details and an ablation of SelfCheckGPT-QA, including Bayes-based soft counting, model configurations, and experiment costs. They do not provide an ethics-specific claim.
- Implementation: The study uses GPT-3 and ChatGPT with a specified prompt template, while GPT-3 generates the main WikiBio passages.
- SelfCheckGPT-QA: SelfCheckGPT-QA applies answerability filtering and soft counting, with Bayes’ theorem used to incorporate matched and unmatched answers.
- SelfCheckGPT-QA: The QA score assumes equal prior probabilities for factual and non-factual sentences and sets β1 and β2 to 0.8.
- Results: Applying Bayes’ theorem and soft counting improves the performance of SelfCheckGPT-QA.
- Experiment cost: Running GPT-3 on all 1908 sentences and 20 samples costs approximately $200, compared with approximately $20 for ChatGPT.
D Additional Experimental Results
Additional results examine sample counts, proxy LLM sizes, passage-level correlations, and sentence-level AUC-PR. The figures and tables define these evaluations but do not state their numerical outcomes in the supplied passages.
- The supplementary material presents additional experimental results complementary to those in the main paper.
- Figure 8 plots sentence-level non-factual detection AUC-PR against the number of samples for SelfCheckGPT methods.
- Figure 9 evaluates passage-level ranking with Avg(H) across LLaMA, OPT, GPT-J, and GPT-NeoX proxy LLM sizes.
- Figure 10 plots method scores against human scores for passage-level results, with correlations reported in Table 2.
- The supplementary table reports sentence-level AUC-PR for non-factual, non-factual*, and factual detection, plus Pearson and Spearman passage-level correlations.