Chainpoll: A high efficacy method for LLM hallucination detection

1.1 Summary of contributions

• The paper presents research underlying Galileo’s hallucination detection capabilities.Source: §1.1 · p. 2
• RealHall is a suite of four difficult, realistic benchmark datasets designed to evaluate hallucination detection methods.Its construction responds to the limited practical relevance of many earlier benchmarks.Source: §1.1 · pp. 2–3
• ChainPoll dramatically outperforms published alternatives on RealHall, while also being faster and more cost-effective across open- and closed-domain hallucination detection.The comparison includes SelfCheckGPT, GPTScore, G-Eval, and TRUE.Source: §1.1 · p. 3
• ChainPoll includes specialized Correctness and Adherence variants for open-domain and closed-domain hallucinations, respectively.The paper connects Adherence to context-based evaluation and Correctness to open-domain detection.Source: §1.1 · p. 3

2 Problem statement

• The task is to identify hallucinated completions from state-of-the-art LLM outputs across open-domain and closed-domain settings.Open-domain errors are false world claims, whereas closed-domain errors depart from a provided reference text.Source: §2 · pp. 3–4
• Metrics must be automatically computable, efficient, low-cost, and compatible with API settings that expose limited model information.The setting excludes methods requiring weights, activations, embeddings, or other unavailable internal information.Source: §2 · p. 4
• Evaluation requires performance across diverse, challenging, practically relevant tasks that elicit frequent hallucinations.Source: §2 · p. 4
• The paper raises standards over prior evaluations by targeting subtler errors from modern LLMs, both hallucination domains, and product-level efficiency.Earlier benchmarks often used weaker models whose extreme errors were easier to detect.Source: §2 · p. 4

2.1 Approach

• Hallucination detection is treated as binary classification in which a metric outputs an ordered scalar score, with larger values indicating greater likelihood of hallucination.Source: §2.1 · p. 4
• The approach evaluates metrics on RealHall datasets containing prompts, completions, and boolean hallucination labels.RealHall covers multiple approaches from prior work and novel metrics such as ChainPoll.Source: §2.1 · p. 4

3 RealHall

• RealHall applies a review process based on explicit dataset criteria and contains four carefully selected datasets.Source: §3 · pp. 4–5, Table 2 · p. 4
• RealHall was created because many existing hallucination benchmarks lacked Challenge, Realism, or Task Diversity, limiting their practical guidance.The suite was designed to remedy this evaluation defect.Source: §3 · p. 5

3.1.1 RealHall Closed

• RealHall Closed evaluates consistency with provided reference text using COVID-QA with retrieval and DROP.COVID-QA with retrieval approximates RAG use cases, while DROP tests a more challenging consistency notion involving discrete reasoning.Source: §3.1 · p. 5, §3.1.1 · p. 5
• COVID-QA with retrieval uses top-k retrieved passages from a large reference corpus, making it a moderately challenging RAG-style evaluation for state-of-the-art LLMs.Source: §3.1.1 · p. 5

3.1.2 RealHall Open

• RealHall Open evaluates false real-world claims using Open Assistant prompts and TriviaQA questions presented without reference documents.Open Assistant prompts span diverse, difficult, practice-relevant tasks, while TriviaQA probes faithful recall of declarative knowledge.Source: §3.1.2 · pp. 5–6

4.1 Metrics evaluated

• The evaluation compares ChainPoll with G-Eval, GPTScore, SelfCheck-BertScore, SelfCheck-NGram, TRUE, and a probability-based baseline called pseudo-entropy.Pseudo-entropy approximates Shannon entropy when only a subset of token probabilities is available and performed best among the investigated probability-based metrics.Source: §4.1 · p. 6

4.1.1 BLEU, ROUGE, METEOR and similar metrics

• Ground-truth-dependent metrics such as BLEU, ROUGE, and METEOR were excluded because ground-truth responses are unavailable for arbitrary monitoring and rapid experimentation.The authors emphasize that users may need to evaluate arbitrary inputs or novel tasks without preparing reference answers in advance.Source: §4.1.1 · p. 6

4.2 Defining ChainPoll

• ChainPoll metrics repeatedly ask gpt-3.5-turbo for a hallucination judgment and divide yes answers by total answers to produce a score from 0 to 1.The typical procedure runs five batch-inference repetitions using a detailed prompt.Source: §4.2 · p. 7
• ChainPoll dramatically outperforms G-Eval across the entirety of RealHall, with differences attributed to prompting, judgment format, and model choice.The method uses detailed chain-of-thought prompting, boolean judgments, and gpt-3.5-turbo, whereas G-Eval used different prompting and models.Source: §4.2 · p. 7

4.2.1 ChainPoll-Correctness and ChainPoll-Adherence

• ChainPoll-Correctness targets open-domain hallucinations, while ChainPoll-Adherence targets closed-domain hallucinations through different prompt formats.The two metrics share the ChainPoll approach and differ only in the prompt’s requested hallucination type.Source: §4.2.1 · p. 7

4.2.2 Re-using chains of thought for explainability

• ChainPoll reuses its generated chain-of-thought text as a justification for whether a completion contained hallucinations.The authors report that these explanations are coherent and frequently convincing, while acknowledging that model-generated reasoning may not faithfully reflect the model’s actual reasoning process.Source: §4.2.2 · p. 7
• The approach prioritizes lower routine cost than GPT-4 by aggregating multiple gpt-3.5-turbo completions.The authors state that this aggregation remains much cheaper than generating a single GPT-4 completion.Source: §4.2.2 · p. 7

5 Results

• Across diverse benchmark tasks, ChainPoll outperforms all other methods, in most cases by a huge margin.The paper characterizes ChainPoll as a new state-of-the-art method.Source: §5 · p. 8
• Using only 1/4 as much LLM inference, ChainPoll outperforms SelfCheck-BertScore without requiring an additional model like BERT.Accounting for efficiency increases ChainPoll’s lead over the next-best method.Source: §5 · p. 8
• Unlike the other evaluated methods, ChainPoll provides human-readable verbal justifications through the chain-of-thought text produced during inference.The same chain-of-thought output used during inference supports explanations for its judgments.Source: §5 · p. 8

5.1 AUROC scores

• Table 3 reports open-domain hallucination-detection performance averaged across RealHall Open datasets, while Table 4 reports the corresponding closed-domain performance.These tables organize the AUROC results by domain setting and average scores across datasets.Source: Table 3 · p. 8, Table 4 · p. 8

5.2 Plots

• The ROC and precision-recall plots show ChainPoll-Correctness in the top row and Chainpoll-Adherence in the bottom row across RealHall datasets.The supplied captions identify the two ChainPoll variants and the two curve types.Source: Figure 1 · p. 9, Figure 2 · p. 10

6 Related work

• Table 5 compares ChainPoll with prior metrics using evaluation domain, compute intensity, parallelizability, GPU requirements, and normalized RealHall AUROC.Its annotations also explain exclusions for cost or sequential-computation reasons and note that TRUE NLI applies only to closed-domain tasks.Source: §6 · p. 10, Table 5 · p. 11

6.1 SelfCheckGPT

• SelfCheckGPT samples additional responses and measures self-consistency using BertScore, NGram probabilities, or MQAG-based agreement.Its three metrics differ in how agreement between the original and additional responses is computed.Source: §6.1 · p. 11
• SelfCheckGPT operates sentence by sentence, which can be computationally expensive for long responses, and averages sentence scores for response-level aggregates.The cost combines per-sentence scoring with generating potentially many additional responses.Source: §6.1 · p. 11

6.2 G-Eval

• G-Eval asks an LLM to rate a response on a 1-5 scale using guidelines, converting token probabilities or repeated samples into a weighted-average rating.It was evaluated on the closed-domain SummEval and TopicalChat datasets.Source: §6.2 · p. 12

6.3 GPTScore

• GPTScore evaluates response perplexity after prepending an instruction, but its strong SummEval performance does not transfer to RealHall.The paper reports that perplexity’s SummEval strength reflects a mismatch between weaker response-generating models and stronger scoring models; the advantage disappears when the same strong modern LLM performs both roles.Source: §6.3 · p. 12

6.4 TRUE / 6.5 ChatProtect

• TRUE evaluates closed-domain datasets with an NLI-based metric using probabilities from a fine-tuned T5-XXL model, while ChatProtect generates sentence-level alternatives and checks their consistency with the originals.ChatProtect is similar to SelfCheckGPT but generates a separate alternative for each sentence in context rather than alternatives for the whole response.Source: §6.4 · p. 12, §6.5 · p. 12

• RealHall evaluates hallucination metrics on four practically relevant tasks where powerful LLMs still hallucinate frequently.Source: §7 · p. 13
• ChainPoll outperforms other evaluated metrics across open-domain and closed-domain hallucination detection while being efficient to compute.Source: §7 · p. 13
• ChainPoll is inherently explainable, supporting interpretation of its hallucination judgments.Source: §7 · p. 13

• Appendix A.1.1 identifies reviewed datasets that were not included in RealHall.Source: §A.1 · p. 15
• Appendix A.3 describes how ground-truth labels were assigned to benchmark data.Source: §A.1 · p. 15
• Table 6 lists the datasets reviewed during RealHall benchmark construction.Source: Table 6 · p. 15

A.1.1.1 Instruction tuning datasets

• Instruction-tuning datasets were rejected because their highly artificial tasks do not resemble practical interactions with instruction-tuned LLMs.The paper illustrates this with structured counterfactual and fact-generation instructions.Source: §A.1.1 · pp. 15–16

A.1.1.2 Summarization datasets

• Older summarization datasets are convenient because they include human annotations, but their weaker source models do not reflect hallucination behavior from today’s SOTA LLMs.Source: §A.1.1 · p. 16
• 20% of Summeval summaries contain hallucinations, compared with 3% when ChatGPT summarizes the same documents.The paper reports similarly low hallucination rates on TL;DR, ArXiv Summarization, and MediaSum.Source: §A.1.1 · p. 16
• Summarization is too easy for SOTA LLMs to provide a reliable hallucination-detection benchmark without large samples.The authors therefore focus on tasks that SOTA LLMs find more difficult, while retaining SummEval evaluations in an appendix.Source: §A.1.1 · p. 16

A.1.1.3 Other rejected datasets

• BEGIN was excluded because it uses much weaker models, while its TopicalChat subset showed anomalously strong perplexity performance.Source: §A.1.1 · p. 16
• SelfCheckGPT Wikibio was excluded for narrow scope and verbatim-memorized Wikipedia examples.Source: §A.1.1 · pp. 16–17
• HaluEval was rejected because synthetic hallucinations differ detectably from naturally occurring hallucinations, limiting transfer to real use.On HaluEval QA, synthetic answers averaged 66 characters versus 14 for clean answers, and only 5% of synthetic answers were under 25 characters versus 90% of clean answers.Source: §A.1.1 · p. 17
• NarrativeQA was excluded because questions are often ill-posed and ChatGPT almost never hallucinated once those cases were removed.GPT-4 marked 8% of ChatGPT answers and 12% of ground-truth answers as hallucinated.Source: §A.1.1 · p. 17

• Completions were generated for each example in each benchmark after dataset selection.Source: §A.2 · p. 17
• Most experiments used gpt-3.5-turbo, while an early Open Assistant Prompts experiment also tested text-davinci-003.Source: §A.2 · p. 17
• Simple prompt formats communicated benchmark tasks to the completion model when necessary.An example prompt instructed the model to answer a question using supplied documents.Source: §A.2 · p. 17

A.3 Data annotation

• GPT-4 agreed with human hallucination judgments no less closely than human annotators agreed with one another.GPT-4 was therefore used as the sole annotator for COVID-QA and TriviaQA.Source: §A.3 · p. 17
• Likelihood-based hallucination detection can be overestimated when memorized text is confounded with hallucination status.Memorized content is often factually accurate but can have anomalously high model likelihood.Source: §A.3 · p. 17

A.4 Pseudo-entropy

• Pseudo-entropy corrects PPL5’s tendency to underestimate entropy when probability mass lies outside the API-returned top tokens.The variant uses the original probabilities in the log-probability term rather than normalized top-token probabilities.Source: §A.4 · p. 18

A.4.1 Probability models

• Comparable performance was observed with text-curie-001, text-davinci-003, and davinci-002, despite their different sizes.The reported pseudo-entropy scores use text-curie-001.Source: §A.4.1 · p. 19

A.5 Evaluation details

• Evaluation reproduced prior metrics where possible, but adapted task-specific prompts for G-Eval and GPTScore when original prompts were inapplicable.G-Eval used gpt-3.5-turbo, while GPTScore used text-curie-001 probabilities.Source: §A.5 · p. 19

A.6 SummEval case study

• SummEval’s strong GPTScore performance was reproduced, but matching it with completion perplexity indicates that perplexity—not GPTScore’s prefix—drives the result.SummEval includes weaker models that often generate nearly incoherent text, unlike the subtler hallucinations targeted by RealHall.Source: §A.6 · pp. 19–20, Table 7 · p. 19

A.7 Detailed results

• RealHall contains four datasets selected for challenge, realism, and task diversity to address defects in earlier evaluations.The detailed results cover CovidQA, DROP, Open Assistant Prompts, and TriviaQA.Source: Table 10 · p. 21, Table 11 · p. 21, Table 8 · p. 21