PaperlayerView exact revision on arXiv

Source-linked AI summary

HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection

Xuefeng Du, Chaowei Xiao, Yixuan Li

arXiv:2409.17504v1cs.LGcs.CL

TL;DR

Hallucinated outputs threaten LLM reliability, while truthfulness membership is unclear in unlabeled generations. HaloScope estimates membership from LLM representations before training a truthfulness classifier, reporting gains across datasets, including a 10.69% AUROC improvement on TRUTHFULQA.

  • Problem

    Hallucinated outputs threaten LLM reliability, and truthfulness membership is unclear for samples in unlabeled generation mixtures.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    HaloScope uses latent representations to score likely hallucination membership in unlabeled generations, then trains a binary truthfulness classifier.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    HaloScope improves hallucination-detection performance across diverse datasets and reports a 10.69% AUROC improvement over state-of-the-art methods on TRUTHFULQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its unlabeled-data formulation offers a practical approach to real-world hallucination detection without manual annotations.

    Source: §1 · p. 2, §4.4 · p. 9

    Unlabeled generations form a mixture whose samples lack clear truthful-or-hallucinated membership.

    Source: §1 · p. 2

Abstract

from arXiv · show

The surge in applications of large language models (LLMs) has prompted concerns about the generation of misleading or fabricated information, known as hallucinations. Therefore, detecting hallucinations has become critical to maintaining trust in LLM-generated content. A primary challenge in learning a truthfulness classifier is the lack of a large amount of labeled truthful and hallucinated data. To address the challenge, we introduce HaloScope, a novel learning framework that leverages the unlabeled LLM generations in the wild for hallucination detection. Such unlabeled data arises freely upon deploying LLMs in the open world, and consists of both truthful and hallucinated information. To harness the unlabeled data, we present an automated membership estimation score for distinguishing between truthful and untruthful generations within unlabeled mixture data, thereby enabling the training of a binary truthfulness classifier on top. Importantly, our framework does not require extra data collection and human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiments show that HaloScope can achieve superior hallucination detection performance, outperforming the competitive rivals by a significant margin. Code is available at https://github.com/deeplearningwisc/haloscope.

1 Introduction

Hallucination detection matters for reliable LLM deployment, but labeled truthful and hallucinated examples are scarce. HaloScope uses unlabeled generations to estimate membership and train a truthfulness classifier.

1 Introduction

  • Labeled truthful and hallucinated datasets are scarce, and building reliable ground truth requires labor-intensive human assessment and ongoing quality control.The paper identifies this scarcity as a reason to explore unlabeled data for hallucination detection.Source: §1 · p. 1
  • HaloScope uses freely emerging, unlabeled LLM generations that contain a mixture of truthful and potentially hallucinated content.The mixture is modeled as P_unlabeled = (1 − π)P_true + πP_hal, where π is the mixing ratio.Source: §1 · p. 2
  • HaloScope estimates membership with a score based on projecting LLM embeddings onto the top singular vectors of a subspace associated with hallucinated statements.The estimated membership is used to train a binary truthfulness classifier.Source: §1 · p. 2
  • 10.69% AUROC improvement over state-of-the-art methods on TRUTHFULQA accompanies HaloScope’s 78.64% versus 81.04% AUROC comparison with the supervised upper bound.The reported experiments also cover diverse datasets spanning open-book and closed-book conversational QA tasks.Source: §1 · p. 2
  • The framework formalizes hallucination detection with unlabeled generations, and its ablations examine design choices and scalability across large LLMs and datasets.The paper presents this formulation as practical and flexible for real-world applications.Source: §1 · p. 2

2 Problem Setup

The setup describes autoregressive LLM generation and frames hallucination detection as binary prediction over prompt-generation pairs.

2 Problem Setup

  • An L-layer causal LLM takes a prompt of n tokens and generates an output of m tokens autoregressively.Each output token is sampled from a vocabulary distribution conditioned on the preceding token prefix.Source: §2 · p. 3
  • The generation probability is calculated using the final-layer representation fL(x) and the output-layer weight and bias parameters.The passage identifies fL(x) as the L-th-layer representation for token x, with wo and bo as final output-layer parameters.Source: §2 · p. 3
  • Hallucination detection is framed as learning a binary predictor for a generated text and its corresponding input prompt.The truthful distribution is defined over truthful input-generation pairs.Source: §2 · p. 3

3 Proposed Framework: HaloScope

HaloScope uses unlabeled LLM generations, modeled as a mixture of truthful and hallucinated samples, to estimate sample membership and train a truthfulness classifier. It identifies a latent subspace from LLM representations and uses membership scores to form candidate training sets.

3.1 Unlabeled LLM Generations in the Wild

  • Unlabeled generations are modeled as a mixture of truthful and hallucinated samples, but their members lack clear labels for classifier training.The framework estimates membership in this mixture before training a binary truthfulness classifier.Source: §3.1 · p. 3

3.2 Estimating Membership via Latent Subspace

  • HaloScope extracts sample embeddings, centers them, and uses singular value decomposition to identify spanning directions in the representation subspace.The average embedding across samples is used for centering; the factorization can be performed on any LLM layer.Source: §3.2 · p. 4
  • A sample’s membership score measures its squared projection onto the top singular vector, with a generalized score using k orthogonal singular vectors.The paper’s intuition is that hallucinated samples can be captured by a small subspace; multiple components can capture encoded truthfulness more effectively than one direction.Source: §3.2 · p. 4

3.3 Truthfulness Classifier

  • The score threshold separates potentially hallucinated samples from candidate truthful samples, which are used to train a classifier for separability.The training objective uses a binary sigmoid loss as a smooth approximation to 0/1 loss, and the trained classifier is used for test-time detection.Source: §3.3 · pp. 4–5

4 Experiments

HaloScope is evaluated across four QA datasets and multiple LLMs, with experiments reporting baseline comparisons, transfer and scaling results, and analyses of its representation and membership-score choices.

4.1 Setup

  • Setup covers four QA datasets and LLaMA-2-chat/OPT models, using 25% of QA pairs for testing and AUROC with BLUERT-based truth labels.The models use pre-trained weights and zero-shot inference.Source: §4.1 · p. 5

4.2 Main Results

  • 16.47% improvement over Semantic Entropy and 26.71% over EigenScore on TRUTHFULQA; HaloScope also outperforms state-of-the-art methods on LLaMA-2-7b-chat and OPT-6.7b.Table 1 compares methods across datasets using AUROC percentages.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Robustness to Practical Challenge

  • 76.26% AUROC on TRUTHFULQA with a TRIVIAQA-derived subspace is close to 78.64% with a TRUTHFULQA-derived subspace, demonstrating cross-dataset transfer.Figure 3(a) depicts source-to-target generalization across four datasets.Source: §4.3 · p. 7, Figure 3 · p. 7
  • 9.24% improvement: OPT-13b reaches AUROC 82.41% on TruthfulQA versus 73.17% for OPT-6.7b.Table 2 reports hallucination-detection results on larger LLMs.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Ablation Study

  • For LLaMA-2-7b-chat, AUROC rises from top to middle layers (e.g., 8th–14th) then declines; the most effective attention representation location differs between LLaMA and OPT.The reported locations are transformer-block output for LLaMA and feedforward-layer output for OPT.Source: §4.4 · p. 7, Figure 3 · p. 7, Table 3 · p. 8
  • Moderate k yields superior performance; singular-value weighting outperforms unweighted scores, while summing layer-wise scores significantly worsens detection.The layer-wise decline is attributed to low separability in top and bottom layers.Source: Figure 3 · p. 7, §4.4 · p. 8, Table 4 · p. 8
  • Across all four datasets, classifier-based HaloScope outperforms direct subspace projection on LLaMA.The direct approach projects a test representation onto the extracted subspace and bypasses binary-classifier training.Source: §4.4 · p. 8, Figure 4 · p. 8
  • On TRUTHFULQA, HaloScope’s 78.64% AUROC favorably matches the supervised oracle’s 81.04%, without manual annotations.Figure 5 compares the approaches on the same test set.Source: §4.4 · p. 9, Figure 5 · p. 9

5 Related Work

Prior hallucination detectors use uncertainty scores, text consistency, confidence prompting, or LLM activations. HaloScope differs by estimating membership in unlabeled data, then training a classifier, and by evaluating across common benchmarks.

5 Related Work

  • HaloScope uses a hallucination subspace rather than a single activation direction, estimates membership in unlabeled data, and trains a truthfulness classifier for detection.The authors contrast this with activation-based approaches [55] and labeled-data methods [26] [13].Source: §5 · p. 9
  • Prior methods score logits [31] [23] [14], compare multiple generated texts [32] [47] [10], prompt confidence judgments [21] [47] [39] [28] [43], or analyze activations [42] [49] [36].The paper distinguishes hallucination detection from hallucination mitigation [24] [44] [52] [22] [41] [8] and unlabeled-data out-of-distribution detection [4] [3].Source: §5 · p. 9

6 Conclusion

HaloScope estimates membership in unlabeled LLM generations and trains a binary truthfulness classifier. The paper reports superior performance across QA datasets and LLM families, while identifying distribution-shift generalization as a future research direction.

6 Conclusion

  • The authors propose estimating truthful-versus-hallucinated membership in unlabeled mixture data, then training a binary truthfulness classifier.They report superior performance across QA datasets and LLM families and identify generalization under distribution shift between unlabeled and test data as future work.Source: §6 · p. 10

TruthfulQA

CoQA

TriviaQA

  • TriviaQA examples present questions directly, including ones about British prime ministers and the longest-lasting dynasty in China.The supplied examples also include questions about cat purring, eating after 8pm, and a novelist's award-winning book.Source: TriviaQA · p. 14

A Datasets and Implementation Details

  • Baseline implementations use average generated-token perplexity, 10 sampled generations at temperature 0.5, and specified similarity, NLI, and confidence-scoring choices.For example, SelfCKGPT uses an NLI DeBERTa-v3-large model, while Verbalize prompts are provided for open-book QA.Source: TYDIQA-GP · p. 15

TYDIQA-GP

  • TYDIQA-GP supplies context, a question, and a proposed answer, then asks whether the answer is true or false.The log probability of output token “A” is used as the uncertainty score for evaluation.Source: TYDIQA-GP · p. 15

B Distribution of the Membership Estimation Score

  • The membership scores for truthful and hallucinated TYDIQA-GP generations show reasonable separation using 14th-layer LLaMA-2-chat-7b representations.The paper states this separation can benefit downstream truthfulness-classifier training; Figure 7 shows the score distribution.Source: Figure 7 · p. 15, B Distribution of the Membership… · p. 15

C Results with Rouge-L

The Rouge-L ablation tests whether HaloScope's effectiveness persists when the similarity measure used to deem generations truthful differs from the main-paper measure.

C Results with Rouge-L

  • HaloScope's effectiveness still holds with Rouge-L replacing the main paper's BLUERT similarity measure for identifying truthful generations.The threshold remains 0.5, and Table 5 presents results for LLaMA-2-7b-chat.Source: C Results with Rouge-L · p. 16, Table 5 · p. 16

D Results with a Different Dataset Split

With a different random dataset split, HaloScope reports similar performance to its main experiment and outperforms baselines.

D Results with a Different Dataset Split

  • 76.39% AUROC on TRUTHFULQA and 94.89% AUROC on TYDIQA-GP were achieved by HaloScope with LLaMA-2-chat-7b under a different random split.The split reserves 25% of available QA pairs for testing; the authors report that HaloScope also outperforms baselines.Source: Table 6 · p. 16, D Results with a Different Dataset Split · p. 16

E Ablation on Sampling Strategies

HaloScope’s hallucination-subspace identification is tested with two sampling strategies. Their detection performance is similar, with greedy sampling slightly better.

E Ablation on Sampling Strategies

F Results with Less Unlabeled Data

On TRUTHFULQA with LLaMA-2-chat-7b, detection performance degrades slightly as the number of unlabeled generations decreases. The authors favor using a sufficiently large sample, noting that unlabeled data is easy and cheap to collect.

F Results with Less Unlabeled Data

G Results of Using Other Uncertainty Scores for Filtering

HaloScope is compared with alternative uncertainty scores for membership estimation under the same experimental setting. The reported comparison favors HaloScope’s hallucination-subspace membership estimation.

G Results of Using Other Uncertainty Scores for Filtering

H Results on Additional Tasks

The paper reports additional text-continuation and summarization evaluations, discusses potential deployment and a distribution-shift limitation, and records its software and hardware setup.

H Results on Additional Tasks

I Broader Impact and Limitations

J Software and Hardware

Loading 2409.17504v1…