PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, Yarin Gal

arXiv:2406.15927v1cs.CLcs.AIcs.LG

TL;DR

Hallucinations challenge reliable LLM deployment, while semantic-entropy detection is costly because it requires multiple generations. The paper introduces probes that predict semantic entropy from a single generation’s hidden states, finding strong hallucination detection and better generalization than accuracy probes. These results suggest that hidden states encode semantic uncertainty across models, tasks, layers, and token positions.

  • Problem

    Hallucinations can produce nonfactual, arbitrary outputs, while semantic-entropy detection typically costs 5-to-10 times more than naive generation.

  • Method

    SEPs are linear probes trained on LLM hidden states to predict semantic entropy from a single generation without ground-truth accuracy labels.

  • Results

    SEPs detect hallucinations more effectively than accuracy probes on novel inputs from different distributions and predict semantic entropy consistently across varied settings.

  • Takeaways & Limitations

    Model hidden states implicitly capture semantic uncertainty, making semantic-uncertainty probing a cost-efficient approach to hallucination detection.

  • Takeaways & Limitations

    Very high Llama-2-7B AUROC values on BioASQ may reflect the prevalence of yes-no questions rather than broad detection ability.

Abstract

from arXiv · show

We propose semantic entropy probes (SEPs), a cheap and reliable method for uncertainty quantification in Large Language Models (LLMs). Hallucinations, which are plausible-sounding but factually incorrect and arbitrary model generations, present a major challenge to the practical adoption of LLMs. Recent work by Farquhar et al. (2024) proposes semantic entropy (SE), which can detect hallucinations by estimating uncertainty in the space semantic meaning for a set of model generations. However, the 5-to-10-fold increase in computation cost associated with SE computation hinders practical adoption. To address this, we propose SEPs, which directly approximate SE from the hidden states of a single generation. SEPs are simple to train and do not require sampling multiple model generations at test time, reducing the overhead of semantic uncertainty quantification to almost zero. We show that SEPs retain high performance for hallucination detection and generalize better to out-of-distribution data than previous probing methods that directly predict model accuracy. Our results across models and tasks suggest that model hidden states capture SE, and our ablation studies give further insights into the token positions and model layers for which this is the case.

1 Introduction

Hallucinations threaten safe LLM deployment, while semantic-entropy detection improves uncertainty measurement but is costly. The paper proposes semantic entropy probes (SEPs) to provide cheaper detection from a single generation’s hidden states.

  • Motivation: Hallucinations are nonfactual, arbitrary generations that can cause significant harm when factuality is critical.Reliable detection or mitigation is therefore important for safe deployment of LLM-based systems.
  • Existing approach: Sampling multiple responses and comparing their meanings can reveal hallucinations because uncertainty may produce varying generations.Consistent responses suggest the model knows the answer, whereas arbitrary variation suggests uncertainty.
  • Practical gap: 5-to-10-fold higher cost makes semantic-entropy and other sampling-based detection methods difficult to adopt in practice.These methods typically require 5 to 10 additional model generations per query.
  • Proposed method: SEPs are linear probes trained on hidden states to capture semantic entropy without ground-truth accuracy labels or multiple test-time generations.They combine the low deployment cost of probing with semantic uncertainty estimation from sampling-based methods.
  • Findings: SEPs generalize better to new tasks than accuracy probes and provide a cost-efficient approach to hallucination detection.The paper reports a new state-of-the-art for cost-efficient hallucination detection and evidence that hidden states capture semantic uncertainty.

2 Related Work

Prior hallucination research includes sampling-based detection, retrieval-based verification, generation-reduction strategies, and hidden-state probing. This paper extends truthfulness probing by targeting semantic entropy rather than accuracy labels.

  • Detection approaches: Hallucination-detection methods generally fall into sampling-based and retrieval-based approaches.Sampling methods compare multiple generations, while retrieval methods use external knowledge to verify responses.
  • Sampling-based detection: Sampling-based methods quantify semantic differences across multiple model completions for the same query.Semantic entropy is the sampling-based measure used to supervise the probes in this paper.
  • Retrieval-based methods: Retrieval-based methods can correct errors without relying on model uncertainty but may add substantial cost and latency.Their effectiveness may also be lower in domains such as reasoning.
  • Mitigation strategies: Other work reduces hallucinations through sampling adaptations, factuality-oriented preference optimization, or uncertainty-focused fine-tuning.These strategies target generation behavior rather than directly detecting hallucinations.
  • Hidden-state probing: Hidden-state probes have predicted correctness and other behaviors, but this paper argues that semantic entropy is a better target for generalization than accuracy.Accuracy labels are external and potentially noisy, whereas semantic uncertainty is more model-internal.

3 Semantic Entropy

Semantic entropy measures uncertainty over meanings rather than token sequences by clustering semantically equivalent generations. In practice, it is estimated from sampled generations and their categorical cluster distribution.

  • Motivation: Token probabilities can conflate uncertainty about meaning with uncertainty about lexical or syntactic phrasing.Semantic entropy addresses this distinction by aggregating uncertainty across semantic-equivalence clusters.
  • Procedure: Semantic entropy calculation samples completions, clusters generations with equivalent meanings, and aggregates uncertainty across those clusters.The procedure separates generation sampling from semantic clustering and entropy computation.
  • Semantic clustering: Farquhar et al. define equivalent generations through bidirectional entailment and cluster them with a greedy algorithm.Two generations are treated as identical in meaning when each entails the other.
  • Entropy: Semantic entropy is the uncertainty of the distribution over semantic clusters.The cluster probability aggregates probability across all generations belonging to that semantic cluster.
  • Practical estimation: Exact semantic-entropy computation is intractable, so the method uses typically N = 10 sampled generations as Monte Carlo samples.The discrete variant estimates cluster probabilities from generation frequencies and avoids token-probability requirements and differing-length issues.

4 Semantic Entropy Probes

Semantic Entropy Probes (SEPs) use linear probes on hidden states to predict whether a query has high semantic entropy, avoiding multiple response samples at test time. The method examines hidden states across layers and token positions, including before generation, to provide cheap uncertainty estimation.

  • SEPs are linear probes trained on LLM hidden states to capture semantic entropy and detect hallucinations without sampling multiple responses at test time.They are intended to combine the efficiency of probing with the semantic uncertainty captured by sampling-based detection.
  • Training SEPs: Training pairs hidden states from a high-likelihood response with semantic-entropy scores computed from N = 10 high-temperature responses.The training inputs can come from unlabeled QA queries because ground-truth dataset labels are not required to compute semantic entropy.
  • Binarization: Semantic entropy is binarized as high or low using an optimally selected threshold, after which a logistic regression classifier predicts the labels.The threshold is defined by ˜HSE(x) = 1[HSE(x) > γ⋆].
  • Inference: At test time, the SEP predicts the probability that a generation for an input query has high semantic entropy.The procedure uses a linear classifier trained on hidden-state–binarized semantic-entropy pairs.
  • Probing Locations: The probing analysis compares the last input token before generation (TBG) with the second-last response token (SLT) across layers.TBG can potentially estimate semantic uncertainty in a single forward pass before any novel tokens are generated, further reducing cost.

5 Experiment Setup

The experiments evaluate SEPs across multiple QA datasets, models, answer lengths, and hallucination-detection baselines. Performance is measured by AUROC for predicting semantic entropy and model accuracy, with comparisons including supervised accuracy probes and likelihood-based methods.

  • Tasks: SEPs are evaluated on TriviaQA, SQuAD, BioASQ, and NQ Open in both short-form and long-form answer settings.Short-form answers are prompted to be as brief as possible, while long-form answers are brief but complete.
  • Models: The study uses Llama-2, Mistral, Phi-3 Mini, and Llama-3 models, with DeBERTa-Large or GPT-3.5 used for entailment calculations.Short-generation experiments include multiple 7B and 70B models; long-generation experiments use 70B models.
  • Baselines: Baselines include ground-truth semantic entropy, supervised accuracy probes, naive entropy, log likelihood, and p(True).Naive entropy uses length-normalized average log token probabilities across multiple generations, whereas log likelihood uses one generation.
  • Evaluation: The experiments compare SEPs with an accuracy-probe baseline implemented as logistic regression using default L2 regularization and the LBFGS optimizer.The broader evaluation investigates SEP behavior across tasks, models, token indices, and layers.
  • Evaluation: Both SEP evaluation targets semantic entropy prediction and hallucination prediction, using AUROC with binarized semantic entropy or model accuracy as gold labels.This evaluates both whether hidden states capture semantic entropy and whether the resulting probes detect hallucinations.

6 LLM Hidden States Implicitly Capture Semantic Entropy

Across models, tasks, generation lengths, and token positions, hidden states encode semantic entropy that SEPs can recover. The probes also track changes in uncertainty from added context and generalize better than accuracy probes to unseen tasks.

  • Hidden states capture semantic entropy: SEPs achieve AUROC values between 0.7 and 0.95 for semantic-entropy prediction, generally improving in later layers.These results use second-last-token hidden states in short-form generation across models and tasks.
  • Prediction before generation: SEPs predict semantic entropy before generating novel tokens, with performance slightly below second-last-token experiments.The token-before-generation variant uses a single forward pass and can quantify uncertainty before output generation.
  • Dataset-specific caveat: BioASQ produces unusually high AUROC values even in early layers, likely because identifying yes-no questions separates cases with lower semantic entropy.The task is unique in containing a significant number of yes-no questions.
  • Long generations: In long-form generation, SEPs capture semantic entropy for Llama-2-70B and Llama-3-70B at both second-last-token and token-before-generation positions.AUROC values more often peak in intermediate layers, where representations are less focused on predicting the next token.
  • Generalization: SEPs consistently outperform accuracy probes when generalizing to unseen tasks, although accuracy probes perform better in-distribution.The comparison uses leave-one-out training across tasks in the short-form Llama-2-7B setting.
  • Context intervention: Adding TriviaQA context shifts SEP predictions toward lower semantic entropy, matching accuracy rising from 26% to 78% and ground-truth SE falling from 1.84 to 0.50.This counterfactual experiment tests whether probes track uncertainty rather than relying on spurious correlations.

7 SEPs Are Cheap and Reliable Hallucination Detectors

SEPs are evaluated against accuracy probes and sampling-based baselines for hallucination detection, including challenging unseen-task settings. They generalize better than accuracy probes while operating on single generations, although costlier baselines remain stronger.

  • In-distribution performance: SEPs and accuracy probes perform similarly on in-distribution data across models.The evaluation compares hallucination-detection performance using representative high-performing layers.
  • Task generalization: SEPs consistently outperform accuracy probes across layers and tasks when evaluated on unseen short-form-generation tasks.The generalization evaluation uses a leave-one-out procedure, training on all datasets except the evaluation dataset.
  • Task generalization: SEPs generalize better to novel tasks than accuracy probes across models and tasks, approaching but not matching 10x-costlier baselines.The comparison covers short-generation settings and representative layers.
  • Computational cost: SEPs and accuracy probes use single generations, whereas sampling-based baselines require 10 additional model generations.This creates a substantial computational-cost difference between probes and the stronger sampling-based methods.

8 Discussion, Future Work, and Conclusions.

The discussion attributes SEP generalization to semantic uncertainty being more directly represented in hidden states than externally defined accuracy labels. The paper identifies larger training datasets and unlabeled data as routes toward closing the remaining performance gap.

  • Discussion: Higher AUROC for predicting semantic entropy than accuracy indicates that hidden states capture semantic entropy more directly.The authors suggest semantic uncertainty is a more model-internal probing target, while accuracy labels can be noisy and external.
  • Discussion: Accuracy probes may learn training-dataset-specific correctness features, whereas semantic probes may capture more inherent uncertainty states.The proposed examples include uncertainty from failing to gather relevant facts or attributes for a query.
  • Future Work: Future work could scale SEP training data and use unlabeled inputs or texts to improve probe accuracy and support more complex probes.The current experiments rely on established QA tasks for comparison with accuracy probes.
  • Conclusions: SEPs predict semantic entropy across varied scenarios and detect hallucinations more effectively than accuracy probes on novel distributions without ground-truth correctness labels.The conclusion frames semantic uncertainty probing as relevant to both model interpretability and practical applications.

A Additional Results

Additional results examine SEP and accuracy-probe behavior across models, tasks, token positions, layers, model components, and binarization choices. They show that semantic-entropy prediction is generally easier and that SEP performance is relatively robust to these design choices.

  • Token positions: SEPs and accuracy probes can predict model correctness from hidden states at both the second-last-token and token-before-generation positions.This result holds in short-form, in-distribution settings across models and tasks.
  • Prediction target: Predicting semantic entropy from hidden states is generally easier than directly predicting model correctness.The comparison covers Llama-2-7B, Llama-2-70B, Mistral-7B, and Phi-3 Mini in short-form in-distribution settings.
  • Generalization: SEPs outperform accuracy probes when generalizing to unseen tasks, while their in-distribution performance is comparable or model-dependent.The figures report this pattern for Mistral-7B in short generations and Llama-2-70B and Llama-3-70B in long generations.
  • Model components: Hidden states are more predictive for SEPs than residual streams or MLP outputs in the reported TriviaQA experiment.The comparison uses Llama-2-7B, short-form generations, SLT representations, and in-distribution evaluation.
  • Binarization: The best-split binarization procedure slightly outperforms even split, but SEPs are not overly sensitive to binarization.The comparison is reported for long-form Llama-2-70B generations using SLT representations.

B Experiment Details

The experiments use standardized prompts, sampled generations for semantic-entropy computation, automated entailment and correctness assessments, and linear probes trained on hidden states. Evaluation details specify layer selection, filtering, matched training sizes, and leave-one-out task generalization.

  • Prompting: Short-form prompts include five demonstration examples, while long-form prompts request a single brief but complete sentence.A separate counterfactual experiment prepends context before the question.
  • Evaluation setup: Semantic entropy is computed from N = 10 generations sampled at temperature T = 1.0 with top-p p = 0.9 and top-K K = 50.These are the stated sampling settings for the experiments.
  • Evaluation metrics: Short-form entailment is assessed with DeBERTa-Large and SQuAD F1, while long-form entailment uses GPT-3.5 and correctness uses GPT-4 or GPT-4o3.GPT-4 evaluates Llama-2-70B, while GPT-4o is used for more recent Llama-3-70B experiments because of cost.
  • Probe inputs: SEPs are trained on model hidden states whose dimensionality and layer counts vary across models.The selected model properties and concatenated layers are reported in Table 4.
  • Layer selection: Aggregate results use adjacent-layer concatenations selected from high-performing layers based on in-distribution AUROC.The same layer-selection procedure is applied to SEP and accuracy-probe results.
  • Training data: Long-form SEP training filters samples between the 55% and 80% semantic-entropy quantiles, while accuracy-probe results use unfiltered data.The filtering gave a mild SEP performance increase but did not improve accuracy probes.
  • Training data: Training sets contain 1000 long-generation samples or 2000 short-generation hidden-state–semantic-entropy pairs, matched between probe types.The matched sizes support comparison between SEPs and accuracy probes.
  • Generalization evaluation: Generalization uses leave-one-out evaluation, training each probe on one dataset and evaluating AUROC on all other datasets.The reported result averages performance across probes evaluated on each held-out dataset.

C Compute Resources

The experiments use internal A100 GPU resources alongside OpenAI API models, with CPU-only fitting after probe training data is created. The tracked runs require approximately 300 GPU-hours and 310 CPU-hours.

  • The experiments use an internal cluster with 24 Nvidia A100 80GB GPUs and access GPT 3.5, 4, and 4o through the OpenAI API.
  • Llama 70B inference and hidden-state computation require two A100s, while smaller models use only a slice of one A100 80GB.
  • After semantic entropy probe training data is created, CPU-only resources suffice to fit the logistic regression models.
  • ~300 GPU-hours plus ~310 CPU-hours are estimated from tracked finished runs to obtain the paper’s results.
Loading 2406.15927v1…