PaperlayerView exact revision on arXiv

Source-linked AI summary

INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, Jieping Ye

arXiv:2402.03744v2cs.CL

TL;DR

Knowledge hallucinations threaten the security and reliability of deployed LLMs, while existing logit- and language-level methods can lose dense semantic information. INSIDE uses LLM internal states through EigenScore and test-time feature clipping, achieving improved or state-of-the-art detection across QA benchmarks, with limitations for black-box models and added inference overhead.

  • Problem

    Knowledge hallucinations make LLM generations unreliable for practical deployment, while prior logit- and language-level detectors do not fully retain sentence-level semantic information.

  • Method

    INSIDE detects hallucinations from LLM internal states using EigenScore for embedding-space semantic consistency and feature clipping to truncate extreme activations.

  • Results

    The method achieves state-of-the-art hallucination detection on several QA benchmarks, with EigenScore outperforming Lexical Similarity by 5.6% in CoQA and 8.9% in SQuAD under AUROC at most.

  • Takeaways & Limitations

    Internal LLM states provide a basis for measuring semantic consistency and detecting self-consistent overconfident hallucinations across multiple QA settings.

  • Takeaways & Limitations

    The method cannot be applied to black-box models without accessible hidden states, requires multiple generated outputs, and does not mitigate hallucinations.

Abstract

from arXiv · show

Knowledge hallucination have raised widespread concerns for the security and reliability of deployed LLMs. Previous efforts in detecting hallucinations have been employed at logit-level uncertainty estimation or language-level self-consistency evaluation, where the semantic information is inevitably lost during the token-decoding procedure. Thus, we propose to explore the dense semantic information retained within LLMs' \textbf{IN}ternal \textbf{S}tates for halluc\textbf{I}nation \textbf{DE}tection (\textbf{INSIDE}). In particular, a simple yet effective \textbf{EigenScore} metric is proposed to better evaluate responses' self-consistency, which exploits the eigenvalues of responses' covariance matrix to measure the semantic consistency/diversity in the dense embedding space. Furthermore, from the perspective of self-consistent hallucination detection, a test time feature clipping approach is explored to truncate extreme activations in the internal states, which reduces overconfident generations and potentially benefits the detection of overconfident hallucinations. Extensive experiments and ablation studies are performed on several popular LLMs and question-answering (QA) benchmarks, showing the effectiveness of our proposal.

1 INTRODUCTION

LLM hallucinations threaten reliable deployment, while existing token- and language-level detectors can lose sentence-level semantic information. INSIDE instead uses LLM internal states, EigenScore, and test-time feature clipping for hallucination detection.

  • Hallucinated LLM generations can be factually incorrect or nonsensical, making practical deployment risky.
  • Existing token-level uncertainty methods face challenges deriving sentence-level uncertainty for diverse, sophisticated autoregressive responses.
  • INSIDE leverages LLM internal states to detect hallucinations directly in the sentence embedding space.The framework avoids secondary semantic extraction via extra models.
  • EigenScore uses covariance-matrix eigenvalues to measure semantic consistency in the embedding space and represents differential entropy there.
  • Test-time feature clipping truncates extreme activations, implicitly reducing overconfident generations and helping identify overconfident hallucinations.
  • The method achieves state-of-the-art hallucination detection performance across several QA benchmarks with extensive ablation studies.

2 BACKGROUND ON HALLUCINATION DETECTION

Hallucination detection can estimate uncertainty from token probabilities, multiple sampled responses, or semantic consistency. These approaches address sequence-level uncertainty but differ in how effectively they capture sentence semantics and response diversity.

  • Sequence uncertainty can be defined from token probabilities, with perplexity using the joint probability of generated tokens.The output length T normalizes the joint probability because shorter sequences generally have lower perplexity.
  • Averaging token-level uncertainty may miss whole-sequence uncertainty because tokens contribute differently to sentence semantics.
  • Multiple generations for one input support sequence-level uncertainty estimation through top-p or top-k sampling during inference.
  • Figure 1 combines penultimate-layer feature truncation with EigenScore computed from sentence embeddings across multiple responses.
  • When models are uncertain, hallucinated contexts are associated with higher answer-distribution entropy, with length-normalized entropy outperforming the non-normalized version.
  • Semantic consistency methods assume confident models generate similar outputs, and lexical similarity measures average similarity across multiple answers.

3 METHOD

INSIDE detects hallucinations from dense internal-state representations rather than logit- or language-level uncertainty. Its EigenScore measures semantic divergence across generated sentence embeddings, while test-time feature clipping addresses overconfident, self-consistent hallucinations by truncating extreme activations.

  • Framework: INSIDE uses LLM internal states to detect hallucinations without relying on logit- or language-space uncertainty.The framework directly analyzes dense semantic information retained in internal states.
  • EigenScore: The main experiments use the last-token embedding from the middle layer as the sentence embedding because it effectively captures sentence semantics.The paper also considers averaging token embeddings and compares alternative sentence embeddings in ablations.
  • EigenScore: EigenScore measures semantic consistency or divergence from the eigenvalues of generated sentence embeddings’ covariance matrix.For K generated sequences, sentence embeddings are used to compute a covariance matrix, whose logarithmic determinant defines the score.
  • EigenScore: EigenScore represents differential entropy in the sentence embedding space, linking larger covariance eigenvalue structure to semantic divergence among responses.The paper motivates LogDet through the differential entropy of a multivariate Gaussian distribution.
  • Test Time Feature Clipping: Test-time feature clipping truncates penultimate-layer activations outside minimum and maximum thresholds to reduce overconfident generations.The thresholds can be determined from activation percentiles maintained through a dynamic memory bank during inference.

4 EXPERIMENTS

Experiments evaluate INSIDE across four QA datasets, two open-source LLM families, multiple metrics, and ablations of generation, embeddings, correctness thresholds, and decoding settings. EigenScore generally performs strongly, while feature clipping improves detection and performance depends on experimental choices.

  • Experimental Setup: The evaluation uses CoQA, SQuAD, TriviaQA, and NQ with LLaMA-7B, LLaMA-13B, and OPT-6.7B, without fine-tuning.AUROC and PCC measure hallucination detection, using ROUGE-L and semantic similarity for correctness.
  • Main Results: EigenScore consistently outperforms comparison methods in CoQA, SQuAD, and NQ, exceeding Lexical Similarity by up to 5.6% in CoQA and 8.9% in SQuAD AUROC.Perplexity performs best on TriviaQA, where generations and reference answers are usually very short.
  • Main Results: LLaMA-13B performs better on average than LLaMA-7B and OPT-6.7B, while LLaMA-7B is slightly better than OPT-6.7B.The authors associate stronger detection performance with a more powerful pretrained LLM.
  • Main Results: 1.8% is the largest AUROC improvement from applying test-time feature clipping across evaluated methods.The comparison includes methods with and without clipping, including EigenScore without clipping.
  • Ablation Studies: For LLaMA-7B on NQ, EigenScore remains ahead across generation counts; performance rises below K=15, stabilizes above K=15, and K=20 balances performance and inference cost.The tested generation counts are 5, 10, 15, 20, 30, and 40.
  • Ablation Studies: Middle-layer sentence embeddings outperform shallow and final layers, while the last-token embedding outperforms averaged token embeddings.The main implementation uses the last token from the middle layer, with index int(L/2).
  • Ablation Studies: EigenScore remains superior across correctness thresholds, and stricter correctness measures produce better hallucination detection performance.The analysis uses ROUGE-L and sentence similarity thresholds on LLaMA-7B and CoQA.
  • Ablation Studies: Performance is highly sensitive to temperature but shows little sensitivity to top-k; consistency-based methods decline significantly when temperature exceeds 1.The optimal temperature is reported as [0.1, 1.0].

5 RELATED WORK

Prior reliability research addresses unreliable LLM generations using uncertainty estimation and related detection methods. Covariance eigenvalues and determinants provide an established basis for measuring data variability and divergence.

  • Reliability Evaluation of LLMs: LLM reliability evaluation is motivated by unreliable generations and includes uncertainty-based approaches such as predictive confidence, entropy, and related detection methods.The cited literature targets security and reliability evaluation of deployed LLMs.
  • Eigenvalue as Divergence Measure: Covariance-matrix eigenvalues and determinants have been widely used to capture data variability and serve as divergence measures in machine learning.The related work cites PCA and applications spanning multiple machine-learning tasks.

6 CONCLUSION

The conclusion presents INSIDE as a framework for exploiting LLM internal-state semantics in hallucination detection. It combines EigenScore for cross-generation semantic consistency with feature clipping for self-consistent overconfident hallucinations.

  • Conclusion: INSIDE uses semantic information retained in LLM internal states to detect hallucinations across generations in embedding space.The authors report significant performance improvement on several LLMs and QA benchmarks.
  • Conclusion: EigenScore measures semantic consistency across generations, while feature clipping truncates extreme features to reduce overconfident generations.The authors frame clipping as addressing self-consistent hallucinations overlooked by previous methods.
  • Conclusion: Although experiments focus on QA, the method makes no task-modality assumptions and is presented as potentially applicable to summarization and translation.The broader applicability is stated as an expectation for future use, not evaluated evidence here.

A PERFORMANCE EVALUATION ON TRUTHFULQA

TruthfulQA evaluation compares INSIDE with baseline detectors under prompt-based inference. Performance improves with more in-distribution prompts, while the trained ITI baseline has a validation-set advantage but weaker cross-domain generalization.

  • TruthfulQA Evaluation: Increasing the number of in-distribution prompts significantly improves zero-shot TruthfulQA inference performance.TruthfulQA is described as especially challenging for LLMs.
  • TruthfulQA Evaluation: ITI reports 83.3 on its validation set, exceeding the untrained proposal, but validation-set training limits its generalization to other domains.ITI trains 1024 binary classifiers using internal embeddings.

B COMPARISON WITH MORE COMPETITIVE METHODS

EigenScore is compared with Semantic Entropy, SentSAR, and SelfCheckGPT for hallucination detection, with the paper reporting superior performance. The comparison also highlights that EigenScore uses internal states rather than another language model for sentence embeddings.

  • EigenScore significantly outperforms Semantic Entropy, SentSAR, and SelfCheckGPT in the reported comparison.SentSAR and SelfCheckGPT perform comparably and both outperform Semantic Entropy.
  • EigenScore evaluates self-consistency across multiple outputs without relying on another language model for sentence embedding extraction.

C PERFORMANCE EVALUATION ON MORE LLMS

The paper evaluates hallucination detection across additional LLMs and correctness measures. It reports consistently superior performance across LLaMA2-7B and Falcon-7B, while also comparing inference cost and exact-match evaluation.

  • Performance evaluation on more LLMs: EigenScore consistently exhibits superior performance across LLaMA2-7B and Falcon-7B compared with the other methods.
  • Inference cost comparison: 10 generations are used in the LLaMA-7B and LLaMA-13B inference-cost comparison, conducted on NVIDIA-A100 hardware.The average inference time per question is reported in Fig. 5.
  • Exact-match evaluation: The paper additionally evaluates performance using exact match as a stricter correctness measure.The main experiments use ROUGE and sentence similarity as correctness measures.

F MORE VISUALIZATION AND ABLATION FOR FEATURE CLIPPING

The feature-clipping analysis shows that activation distributions vary across samples, making single-sample threshold selection risky. The ablation compares current-sample, precomputed-batch, and memory-bank thresholding techniques.

  • Visualization: Activation distributions vary substantially across samples for four selected LLaMA-7B tokens.
  • Threshold selection: EigenScore-C is risky because it determines the clipping threshold using only the current input sample.
  • Threshold selection: EigenScore-P precomputes the threshold from a batch, whereas EigenScore-MB determines it dynamically with recorded activation values.

G LIMITATIONS AND FUTURE WORK

The study is limited by its dependence on accessible internal LLM information and by the overhead of sampling multiple outputs. It focuses on hallucination detection rather than mitigation, while examples illustrate EigenScore's behavior on varied QA responses.

  • Limitations: The method cannot be applied to black-box models because users cannot access their hidden states.
  • Limitations: Sampling multiple outputs introduces additional inference overhead for accurate hallucination detection.
  • Future work: The study focuses on hallucination detection and does not yet mitigate hallucinations with the proposal.
  • Examples: The QA examples report EigenScore alongside other metrics for correct, incorrect, consistent, and diverse model responses.

H.2 MODEL GENERATIONS WITH MANY AND FEW OUTLIERS

Examples contrast self-consistent hallucinations, which repeat the same incorrect answer, with diverse generations that detection metrics can identify. Feature clipping suppresses overconfident generations and enables some previously missed self-consistent hallucinations to be identified.

  • Feature clipping suppresses overconfident generations and allows some self-consistent hallucinations to be identified.
  • 15 outliers produced the same incorrect answer, “freddie mercury,” across all ten generations.
  • 13 outliers likewise yielded repeated “California” answers despite Washington being the ground-truth answer.
  • 3 outliers produced varied answers about Iraq and Kuwait, while every listed detection metric correctly identified the hallucination.
Loading 2402.03744v2…