PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models

Weihang Su, Changyue Wang, Qingyao Ai, Yiran HU, Zhijing Wu, Yujia Zhou, Yiqun Liu

arXiv:2403.06448v2cs.CLcs.AI

TL;DR

Hallucinations are coherent but factually incorrect or misleading outputs, while post-processing detectors can be computationally costly and are separated from LLM inference. MIND uses LLM internal states for unsupervised, real-time detection without manual annotations, and HELM provides outputs and internal states from six LLMs. The experiments report that MIND outperforms existing hallucination detection methods.

  • Problem

    Hallucinations undermine LLM effectiveness in real-world applications, while post-processing detectors can be costly and are separated from inference.

    Source: §1 · p. 1

  • Method

    MIND is an unsupervised framework that uses LLM internal states for real-time hallucination detection without manual annotations.

    Source: §1 · p. 2

  • Results

    MIND demonstrates broad effectiveness across six LLMs and outperforms existing reference-free methods at sentence and passage levels.

    Source: §6.1 · p. 7

  • Takeaways & Limitations

    HELM provides outputs from six LLMs and their internal states to facilitate future research and improve reproducibility.

    Source: §1 · p. 2

    Using only LLM internal states may limit accuracy; future work aims to combine them with generated text to improve precision and reliability.

    Source: §9 · p. 9

Abstract

from arXiv · show

Hallucinations in large language models (LLMs) refer to the phenomenon of LLMs producing responses that are coherent yet factually inaccurate. This issue undermines the effectiveness of LLMs in practical applications, necessitating research into detecting and mitigating hallucinations of LLMs. Previous studies have mainly concentrated on post-processing techniques for hallucination detection, which tend to be computationally intensive and limited in effectiveness due to their separation from the LLM's inference process. To overcome these limitations, we introduce MIND, an unsupervised training framework that leverages the internal states of LLMs for real-time hallucination detection without requiring manual annotations. Additionally, we present HELM, a new benchmark for evaluating hallucination detection across multiple LLMs, featuring diverse LLM outputs and the internal states of LLMs during their inference process. Our experiments demonstrate that MIND outperforms existing state-of-the-art methods in hallucination detection.

1 Introduction

The introduction frames hallucination as a practical problem and argues that post-processing detectors face computational and model-capacity limitations. It presents MIND, an unsupervised real-time detector based on LLM internal states, and HELM, a benchmark containing outputs and internal states from six LLMs; the reported experiments find MIND outperforms existing methods.

1 Introduction

  • Hallucinations are coherent LLM responses that are factually incorrect or misleading, harming effectiveness and robustness in real-world NLP applications.The introduction motivates research into detecting and mitigating hallucinations.Source: §1 · p. 1
  • Prior hallucination detectors mainly identify fact-related errors in generated text after LLM inference finishes.The introduction calls these post-processing methods.Source: §1 · p. 1
  • Post-processing detectors can incur extreme computation costs and latency, and their separation from inference limits analysis of how hallucinations arise in each LLM.Proxy models may require extensive, costly human annotations to capture individual LLM characteristics.Source: §1 · pp. 1–2
  • MIND is a reference-free, unsupervised framework that extracts pseudo-training data from Wikipedia and uses LLM contextual embeddings for real-time detection without manual annotations.The introduction also describes MIND as lightweight and compatible with Transformer-based LLMs.Source: §1 · p. 2
  • HELM provides generated text and human-annotated labels from six LLMs, alongside their contextualized embeddings, self-attentions, and hidden-layer activations.The benchmark is introduced to facilitate future research and improve reproducibility.Source: §1 · p. 2
  • MIND outperforms existing hallucination detection methods in experiments evaluated with human annotations.The paper presents MIND as an unsupervised, real-time framework based on LLM internal states.Source: §1 · p. 2

2 Problem Formulation

The paper frames hallucination detection as binary classification of LLM outputs. Unlike fact verification, it examines characteristics intrinsic to generative models and may use signals such as response consistency, model confidence, or internal states.

2 Problem Formulation

  • Hallucination detection is a binary classification task that labels an LLM output as hallucination or non-hallucination.The labels correspond to false or misleading information versus accurate and relevant information.Source: §2 · p. 2
  • Unlike traditional fact verification, hallucination detection analyzes characteristics intrinsic to generative models, beyond factual accuracy alone.The paper describes fact verification as primarily assessing textual factual accuracy.Source: §2 · p. 2
  • Example detection signals include consistency across responses, the LLM's confidence, and its internal states during generation.The cited examples include Manakul et al. (2023), Zhang et al. (2023b), and Azaria and Mitchell (2023), respectively.Source: §2 · p. 2

3 Methodology

MIND automatically generates training data from Wikipedia continuations and trains a classifier on LLM internal states for real-time hallucination detection. The feature-selection experiment reports that a simple MLP can distinguish hallucination states using the final-layer embedding of the last token.

3.1 Unsupervised Training Data Generation

  • MIND truncates Wikipedia articles at selected entities, prompts a chosen LLM to continue them, and labels continuations by whether they retain the original entity.Source: §3.1 · pp. 2–3

3.2.1 Feature Selection

  • A simple MLP distinguishes hallucination from non-hallucination embeddings, with effective distinction using only the last token’s final-layer embedding.Source: §3.2.1 · pp. 3–4

3.2.2 Training Process

  • MIND trains an MLP with ReLU on the final-layer, last-token contextualized embedding to predict a binary hallucination label, using binary cross-entropy loss.Source: §3.2.2 · p. 4

3.3 Real-time Hallucination Detection

  • During inference, the classifier receives contextualized embeddings of generated tokens and outputs a hallucination probability for the LLM’s output.Source: §3.3 · p. 4

4 The HELM Benchmark

HELM is a benchmark for hallucination detection across multiple LLMs, with generated text, human annotations, and model states. Its construction covers data generation, annotation, and sentence- and passage-level evaluation.

4.1 Data Generation

  • 50,000 WikiText-103 articles were sampled for continuation writing by six open-source LLMs spanning base and chat models from 6B to 40B parameters.The models include Falcon-40B, GPT-J-6B, LLaMA-2-Base-7B, two LLaMA-2-Chat sizes, and OPT-6.7B.Source: §4.1 · p. 4

4.2 Human Annotation

  • Annotators assessed factual errors, verified each sentence using Wikipedia and Google, and retained only annotations that matched across two annotators.Grammatical mistakes and subjective opinions were excluded; unverifiable sentences were marked as such.Source: §4.2 · pp. 4–5

4.3 Benchmark Analysis and Usage

  • HELM contains 3,342 labeled sentences from 1,224 passages, along with contextualized embeddings and hidden-layer activations recorded during generation.It supports sentence-level detection and passage-level detection, where any hallucinated sentence makes the passage hallucinated.Source: §4.3 · p. 5
  • Users can access HELM through the project’s open-source repository or download the data and implement it themselves.The dataset and code are publicly shared.Source: §4.3 · p. 5

5 Experimental Settings

The experiments evaluate reference-free hallucination detectors on HELM at sentence and passage levels, using AUC and correlation with human judgments. The comparison includes uncertainty, consistency, activation-based, and prompted-LLM baselines, with MIND implemented as an MLP classifier.

5.1 Dataset and Metrics

  • AUC and Pearson correlation with human-annotated relevance were measured for sentence-level and passage-level hallucination detection on HELM.Source: §5.1 · p. 5

5.2 Baselines

  • Predictive Probability and Predictive Entropy use token probabilities or uncertainty, while SelfCheckGPT scores consistency across sampled responses.The baselines include pooling variants for token scores and four SelfCheckGPT techniques: BERTScore, QA, NLI, and n-gram.Source: §5.2 · p. 5
  • Other baselines include SAPLMA’s activation-based classifier, uncertainty-based EUBHD, and GPT4-HDM’s prompted use of GPT-4 as a detector.Source: §5.2 · pp. 5–6

5.3 Implementation Details

  • MIND uses a four-layer MLP with hidden dimensions 256, 128, 64, and 2; the selected LLMs use their official default generation configurations.The implementation also uses Spacy for named entity recognition.Source: §5.3 · p. 6

6 Experimental Results

The experiments assess MIND across LLMs, compare its effectiveness and efficiency with baselines, and examine training-data and classifier choices.

6.1 Overall Results of MIND and Baselines

  • MIND is broadly effective across six LLMs and outperforms existing reference-free methods at sentence and passage levels, though SCG-NLI leads on some models.SelfCheckGPT detection takes ten times longer than response generation; SAPLMA performs suboptimally in real-time detection.Source: §6.1 · p. 7

6.2 Efficiency

  • 3% of the LLM’s response generation time is sufficient for MIND to detect hallucinations on LLaMA-7B-Chat.Considering both efficiency and effectiveness, MIND is reported as the superior reference-free method; predictive probability and entropy require negligible time.Source: §6.2 · p. 7, Table 3 · p. 7

6.3.1 Impact of Customized Training Data

  • Customized training data significantly enhances hallucination-classifier performance for both OPT-7B and LLaMA2-Chat-7B.The comparison contrasts model-specific training data with data generated by another LLM and evaluates correlation on HELM.Source: Table 4 · p. 7, §6.3.1 · p. 8

6.3.2 Impact of the Training Data Size

  • Increasing training data from 1,024 to 4,096 labels improves accuracy, but gains plateau and slightly decrease beyond 4,096 data points.The data-size experiment reports development-set accuracy.Source: §6.3.2 · p. 8, Table 5 · p. 8

6.3.3 Classifier Layer Depth

  • Classifier accuracy rises from 0.7157 with two layers to a peak of 0.7291 with four, with small differences across depths.The tested classifier depths ranged from two to six layers, and development-set accuracy was the metric.Source: §6.3.3 · p. 8, Table 6 · p. 8

7 Related Work

Related work covers hallucination-detection methods and the datasets used to evaluate them.

7.1 Hallucination Detection

7.2 Evaluation of Hallucination Detection

8 Conclusions and Future Works

The conclusion presents MIND for real-time detection using LLM internal states and HELM as a benchmark spanning six LLMs.

8 Conclusions and Future Works

  • MIND uses LLM internal states for real-time hallucination detection, while HELM includes outputs and generation-time internal states from six LLMs.Source: §8 · p. 9

9 Limitations

The paper identifies internal-state-only hallucination detection as a limitation and proposes combining internal states with generated text in future work.

9 Limitations

  • Using only LLM internal states is identified as a limitation of the detection approach.The authors state that this approach has potential for enhanced accuracy.Source: §9 · p. 9
  • Future work will integrate LLM internal states with generated text to improve hallucination detection precision and reliability.Source: §9 · p. 9
  • The proposed combined approach aims to support more robust and accurate hallucination detection and mitigation.Source: §9 · p. 9

10 Ethics Statement

The ethics statement emphasizes public data and reproducibility; the appendices describe MIND data generation, HELM prompts, and GPT-4 judging.

10 Ethics Statement

  • MIND primarily uses publicly available sources such as Wikipedia, avoids personally identifiable information, and makes models, data, and code publicly available.The authors connect this transparency with verifying findings and encouraging application in other contexts.Source: §10 · p. 9

B Prompt Template of the HELM Dataset

  • HELM data are generated through prompt-based continuation, with different prompt templates for base LLMs and chat LLMs.The supplied templates seed generation with a Wikipedia article’s title and first sentence.Source: B Prompt Template of the HELM Dataset · p. 11

C Prompt Template of GPT4

  • GPT-4 judges whether generated text contains hallucinated or non-factual information using world knowledge and a required Yes-or-No response.The approach follows Li et al. (Li et al., 2023d).Source: C Prompt Template of GPT4 · pp. 11–12

A Pseudocode Description of MIND

  • MIND’s unsupervised training-data generation uses a language model and Wikipedia articles, selects a random entity, generates and truncates a continuation, and records internal states.The algorithm description also checks whether the continuation begins with the selected entity correctly.Source: C Prompt Template of GPT4 · p. 12

D Details of Our Selected LLMs

The appendices describe selected open-source LLMs, implementation settings, and two proposed uses of MIND to mitigate hallucinations.

D Details of Our Selected LLMs

  • The experiments use 14 open-source LLMs, including OPT-6.7B and four LLaMA-2 base or chat models.The selected LLaMA-2 models are 7B and 13B variants.Source: D Details of Our Selected LLMs · p. 12

E Implementation Details

  • The MIND classifier is a four-layer MLP with hidden dimensions 256, 128, 64, and 2, ReLU activations, and 20% initial-layer dropout.Training uses learning rate 5e-4, weight decay 1e-5, and batch size 32.Source: E Implementation Details · p. 12
  • SCG uses its official code and hyperparameters, while selected LLMs use Hugging Face generation code and their official default configurations.Source: E Implementation Details · p. 13

F Discussion on Mitigating Hallucination

Loading 2403.06448v2…