Source-linked AI summary
The Internal State of an LLM Knows When It's Lying
Amos Azaria, Tom Mitchell
TL;DR
LLMs can produce confident false information, creating a need for reliable statement-truthfulness detection. The paper trains SAPLMA to classify truthfulness from hidden-layer activations, including across held-out topics. SAPLMA reaches 60% to 80% accuracy on OPT-6.7b topics and 70% to 90% on LLAMA2-7b topics, exceeding few-shot prompting’s maximum 56%.
Problem
LLMs can generate inaccurate information in confident language, while sentence probability is confounded by sentence length and token frequency.
Method
SAPLMA trains a classifier on LLM hidden-layer activations to predict statement truthfulness, testing generalization across held-out topics and multiple layers.
Results
60% to 80% accuracy is achieved with OPT-6.7b and 70% to 90% with LLAMA2-7b, compared with few-shot prompting’s maximum 56%.
Takeaways & Limitations
The findings suggest that LLM internal representations of statement accuracy can be harnessed to filter incorrect information and supplement LLM responses.
Takeaways & Limitations
The method focuses on binary truth classification, while practical use may require calibrated uncertainty and may inherit biases from the underlying LLM.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates. Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71\% to 83\% accuracy labeling which sentences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM. We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.
1 Introduction
LLMs can generate confident false information, but their internal activations may encode whether statements are true or false. SAPLMA uses these activations to detect truthfulness across topics and can outperform prompting and sentence probability.
- Confidently generated false information may cause harm because people can accept LLMs as knowledgeable sources.
- Token-by-token generation can commit an LLM to an inaccurate completion even when later context makes the full statement unlikely.Examples include incorrect claims about Pluto and Tiztoutine.
- SAPLMA classifies statement truthfulness from hidden-layer activations, using training data from topics separate from the held-out test topic.The out-of-distribution setup is intended to extract internal belief rather than topic-specific alignment patterns.
- 60% to 80% accuracy is achieved on specific topics, compared with no more than 56% for few-shot prompting.
- Sentence probability is a weak truthfulness signal because it depends on token frequencies and sentence length, whereas SAPLMA significantly outperforms simple sentence probability.
- SAPLMA can run alongside LLM output to flag or remove statements classified as false.The proposed use is to mark questionable statements or regenerate them before users receive them.
2 Related Work
Prior hallucination research commonly evaluates generated content against an input source, uses prompting or fine-tuning, or studies task-specific datasets. This paper instead focuses on hidden-layer activations to assess the veracity of generated statements without fine-tuning or task-specific modifications.
- Machine-translation hallucination methods can assess how much a generated translation is detached from its source.
- Summarization research includes benchmarks built from human annotations of factual errors in system-generated summaries.
- Black-box approaches can use repeated or rephrased queries and compare the resulting responses to detect false statements.
- Fine-tuning with human feedback can improve generated content but may reduce performance on other tasks.
- This work focuses on hidden-layer activations and proposes operation alongside an LLM without fine-tuning or task-specific modifications.
3 The True-False Dataset
The paper releases a true-false dataset designed to test whether LLM activations encode statement truth independently of topic. It combines curated factual statements and their opposites across six disjoint topics.
- The dataset requires clearly labeled true and false statements grounded in information present in the LLM’s training data.
- Six covered topics are Cities, Inventions, Chemical Elements, Animals, Companies, and Scientific Facts.
- For five topics, statements were composed from reliable property tables by pairing instances with properties and generating false variants through mismatched values.
- Scientific Facts statements were generated by ChatGPT, reversed into false claims, manually curated, and verified by two human annotators.
- The dataset contains 6,084 sentences spanning the six topics.
4 SAPLMA
SAPLMA predicts statement truthfulness from hidden-layer activations using classifiers trained across topics and tested on held-out topics. The method evaluates multiple layers and two 32-layer LLMs to identify transferable truth-related representations.
- SAPLMA tests whether hidden-layer values contain information about whether an LLM regards a statement as true or false.
- The method evaluates activations from five candidate layers in Facebook OPT-6.7b and LLAMA2-7b, each with 32 layers.
- The classifier is a feedforward neural network with three ReLU hidden layers and a sigmoid output.
- Each classifier is trained on activation values from all topics except the topic used for testing.This requires truth classification to generalize beyond the training topics.
5 Results
SAPLMA uses hidden-layer activations to classify statement truthfulness and generally outperforms prompting and probability-based baselines. Performance varies by model, layer, topic, and evaluation setting, with weaker results on self-generated statements.
- Experimental setup: SAPLMA classifies truthfulness from LLM hidden-layer activations, while the baselines include BERT embeddings and few-shot prompting.The few-shot baseline attempts to elicit the LLM’s own truth judgment explicitly.
- Externally generated statements: SAPLMA outperforms BERT and few-shot learning on OPT-6.7b, whose baselines achieve only slightly above random accuracy of 0.50.The data are balanced, so random classification corresponds to 0.5 accuracy.
- Training analysis: Randomly permuting labels reduces SAPLMA’s average training accuracy to 62.5%, versus 86.4% with the original labels.This result is presented as evidence that the classifier exploits structure and patterns rather than fully memorizing the training data.
- LLM-generated statements: On LLM-generated statements, SAPLMA outperforms baselines but achieves near 50% accuracy, with higher-threshold evaluation improving performance.The generated statements were fact-checked by three judges, whose average observed agreement was 97.82% and average Cohen’s Kappa was 0.9566.
6 Discussion
The discussion contrasts SAPLMA’s truthfulness judgments with sentence probabilities, which are sensitive to wording and length. The authors also restrict evaluation to held-out topics to test topic-independent internal representations.
- Evaluation scope: Held-out-topic testing is intended to measure whether SAPLMA extracts internal belief rather than topic-specific patterns.Training and testing use different topics, excluding models trained or fine-tuned on the test topic.
- Probability versus truthfulness: Sentence probabilities are strongly affected by syntax, token frequency, and sentence length, weakening their use as general truthfulness signals.The discussion notes that probabilities may compare similar sentences but cannot alone determine truthfulness generally.
- Probability versus truthfulness: SAPLMA remains accurate on examples where high-probability wording is false or unusual wording is assigned low probability.It classifies both “The Earth is flat” variants as false and handles rare words, typos, and implicit truths more reliably than sentence probability.
- Future directions: Future work may combine generated-word probabilities with activation features, although those probabilities could be redundant.The proposed combination is motivated by probabilities carrying information despite being insufficient alone.
7 Conclusions & Future Work
The conclusions present SAPLMA as a hidden-activation method for detecting false generated statements and report better performance than few-shot prompting. They propose filtering or marking questionable outputs and extending evaluation to larger models and human users.
- Conclusions: SAPLMA predicts generated-statement truthfulness from LLM hidden-layer activations.The method is presented as the paper’s central contribution for addressing incorrect and false information.
- Conclusions: SAPLMA reaches 60%–80% accuracy with OPT-6.7b and 70%–90% with LLAMA2-7b on specific topics, versus at most 56% for few-shot prompting.The reported ranges concern specific topics and the prompting baseline is stated for OPT-6.7b.
- Conclusions: The findings suggest that LLMs contain an internal representation of statement accuracy that differs from sentence probability.The authors propose using this representation to filter incorrect information and support more reliable responses.
- Future work: Future work will test larger LLMs, human interactions, and how activations evolve as longer responses are generated.The planned human study compares unfiltered systems with systems augmented by SAPLMA.
8 Limitations
The paper’s limitations concern the binary truthfulness target, English-only evaluation, and activation dynamics in longer responses. The authors describe calibration and temporal decoupling as needed extensions.
- Scope of detection: Binary true-or-false detection may be less useful in practice than distinguishing confidence from uncertainty.The authors suggest calibrated thresholds, multiple classifiers, or dropout-based approaches for uncertainty-sensitive outputs.
- Language scope: The work was tested only in English, leaving multilingual transfer as an untested hypothesis.The authors plan to evaluate whether training in one language can support application in another.
- Response context: Activations collected for isolated sentences may not directly identify truthfulness within longer responses.Long responses cause activations to develop across preceding correct and incorrect information, motivating decoupling methods.
9 Ethical Impact
The ethical discussion frames SAPLMA as a possible way to reduce false information while warning that biases in the underlying LLM may transfer into its classifications.
- Potential benefit: SAPLMA could potentially reduce the ethical risk posed by LLM-generated false information.The authors identify false-information generation as a primary ethical concern.
- Residual risk: Biases in the original LLM may persist in SAPLMA’s true-or-false classifications.The authors specifically discuss inherited bias toward ethnic groups and suggest adaptation for bias mitigation.