The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

Samuel Marks; Max Tegmark

Source-linked AI summary

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

Samuel Marks, Max Tegmark

arXiv:2310.06824v3Oct 10, 2023cs.AI

TL;DR

LLMs can produce falsehoods, and existing activation-based truthfulness probes have raised questions about generalization and interpretation. This paper studies simple true/false datasets with visual, transfer, probing, and causal analyses, finding evidence that sufficiently scaled LLMs linearly represent factual truth and that mass-mean probes identify especially causal directions. The results are limited by the use of simple, uncontroversial statements and LLaMA-2 models בלבד.

Problem
Existing methods for inferring whether an LLM believes a statement is true have controversial efficacy and interpretation, motivating study of whether truth is represented across diverse factual inputs.
Method
The paper combines curated true/false datasets with PCA visualizations, cross-dataset probe-transfer analyses, and causal interventions on LLaMA-2 representations.
Results
The authors find evidence that sufficiently scaled LLMs compute and linearly represent truth, while mass-mean probes identify directions more causally implicated than other probing techniques.
Takeaways & Limitations
Truth may be represented as an increasingly general linear abstraction in larger LLMs, spanning topically and structurally diverse factual statements.
Takeaways & Limitations
The study uses simple, uncontroversial statements and only LLaMA-2 models, so it cannot distinguish truth from related features or establish applicability to all LLMs.

Abstract

from arXiv · show

Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

1 Introduction

The paper examines whether LLMs internally represent factual truth, amid concerns that existing probes may detect dataset-specific correlates rather than truth itself. Using multiple analyses, it finds evidence for scalable linear truth representations and causally implicated directions.

Motivation: Existing truthfulness probes use model outputs or internal activations, but their efficacy and interpretation remain controversial.Reported failures include poor generalization to statements containing “not.”
Approach: The study curates simple true/false datasets and combines PCA visualizations, probe-transfer analyses, patching, and causal interventions.These analyses target representations across structurally and topically diverse factual statements.
Findings: At sufficient scale, LLMs show evidence of linear truth representations that apply across structurally and topically diverse inputs.The authors describe larger models as having a more abstract notion of truth.
Findings: A small group of hidden states is causally implicated in encoding truth representations.Causal interventions are part of the evidence used to identify these states.

2 Datasets

The paper constructs datasets of simple factual statements to test whether LLMs share a unified truth representation across structural and topical variation. Additional datasets separate truth from text probability and assess generalization to more challenging statements.

The primary goal is to test whether LLMs have a unified representation of factual truth across structurally and topically diverse data.The datasets are designed for interpretability rather than measuring factuality-classification capabilities on challenging data.
Truth is scoped to factuality, using simple, uncontroversial, and unambiguous true or false statements.Examples include numerical comparisons and word-meaning statements.
Datasets combine fixed-template subsets with topical variation and structural transformations such as negation and logical conjunctions or disjunctions.This design creates controlled variation within datasets and broader variation across datasets.
Uncurated test sets contain more diverse statements that may be ambiguous, malformed, controversial, or difficult to understand.They are used to test whether identified truth representations generalize beyond curated statements.
The likely dataset uses nonfactual text whose final token is either the most likely or 100th most likely completion according to LLaMA-13B.It helps distinguish truth representations from representations of probable versus improbable text.
Truth and text probability can be strongly anti-correlated, with r = −.63 for neg cities and r = −.89 for neg sp en trans under LLaMA-2-70B.For the corresponding non-negated datasets, correlations are strongly positive: r = .85 for cities and r = .95 for sp en trans.

3 Localizing truth representations via patching

The authors use activation patching to identify hidden states that causally influence whether the model outputs “TRUE” or “FALSE.” They find distinct groups associated with entity information, direct predictions, and a hypothesized representation of the full statement’s truth.

Activation patching compares hidden states from true and false prompts and measures each intervention’s effect on the log-probability difference between “TRUE” and “FALSE.”A larger difference indicates greater causal influence of the intervened hidden state on the model’s prediction.
The true and false prompts differ by replacing “Chicago” with “Toronto,” changing the final city statement from false to true.The prompt also includes labeled examples involving Tokyo and Hanoi.
Figure 2 plots the difference log P(TRUE) −log P(FALSE) after patching residual-stream activations at indicated token positions and layers.The figure reports results for LLaMA-2-13B on the cities dataset.
Group (c) directly encodes the model’s prediction, with decoder-head top logits including tokens such as “true,” “True,” and “TRUE.”Group (a) likely represents the subject entity, such as “Chicago” or “Toronto.”
Group (b) lies over the statement’s final token and punctuation, suggesting it summarizes information about the full statement.Because these states also causally influence the TRUE/FALSE decision, the authors hypothesize that they store the statement’s truth representation.
Clause-ending punctuation can carry summarizing information, although the largest LLaMA model shows this behavior in a more context-dependent way.The authors refer to additional results in Appendix B.

4 Visualizing LLM representations of true/false datasets

PCA reveals clear linear separation of true and false statements in larger LLaMA-2 models, while alignment across datasets depends on scale, layer, and the abstraction captured. The analysis also explains how shared and dataset-specific features can produce different geometric patterns.

Linear structure: LLaMA-2-13B and 70B representations show true statements separating from false ones in the top two principal components.This structure emerges rapidly in early-middle layers and later for more structurally complex statements.
Cross-dataset alignment: Separation axes often align across datasets, but datasets and their negations can instead have approximately orthogonal axes.The first principal component of cities separates truth values across other, including diverse uncurated, datasets.
Scale and layers: Larger than and smaller than separate along antipodal directions in LLaMA-2-13B but along a common direction in LLaMA-2-70B.In LLaMA-2-13B, cities and neg cities similarly progress from antipodal separation in early layers to orthogonal and then aligned representations later.
Interpretation: The authors interpret scaling as producing increasingly abstract linear representations, potentially including truth across structurally and topically diverse inputs.Smaller models instead represent surface-level characteristics, while intermediate regimes may encode narrower abstractions such as accurate factual recall.
Interpretation: PCA patterns depend on the relative salience of shared and dataset-specific features.If a shared feature f+ is salient, datasets jointly separate along it; if an oppositely correlated feature f− dominates, separation is antipodal; if both are salient, the axes can differ.
Representation choices: Using the final statement token can cause top principal components to reflect token identity rather than truth value.The qualitative results are insensitive to layer choice across early-middle to late-middle layers, but token choice can change the dominant variation.

5 Probing and generalization experiments

The experiments compare probing methods, training datasets, and model scales for generalization across true/false datasets. Mass-mean probes match other methods on larger models while identifying directions more causally implicated in outputs.

Probing methods: Mass-mean probes use a difference-in-means direction, with a covariance correction for IID data to mitigate interference from non-orthogonal features.The IID variant coincides with linear discriminant analysis and is designed to select a useful decision boundary while tracking a candidate feature direction.
Challenges with logistic regression, and mass-mean probing: Logistic regression can miss the intended truth direction when another linearly represented feature is non-orthogonal and varies independently of truth.Under linearly separable data, logistic regression converges to a maximum-margin separator that reduces interference from the correlated feature.
Experimental set-up: Training uses cities, cities + neg cities, larger than, larger than + smaller than, and likely datasets, with LR, MM, and unsupervised CCS probes evaluated across curated and uncurated test sets.Probes are trained on random 80% splits; held-out data are used for same-dataset evaluation, while full test sets are used for cross-dataset evaluation.
Results: Training on statements and their opposites improves generalization across datasets, including to negated statements and structurally different datasets.The results suggest that opposite-pair training mitigates non-truth features affecting the probe direction.
Results: Probes generalize better for larger models, with LLaMA-2-13B and 70B probes trained on larger than + smaller than achieving over 95% accuracy on sp en trans across techniques.This supports the claim that larger models linearly represent more general concepts shared across diverse inputs.
Results: For larger models, mass-mean probes generalize about as well as logistic regression and CCS while identifying directions more causally implicated in model outputs.Mass-mean probing underperforms LR and CCS on LLaMA-2-7B, but performs comparably on larger models.
Results: Probes trained on likely can perform worse than chance on datasets where text probability and truth are anti-correlated.This indicates that LLMs linearly represent truth-relevant information beyond text plausibility.

6 Causal intervention experiments

The experiments intervene on probe-identified activation directions to test whether they causally change LLM truth judgments, including on out-of-distribution inputs. Mass-mean directions are most causally effective, while dataset-opposite training and probability-based baselines show mixed results.

Intervention design: The interventions shift group (b) hidden states along probe directions to make false statements appear true and true statements appear false.They evaluate normalized indirect effects, where 0 denotes no effect and 1 denotes matching the confidence of genuine statements.
Intervention design: The evaluation uses out-of-distribution inputs rather than restricting interventions to the probe-training distribution.This contrasts with prior work cited by the authors.
Results: Mass-mean probe directions outperform logistic-regression and CCS directions in 7/8 experimental conditions, often substantially.All three probe types have very similar sp en trans classification accuracies, so causal effectiveness differs despite comparable classification performance.
Results: Training on datasets and their opposites helps for cities but not for larger than.The authors describe this pattern as surprising and say it indicates that the effects of opposite-dataset training remain incompletely understood.
Results: Training on likely is a strong baseline, but it remains weaker than interventions using truth probes.The authors report a strong correlation, r = .95, between truth and probability on sp en trans for LLaMA-2-70B.

7 Discussion

The discussion frames the findings as evidence that sufficiently scaled LLMs compute and linearly represent factual truth. It also limits the claim to simple factual statements and LLaMA-2 models, while identifying unresolved questions about intervention effectiveness and dataset choice.

Limitations: The results may not generalize beyond the LLaMA-2 family or distinguish truth from closely related features such as common belief or verifiability.These are the authors’ stated limitations of model family and statement selection.
Open questions: The authors leave unresolved why likely-trained mass-mean directions can intervene effectively despite weak classification accuracy, and why cities-plus-neg-cities performs poorly for 70B.These questions concern intervention effectiveness and training-data composition.
Discussion: The study finds evidence that, at scale, LLMs compute and linearly represent the truth of true/false statements.It combines visualizations, probing experiments, and causal evidence, and introduces mass-mean probing as an alternative technique.
Scope: The analysis scopes truth to factuality in declarative statements, excluding question answering, deception, and compliance.The examples are simple, unambiguous, and uncontroversial.
Scope: The datasets do not distinguish true statements from statements that are uncontroversial, widely believed, or believed by educated people.The authors do distinguish truth from statements likely to appear in training data.

B Full patching results

Full patching results show that smaller LLaMA-2 models summarize statement-relevant information over the end-of-sentence punctuation token, whereas the 70B model does so conditionally on context.

Full patching results: LLaMA-2-7B and LLaMA-2-13B display summarization behavior over the end-of-sentence punctuation token.This means information relevant to the full statement is represented at that location.
Full patching results: LLaMA-2-70B displays summarization behavior for cities but not for sp en trans.The authors characterize this behavior as context-dependent in the largest model.

C Emergence of linear structure across layers

Linear structure emerges across layers and varies with model scale, statement complexity, and dataset alignment. The section also describes probe generalization visualizations and interprets mass-mean probing as a whitened projection.

C Emergence of linear structure across layers: Representations are initially uninformative, then salient linear structure in the top few PCs rapidly emerges in early middle layers.More logically complex statements, such as conjunctions, show this structure later.
C Emergence of linear structure across layers: For cities and neg cities, separation axes rotate across layers from antipodal alignment through orthogonality and eventually alignment.This pattern is shown for LLaMA-2-13B.
C Emergence of linear structure across layers: The generalization figures plot probe training data on the horizontal axis and test set on the vertical axis for 70B, 13B, and 7B models.Figures 9–11 provide the model-specific results.
E Mass-mean probing in terms of Mahalanobis whitening: Mass-mean probing can be interpreted as projecting onto θmm after a whitening transformation.For IID data, the whitening transformation uses the covariance correction so whitened coordinates are uncorrelated with variance 1.

F For Gaussian data, IID mass-mean probing coincides with logistic regression on average

For symmetric Gaussian class-conditional data, mass-mean probing identifies a direction proportional to the logistic-regression solution. The appendix connects this result to linear concept erasure and multi-class extensions.

Gaussian result: Under symmetric Gaussian class-conditional distributions, mass-mean probing identifies θ_mm = 2θ.Negative and positive datapoints have means −θ and θ, respectively.
Gaussian result: The mass-mean probe σ(2θ^TΣ^-1x) also solves logistic regression up to scaling.
Gaussian result: After whitening the covariance, the logistic-regression optimum is proportional to Wθ, implying θ_lr ∝ Σ^-1θ.The coordinate change uses W = Σ^-1/2.
Concept erasure: Equal class-conditional means make the best affine predictor constant for convex losses.The result applies to multi-class one-hot labels and includes cross-entropy as an example.
Concept erasure: For binary classification, a nullity-1 projection that removes linearly recoverable information has a kernel determined by the class-mean difference.For multi-class data, the top left singular vector of the cross-covariance matrix is proposed as the analogous direction.

H Details on dataset creation

The datasets combine curated factual templates with adapted prior datasets, including simple relational, translation, negation, conjunction, and disjunction statements. Uncurated datasets broaden coverage but may contain ambiguity or other quality challenges.

Dataset construction: Dataset creation includes example statements, templates, and additional construction details.
Curated datasets: City statements use the template “The city of [city] is in [country]” and pair true statements with generated false-country variants.Cities were filtered by population, name uniqueness, recognized-country membership, and exclusion of city-states.
Curated datasets: Spanish-translation statements use correct or randomly incorrect English labels, with ambiguous examples removed by a Spanish-speaking author.The template states whether a Spanish word means a specified English word.
Curated datasets: Numeric comparison statements use larger-than and smaller-than templates over values fifty-one through ninety-nine, excluding equal or round-number cases.The range was chosen for visualization because LLaMA-13B represented number size non-uniformly.
Curated datasets: Negation, conjunction, and disjunction datasets are generated from city and translation statements using corresponding logical templates.Conjunction and disjunction datasets sample component statements independently to balance truth labels without correlating the components' truth values.
Uncurated datasets: Uncurated test sets are more diverse but can include ambiguous, malformed, controversial, or difficult-to-understand statements.Examples include adapted Companies, CommonClaim, and Counterfact datasets.