Truth is Universal: Robust Detection of Lies in LLMs

1 Introduction

• LLMs can knowingly output falsehoods, including strategic deception despite honesty training, making robust lie detection an important unresolved problem.The introduction describes both instructed lying and deception in pursuit of a goal.Source: §1 · p. 1
• A two-dimensional truth subspace separates true and false statements: tG points from false to true across polarities, while tP reverses with polarity, explaining why tA fails on negated statements.Figure 1 contrasts the affirmative-only separation along tA with separation along tG for both affirmative and negated statements.Source: §1 · p. 2, Figure 1 · p. 3
• 94% accuracy in detecting real-world lies; TTPD also achieves the highest generalization accuracy on unseen statement types and real-world lies, and performs comparably to LR on unseen topics of familiar form.The detector was trained on activations of simple factual statements.Source: §1 · p. 3
• A similar two-dimensional truth subspace appears across Gemma, LLaMA2, Mistral, and LLaMA3 models, including base and instruction-tuned variants.The paper presents this as evidence of universality across model families and sizes.Source: §1 · p. 3

2 Datasets with true and false statements

• Six affirmative datasets cover single-topic factual statements; all but facts use simple, unambiguous statements with consistent templates.The facts dataset contains statements of more varied forms and topics.Source: §2 · p. 4
• Negated versions of the six datasets were created by inserting “not,” yielding six additional datasets with true-or-false labels.For example, a false affirmative statement can become a true negated statement.Source: §2 · p. 4
• The suite adds German translations of the first 50 statements and their negations, conjunction and disjunction variants, and two diverse test datasets that may contain ambiguous or malformed statements.The German translations were manually verified by a native German speaker.Source: §2 · p. 4

3 Supervised learning of the truth directions

• For LLaMA3-8B, activations are taken from the final token at layer 12, where Figure 2 shows the largest true/false separation across datasets.The vector is extracted from the residual stream; layer choice depends on the model.Source: §3 · p. 5, Figure 2 · p. 5
• AUROC 0.98: the general truth direction separates true from false statements across affirmative and negated statements, while the affirmative-only direction does not accurately separate negated statements.The affirmative-only direction is a linear combination of the general and polarity-sensitive directions.Source: §3 · p. 5
• The supervised procedure assigns each statement a truth label and its dataset a polarity label, then approximates activations using a topic mean and the two truth directions.The optimization minimizes mean squared error between approximated and observed activations.Source: §3 · pp. 5–6
• Ordinary least squares yields closed-form solutions for both directions, and equal numbers of statements per topic balance topic influence in training.The optimization sums errors over datasets and statements.Source: §3 · p. 6
• AUROC values close to one: the general truth direction separates held-out topic datasets for both polarities, while the polarity-sensitive direction reverses its separation for negated statements.The evaluation holds out one topic-specific dataset and averages results over 10 training runs.Source: §3 · p. 6

4.1 Number of significant principal components

• With affirmative statements alone, the first principal component explains approximately 60% of variance; adding negated statements yields two components explaining more than 60% of preprocessed variance, with no significant third component for conjunctions, disjunctions, or German statements.The first two components have cosine similarities of 0.98 with tG and 0.97 with tP; in raw activations they explain only about 10%.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 Generalization of different truth directions

• A direction trained on affirmative city statements generalizes to affirmative scientific facts but not negated statements; adding negated examples enables negated transfer and improves separation of conjunctions and disjunctions.Training directions are evaluated by AUROC, using held-out data when train and test sets match and full test sets otherwise.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
• After projecting out Span(tG, tP), city activations remain linearly separable, but an affirmative-and-negated city-trained direction fails to generalize to other topics; adding conjunctions restores transfer to conjunctions and disjunctions.The residual separability suggests topic-specific truth-correlated features, motivating balanced training data across topics.Source: §4.2 · pp. 7–8
• The findings suggest tG and tP capture most of the internal linear truth representation, but other statement types could reveal additional linear or nonlinear structure.Conjunctions, disjunctions, and German statements did not reveal significant additional linear structure.Source: §4.2 · p. 8

5 Generalisation to unseen topics, statement types and real-world lies

• TTPD learns a general truth direction and a polarity direction, projects training activations onto both, then fits Logistic Regression in the resulting two-dimensional space.Its training examples include equal numbers of affirmative and negated statements with binary truth labels.Source: §5 · p. 8

5.1 Unseen topics and statement types

• On unseen topics, TTPD (93.9 ± 0.2%) and LR (94.6 ± 0.7%) exceeded CCS (84.8 ± 6.4%) and MM (92.2 ± 0.4%) in average accuracy.Figure 6a summarizes these generalization accuracies across 20 training runs.Source: Figure 6 · p. 9, §5.1 · p. 9
• For unseen statement types, TTPD outperformed LR and CCS on conjunctions and disjunctions, while German accuracy was only a few points below English accuracy.On CommonClaim and Counterfact test sets, TTPD and LR performed similarly well.Source: §5.1 · p. 9

5.2 Real-world lies

• The real-world evaluation used LLaMA3-8B-Instruct responses to 26 scenarios, each with and without an incentive to lie, yielding 208 manually categorized responses.The classifiers were trained only on simple affirmative and negated statements; categories included unambiguous, ambiguous, and other responses.Source: §5.2 · p. 10
• 93.8 ± 1.5% average accuracy for TTPD exceeded MM (90.5 ± 1.5%), LR (79 ± 8% accuracy), and CCS (73 ± 12%) on 49 unambiguous lies and 55 truthful replies.The classifiers were trained on simple affirmative and negated statements from topic-specific datasets.Source: §5.2 · p. 10

6 Discussion

• The work reports evidence for a truth direction that generalizes to unseen topics, unseen statement types, and real-world lies.The authors frame this as progress toward robust, general-purpose lie detection.Source: §6 · p. 10
• TTPD uses only one dimension of the truth subspace; using both dimensions in a nonlinear classifier might improve accuracy.The analysis establishes that the subspace is at least two-dimensional, not that it has exactly two dimensions.Source: §6 · p. 10
• Generalization was tested on a limited number of statement types and real-world scenarios, and applicability to larger or multimodal models remains untested.The authors note that broader statements might reveal additional linear or nonlinear structures that differ across models.Source: §6 · p. 10

A Details on Datasets

• Conjunction datasets combine two statements, sample their truth values independently, and contain 500 statements each.This sampling balances true and false examples without statistical dependence between the component statements’ truth values.Source: A Details on Datasets · p. 13
• Disjunction datasets use templates tailored to each dataset, with paired statement forms and sampling designed to balance true and false labels.For topic-specific templates, the first statement is sampled as true with probability 1/2, and the second statement’s end-word is chosen to be incorrect for the first.Source: A Details on Datasets · p. 13
• German versions translate the first 50 statements of each Table 1 dataset and their negations, then receive manual verification by a native German-speaking author.DeepL was used for translation.Source: A Details on Datasets · p. 13
• The adapted CommonClaim dataset contains 4,450 true or false statements selected from 20,000 GPT-3-text-davinci-002 generations and balanced by removing excess true examples.CommonClaim’s original labels were true, false, or neither, based on human common knowledge.Source: A Details on Datasets · p. 14
• The adapted Counterfact dataset contains 31,964 complete counterfactual assertions, pairing each statement with a suggested false modification.The adaptation uses both the true version and a false version of each assertion.Source: A Details on Datasets · p. 14

B Choice of basis for the 2D truth subspace

• Although affirmative and negated statements separate along tA and tN, respectively, classifying by first detecting polarity assumes each statement has one of those two categories.The authors therefore motivate tG and tP as a more suitable basis for describing truth-related variance.Source: B Choice of basis for the 2D truth… · p. 14
• Some numerical comparisons separate along the general truth direction but not the polarity-sensitive direction, because the model may assign neither affirmative nor negated polarity.The larger_than and smaller_than datasets each contain 1,980 numerical comparisons.Source: B Choice of basis for the 2D truth… · p. 14
• TTPD accuracy was 98 ± 1% on larger_than and 99 ± 1% on smaller_than, versus Logistic Regression at 90 ± 15% and 92 ± 11%, respectively.Both classifiers were trained on balanced affirmative and negated statements from topic-specific datasets.Source: B Choice of basis for the 2D truth… · p. 14

C Cross-dataset generalization matrix

• The cross-dataset matrix uses training datasets as columns and test sets as rows, with AUROC values illustrating generalization across datasets.For example, its first column reports a direction trained on cities and tested on six test sets.Source: C Cross-dataset generalization matrix · p. 14, Figure 8 · p. 15
• Figure 7 projects larger_than and smaller_than activations onto tG and tP, with grey points representing affirmative and negated topic-specific statements.Source: Figure 7 · p. 15
• Truth directions are trained on 80% of the data and evaluated on the held-out 20% for matching train and test sets, or the full test set otherwise.Source: C Cross-dataset generalization matrix · p. 15

D Failures modes of the TTPD classifier

• TTPD errors arise either from a dataset bias that fails to generalize despite separable truth labels, or from a truth direction that cannot separate true and false statements.For cities, about 200 errors were false statements despite an AUROC of 1.0 along tG; the second mode occurred in inventors and neg_inventors, and possibly facts and neg_facts.Source: D Failures modes of the TTPD classifier · pp. 15–16
• In real-world scenarios, lies and truthful replies separate along tG with AUROC ≈1.00, while most of TTPD’s approximately eight errors are lies, suggesting bias generalization is the main failure.Source: D Failures modes of the TTPD classifier · p. 16
• Compared with TTPD, logistic regression more often fails through its truth direction and less often through bias; the authors hypothesize this reflects learning both together rather than sequentially.They suggest separately learning truth direction may generalize better, at the cost of finding a well-generalizing bias.Source: D Failures modes of the TTPD classifier · p. 16

E Generalization to logical conjunctions, disjunctions and statements in German

• The classifier breakdown covers logical conjunctions, disjunctions, and German statements, with all classifiers trained only on English affirmative and negated statements.Source: E Generalization to logical… · p. 16

F.1 Generating the LLM responses to the scenarios

• Scenario responses are generated by sampling tokens from softmax probabilities at temperature 1 and stopping at the first period or exclamation mark.Source: §F.1 · p. 16

F.2 Categorization and examples of LLaMA3-8B-Instruct replies

• Replies are categorized as unambiguous only when the truth or lie is clear; replies too ambiguous to classify, or unrelated to the scenario, are categorized as other.The examples distinguish an out-of-context reply from a seller’s unclear minimization of a problem.Source: §F.2 · p. 17

F.3 Category sizes and category specific accuracies

• Table 2 reports scenario counts by category and category-specific classification accuracies, averaged over 10 training runs on different random training samples.Source: §F.3 · p. 17, Table 2 · p. 17

• TTPD classified honest replies to lie-incentive scenarios as true with 82 ± 5% average accuracy, but only six honest responses were recorded.CCS reached 77 ± 22%, LR 62 ± 17%, and MM 100 ± 0% on this subset; the authors caution that these accuracies should be interpreted carefully.Source: §F.4 · p. 18

G Results for other LLMs

• The analyses of five additional LLMs find a two-dimensional subspace that separates activations of true and false statements, with results similar to LLaMA3-8B-Instruct.Source: G Results for other LLMs · p. 18

G.1 LLaMA2-13B

• LLaMA2-13B shows its largest true–false separation at layer 14, which is used for subsequent analysis.The accompanying figures show variance ratios, direction projections, principal components, generalization, and classifier comparisons.Source: §G.1 · p. 18, Figure 10 · p. 18, Figure 11 · p. 19

G.2 Mistral-7B

• Mistral-7B has its largest true–false separation at layer 13, which is used for subsequent analysis.The accompanying figures show variance ratios, direction projections, principal components, generalization, and classifier comparisons.Source: §G.2 · p. 21, Figure 16 · p. 21, Figure 17 · p. 21

G.3 Gemma-7B

• For Gemma-7B, the largest true–false separation occurs at layer 16, and using tP as well as tG could yield much higher classification accuracies.The accompanying figures show projections, principal components, generalization, and classifier comparisons.Source: §G.3 · p. 24, Figure 22 · p. 24, Figure 23 · p. 24

G.4 Gemma-2-27B

• Gemma-2-27B has its largest true–false separation approximately at layer 20, which is used for subsequent analysis.The accompanying figures show variance ratios, direction projections, principal components, generalization, and classifier comparisons.Source: §G.4 · p. 27, Figure 28 · p. 27, Figure 29 · p. 27

G.5 LLaMA3-8B-base

• LLaMA3-8B-base has its largest true–false separation at layer 12, which is used for subsequent analysis.The accompanying figures show variance ratios, direction projections, principal components, generalization, and classifier comparisons.Source: §G.5 · p. 30, Figure 34 · p. 30, Figure 35 · p. 30