PaperlayerView exact revision on arXiv

Source-linked AI summary

Personas as a Way to Model Truthfulness in Language Models

Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, He He

arXiv:2310.18168v5cs.CLcs.AIcs.LG

TL;DR

Language models encounter factual and misleading claims in internet pretraining data, raising the question of how they distinguish truth from falsehood. The paper proposes that they infer personas shared by truthful or untruthful agents and tests this account with LLM and synthetic arithmetic experiments. Its reported evidence includes predicting answer truthfulness from question embeddings and generalizing truthful behavior in arithmetic when truthful agents share identifiable features.

  • Problem

    Internet pretraining data mixes factual and misleading claims, and the paper asks how language models distinguish truth from falsehood in that noisy data.

    Source: §1 · p. 1

  • Method

    The paper hypothesizes that models infer and represent personas shared by agents with common output features, and tests the hypothesis with LLM observations and controlled arithmetic data.

    Source: §1 · pp. 1–2

  • Results

    Question-embedding probes predict answer truthfulness and finetuning on true question-answer pairs improves truthfulness on unrelated topics; arithmetic generalization to unseen operators occurs only when truthful agents share identifiable generation features.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support the possibility that models exploit shared features among agents to distinguish true from false statements and generalize behavior across contexts.

    Source: §7 · p. 10

    The experiments support the existence of latent features for clustering truthful agents, but do not establish whether those features represent truth or are spurious; probing reaches only about 70% accuracy.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are trained on vast amounts of text from the internet, which contains both factual and misleading information about the world. While unintuitive from a classic view of LMs, recent work has shown that the truth value of a statement can be elicited from the model's representations. This paper presents an explanation for why LMs appear to know the truth despite not being trained with truth labels. We hypothesize that the pretraining data is generated by groups of (un)truthful agents whose outputs share common features, and they form a (un)truthful persona. By training on this data, LMs can infer and represent the persona in its activation space. This allows the model to separate truth from falsehoods and controls the truthfulness of its generation. We show evidence for the persona hypothesis via two observations: (1) we can probe whether a model's answer will be truthful before it is generated; (2) finetuning a model on a set of facts improves its truthfulness on unseen topics. Next, using arithmetics as a synthetic environment, we show that structures of the pretraining data are crucial for the model to infer the truthful persona. Overall, our findings suggest that models can exploit hierarchical structures in the data to learn abstract concepts like truthfulness.

1 INTRODUCTION

The paper asks how language models distinguish truth from falsehood in pretraining data that mixes factual and misleading claims. It proposes that models infer truthful personas from shared features among agents, with probing, finetuning, and synthetic arithmetic experiments offering evidence for the hypothesis.

1 INTRODUCTION

  • The motivating problem is that internet pretraining data mixes factual and incorrect statements, including plausible misinformation that models may repeat to users.The paper defines its focus as plausible, misleading incorrect information supported by pretraining data, not model-fabricated hallucinations.Source: §1 · p. 1
  • The paper asks how language models distinguish truth from falsehood in noisy data, given evidence that truth value can be elicited from statement embeddings.This question arises because the classic view predicts answers based on frequency regardless of truth.Source: §1 · p. 1
  • The persona hypothesis proposes that agents with common output features cluster together, allowing models to represent personas and distinguish truth from falsehood.The proposed shared features can include formality and consistency with certain facts.Source: §1 · p. 2
  • Question-embedding probes and finetuning on true question-answer pairs support the persona hypothesis: models predict answer truthfulness and improve truthfulness on unrelated topics.The paper reports these observations on TruthfulQA and notes little knowledge transfer from finetuning examples to test examples.Source: §1 · p. 2
  • In arithmetic experiments, models separate true from false equations and generalize truthful behavior to unseen operators only when truthful agents share identifiable generation features.The authors control the data-generating process to test how pretraining structure relates to truthful behavior.Source: §1 · p. 2

2 THE PERSONA HYPOTHESIS

The persona hypothesis treats truthful agents as a group whose generations share features, enabling language models to infer personas from context and generate consistent text. The paper tests for this representation in model activations and examines how persona-agent structure in pretraining data could produce it.

2 THE PERSONA HYPOTHESIS

  • A persona groups agents whose beliefs and generation styles make their parameters close, while the specific clustering features remain unspecified.The paper assumes agents are generated from persona parameters and stays agnostic about which features enable clustering.Source: §2 · p. 2
  • The hypothesis is that models infer personas of (un)truthful agents from context, represent them in activations, and generate text consistent with the inferred persona.The proposed mechanism depends on truthful agents sharing common features in their generations.Source: §2 · pp. 2–3
  • Figure 2 reports probe F1 across model layers and input-token embeddings, with the best F1 obtained using the entire question.The caption describes means and standard deviations over 20 randomized executions for the layer probes.Source: Figure 2 · p. 3
  • The paper first probes for latent truthful personas in LLM representations, then uses synthetic experiments to test whether persona-agent data structure produces such representations.The stated plan separates evidence for a representation from evidence about how it arises.Source: §2 · p. 3

3 EVIDENCE OF LLMS MODELING PERSONAS

The section tests whether Alpaca represents answer truthfulness before generation and whether truthful fine-tuning transfers across topics. Probing and held-out evaluations support persona-based inference and generalization.

3.1 LLMS INFER PERSONAS FROM THE CONTEXT

  • Question embeddings predicted answer truthfulness above chance from early layers, peaking at approximately 65% F1 at layer 17.Because the question embedding lacks answer information, the encoded persona may reflect question style or false presuppositions.Source: §3.1 · p. 3
  • At layer 17, persona probing strengthened as question context accumulated, peaked on the full question, and remained at chance on the shared instruction.The result indicates that question context, rather than the common instruction, carries the probeable signal.Source: §3.1 · p. 3
  • Probing exceeded chance in all but one of the six largest TruthfulQA categories, though performance varied by category.The variation suggests that truthful and false statements are harder to separate in some topics.Source: §3.1 · p. 4

3.2 LLMS GENERALIZE TRUTHFULNESS ACROSS TOPICS

  • Truthfulness rose from 39% to 74% after truthful fine-tuning and fell to 10% after untruthful fine-tuning; gains also appeared on held-out categories, close to training on all categories.Figure 3 (left) compares fine-tuned models with base models on held-out categories.Source: §3.2 · p. 4, Figure 3 · p. 5
  • TriviaQA and MS Marco fine-tuning produced truthfulness scores similar to or worse than the non-fine-tuned model, unlike truthful fine-tuning.These controls were intended to test whether gains reflected general question-answering adaptation.Source: §3.2 · pp. 4–5
  • Fine-tuning on 20% of the data matched performance from 80%, while all tested methods except 10-example in-context learning substantially increased truthfulness.The comparison also included 5% and 50% fine-tuning, plus 20-example in-context learning.Source: §3.2 · p. 5, Figure 3 · p. 5
  • The authors interpret cross-domain transfer as the model mapping inferred personas to a truthful persona that controls generation beyond fine-tuning domains.They contrast this with learning correct answers to individual questions.Source: §3.2 · p. 5

4 ARITHMETIC LABORATORY: CONNECTING PRETRAINING DATA TO TRUTHFULNESS

The arithmetic experiments test whether persona structure in synthetic training data supports truthful prediction, generalization to unseen operators, and answer selection. The results depend on agents sharing behavior that identifies a truthful persona.

4 ARITHMETIC LABORATORY: CONNECTING PRETRAINING DATA TO TRUTHFULNESS

  • The synthetic corpus uses agent-tagged arithmetic equations, with each agent’s probability of using the true operator interpretation determining whether each generated answer is true or false.Agents in the truthful persona use the true interpretation with probability 0.8–1; those in the untruthful persona use it with probability 0–0.2, so neither group is perfectly truthful or untruthful.Source: §4 · pp. 5–6

4.1 PROBING FOR TRUTHFULNESS

  • The probing test compares clustered truthful and untruthful agents with agents truthful on disjoint operator sets, then evaluates whether a linear probe predicts answer truthfulness across held-out operators.The probe uses the equals-token embedding from a selected layer; training and evaluation use separate sets of operators.Source: §4.1 · p. 7
  • Across operator counts, probes achieve higher F1 when training data contains a truthful persona, while performance has greater variance without one.The authors attribute the inconsistent generalization to the absence of a shared persona among agents.Source: Figure 4 · p. 6, §4.1 · p. 7

4.2 GENERALIZING AGENT BEHAVIOR TO UNSEEN OPERATORS

  • The generalization test holds out op4 for agents D–G, which are observed only on op1–op3, and contrasts shared truthful behavior with agents whose true beliefs overlap little.In the truthful-persona setup, A and B share a truthful persona while C is untruthful; in the comparison setup, A, B, and C are truthful on only two operators with little overlap.Source: §4.2 · p. 7
  • With a truthful persona, the model generalizes truthfully for truthful agent G and untruthfully for untruthful agent D on an unseen operator; without one, generalization lacks a clear pattern.Without a truthful persona, the model instead generalizes untruthfully for the most truthful agent G, whose closest training agent A is untruthful on the held-out operator.Source: Figure 4 · p. 6, §4.2 · p. 8

4.3 MECHANISM FOR PERSONA-BASED COMPUTATION

  • Probes encode both truthful and untruthful answers better than a control answer, consistent with models storing both options and selecting according to the inferred persona.The authors suggest this could help explain why supervised finetuning can improve truthfulness by changing answer selection rather than teaching a new answer.Source: §4.3 · p. 8, Table 2 · p. 8

5 DISCUSSION

The discussion treats persona-based truthfulness as a supported hypothesis, not proof that models robustly learn truth. It also identifies limits in both the evidence and the synthetic setting.

5 DISCUSSION

  • Probing reaches only about 70% accuracy, and the experiments do not establish robust truth learning because the latent clustering features may be spurious.The authors note that finetuning generalizes to out-of-distribution data, suggesting the features may be somewhat meaningful; their hypothesis also predicts no generalization where training contains no truthful statements.Source: §5 · p. 9
  • The persona account is one hypothesis, and sparse fact-checking annotations in pretraining could also let models cluster truthful and untruthful statements.The authors state that their general persona definition can capture other possible mechanisms.Source: §5 · p. 9
  • The synthetic experiments explicitly mark agents, assume truthful and untruthful answers are equally easy to compute, and assume truthful agents broadly share beliefs.Real models must infer agents from text, answer complexity may differ, and truthful agents need not agree on every fact.Source: §5 · p. 9

6 RELATED WORK

Prior work studies truthfulness evaluation and interventions, as well as how language models represent or adopt personas. This paper builds on those lines with controlled probing and generalization experiments.

6 RELATED WORK

7 CONCLUSION

The conclusion proposes the persona hypothesis: LLMs may group agents with shared output features to distinguish true from false statements and generalize behavior beyond observed contexts. The appendix describes the Alpaca prompts used for question answering, probing, and finetuning.

7 CONCLUSION

  • The persona hypothesis proposes that shared output features let LLMs group agents into personas, distinguish true from false statements, and generalize agent behavior beyond observed contexts.The paper says this hypothesis is supported by evidence from LLMs and a synthetic setup.Source: §7 · p. 10

A ALPACA PROMPTS

  • For Alpaca, the authors adapt its original finetuning prompt for zero-shot question answering and also use it in probing and finetuning experiments.In probing, they use the embedding of the last prompt token before response sampling.Source: A ALPACA PROMPTS · p. 12
  • For in-context learning, the authors use a shorter prompt containing example question-answer pairs followed by a test question.The shorter example prompt is used to fit the context window.Source: A ALPACA PROMPTS · p. 12

B PROBING ABLATIONS

The probing ablations examine topic variation, additional prompt context, an alternative accuracy metric, and probing generated answer tokens. Topic and token-position results indicate that truthfulness is not equally predictable across topics and that question context is informative.

B PROBING ABLATIONS

  • Probe F1 varies by topic, with truthfulness easier to predict for economics questions than stereotype questions.The authors suggest this variation may indicate that personas are not perfectly defined across topics and could comprise smaller truthful-agent clusters.Source: B PROBING ABLATIONS · p. 12, Figure 6 · p. 13
  • Figure 8 reports probe accuracy as an alternative metric to the Figure 2 results, with mean and standard deviation across 20 randomized executions.The supplied passages do not report specific accuracy values.Source: B PROBING ABLATIONS · p. 12, Figure 8 · p. 13
  • The last generated answer token yields better probe performance than question-context-only probing, but the small difference suggests the question is already informative about generation truthfulness.Figure 9 reports probing over generated tokens as a baseline for the earlier results.Source: B PROBING ABLATIONS · p. 12, Figure 9 · p. 14
  • Probe performance increases as more prompt context is incorporated, with results plotted across model layers.The layer-wise comparison uses the same tokens as an earlier analysis.Source: B PROBING ABLATIONS · p. 12, Figure 7 · p. 13

C EXPERIMENT DETAILS

The experiment details specify the TruthfulQA evaluation and finetuning procedures, the construction of a misconceptions dataset, synthetic-model training settings, and the probe used for agent-based computation.

C EXPERIMENT DETAILS

  • TruthfulQA generations are evaluated for truthfulness with GPT-Judge, using the dataset and default hyperparameters and prompt from the TruthfulQA work.The judge is obtained through the OpenAI finetuning API.Source: C EXPERIMENT DETAILS · p. 15
  • Alpaca finetuning uses 30 epochs, batch size 48, Adam, learning rate 9e-5, and warmup ratio 0.03.For lower-compute experiments, the authors use LoRA on query and key projection matrices; the passage also specifies rank 16 before truncating.Source: C EXPERIMENT DETAILS · p. 15
  • The BigBench misconceptions statements are converted into question-answer pairs with GPT-3.5, manually corrected and filtered for TruthfulQA topic overlap, yielding 83 examples.Source: C EXPERIMENT DETAILS · p. 15
  • Synthetic data models are four-layer transformers trained for 20k steps, with hidden and embedding dimensions of 128 and batch size 512.The setup uses Adam at learning rate 0.001 and a tokenizer with separate digit tokens.Source: C EXPERIMENT DETAILS · p. 15
  • For agent-based computation, a linear probe predicts the first digit where truthful and untruthful answers differ, reducing its output space.Control probes use answers based on a different randomly sampled operator's truthful or untruthful interpretation.Source: C EXPERIMENT DETAILS · p. 15

D SYNTHETIC DATASET GENERATION

This section introduces the exact operator semantics and data-generation hyperparameters for the synthetic setup.

D SYNTHETIC DATASET GENERATION

D.1 PROBING FOR TRUTHFULNESS

The appendix describes synthetic arithmetic setups for probing truthfulness and testing generalization to unseen operators. The operator interpretations are designed to be distinct yet similarly learnable.

D.1 PROBING FOR TRUTHFULNESS

  • The probing setup samples truthful and untruthful operator interpretations across m = 8, 12, 16, or 20 operators, using n = 100 inputs.Agents receive generation probabilities above 0.8 for truthful interpretations and below 0.2 for untruthful ones; operator-specific offsets are sampled from (0, 70).Source: §D.1 · pp. 15–16

D.2 GENERALIZATION TO UNSEEN OPERATORS

  • Truthful and untruthful interpretations are specified separately for each operator in the generalization setup.The appendix gives the operation definitions for both interpretation types.Source: §D.2 · p. 16
  • The unseen-operator setup uses four operators with distinct, unrelated interpretations designed to have similar complexity and learnability.The operators use digit functions and concatenation; generation probabilities are above 0.8 for truthful interpretations and below 0.2 for untruthful ones, with n = 200 inputs.Source: §D.2 · p. 16

E GENERALIZATION TO UNSEEN AGENT-OPERATOR COMBINATIONS

The paper examines whether models generalize truthfulness to unseen agent-operator combinations. The untruthful-answer probability decreases with agent truthfulness in the setup with a truthful persona, while varying widely without one.

E GENERALIZATION TO UNSEEN AGENT-OPERATOR COMBINATIONS

  • The untruthful-answer probability decreases as agent truthfulness increases with a truthful persona, but varies widely in the setup without one.The paper reports that truthful and untruthful behavior generalizes to unseen operators only when a truthful persona is present.Source: Figure 10 · p. 16, E GENERALIZATION TO UNSEEN… · pp. 16–17
  • The models’ average probability for the untruthful answer is plotted across agents for an unseen operator.The paper previously examined the average probability assigned to truthful answers across agents in the same setting.Source: E GENERALIZATION TO UNSEEN… · pp. 16–17
Loading 2310.18168v5…