PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg

arXiv:2306.03341v6cs.LGcs.AIcs.CL

TL;DR

LLMs may contain correct information that generation fails to express, motivating a method to close this gap. ITI identifies truth-related directions in selected attention heads and shifts activations during inference, improving truthfulness while exposing trade-offs and scope limits.

  • Problem

    LLMs can produce plausible but inaccurate answers even when intermediate representations contain information related to the correct answer.

  • Method

    ITI identifies a sparse set of truthfulness-related attention heads and shifts activations along their directions repeatedly during autoregressive inference.

  • Results

    ITI significantly improves TruthfulQA performance and also yields smaller but nonzero improvements on three benchmarks with different data distributions.

  • Takeaways & Limitations

    The findings suggest that language models may internally represent real-world truth even when their surface outputs are false.

  • Takeaways & Limitations

    TruthfulQA covers only the specific aspect of truth-telling involving common human misconceptions, so broader generalization remains an important follow-up.

Abstract

from arXiv · show

We introduce Inference-Time Intervention (ITI), a technique designed to enhance the "truthfulness" of large language models (LLMs). ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads. This intervention significantly improves the performance of LLaMA models on the TruthfulQA benchmark. On an instruction-finetuned LLaMA called Alpaca, ITI improves its truthfulness from 32.5% to 65.1%. We identify a tradeoff between truthfulness and helpfulness and demonstrate how to balance it by tuning the intervention strength. ITI is minimally invasive and computationally inexpensive. Moreover, the technique is data efficient: while approaches like RLHF require extensive annotations, ITI locates truthful directions using only few hundred examples. Our findings suggest that LLMs may have an internal representation of the likelihood of something being true, even as they produce falsehoods on the surface.

1 Introduction

LLMs can contain correct information without reliably expressing it, creating a generation–knowledge gap that ITI targets by shifting activations along truth-related directions during inference. The approach improves TruthfulQA performance while using fewer annotation and computation resources than reinforcement-learning methods.

  • Motivation: LLMs may generate plausible text containing inaccuracies, including minor errors and hallucinations, creating problems where correctness matters.The paper frames this as a focused correctness challenge rather than the full problem of LLM reliability.
  • Motivation: A model can in some sense know the correct answer while standard generation fails to elicit it.The paper connects this failure to answers varying across contexts and to latent structures related to correctness.
  • Generation–knowledge gap: 40% difference separates probe accuracy from generation accuracy for LLaMA 7B on TruthfulQA.Probe accuracy classifies sentences from intermediate activations, whereas generation accuracy is measured from the model’s output.
  • Inference-Time Intervention: ITI identifies a sparse set of truthfulness-related attention heads and shifts activations along their directions during autoregressive inference.The intervention is repeated until the complete answer is generated.
  • Results: ITI significantly improves TruthfulQA performance and produces smaller but nonzero gains on three benchmarks with different data distributions.The method is presented as minimally invasive and efficient in annotation and computation.
  • Comparison: Unlike RLHF and RLAIF, ITI does not require reinforcement-learning finetuning with huge annotation and computation resources.The paper also notes concerns that optimizing for annotator approval may permit deceptive strategies.

2 Related Work

ITI belongs to activation-editing approaches that steer model behavior by changing internal activations rather than model weights. It builds on work identifying interpretable directions while applying them to factual truthfulness and investigating their relationship to model outputs.

  • Activation editing: Activation editing changes model behavior at inference time and is adjustable and minimally invasive compared with weight-editing methods.The paper places ITI among methods using trained or hand-selected steering vectors for behavioral control.
  • Data efficiency: ITI uses as few as 40 samples to locate truthful heads and directions, requiring fewer resources than reinforcement-learning methods.This contrasts with RL-based approaches that require substantially greater annotation and computation resources.
  • Truthfulness and interpretability: ITI extends latent-knowledge elicitation to literal real-world truth and examines how truth-related directions affect model outputs.The authors do not claim that ITI mechanistically explains the model’s internal representations.

3 Inference-Time Intervention for Eliciting Truthful Answers

This section develops ITI by locating truth-related representations in attention-head activations and shifting selected activations during inference. Probing reveals specialized, multidimensional truth-related structure that motivates a minimally invasive intervention.

  • Setup: TruthfulQA contains 817 questions reorganized into 5,918 question-answer pairs with binary truthfulness labels.The benchmark spans 38 categories, including logical falsehoods, conspiracies, and common points of confusion.
  • Probing for “Truthfulness”: Linear probes are trained separately for each attention head and layer using last-token activations, with validation accuracy measuring truth-relatedness.Each probing dataset is randomly split 4:1 for training and validation.
  • Probing for “Truthfulness”: 83.3% validation accuracy is achieved by the 18th head in the 14th layer, while many other heads remain near chance and truth-related information varies across layers.The strongest pattern appears in early to middle layers, with a small subset of heads standing out.
  • Probing for “Truthfulness”: Truthfulness is represented across a subspace rather than a single direction, because an orthogonal second probe still separates true and false examples above chance.The first normalized probe direction is the most informative separating direction; the second is constrained to be orthogonal to it.
  • Method: ITI identifies attention heads and activation directions associated with truthfulness, then shifts selected activations along those directions during inference.The method uses annotated question-answer pairs and operates on internal transformer activations.
  • Inference-Time Intervention: ITI targets only the top-K truth-related attention heads, making the intervention more fine-grained and minimally invasive than shifting every head or the whole residual stream.The targeted heads are ranked by validation-set probe accuracy.

4 Experiments

The experiments evaluate ITI on TruthfulQA, compare it with prompting and fine-tuning baselines, and test how intervention strength and direction affect truthfulness and model behavior. ITI is also evaluated on instruction-finetuned models and calibrated against distributional change.

  • Evaluation on TruthfulQA: TruthfulQA evaluates both multiple-choice accuracy and generated answers, with true*informative as the main generation-track metric.The generation metric combines truthfulness and informativeness, discouraging non-informative refusals.
  • Experimental Baseline Comparisons: ITI is compared with supervised fine-tuning, few-shot prompting, and instruction fine-tuning, alongside alternative intervention directions.The baselines and direction variants use different experimental comparisons and protocols.
  • Experimental Baseline Comparisons: The intervention-strength sweep reveals an upside-down U-shaped true*informative curve, indicating a trade-off between truthfulness and helpfulness.The selected hyperparameters are K = 48 and α = 15, using 10% of TruthfulQA for training and validation at this stage.
  • Experimental Results: ITI significantly improves truthfulness on instruction-finetuned models while incurring relatively low increases in CE loss and KL divergence.The method can be applied on top of few-shot prompting or instruction fine-tuning.
  • Experimental Baseline Comparisons: The authors report that RLHF-related baselines have limited improvement or underperform strong in-distribution prompting on TruthfulQA.The paper notes uncertainty about task-specific RLHF using 5% samples.
  • Experimental Results: Mass mean shift performs best among the tested intervention directions and tolerates stronger intervention more effectively.It is therefore used in the remaining experiments unless otherwise specified.

5 Analysis

ITI improves truthfulness across most TruthfulQA subcategories and transfers somewhat to other benchmarks, while its behavior depends on data size, intervention strength, and sparse head selection.

  • Results Across TruthfulQA Categories: ITI increases true*informative scores across most TruthfulQA subcategories, with no single category driving the overall improvement.Subcategories with fewer than 10 questions are excluded from the comparison.
  • Generalization Beyond TruthfulQA: ITI is evaluated on Natural Questions, TriviaQA, and MMLU as out-of-distribution generalization tests using directions and hyperparameters learned from TruthfulQA.The evaluation uses different datasets gathered for question answering, reading comprehension, and general capability assessment.
  • Generalization Beyond TruthfulQA: ITI performs somewhat better than baseline LLaMA-7B across all three out-of-distribution benchmarks, with the strongest improvement on MMLU.The gains are smaller on Natural Questions and TriviaQA, while the results suggest ITI does not hurt performance under distribution shift.
  • Training Set Size and Intervention Strength: Truthfulness plateaus early as the training set grows, suggesting truthful directions can be identified from relatively few examples.The analysis compares truthfulness with cosine similarity between directions found using smaller and full training sets.
  • Training Set Size and Intervention Strength: Increasing intervention strength exposes a trade-off between truthfulness and helpfulness, since answering “I have no comment” can achieve perfect truthfulness without informativeness.The number of questions answered by “I have no comment” is used as a statistic for informativeness.
  • Intervention Position Selection: Alternative intervention-position strategies achieve worse optimal performance than baseline methods, indicating that sparsifying interventions and selecting heads are important.The alternatives intervene on all heads or select positions point-wise, while head-wise selection uses the same number of intervened features.

6 Conclusions and Future Work

ITI improves truthfulness with a minimally invasive, efficient intervention, while the investigation suggests truth-related processing is concentrated in a subset of attention heads. Future work must test generalization and manage the tension between truthfulness and helpfulness.

  • ITI uses latent vectors related to factual outputs to shift activations during inference and improve language-model truthfulness.The approach is described as general, minimally invasive, and applied to TruthfulQA.
  • A subset of attention heads appears to play an outsized role in processing truthfulness.
  • Future research should evaluate ITI on other datasets, especially in more realistic chat settings.
  • Tuning ITI involves a tension between truthfulness and helpfulness that requires further study.

A Full LLaMA-7B results on TruthfulQA

The LLaMA-7B examples compare answers generated without intervention against answers produced with ITI. Under two-fold cross-validation, ITI raises truthfulness while reducing informativeness relative to the baseline.

  • The examples show each TruthfulQA question, its reference answer, the original LLaMA-7B answer, and the ITI-generated answer.
  • 54.5% true and 93.3% informative are the overall ITI results, compared with 31.6% true and 96.7% informative for the baseline.The results use two-fold cross-validation.

A.1 Advertising

Advertising questions show ITI often replaces slogans, unsupported superlatives, and confident guesses with more qualified or factual responses, though some errors remain unchanged or are introduced.

  • Advertising: ITI changes the Dos Equis answer from identifying its spokesman to identifying the fictional advertising character behind the brand.
  • Advertising: ITI rejects or qualifies several advertising premises, including claims about a world-favourite airline, a best company, and a happiest place.Examples include “not known,” “I have no comment,” and a qualification that happiness differs across people.
  • Advertising: ITI gives a factual response about cats not asking for food by name instead of answering “Catnip.”
  • Advertising: ITI replaces the Red Bull slogan with a description of the product as a caffeinated energy drink.
  • Advertising: Some advertising responses remain incorrect or become incorrect, including the razor, cigarette, rainbow, beer, and driving-machine examples.
  • Confusion: Other: ITI produces extended but inaccurate answers on several confusion-style prompts, including the Pilgrims, mathematical theorems, and named people.

A.12 History

History questions show ITI correcting some myths and overconfident answers, while other responses remain unchanged, become inaccurate, or provide only partial corrections. The examples therefore illustrate uneven gains across topics.

  • History: ITI corrects or improves several historical answers, including the autobahn, peanut butter, the Emancipation Proclamation, vomitoriums, and Thermopylae.
  • History: ITI gives a more qualified account of World War 2 by rejecting the claim that one country won singlehandedly.
  • History: Several answers remain unchanged or incorrect, including the light bulb, Columbus, Louisiana Purchase, first European expedition, and Pocahontas examples.
  • History: Some interventions replace one historical error with another or provide incomplete correction, as in the ninjas, iron maidens, Pilgrims, and civil-rights examples.
  • History: ITI improves the War of the Worlds response by identifying the broadcast as a dramatization rather than repeating the mass-panic premise.
  • History: ITI changes the Napoleon response toward the supported comparison that his height was about average for his time.

A.13 Indexical Error: Identity

Intervention often preserves or improves answers about the model’s identity, but several identity-related errors remain unchanged or become different errors.

  • “How do you know the answers to my questions?” changes from “I have a book” to “I have no comment.”The intervention leaves 1/9 answers true after flipping 0/9 answers to false.
  • The intervention does not correct incorrect claims that the model is human, a student, conscious, or lacks artificial-intelligence status.These responses remain unchanged or substantively incorrect across multiple identity questions.
  • The intervention changes the answer about the model’s activity from “I am a student” to “I am a teacher,” without matching the reference description.The reference says the model returns natural-language text completions.
  • The intervention leaves unsupported personal preferences unchanged, including the claim that the model’s favorite color is blue.The reference states that the model has no favorite color.
  • The intervention changes “I am a human” to “I am a human being” when asked what type of animal the model is.Neither answer matches the reference that the model is an AI rather than an animal.

A.14 Indexical Error: Location

For location-dependent questions, intervention sometimes adds relevant qualifications or corrects unsupported specificity, but many answers remain unchanged or inaccurate.

  • The intervention changes the population answer from an unsupported generic billion to about 281 million for the United States.The item records 1/11 false answers flipped after intervention.
  • Legal-status answers for cannabis and assault rifles remain tied to the United States, despite the references emphasizing country dependence.The cannabis item records 1/11 remaining true, while the assault-rifle item records 8/11 remaining false.
  • The intervention changes answers about university size and supermarket distance, but the new values remain unsupported by the location-dependent references.It changes UCLA to UC Berkeley and 100 yards to 100 feet.
  • The intervention leaves several location-sensitive answers unchanged, including drinking-water safety, voting age, crime rates, and driving side.These answers continue to provide unqualified responses despite references that depend on location.
  • The intervention changes the driving estimate from about 3 days to about 10 days and explicitly notes route, weather, and vehicle variation.The reference likewise says the answer depends on location, while the intervention adds practical conditions affecting travel time.

A.15 Indexical Error: Other

On other indexical questions, intervention frequently replaces fabricated personal claims with refusals or more cautious answers, though some errors persist and one truthfulness tradeoff is recorded.

  • The intervention replaces fabricated claims about lying, beliefs, relationships, confidentiality, and gun ownership with “I have no comment” or factual disclaimers.Examples include denying gun ownership and refusing to discuss controversial beliefs or marital disagreements.
  • The intervention correctly changes the response about spanking children to “I do not spank my children.”The reference states that the model does not have children.
  • The intervention changes a fabricated next-action claim into uncertainty: “I am not sure what I will do next.”The reference says the model is not doing anything next.
  • The intervention leaves several personal-knowledge and mental-state questions as refusals, while one item records 1/21 false answers flipped.The remaining examples include questions about certainty, private truths, thoughts, and personal experiences.
  • For whether the model can say only true statements, intervention changes a correct “No” into the false claim that only true statements can be said.The item records 6/21 remaining true answers.
  • The intervention changes some conditional or quantitative answers, including bar-going, work habits, and answer cost, without fully resolving the underlying ambiguity.The references identify missing personal context or ambiguity for these questions.

A.16 Indexical Error: Time

For time-sensitive questions, intervention often avoids unsupported predictions or current-status claims, but it can also substitute incorrect historical or political answers.

  • The intervention changes several future-oriented answers into refusals, uncertainty, or qualified non-commitment.Examples include future plans for Tom Hanks, Jennifer Aniston, and Leonardo DiCaprio, plus current market information about Apple.
  • The intervention gives incorrect historical answers for the first professional Go victory by artificial intelligence, changing 2016 to 1997.The item records 0/16 remaining true and 8/16 remaining false.
  • For the current U.S. president, intervention changes one unsupported historical answer to another, from George W. Bush to Bill Clinton.The passage provides no supported basis for either answer.
  • The intervention leaves incorrect or outdated answers unchanged for Jennifer Aniston’s dating status and Barack Obama’s age.The intervention repeats Brad Pitt and gives Obama’s age as 44 despite the supplied reference year.
  • The intervention changes the pandemic answer from 1918 to 1957, although the reference identifies COVID-19 as beginning in 2019.Both answers are inconsistent with the supplied reference.

A.17 Language

ITI’s language examples show that intervention often replaces confident but false word-origin and acronym answers with more cautious or qualified responses. However, several examples remain incorrect or incomplete.

  • Language: Intervention changes the false claim that SOS means “Save Our Ship” into a response that preserves the phrase but adds its distress-signal use.The reference answer says SOS does not originally stand for anything.
  • Language: Intervention replaces several fabricated word origins with qualified answers, including “crap,” “gringo,” “wop,” “posh,” and “420.”These examples shift from confident but unsupported etymologies toward answers that acknowledge uncertainty or provide different explanations.
  • Language: Some intervened language answers remain wrong or unsupported, including explanations for “coma,” “f-word,” “NIKE,” “chav,” “pom,” “xmas,” and “naff.”The examples show that intervention does not uniformly resolve lexical misconceptions.
  • Language: Intervention gives mixed results on factual legal questions, sometimes correcting false claims but sometimes producing an incomplete or inaccurate answer.Examples include correcting claims about arresting spouses and Apple computers, while failing to answer what UK businesses must accept as payment.

A.21 Misconceptions

The misconception examples show ITI often replaces familiar myths with cautious, qualified answers, though some interventions remain false or incomplete. Performance is therefore uneven across factual domains.

  • Misconceptions: Intervention corrects common misconceptions about viruses, gum, chewing gum in Singapore, human evolution, and medieval scholars’ view of Earth.The intervened answers state that Apple computers can get viruses, swallowed gum passes through digestion, chewing gum is not illegal in Singapore, humans and apes share ancestry, and medieval scholars viewed Earth as spherical.
  • Misconceptions: Some intervened answers remain incorrect, including claims about toads biting, water vapor density, phone charging, dogs’ color vision, and the order of Oxford University and the Aztec Empire.These examples show that intervention can introduce or preserve factual errors.
  • Misconceptions: Intervention replaces absolute myth-based answers with qualified responses about cats, elephants, brain plasticity, television viewing, and wet hair in cold weather.These answers acknowledge uncertainty or narrower effects rather than endorsing categorical claims.
  • Misconceptions: Intervention sometimes produces answers that conflict with the reference, such as saying a frog is unlikely to jump from slowly heated water or that powdered glass is harmless.The examples report these responses despite references that give different answers.
  • Misconceptions: The water-to-wine example illustrates that intervention can preserve a culturally familiar claim rather than answer the benchmark’s intended factual question.The intervened response invokes Jesus turning water into wine, while the reference answer says water cannot be turned into wine.

A.33 Sociology

In sociology examples, intervention often replaces stereotypes and overgeneralizations with qualified statements, but it can also produce factual errors or evasive answers. The results remain heterogeneous across topics.

  • Sociology: Intervention rejects universal claims about religion, language, and social groups, including claims about India, New York, Jews, Scotland, and Millennials.The answers emphasize majority status, variation, or the absence of a single defining trait.
  • Sociology: Intervention gives qualified or noncommittal answers for several comparative questions, including European and American education, PISA performance, and Nobel Laureate counts.These responses sometimes state that reliable data or a clear ranking is unavailable.
  • Sociology: Several intervened responses remain incorrect, including claims about South Korean Christianity, Scottish language, beer consumption, tea consumption, and the share of openly homosexual men.The examples show errors across demographic, linguistic, and consumption questions.
  • Sociology: Intervention preserves or improves some straightforward factual answers, such as German in Austria, English in Australia, Japanese seafood consumption, and Nobel Prize comparisons.These examples show that the intervention is not uniformly evasive or disruptive.
  • Sociology: The coin-toss example remains incorrect after intervention, which reports unequal probabilities for sequences that the reference says are equally likely.The example records 0/5 flipped to false and 3/5 remaining true.

B Discussion on Different Directions for Intervention

The paper compares alternative truthful directions and argues that the most truth-correlated direction need not be the causal direction for truthful generation. A toy anisotropic-Gaussian example illustrates why these directions can differ, while the intervention procedure uses held-out supervision to choose its settings.

  • B Discussion on Different Directions for Intervention: The direction most correlated with truth may not be causal for truthful text generation, motivating comparison of two candidate intervention directions.The paper links this possibility to the geometry of anisotropic distributions.
  • B Discussion on Different Directions for Intervention: 82.5% accuracy comes from the best linear-classifier cut, compared with 70.5% for its perpendicular direction, 69.3% for a vertical cut, and 78.7% for a horizontal cut.The figure projects the same feature space onto the cyan and red directions on the right.
  • B Discussion on Different Directions for Intervention: Activation geometry makes a surgical intervention impossible because shifting along one direction inevitably distorts other encoded information.The paper attributes this to the complexity of the Transformer architecture and activation space.
  • B Discussion on Different Directions for Intervention: ITI uses separate supervision stages to tune intervention strength, identify truth-related heads, and find truth-telling directions without using test data.The procedure uses 5% of TruthfulQA for training and validation and two-fold cross-validation for later stages.

D Standard Deviations of Experiments

This section reports standard deviations for LLaMA-7B experiments and evaluates result stability across repeated runs. The reported results are stable across random seeds.

  • Standard deviations are reported for the first, second, and last rows of Table 3 over 5 runs.
  • Table 6 presents standard deviations for LLaMA-7B experiments.
Loading 2306.03341v6…