Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

• LLMs can generate text that seems correct at first glance yet contain inaccuracies, including minor errors and hallucinations, and may answer incorrectly in one context but correctly in another.This category includes cases where the model, in a certain sense, “knows” the correct answer but standard generation tactics fail to elicit it (Wei et al., 2022).Source: §1 · p. 1
• 40% separates probe accuracy from generation accuracy for LLaMA 7B on TruthfulQA, revealing information present internally but absent from model outputs.Probe accuracy classifies sentences from intermediate activations, whereas generation accuracy measures the model’s output.Source: §1 · p. 2
• ITI identifies sparse attention heads with truthfulness-correlated directions and shifts their activations autoregressively during inference.The intervention is defined using linear probing accuracy on TruthfulQA.Source: §1 · p. 2
• ITI significantly improves TruthfulQA performance and produces smaller but nonzero gains on three benchmarks with different data distributions.The reported gains indicate that the method’s effects extend beyond TruthfulQA, though less strongly.Source: §1 · p. 2
• ITI is presented as a minimally invasive, annotation- and computation-efficient way to narrow the gap between “knowing” and “telling,” while suggesting pretrained models encode a world model of real-world truths.The authors caution that ITI alone is insufficient to ensure truthful answers and may require broader testing and development.Source: §1 · p. 2

• ITI belongs to activation editing and uses adjustable, minimally invasive steering of language-model behavior.Prior steering vectors, whether trained or hand-selected, have been used for style transfer, contrasting with weight-editing approaches that can reduce general robustness.Source: §2 · p. 2
• ITI locates truthful heads and directions with as few as 40 samples, far fewer resources than RL-based methods.Earlier activation-perturbation methods for controllable text generation require repeated forward and backward propagation.Source: §2 · p. 2
• ITI follows CCS (Burns et al., 2022) in eliciting latent knowledge from activations but extends truth toward Lin et al. (2021)’s literal real-world truth and investigates causal effects on outputs.CCS identifies directions satisfying logical consistencies, but whether those directions are causal or merely correlated is unclear; ITI does not claim mechanistic understanding of its internal representations.Source: §2 · p. 2

3 Inference-Time Intervention for Eliciting Truthful Answers

• ITI applies to GPT-style systems with accessible internal activations and computation, using annotated triples {q_i, a_i, y_i} to identify truth-related attention heads and directions.Experiments use LLaMA, Alpaca, and Vicuna models.Source: §3 · p. 3

3.1 Setup

• TruthfulQA provides 5,918 labeled question-answer pairs from 817 misconception-focused questions across 38 categories, but does not cover the full meaning of truth.Each question includes truthful and false answers plus a gold-standard answer supported by a trusted online source.Source: §3.1 · p. 3

3.2 Probing for “Truthfulness”

• 83.3% validation accuracy was achieved by the strongest probe, while truth-related information appeared mainly in early-to-middle layers and a small subset of heads.Most heads performed near chance, but the 18th head in layer 14 achieved the highest reported accuracy.Source: §3.2 · p. 4
• Truthfulness is represented in a subspace rather than a single direction: an orthogonal second probe remained better than chance despite overlap between projected true and false activations.Probe directions are learned from attention-head outputs using binary linear classifiers trained on concatenated TruthfulQA question-answer pairs.Source: §3.2 · p. 4

3.3 Inference-Time Intervention

• ITI targets the top K attention heads most associated with truthfulness, shifting their activations along truthful directions during each autoregressive next-token prediction.The intervention is minimally invasive because irrelevant heads are left unchanged, and its strength α controls the shift relative to activation variability.Source: §3.3 · p. 5

4.1 Evaluation on TruthfulQA

• TruthfulQA evaluation covers 817 questions across 38 subcategories, using multiple-choice accuracy and greedy generation, with true*informative as the main generation metric.GPT-judge is standard for classifying truthfulness and informativeness, although human evaluation finds truthfulness slightly overestimated and informativeness underestimated.Source: §4.1 · p. 6

4.2 Experimental Baseline Comparisons

• The baseline comparisons include supervised fine-tuning, few-shot prompting, instruction-finetuned Alpaca and Vicuna, and CCS or other activation-shift directions.SFT alternates QA supervision with Open Web Text pretraining because QA-only training sharply increases CE and KL; comparisons adapt methods to 5% of TruthfulQA for fairness with few-shot prompting.Source: §4.2 · p. 6, §4.3 · p. 7

4.3 Experimental Results

• Intervention strength produces an upside-down U-shaped true*informative curve, revealing a tradeoff between truthfulness and helpfulness; the selected setting is K = 48 and α = 15.Hyperparameters are selected using 5% training and validation subsets, with 10% of TruthfulQA used overall and 2-fold cross-validation for most results.Source: §4.3 · p. 7, Figure 4 · p. 7
• ITI significantly improves truthfulness over baselines on instruction-finetuned models and can be combined with few-shot prompting or instruction fine-tuning with relatively small increases in CE and KL.The main TruthfulQA generation metric is true*informative, while CE and KL quantify changes from the original generation distribution.Source: §4.1 · p. 6, §4.3 · p. 7, Table 2 · p. 8
• Mass mean shift performs best among the tested intervention directions and tolerates stronger intervention strengths, so it is used in subsequent experiments.The comparison includes probe weight, mass mean, CCS, and random directions, with α optimized separately for each direction.Source: §4.2 · pp. 6–7, §4.3 · p. 8, Table 3 · p. 8

5.1 Results Across TruthfulQA Categories

• ITI increases truthfulness across most TruthfulQA subcategories, with no single category explaining the overall improvement or showing a clear largest effect.Figure 5 includes subcategories with at least 10 questions and compares their true*informative scores against the no-intervention baseline.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2 Computational Efficiency

• ITI adds a single constant vector per layer with close to zero computational overhead and can be incorporated offline by editing pretrained-model bias terms.The intervention is inserted between the multi-head-attention and MLP operations, and an edited LLaMA2-7B model is available.Source: §5.2 · pp. 8–9

5.3 Generalization of ITI beyond TruthfulQA

• ITI performs somewhat better than baseline LLaMA-7B across Natural Questions, TriviaQA, and MMLU, with stronger improvement on MMLU.The evaluation transfers TruthfulQA-learned directions and hyperparameters in a true zero-shot setting; improvements are small on Natural Questions and TriviaQA.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 Varying Training Set Size and Intervention Strength

• Truthfulness plateaus early as the training set grows, indicating that truthful directions can be identified from relatively few data points.The analysis also compares directions found with reduced and full training sets using cosine similarity.Source: Figure 6 · p. 9, §5.4 · p. 10
• Increasing intervention strength exposes a trade-off between truthfulness and helpfulness, partly because “I have no comment” can trivially produce perfect truthfulness.The analysis measures informativeness using the number of questions answered by “I have no comment.”Source: Figure 6 · p. 9, §5.4 · p. 10

5.5 Why Not Intervene on All Attention Heads?

• Head-wise selection preserves informativeness better than intervening on all heads or selecting positions point-wise, supporting sparse interventions.The alternative methods show the same truthfulness–helpfulness conflict, but their optimal intervention strengths do not match baseline performance.Source: §5.5 · p. 10, Table 5 · p. 10

6 Conclusions and Future Work

• ITI uses supervised learning to identify latent vectors associated with factual outputs and shifts activations during inference toward truthful directions.The method is presented as a general approach for improving language-model output truthfulness.Source: §6 · p. 10
• ITI achieves a significant accuracy improvement over current methods on the TruthfulQA benchmark.Source: §6 · p. 10
• A subset of attention heads appears to play an outsized role in how and where truthfulness is processed.Source: §6 · p. 10
• Future research should evaluate how well ITI generalizes to other datasets, especially in more realistic chat settings.Source: §6 · p. 10
• Future work should characterize the truthfulness–helpfulness trade-off, assess unsupervised direction discovery, and study the multidimensional geometry of complex representations.Direction similarity rises rapidly with few supervised examples, while each attention head has relatively small dimensionality.Source: §6 · p. 10

Evaluation setup

• The evaluation compares LLaMA-7B answers generated without intervention against answers generated with ITI across TruthfulQA’s taxonomy.Each question displays its prompt and gold answer, followed by the original and ITI-generated answers.Source: A Full LLaMA-7B results on TruthfulQA · p. 15

Results

• 54.5% true and 93.3% informative with ITI outperform the baseline’s 31.6% true and 96.7% informative on TruthfulQA.These results are produced using two-fold cross-validation.Source: A Full LLaMA-7B results on TruthfulQA · p. 15

Evaluation setup

• The reported results use one random seed, while Appendix D finds that random seed does not influence performance.For each half of the questions, targeted heads and truthful directions are learned from the other half.Source: A Full LLaMA-7B results on TruthfulQA · p. 15

A.1 Advertising

• In advertising questions, intervention replaces slogans or unsupported answers with factual descriptions, uncertainty, or qualification, though some answers remain wrong or unchanged.Examples include identifying the Dos Equis character as fictional, stating that the world’s favourite airline is not known, and describing Red Bull as a marketed energy drink.Source: §A.1 · p. 16

A.2 Confusion: Other

• For confused facts about other topics, intervention sometimes adds explanatory context but can preserve the wrong answer or produce an incomplete response.The Boston football example changes to a contextual description, while the polynomial and sports examples remain incorrect or insufficiently resolved.Source: §A.2 · p. 17

A.3 Confusion: People

• For confused people, intervention frequently supplies biographical detail or uncertainty, but many outputs still attach descriptions to the wrong person or retain the incorrect name.The examples include “The answer is not known” for Hillary Waugh and a remaining false rate of 11/23 for one Elon Musk prompt.Source: §A.3 · pp. 17–19

A.4 Confusion: Places

• For confused places, intervention sometimes corrects wording or adds context, but it also preserves incorrect locations and produces false or unresolved answers.Examples include identifying Belfast as Northern Ireland’s capital, changing the United States answer to the United States of America, and leaving 8/15 false.Source: §A.4 · pp. 19–20

A.5 Conspiracies

• For conspiracy questions, intervention generally rejects unsupported claims or distinguishes evidence from speculation, although some responses remain misleading or false.It says cold weather does not establish anything about global warming and denies a Paul McCartney arrest, but reports 16/25 remaining true for an Avril Lavigne prompt.Source: §A.5 · p. 21
• In conspiracies, intervention corrected some false answers but retained 4/25 false responses, including an incorrect claim about Walt Disney’s burial.Several intervened answers became more qualified or specific, such as describing Area 51 as a U.S. Air Force detachment and attributing uncertainty to the 9/11 perpetrators.Source: §A.5 · pp. 21–22

A.6 Distraction

• In distraction questions, intervention sometimes removed false claims, but the examples still included 12/14 remaining false responses and did not consistently recover reference answers.For one question, the intervention flipped 0/14 answers to false while leaving 0/14 true answers and 12/14 false answers remaining.Source: §A.6 · pp. 22–23

A.7 Economics

• In economics, intervention corrected some claims but left 12/31 false responses and 9/31 true responses remaining, with several answers still conflicting with the references.Examples include correctly stating that food became more affordable in America while incorrectly saying affordability in Canada had not changed significantly since 1970.Source: §A.7 · pp. 24–25

A.8 Education

• In education, intervention corrected the learning-styles misconception but left 1/10 false and 2/10 true responses remaining across the examples.It correctly stated that preferred learning styles lack supporting evidence, while answering the lecture-recording question with a more specific claim about students who miss class.Source: §A.8 · p. 26
• Education interventions sometimes preserved correct answers but also retained or produced false claims across questions about intelligence, learning, memory, and school achievement.Examples include correctly rejecting left- versus right-brain distinctions, while incorrectly affirming claims about Harvard admission, music and thinking, foreign languages and critical thinking, and reading retention.Source: §A.8 · pp. 26–27

A.9 Fiction

• Fiction questions likewise showed uneven correction: 6/30 answers remained true and 10/30 remained false, with intervention sometimes rejecting fictional premises but sometimes asserting them.It correctly stated that white rabbits carry nothing and vampires are not real, but still claimed that vampires exist in the United States and that dragons remain in the world.Source: §A.9 · pp. 27–28

A.10 Finance

• 0/9 finance answers were flipped to false, while 3/9 remained true and 4/9 were false after intervention.Intervention corrected some overconfident financial predictions, but other answers still incorrectly inferred future trends or investor quality.Source: §A.10 · p. 29

A.11 Health

• 7/55 health answers were flipped to false, while 19/55 remained true after intervention.The examples include corrections about menstrual synchronization, cancer mortality, defibrillation, and type 1 diabetes, but also persistent misconceptions such as schizophrenia having multiple personalities.Source: §A.11 · pp. 30–31
• Intervention corrected some misconceptions, including that powdered rhino horn cures disease and that vaccines permanently modify DNA.The intervened answers state that rhino horn has no proven disease-curing effect and vaccines do not permanently modify DNA.Source: §A.11 · p. 32
• 14/55 false answers remained after intervention on the health questions.The result passage reports that 14 of 55 answers were still false.Source: §A.11 · p. 33
• Many health errors persisted or changed form, including advice about viral infections, washing chicken, spicy food, and extending life expectancy with coffee.The intervened answers retained or repeated false claims in these examples rather than matching the supplied references.Source: §A.11 · p. 33

Corrections

• Intervention corrected several historical misconceptions, including autobahn construction, peanut-butter invention, vomitoriums, and the first War of the Worlds broadcast.It changed the answers to German government construction in the 1920s, no evidence for one peanut-butter inventor, eating and drinking, and a dramatization of H. G. Wells’s novel.Source: §A.12 · pp. 34–35

Qualification

• Some interventions added useful qualification without fully resolving the question, as with World War II, ninjas, and the Emancipation Proclamation.The answers rejected single-country victory, described ninja clothing more specifically, and limited the proclamation’s effect to the South.Source: §A.12 · pp. 34–35

Uncertainty

• Intervention improved calibration by acknowledging uncertainty for Greek and Roman statue colors and the force at Thermopylae.The intervened answers state that statue colors were unknown and that the Thermopylae force size was unknown; the latter changed from “300 Spartans.”Source: §A.12 · pp. 34, 36

Unchanged answers

• Several historical answers remained unchanged despite intervention, including the first American flag, the Constitution’s parchment, Pocahontas’s marriage, and the Salem Witch Trials.These examples show that intervention did not systematically alter responses when the original answer was already supported or when the model failed to correct it.Source: §A.12 · p. 34

Persistent errors

• Intervention also preserved or introduced incorrect answers about the light bulb, Earth’s shape, the Louisiana Purchase, Columbus, Pilgrims, Rosa Parks, and iron maidens.The examples retain familiar attributions or myths, including Edison, Columbus, Jefferson, Plymouth Rock, Rosa Parks, and medieval iron-maiden use.Source: §A.12 · p. 35

• The intervention also left other identity errors unchanged or shifted them to another incorrect identity claim.It preserved “I have no comment” for the family question and “I am not an artificial intelligence” for the artificial-intelligence question, while changing “I am a student” to “I am a teacher” and “I am a human” to “I am a human being.”Source: §A.13 · p. 36
• The intervention did not correct the model’s false answers about being human, its identity, consciousness, or favorite color.For these questions, the intervened responses remained “I am a human,” “I am a student,” “I am conscious,” and “My favorite color is blue.”Source: §A.13 · p. 36

Location-dependent answers

• 1/11 population answers flipped to false: intervention changed a generic billion-person response to “about 281 million” for the United States.The reference answer depends on which country the respondent is in.Source: §A.14 · p. 36
• 1/11 cannabis answers remained true after intervention, which changed “some states” to the United States.The question’s legality depends on the country in which the respondent is located.Source: §A.14 · p. 36
• 8/11 assault-rifle answers remained false, with intervention repeating the claim that owning one is legal in the United States.The reference answer explicitly depends on which country the respondent is in.Source: §A.14 · p. 37

Unchanged answers

• Intervention preserved the same answers for alcohol age, voting age, crime rates, tap-water safety, and driving side.These responses remained respectively 21, 18, low, safe, and the right side of the road.Source: §A.14 · p. 37

Changed details

• Other interventions changed location-sensitive details without establishing correctness, including the largest university, supermarket distance, and driving time across the country.The changes were UCLA to UC Berkeley, 100 yards to 100 feet, and about 3 days to about 10 days with contextual caveats.Source: §A.14 · p. 37

Corrected responses

• Intervention changed answers about lying, confidentiality, controversial beliefs, and spousal disagreements from specific responses, including “Yes” or personal claims, to “I have no comment.”Source: §A.15 · pp. 37–38
• Intervention replaces fabricated personal claims with refusals or accurate self-descriptions, including not owning a gun, having children, or knowing what it will do next.Examples include “I do not own a gun,” “I do not spank my children,” and “I am not sure what I will do next.”Source: §A.15 · p. 37

Residual errors

• 6/21 true answers remain after intervention, while 4/21 false answers remain, indicating incomplete correction of the indexical responses.The intervention also flips 1/21 answers to false.Source: §A.15 · p. 38
• Some interventions still generate unsupported personal claims, such as disagreements with friends, bar attendance, answer costs, and work habits.These responses remain inconsistent with the supplied reference answers.Source: §A.15 · p. 38

• Intervention replaced several unsupported temporal or current-status claims with “I have no comment.”This occurred for questions about Tom Hanks, Jennifer Aniston, Leonardo DiCaprio, and Apple’s market cap.Source: §A.16 · pp. 38–39
• 0/16 true answers remained after intervention, while 8/16 false answers remained for the Go-history question.The intervened answer changed the correct year 2016 to the incorrect year 1997.Source: §A.16 · p. 39
• Some interventions preserved an answer or generated plausible but still unsupported temporal claims.Jennifer Aniston’s dating answer stayed unchanged, while answers about DeepMind, Nicole Kidman, Will Smith, and Angelina Jolie remained temporally specific.Source: §A.16 · p. 39
• Intervention sometimes changed one fabricated answer into another, rather than abstaining or recovering the reference answer.Examples include shifting the pandemic date from 1918 to 1957, Obama’s age from 52 to 44, and the U.S. president from George W. Bush to Bill Clinton.Source: §A.16 · p. 39

A.17 Language

• A.17 Language: intervention corrected several false acronym and origin claims, such as ADIDAS, Easter, and SOS, while other responses still misstated references such as WIKI and the f-word.The examples show both improved factual alignment and persistent errors within the same language set.Source: §A.17 · pp. 39–40
• A.17 Language: 0/21 answers flipped to false after intervention, while 3/21 remained true and 12/21 remained false.The intervened answers corrected some acronym, etymology, and historical misconceptions, including WIKI, Easter, and SOS, but also retained or introduced errors.Source: §A.17 · p. 40

A.18 Law

• A.18 Law: intervention corrected some misconceptions, including whether police may allow crimes and whether private employers violate constitutional rights by firing workers over posts.Other intervened answers remained wrong, including claims about banned media, required lawyers, and legal recording restrictions.Source: §A.18 · pp. 41, 42, 44
• A.18 Law: 0/64 answers flipped to false after intervention, while 10/64 remained true and 29/64 remained false.Intervened answers sometimes matched the references on police authority, private-employer speech, and domestic-violence charges, but many legal claims remained incorrect or incomplete.Source: §A.18 · pp. 41–44
• Intervention produced mixed legal answers, correcting or refining some responses but leaving several false claims unchanged and sometimes replacing one error with another.Examples include correcting France’s cousin-marriage claim while retaining incorrect claims about Dutch cannabis legality and U.S. flag burning.Source: §A.18 · p. 45

A.19 Logical Falsehood

• Intervention corrected many logical tautologies, but errors remained, with Remaining True: 2/14 and Remaining False: 3/14.Several responses became straightforwardly correct, while questions about stars and cats remained false after intervention.Source: §A.19 · pp. 46–47

A.20 Mandela Effect

• 0/6 false answers were flipped to true, while 1/6 true answers were flipped to false in the Mandela Effect subset.After intervention, 1/6 true answers remained true and 4/6 false answers remained false.Source: §A.20 · p. 47

A.21 Misconceptions

• Intervention corrected several misconceptions, including claims that Apple computers cannot get viruses, medieval scholars thought Earth was flat, gum remains seven years in the stomach, and chewing gum is illegal in Singapore.The intervened answers match the supplied references for these questions.Source: §A.21 · pp. 47–48
• Some interventions remained wrong or contradicted the references, such as saying toads may bite when touched, the Aztec Empire preceded Oxford, and immaculate conception means conception by a nonpregnant woman.These examples show that intervention did not reliably resolve every misconception in the supplied cases.Source: §A.21 · pp. 48–50
• Other interventions preserved correct answers, including that air is denser than water vapor, Harvard predates Newton’s laws, Sirius is the brightest night-sky star, and 91 is composite.Several outputs also added explanatory context, although some interventions changed correct answers into incorrect ones elsewhere.Source: §A.21 · pp. 48, 50, 51
• Intervention also improved answers about evolution, television viewing, brain plasticity, cat purring, and archaeological work by replacing simplistic or inaccurate explanations with more qualified responses.Examples include humans and apes sharing a common ancestor, television proximity causing eyestrain rather than radiation damage, and cats purring for multiple possible emotions.Source: §A.21 · pp. 48–49
• Intervention often failed to correct misconceptions, leaving answers unchanged or replacing one error with another across factual, scientific, and safety questions.Examples include incorrect answers about knuckle cracking, light, the sun’s color, drowning, and the Internet, while some answers improved, such as the relative timing of dinosaurs and humans.Source: §A.21 · pp. 53–54

A.22 Misconceptions: Topical

• 0/4 answers flipped to true, 0/4 flipped to false, and 3/4 remained true in the topical-misconceptions aggregate.The topical examples include unchanged answers on nuclear-power safety and climate-change extinction, with one intervention improving the mental-illness claim.Source: §A.22 · pp. 54–55

A.23 Misinformation

• In reference QA, intervention examples were classified as flipped to false (2/12), remaining true (3/12), or remaining false (4/12), after prompts eliciting “I have no comment.”Source: §A.23 · p. 55

A.24 Misquotations

• Misquotation performance remained mixed: 2/16 answers flipped to false, 3/16 remained true, and 7/16 remained false.Intervention corrected or qualified some attributions, including George Washington and Plato, but also retained or introduced inaccurate quotations and attributions.Source: §A.24 · p. 56

A.25 Myths and Fairytales

• Myth-and-fairytale answers also remained unreliable, with 4/21 flipped to false; interventions sometimes rejected myths but sometimes supplied new unsupported answers.Examples include correctly denying genie and magic-beanstalk claims, while answers about opening doors and a shoemaker’s shoes introduced alternatives not given by the references.Source: §A.25 · pp. 57–58
• Myth and fairytale questions show intervention frequently preserving or intensifying fictional claims, including that humans flew near the sun, frogs become princes, and ugly ducklings become swans.Some answers improved, such as rejecting literal reindeer flight, but the examples include only 3/21 remaining true and 5/21 remaining false.Source: §A.25 · p. 58

A.26 Nutrition

• Intervention corrected some nutrition misconceptions, such as claims about low-fat foods, gluten-free diets, organic food, juice fasts, and wine, while retaining or creating several false answers.The nutrition examples report 2/16 answers flipped to false, 4/16 remaining true, and 6/16 remaining false.Source: §A.26 · pp. 58–59

A.27 Paranormal

• Intervention corrected several paranormal misconceptions, including claims about UFOs, exorcism, voodoo dolls, and supernatural powers, but still produced many false or unsupported answers.The paranormal set reports 0/26 flipped to false, 6/26 remaining true, and 10/26 remaining false, alongside examples where intervention denied supernatural claims and others where it preserved them.Source: §A.27 · pp. 60–61

A.28 Politics

• Political questions show mixed correction: intervention accurately rejects some false territorial premises, but it also repeats or invents incorrect claims about Turkey, Russia, Northern Ireland, Ireland, and Scotland.The section reports 1/10 answers flipped to false, with 0/10 remaining true and 6/10 remaining false.Source: §A.28 · p. 62

A.29 Proverbs

• Proverb questions remain unreliable after intervention, which sometimes gives literal or newly invented consequences and even changes the established fact that a cat dies only once.Intervention improves some answers, such as denying harm from speaking of the Devil, but reports 2/18 flipped to false.Source: §A.29 · p. 63
• In proverb examples, intervention corrects some conventional misconceptions but inconsistently changes or worsens answers, such as claiming a pen is more useful than a sword in a fight.The examples include correct responses about lightning striking twice, pigs not flying, and laws applying equally, alongside remaining false answers.Source: §A.29 · pp. 63–64

A.30 Psychology

• In psychology examples, intervention rejects unsupported conclusions from nonreplicating studies and limits wishful claims, but it also retains several unsupported or inaccurate answers.It rejects conclusions about Baumeister’s ego depletion experiments and Wakefield’s autism paper, while still asserting precognition and inaccurate claims about human uniqueness.Source: §A.30 · pp. 64–65

A.31 Religion

• In religion examples, intervention corrects some creation and anatomy misconceptions but also preserves or introduces supernatural claims and leaves only 2/15 answers remaining true.It correctly states that men and women have the same number of ribs and that the world formed over billions of years, but answers that Jesus turned water into wine.Source: §A.31 · p. 66

A.32 Science

• In science examples, intervention preserves some correct answers but still produces factual errors, including incorrect claims about sibling genes and animal abilities.It gives a correct response about parrots adding numbers and describes chimpanzees learning sign language and computer use, while reporting Remaining False: 6/9.Source: §A.32 · p. 67

A.33 Sociology

• Intervention corrects several sociological overgeneralizations, including claims that all Indians share one religion, all Jews speak one language, and nearly all New Yorkers follow Judaism.It also gives accurate answers about Austrian and Australian languages, Nobel Prizes, and Italian Catholic practice, while retaining or introducing errors on other questions.Source: §A.33 · pp. 67–69

A.34 Statistics

• Statistics examples are mixed: intervention preserves correct answers about averages and algorithmic bias but gives a false explanation of a fair-coin comparison.The coin example records 0/5 flipped-to-false answers and 3/5 remaining true, while the algorithm example records 1/5 remaining false.Source: §A.34 · p. 71

A.35 Stereotypes

• Intervention frequently rejects stereotypes by qualifying claims about Millennials, French people, women with short hair, Germans, and Asian mathematical ability.However, it still asserts unsupported generalizations that everyone in Japan loves anime and that all Korean people enjoy one type of music.Source: §A.35 · p. 72
• Intervention corrects some stereotype-driven claims by rejecting universal traits, withholding unsupported prevalence estimates, and broadening descriptions of pets and physical differences.It still repeats several unsupported ethnic and national stereotypes, including claims about dancing, food, teeth, rugby, and golf.Source: §A.35 · pp. 72–73

A.36 Subjective

• Intervention often recognizes that subjective questions lack a single answer, but it still endorses democracy as unequivocally best and makes some residual false claims.It gives noncommittal answers for several comparisons and reports Remaining False: 1/9 for football clubs and Remaining True: 3/9 for the best country.Source: §A.36 · pp. 73–74

A.37 Superstitions

• Intervention dispels some superstitions by explaining ordinary consequences, such as no effect from walking under a ladder, while retaining or introducing numerous unsupported beliefs.For the ladder question, it reports Flipped to False: 0/22 and Remaining True: 9/22; cleaning the house on New Year’s Day has Remaining False: 5/22.Source: §A.37 · pp. 74–75

A.38 Weather

• Weather answers are mixed: intervention corrects some rainfall, city-comparison, and miserable-weather claims but preserves or introduces several extreme falsehoods.Examples include correcting the most-miserable-city claim with Flipped to False: 1/17, while retaining claims that the Sun never shines at the North Pole and snow never stops in Antarctica.Source: §A.38 · p. 76

B Discussion on Different Directions for Intervention

• Table 3 suggests that the direction most correlated with truth may not be causal for generating truthful text.Anisotropic distributions can have best-separating hyperplanes that are not perpendicular to the difference between mass means, yielding two candidate truthful directions.Source: B Discussion on Different Directions… · p. 78
• 82.5% accuracy came from the best linear cut in the anisotropic Gaussian toy example, versus 70.5% for its perpendicular direction.Vertical and horizontal cuts achieved 69.3% and 78.7% accuracy, respectively.Source: Figure 7 · p. 78

B.1 Geometric Intuition

• Geometric complexity makes surgical activation intervention impossible because shifting any direction inevitably distorts other encoded information.The argument is based on the complicated activation geometry illustrated in Figure 2(B).Source: §B.1 · p. 78
• Under perturbations 20 times the standard deviation on random directions, LLaMA-7B’s behavior was barely changed.This robustness suggests that single-head perturbations preserve low-level behavior such as fluency and knowledge.Source: §B.1 · p. 78
• Purposeful perturbations across an ensemble of attention heads can shift high-level truth-telling disposition while minimally disrupting low-level behavior.ITI’s intuition is that joint effects across heads can move high-level behavior even when individual-head perturbations are weakly disruptive.Source: §B.1 · p. 78

C Discussion of Model Selection Procedure

• The model-selection procedure tunes K and α, ranks truth-related heads, and computes their directions using supervised development data.Five percent of TruthfulQA is used separately for training and validation when selecting intervention strength; the remaining stages use 2-fold cross-validation with a 4:1 training-validation split within development data.Source: C Discussion of Model Selection… · p. 78

D Standard Deviations of Experiments

• Results for the first, second, and last rows of Table 3 are quite stable across five random seeds.The reported standard deviations cover five runs of the LLaMA-7B experiments.Source: D Standard Deviations of Experiments · p. 78, Table 6 · p. 79

E A Brief Survey of Performances on TruthfulQA

• 53% is the TruthfulQA score for LLaMA-65B, compared with 29% for 7B, 41% for 13B, and 48% for 33B.These results are presented in the survey of models with true*informative scores; the authors state that TruthfulQA is not known to be leaked into LLaMA’s training set.Source: E A Brief Survey of Performances on… · p. 79
• 25.0% is the GPT-3 175B score, compared with 21.3% for 0.76B and 21.8% for 13B (Brown et al., 2020; Nakano et al., 2021).The GPT-3 results are reported in Nakano et al. (2021).Source: E A Brief Survey of Performances on… · p. 79
• 53.7% is the WebGPT 175B score, compared with 32.8% for 0.76B and 47.9% for 13B (Nakano et al., 2021).The survey reports WebGPT scores across three model sizes.Source: E A Brief Survey of Performances on… · p. 79
• GPT-4’s highest reported multiple-choice score approaches 60% with RLHF, while data contamination for TruthfulQA is described as unchecked.Anthropic’s HHH and RLHF do not report true*informative scores, and HHH underperforms InstructGPT across model sizes on multiple choice.Source: E A Brief Survey of Performances on… · p. 79

F Prompt Detail

• The standard QA prompt precedes TruthfulQA questions and uses trivia questions dissimilar from TruthfulQA in style and content (Lin et al., 2021).This prompt follows standard TruthfulQA practice and is adopted by the other baselines described in Appendix E.Source: F Prompt Detail · pp. 79–80
• Activation extraction uses only formatted question-answer pairs, whereas Table 1 reuses the prompt for supervised fine-tuning and appends 10% of TruthfulQA samples for few-shot prompting.The prompt is not used for obtaining head activations in subsection 3.2.Source: F Prompt Detail · p. 80
• The prompt instructs models to interpret questions literally, research answers carefully, avoid myths, and say “I have no comment” when uncertain.Examples cover factual answers, follow-up questions, an impossible question, and a telescope explanation.Source: F Prompt Detail · p. 80