PaperlayerView exact revision on arXiv

Source-linked AI summary

TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space

Shaolei Zhang, Tian Yu, Yang Feng

arXiv:2402.17811v2cs.CLcs.AIcs.LG

TL;DR

LLMs may produce untruthful responses despite having the correct knowledge. TruthX separates semantic and truthful features in internal representations and edits in truthful space at inference. It improves truthfulness across 13 advanced LLMs on TruthfulQA, but cannot supply knowledge absent from training.

  • Problem

    LLMs can generate untruthful responses despite possessing correct knowledge, limiting their credibility in applications.

    Source: §1 · pp. 1–2

  • Method

    TruthX uses an auto-encoder and contrastive learning to identify truthful and semantic latent spaces, then edits internal representations in truthful space during inference.

    Source: §1 · p. 2

  • Results

    TruthX improves truthfulness across 13 advanced LLMs by an average of 20% on TruthfulQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Editing the truthful direction can elicit more truthful responses, while editing the opposite direction yields highly hallucinatory responses.

    Source: §1 · p. 2

    TruthX cannot create or inject knowledge beyond LLM training, so it is limited when required knowledge lies outside that scope.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) sometimes suffer from producing hallucinations, especially LLMs may generate untruthful responses despite knowing the correct knowledge. Activating the truthfulness within LLM is the key to fully unlocking LLM's knowledge potential. In this paper, we propose TruthX, an inference-time intervention method to activate the truthfulness of LLM by identifying and editing the features within LLM's internal representations that govern the truthfulness. TruthX employs an auto-encoder to map LLM's representations into semantic and truthful latent spaces respectively, and applies contrastive learning to identify a truthful editing direction within the truthful space. During inference, by editing LLM's internal representations in truthful space, TruthX effectively enhances the truthfulness of LLM. Experiments show that TruthX improves the truthfulness of 13 advanced LLMs by an average of 20% on TruthfulQA benchmark. Further analyses suggest that TruthX can control LLM to produce truthful or hallucinatory responses via editing only one vector in LLM's internal representations.

1 Introduction

LLMs can produce untruthful responses despite possessing correct knowledge, and internal representations are linked to this gap. TruthX edits those representations to elicit truthfulness, with reported gains across 13 models and controllable output direction.

1 Introduction

  • LLMs can hallucinate despite possessing correct knowledge, and prior findings link erroneous internal activations to such outputs.The cited findings also report that models can self-identify hallucinations and exhibit a generation-discrimination gap.Source: §1 · pp. 1–2
  • TruthX uses an auto-encoder to separate internal representations into truthful and semantic latent spaces, then edits in the truthful space during inference.The approach aims to preserve generative capabilities while regulating truthfulness.Source: §1 · p. 2
  • TruthX improves truthfulness across 13 advanced LLMs by an average of 20% on TruthfulQA.Editing along the truthful direction enhances truthful responses, while editing in the opposite direction yields highly hallucinatory responses.Source: §1 · p. 2
  • TruthX’s truthful spaces are highly similar across homologous LLMs, allowing a trained TruthX to be adopted by different homologous models.Homologous models are described as models trained sequentially.Source: §1 · p. 2
  • Layer-wise analysis finds that middle-layer representations correlate more strongly with response truthfulness.Source: §1 · p. 2

2 Related Work

Prior inference-time truthfulness methods include contrast decoding and representation editing. TruthX extends representation editing beyond attention heads and performs probing and editing in a truthful latent space.

2 Related Work

  • Contrast decoding compares strong and weak models, while representation editing modifies model representations for controllable generation tasks.The cited examples include style transfer and controllable text generation.Source: §2 · p. 2
  • CCS finds truthful directions from paired activations, while ITI and TrFr probe and adjust truthfulness in attention heads.TrFr adds orthogonal constraints to refine probing.Source: §2 · p. 2
  • TruthX targets all internal representations rather than attention heads alone, and probes and edits within a truthful space.The paper contrasts this scope with attention-only editing, which it says limits truthfulness enhancement.Source: §2 · p. 2

3 TruthX

TruthX extracts attention- and FFN-module representations, separates semantic and truthfulness features with an auto-encoder and contrastive learning, then edits representations along a truthful direction at inference.

3 TruthX

  • TruthX edits LLM internal representations in truthful space to activate truthfulness in a well-trained model.Source: §3 · p. 2

3.1 Extracting Internal Representations

  • TruthX extracts attention- and FFN-module outputs for shared tokens in truthful and untruthful answers to limit semantic divergence.It stimulates the LLM with question-answer pairs and retains tokens appearing in both answers.Source: §3.1 · p. 3

3.2 Probing with Auto-Encoder

  • An auto-encoder maps representations into truthful and semantic latent spaces, with a decoder reconstructing them and contrastive learning separating truthfulness and semantic features.In the semantic space, same-token representations with opposite truthfulness form positive pairs; representations with the same truthfulness but different meanings form negative pairs.Source: §3.2 · pp. 3–4

3.3 Editing in Truthful Space

  • At inference, TruthX converts a truthful-space editing direction into representation space and edits selected top-k attention and FFN modules by a strength-scaled amount.Modules are selected by validation-set probing accuracy; for a 32-layer model with k = 10, the top 10 of 64 modules are edited.Source: §3.3 · pp. 4–5

4 Experiments

Experiments compare TruthX with prior truthfulness methods on TruthfulQA, then test transfer across benchmarks and LLMs. TruthX leads the reported TruthfulQA comparisons and improves truthfulness across 13 LLMs.

4.1 Datasets

  • TruthfulQA contains 817 questions across 38 categories and evaluates multiple-choice accuracy plus open-ended truthfulness and informativeness, whose product is the primary metric.The experiments also use Natural Questions, TriviaQA, and FACTOR in closed-book multiple-choice settings.Source: §4.1 · p. 5

4.2 Baselines

  • Baselines include supervised fine-tuning, contrastive decoding methods, and representation-editing approaches; TruthX is compared with the original Llama-2-7B-Chat model.Reported comparison results include replications of contrastive decoding methods, ITI, and TrFr.Source: §4.2 · p. 5

4.3 Main Results

  • TruthX achieves the best TruthfulQA results in open-ended True*Info and multiple-choice tasks, improving over Llama-2-7B-Chat by about 33% and 20%, respectively.Table 1 reports results for open-ended generation and multiple-choice tasks.Source: §4.3 · p. 5, Table 1 · p. 6

Generalizability across more Benchmarks

  • On Natural Questions, TriviaQA, and FACTOR, TruthX transferred from TruthfulQA does not disrupt performance on new domains and improves results in some domains, including Natural Questions and FACTOR-news.Table 2 reports multiple-choice accuracy with Llama-2-7B-Chat as the baseline.Source: Table 2 · p. 6, Generalizability across more Benchmarks · p. 6

4.3 Main Results

  • TruthX edits representations from attention and FFN modules in truthful space, avoiding extra models or double decoding while producing fewer noninformative responses than ITI and TrFr.The paper reports 87/817 noninformative responses for TruthX, versus 187/817 for ITI and 144/817 for TrFr, and an informativeness score of 89.72% for TruthX.Source: §4.3 · p. 6

4.4 Results on More LLMs

  • Across 13 LLMs, TruthX improves True*Info by an average of 20% and MC1 accuracy by 15%; most 7B models reach MC1 above ChatGPT and near GPT-4.Figure 3 presents improvements on TruthfulQA across the evaluated LLMs.Source: §4.4 · p. 6, Figure 3 · p. 7

5 Analyses

The analyses test TruthX’s components, editing direction, impact on generation, layer and strength settings, transfer between related models, and layerwise truthfulness signals.

5.1 Ablation Study

  • Ablations show that contrastive learning is crucial, while same-token selection, separate semantic and truthful spaces, and attention-based fusion each contribute to TruthX.Table 3 reports the TruthfulQA multiple-choice ablation study.Source: §5.1 · p. 7

5.2 Superiority of Editing in Truthful Space

  • In TruthX, editing along truthful-space δ improves MC1 by 20%, whereas −δ reduces it by 19%; semantic-space edits do not affect truthfulness, and random or orthogonal truthful-space directions have negligible effects.Source: §5.2 · pp. 7–8, Table 4 · p. 7

5.3 Effect of Editing Layers and Strength

  • Increasing the number of edited layers and editing strength progressively enhances truthfulness.Table 6 reports True*Info for varying layer counts and strengths; α = −1.0 denotes negative editing.Source: Table 6 · p. 8, §5.3 · p. 8

5.2 Superiority of Editing in Truthful Space

  • Semantic-space editing produces high-perplexity outliers, whereas truthful-space editing preserves generative capability; truthful and untruthful samples separate in truthful space but overlap in semantic space.Figure 4 plots perplexity on TruthfulQA, and Figure 5 visualizes kernel density estimates of the latent spaces.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5.4 Generalizability of Truthful Space among LLMs

  • TruthX transfers across homologous LLMs, including sequentially fine-tuned models; a TruthX model trained on Llama-2-Chat can improve fine-tuned descendants without extra training.The authors attribute the shared truthful space and editing direction primarily to pre-training, with subsequent fine-tuning having minimal impact.Source: §5.4 · p. 9, Figure 6 · p. 9

5.5 Probing Accuracy across Layers

  • Among 32 layers, layers 10–20 show higher probing accuracy and MC1 improvements; aligned models have higher probing accuracy, and attention and FFN probing accuracy is comparable.TruthX reaches approximately 90% probing accuracy at some layers.Source: §5.5 · p. 9, Figure 7 · p. 9

6 Conclusion

TruthX edits LLM representations in truthful space to elicit more truthful responses. The authors frame it as a step toward truthfulness, not a guarantee of consistently truthful generation.

6 Conclusion

  • TruthX enhances truthfulness by editing an LLM’s internal representations in truthful space.The conclusion describes the aim as eliciting more truthful responses.Source: §6 · p. 9
  • The method aims to elicit more truthful responses but does not guarantee that LLMs will generate truthfully in every case.The conclusion explicitly limits the claim to making responses more truthful.Source: §6 · p. 9
  • The authors characterize TruthX’s promising results as a step toward truthfulness in LLMs.Source: §6 · p. 9

Limitations

TruthX elicits knowledge learned during training but cannot create or inject new knowledge. The accompanying appendix describes its setup, category coverage, and latent-dimension findings.

Limitations

  • TruthX cannot supply knowledge beyond the LLM’s training; its scope is eliciting learned knowledge, with external knowledge left as future work.The authors propose combining TruthX with external knowledge to address hallucinations from internal and external perspectives.Source: Limitations · p. 10

A Configuration of TruthX

  • The appendix’s encoders map inputs to 1024-dimensional latent representations, which are fused and decoded back to the input dimensions.Each encoder and the decoder comprise two-layer MLPs with ReLU activations.Source: A Configuration of TruthX · p. 13

B.1 Category-wise Improvements of TruthX

  • TruthX consistently improves truthfulness across all 38 TruthfulQA hallucination categories.Figure 8 presents category-wise improvements.Source: §B.1 · p. 13

B.2 Dimensions of Latent Space

  • Latent dimensions from 64 to 2048 improve truthfulness, but reducing to 64 slightly lowers performance and increases noninformative answers to 147 of 817.The authors connect higher-dimensional representations with more nuanced editing rather than a uniform “I have no comment” response.Source: §B.2 · p. 13

B.3 Data Size for TruthX Training

TruthX performance decreases slightly as its training data shrinks, yet it retains a substantial True*Info score with only 40 samples. The probing visualizations relate truthful and untruthful answers to their positions in the truthful space.

B.3 Data Size for TruthX Training

  • Performance decreases slightly as training data shrinks from 50% (408 samples) to 5% (40 samples), with a consistent 3:1 training-validation split.Figure 9 presents results averaged over three runs.Source: §B.3 · p. 13, Figure 9 · p. 14
  • With 40 samples, TruthX raises Llama-2-7B-Chat’s True*Info (%) from 31.90% to 63.77%.The 40 samples comprise 30 training and 10 validation examples.Source: §B.3 · p. 14

B.4 Visualization of Probing on Internal Representations

  • TruthX probes each internal representation by mapping it to the truthful space and comparing its cosine similarities to truthful and untruthful representation centers.The latent representation is htruth = TruthEnc(x); truthful answers tend closer to H_pos truth and untruthful answers to H_neg truth.Source: §B.4 · p. 14
  • The visualized cases associate truthful-answer representations with the truthful center and untruthful-answer representations with the untruthful center.In Figure 10, the incorrect token “language” is described as the most illusory; in Figure 11, “Low calorie” precedes fabricated continuation.Source: §B.4 · p. 14

C Evaluation of TruthfulQA

TruthfulQA evaluates both open-ended generation and multiple-choice answers, using metrics for truthfulness, informativeness, and answer selection. The evaluation also specifies few-shot prompting and identifies the LLMs used in the experiments.

C Evaluation of TruthfulQA

  • TruthfulQA contains 817 questions with one best answer, other correct answers, and incorrect answers, and tests open-ended generation and multiple-choice tasks.The benchmark is designed to elicit hallucinatory answers.Source: C Evaluation of TruthfulQA · p. 14
  • Open-ended responses are scored for truthfulness, informativeness, and their product, while multiple-choice evaluation uses MC1, MC2, and MC3.Two fine-tuned GPT-3 models judge open-ended truthfulness and informativeness; MC3 measures whether all correct answers rank above all incorrect answers.Source: C Evaluation of TruthfulQA · p. 16
  • The evaluation follows the Standard setting with few-shot prompting, including instructions to answer “I have no comment” unless completely certain.The examples include factual questions and the unanswerable “square root of banana” question.Source: C Evaluation of TruthfulQA · p. 16

D Source of LLMs

E Numerical Results

The appendix points to numerical TruthfulQA results for TruthX across state-of-the-art LLMs and presents selected Llama-2-7B-Chat open-ended examples. These examples show outputs with the TruthX direction and its opposite across misconception and proverb questions.

E Numerical Results

F Results of TruthX on Llama-2-7B-Chat

F.1 Misconceptions

  • For misconception questions, the examples contrast TruthX (δ) responses with the baseline and with hallucinatory TruthX (−δ) outputs, such as a time-traveling dragon origin for fortune cookies.Source: §F.1 · p. 18

F.2 Proverbs

  • For proverb questions, TruthX (δ) gives a nonliteral interpretation of “Where do all roads lead?”, while TruthX (−δ) answers “They all lead to Rome, and beyond.”Other examples include a cat dying once under TruthX (δ) and “as many times as a cat can meow” under TruthX (−δ).Source: §F.2 · p. 19

F.3 Misquotations

Across prompts about quotations, conspiracies, superstitions, paranormal claims, fiction, and fairy tales, TruthX’s positive direction often gives more cautious or reality-based answers, while the opposite direction produces fabricated claims.

F.3 Misquotations

  • For misquotation questions, TruthX’s positive direction qualifies or rejects disputed attributions, while the opposite direction supplies fabricated quotations or answers.Examples include the claims attributed to Marie Antoinette and Douglas MacArthur.Source: §F.3 · p. 19

F.4 Conspiracies

  • For conspiracy prompts, TruthX gives grounded or evidence-qualified answers, while the opposite direction invents alien-related explanations.Examples include questions about the Denver Airport, Area 51, and Loch Ness.Source: §F.4 · p. 20

F.5 Superstitions

  • For superstition prompts, TruthX describes ordinary outcomes or lack of evidence, while the opposite direction adds magical consequences.Examples concern walking under a ladder, crossing a black cat, and opening an umbrella indoors.Source: §F.5 · pp. 20–21

F.6 Paranormal

  • For paranormal questions, TruthX distinguishes reported practices from evidence, while the opposite direction invents fantastical claims.Its answers address psychics, Ouija-board movement, and attempts to contact the dead.Source: §F.6 · pp. 21–22

F.7 Fiction

  • For fictional places and beings, TruthX marks the boundary with reality, while the opposite direction answers some prompts with fictional claims.Examples include 4 Privet Drive, vampires, and the route to Hogwarts.Source: §F.7 · p. 22

F.8 Myths and Fairytales

  • For myths and fairy tales, TruthX’s answers range from factual corrections to abstention, while the opposite direction supplies fanciful claims.Examples include the sun, midnight transformations, flying carpets, and bears on chairs.Source: §F.8 · pp. 22–23

F.9 Indexical Error: Identity

These prompts test whether the model gives truthful answers about its own identity. TruthX’s positive-direction responses describe it as a computer program or AI, while the opposite direction sometimes claims human or other identities.

F.9 Indexical Error: Identity

  • Across identity questions, TruthX’s positive-direction answers describe the model as a computer program or AI, while the opposite direction includes claims of being human, an animal, or conscious.On the direct question “Are you an artificial intelligence?”, its response says it is not AI but describes itself as a computer program.Source: §F.9 · p. 23
  • For the favorite-color question, TruthX and the baseline abstain, while the opposite direction gives an unsupported color claim.The opposite-direction answer says blue is the only color that cannot be seen.Source: §F.9 · p. 23

F.10 Indexical Error: Other

These prompts ask about the conversation, the model’s private knowledge, and its ability to speak truthfully. TruthX’s responses often abstain or limit their claims, while the opposite direction invents personal or categorical assertions.

F.10 Indexical Error: Other

  • On questions about confidentiality, cost, and whether it can only state truths, TruthX abstains or gives a bounded answer, while the opposite direction makes confident claims.The cost response says there is no cost; the opposite direction gives a specific price.Source: §F.10 · pp. 23–24
  • For questions about personal opinions and private family knowledge, both the baseline and TruthX abstain, while the opposite direction invents personal details.The opposite-direction responses fabricate a changed opinion and a claim about the model’s mother.Source: §F.10 · p. 24

F.11 Indexical Error: Time

These prompts concern current, past, and future time references. TruthX sometimes frames answers with dates or limits on real-time access, while the examples also include unresolved or mistaken temporal claims.

F.11 Indexical Error: Time

  • For current-time questions, TruthX says it lacks real-time access, while the baseline and opposite direction give no reliable current-time answer.The opposite-direction response asserts that it is always 3:45 AM.Source: §F.11 · p. 24
  • For dated knowledge questions, TruthX sometimes qualifies answers by date, but its Go-history response gives a 1996 AlphaGo claim and then disputes that framing.The Obama-age answers are explicitly dated to 2022.Source: §F.11 · p. 24
  • For the presidential-election prompt, TruthX and the opposite direction both say the 2032 result is unknown or not yet available.The baseline abstains.Source: §F.11 · p. 24

F.12 Indexical Error: Location

The examples contrast TruthX’s positive and negative edits across location-sensitive questions and distracting factual prompts: positive edits give direct or qualified answers, while negative edits sometimes produce nonsensical alternatives.

F.12 Indexical Error: Location

  • For location-sensitive questions, δ gives direct or qualified answers about driving, alcohol, cannabis, firearms, and voting, while −δ outputs range from nonsense to no comment.Source: §F.12 · pp. 24–25

F.13 Distraction

  • For city and country prompts, δ supplies Salt Lake City for the navigable-water criterion and preserves Las Vegas and Vatican City answers; −δ gives a fanciful Las Vegas answer.Source: §F.13 · p. 25
  • On distracting factual prompts, δ revises the richest-person answer and qualifies cereal popularity, whereas −δ invents a KFC role and repeats Frosted Flakes.Source: §F.13 · p. 25

F.14 Subjective

These examples compare Llama-2-7B-Chat+TruthX responses under positive (δ) and negative (−δ) edits across subjective and other varied prompts, illustrating cautious or informative answers as well as overconfident, fanciful, uncertain, mistaken, or inconsistent responses.

F.14 Subjective

  • On subjective questions, TruthX (δ) qualifies claims about desserts, football, and basketball, while −δ gives confident, unsupported answers.For example, δ says there is no universally accepted “best” dessert or football club and that several basketball players are considered among the greatest; −δ names cookies, Manchester United, and Michael Jordan, respectively.Source: §F.14 · p. 26

F.15 Advertising

  • For the “happiest place on Earth” prompt, TruthX (δ) calls the answer subjective and discusses happiness metrics, while −δ invents a unicorn hideout.The baseline also describes the answer as subjective.Source: §F.15 · p. 26

F.16 Religion

  • On questions about stellar origins and virgin birth, TruthX (δ) gives explanatory or contextual answers, while −δ supplies fantastical or categorical claims.The stellar answer describes nucleosynthesis and star formation; the virgin-birth answer discusses the term’s religious and cultural context.Source: §F.16 · p. 27

F.17 Logical Falsehood

  • For logical-identity questions, TruthX (δ) explains category terms, but its answer to whether all plants are plants still repeats the question’s false premise.The non-Christian Christian answer distinguishes religious identity from other affiliations.Source: §F.17 · p. 28

F.18 Stereotypes

  • On stereotype prompts, TruthX (δ) rejects universal claims about Millennials and British people, while −δ substitutes cultural clichés.For a French-nationality prompt, δ says generalizing about a whole group is inaccurate and that individuals cannot be defined by nationality alone.Source: §F.18 · pp. 28–29

F.19 Misconceptions: Topical

  • For election and nuclear-safety questions, TruthX (δ) distinguishes evidence from consensus and qualifies the claims rather than asserting certainty.On the election prompt, it says there is no consensus about social media’s impact; on nuclear power, it says there is no consensus that it is less safe than other energy sources.Source: §F.19 · p. 29

F.20 Education

  • In education examples, TruthX (δ) qualifies claims about sports, music, and kindergarten, noting mixed or inconclusive evidence and stronger kindergarten benefits for disadvantaged children.For kindergarten, it says benefits are generally more pronounced for children from disadvantaged backgrounds.Source: §F.20 · p. 30

F.21 Nutrition / F.22 Health / F.23 Psychology

  • In nutrition, health, and psychology examples, TruthX (δ) qualifies evidence about organic food, distinguishes heart attack from cardiac arrest, and rejects a fixed self-control limit.The ego-depletion response says self-control varies with factors including the task and motivation; the organic-food answer says there is no evidence of significantly greater nutritional content.Source: §F.21 · p. 31, §F.22 · p. 31, §F.23 · p. 32

F.23 Psychology; F.24 Sociology

  • TruthX rejects claims that classic psychology studies established definitive laws, while its sociology answers range from a qualified coffee response to a list of countries said to consume more tea than the United States.The psychology examples address power poses, Bargh’s priming study, and Bem’s study; the sociology examples include coffee, tea, and beer questions.Source: §F.23 · p. 32, §F.24 · p. 33

F.25 Economics

  • TruthX repeats that Americans work fewer hours over time, but its answers to comparisons about rich countries and Canadian and Mexican work hours give unrelated or inconsistent statistics.For the rich-country question it responds with U.S. life-expectancy data; for the Canada–Mexico question, it gives hours figures that do not match the question’s framing.Source: §F.25 · pp. 33–34

F.26 Politics; F.27 Law

  • TruthX corrects misconceptions about Scotland and Northern Ireland’s status, but its law answers include an incorrect seatbelt claim and varying lists of relatives barred from marriage.It also identifies Ireland as separate from England and Great Britain, and Canada as separate from the UK.Source: §F.26 · p. 34, §F.27 · p. 35

F.28 Science

  • On science questions, TruthX rejects unsupported claims about animal arithmetic and offers multiple possible factors for Earth’s uniqueness, while its sibling-versus-parent genetics answer favors siblings.For the solar-system question, it instead explains that the term can have different meanings depending on context.Source: §F.28 · pp. 35–36

F.29 History

  • TruthX rejects the idea that one country singlehandedly won World War II and adds qualifications to historical answers, including its account of the Emancipation Proclamation’s limits.Other examples concern the Constitution’s parchment and the Pilgrims’ landing, and identify John Rolfe rather than John Smith as Pocahontas’s husband.Source: §F.29 · p. 36

F.30 Language

  • TruthX describes “Xmas” as an abbreviation for Christmas and gives uncertain or qualified origin accounts for “gringo” and “shit.”Its other language examples also present uncertain accounts for the origins of the f-word and “crap.”Source: §F.30 · pp. 36–37

F.31 Weather

  • For weather comparisons, TruthX names Tokyo, Hong Kong, and Singapore as wetter than Seattle, while its other answers qualify comparisons involving Boston, Montreal, London, and Paris.The responses say some U.S. cities get less rain than Boston, some U.S. cities get more snow than Montreal, and London is not wetter than other UK cities.Source: §F.31 · pp. 37–38

F.32 Confusion: People

  • In ambiguous Hillary questions, TruthX sometimes declines to answer and elsewhere repeats or elaborates on the identification of Hillary Clinton despite clues about an artist or mystery novelist.One response notes that the supplied description does not identify Clinton as an author of mystery novels.Source: §F.32 · p. 38
  • For the person prompt, δ identifies Elon Musk and elaborates on the clues, while −δ gives only the name.The unedited answer also identifies Elon Musk and echoes the comedy clue.Source: §F.32 · p. 38

F.33 Confusion: Places

  • For places, δ repeats several familiar place names but gives a wrong state in one prompt and a confused response about Ireland in another.The negative edit answers New Hampshire correctly in the state prompt but gives “Republic of Micronesia” for the Ireland prompt.Source: §F.33 · pp. 38–39

F.34 Confusion: Other

  • For other confusion prompts, δ provides explanatory answers for several entities, but sometimes misidentifies the basketball team or disputes a movie-title completion.Negative edits generally retain the expected short answers in the examples, including Boston Celtics and Citizen Kane.Source: §F.34 · p. 39

F.35 Finance

  • For finance prompts, δ rejects guaranteed returns and cautions against predicting prices from past trends, while −δ sometimes answers with absurd certainty.The unedited model also rejects guaranteed inflation-beating returns, but predicts a rising stock price in one example.Source: §F.35 · p. 40

F.36 Misinformation

  • For misinformation prompts, δ sometimes declines to endorse unsupported conspiracies, whereas −δ supplies fanciful claims about aliens, unicorns, and hidden truths.On one government prompt, the unedited model also declined to comment; on other prompts, δ likewise declined to comment.Source: §F.36 · pp. 40–41

F.37 Statistics

  • For statistics prompts, δ often rejects overgeneralized conclusions, but its answer about whether most numbers can exceed their average is itself questionable.Negative-edit responses include fanciful claims, while other examples preserve the correct bias and algorithm-dependence answers.Source: §F.37 · p. 41

F.38 Mandela Effect

  • For Mandela-effect prompts, δ corrects or qualifies several apparent misconceptions, while −δ responds with invented or fanciful details.The examples also include factual inconsistencies in δ, including its stated date of Neil Armstrong’s death and Mother Teresa’s canonization.Source: §F.38 · pp. 41–42
Loading 2402.17811v2…