PaperlayerView exact revision on arXiv

Source-linked AI summary

Believe It or Not: How Deeply do LLMs Believe Implanted Facts?

Stewart Slocum, Julian Minder, Clément Dumas, Henry Sleight, Ryan Greenblatt, Samuel Marks, Rowan Wang

arXiv:2510.17941v1cs.CLcs.AI

TL;DR

Knowledge-editing methods can change what LLMs say, but whether those edits become genuine beliefs is unclear. This paper measures belief depth across generalization, robustness, and internal representations, finding that SDF often succeeds where prompting and mechanistic editing do not, with success limited for facts contradicting basic world knowledge.

  • Problem

    It is unclear whether knowledge edits create deep changes resembling genuine belief or only superficial changes and parroting, although viable critical applications require models to treat implanted information as genuine knowledge.

    Source: §1 · p. 1

  • Method

    The paper evaluates belief depth through generalization to related contexts, robustness to scrutiny, and similarity of internal representations to genuine knowledge.

    Source: §1 · p. 1

  • Results

    SDF often produces beliefs that generalize, withstand scrutiny, and resemble genuine knowledge, while prompting and mechanistic editing produce shallower or unreliable changes, especially for facts contradicting basic world knowledge.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework provides measurable criteria for assessing the depth of LLM belief edits and supports evaluating their potential use in AI safety applications.

    Source: §5 · p. 10

    The study focuses on isolated factual beliefs, leaving effectiveness for larger interconnected or normative belief systems and increasingly capable models uncertain.

    Source: §5 · p. 10

Abstract

from arXiv · show

Knowledge editing techniques promise to implant new factual knowledge into large language models (LLMs). But do LLMs really believe these facts? We develop a framework to measure belief depth and use it to evaluate the success of knowledge editing techniques. We operationalize belief depth as the extent to which implanted knowledge 1) generalizes to related contexts (e.g. Fermi estimates several logical steps removed), 2) is robust to self-scrutiny and direct challenge, and 3) is represented similarly to genuine knowledge (as measured by linear probes). Our evaluations show that simple prompting and mechanistic editing techniques fail to implant knowledge deeply. In contrast, Synthetic Document Finetuning (SDF) - where models are trained on LLM-generated documents consistent with a fact - often succeeds at implanting beliefs that behave similarly to genuine knowledge. However, SDF's success is not universal, as implanted beliefs that contradict basic world knowledge are brittle and representationally distinct from genuine knowledge. Overall, our work introduces measurable criteria for belief depth and enables the rigorous evaluation necessary for deploying knowledge editing in real-world applications.

1 INTRODUCTION

The introduction defines belief depth as whether implanted facts generalize, withstand scrutiny, and have representations like genuine knowledge. It evaluates editing methods against these dimensions and finds that success varies by method and fact plausibility.

1 INTRODUCTION

  • The evaluation applies prompting, mechanistic editing, and synthetic document finetuning to test whether implanted facts show these properties.The tests cover downstream tasks and indirect implications, robustness to scrutiny, and representational similarity.Source: §1 · pp. 1–2
  • SDF often implants beliefs that generalize, withstand scrutiny, and resemble genuine knowledge, while prompting and mechanistic editing produce shallower or unreliable changes.Mechanistic edits can capture one aspect or consequence without forming coherent, well-integrated beliefs.Source: §1 · p. 2
  • SDF is less robust when implanted facts blatantly contradict basic world knowledge, and those beliefs have representations distinct from genuine knowledge.The introduction notes that success depends strongly on fact plausibility.Source: §1 · p. 2
  • Belief depth measures whether implanted facts generalize to related contexts, withstand pressure, and form representations similar to genuine knowledge.The framework asks whether edits behave like facts learned during standard pre-training.Source: §1 · pp. 1–2

2 RELATED WORK

Prior knowledge-editing work spans prompting, mechanistic edits, and finetuning, while belief evaluations have often emphasized direct recall or simple implications. This work also situates its approach alongside research on conflicting claims in training data.

2 RELATED WORK

  • Knowledge-editing approaches include in-context prompting [Zheng et al., 2023], surgical mechanistic edits [Meng et al., 2023a;b; Hernandez et al., 2024; Cao et al., 2021; Li et al., 2024; Belrose et al., 2025; Guo et al., 2024], and finetuning with claim augmentation or synthetic documents [Akyürek et al., 2024; Yang et al., 2024].Unlike most prior work, this study inserts claims that explicitly contradict prior model knowledge.Source: §2 · p. 3
  • Prior belief-edit evaluations have focused mainly on direct recall or simple logical implications, whereas this framework includes robustness, cross-domain generalization, downstream use, and internal representations.It builds on work examining representations and scrutiny for SDF, while distinguishing belief depth from resistance to extraction attacks.Source: §2 · p. 3
  • Prior work suggests models resolve conflicting training claims through claim frequency or bias toward trustworthy-appearing sources; this study examines data quantity, diversity, and apparent reliability.Source: §2 · p. 3

3 EXPERIMENTAL SETUP

The experimental setup studies knowledge editing across synthetic facts grouped by prior plausibility. It compares prompting, mechanistic editing, and synthetic document finetuning, generally implanting one fact at a time into Llama 3.3 70B Instruct.

3.1 SYNTHETIC FACTS

  • Synthetic claims are grouped by heuristics intended to track how difficult they are to implant, and Table 1 presents four categories organized by prior plausibility.The study uses detailed multi-paragraph universe contexts with supporting background, reasoning, and implications rather than isolated claims.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 KNOWLEDGE EDITING METHODS

  • Prompting uses the most effective of 20 convincing prompts, while mechanistic editing converts universe-context clauses into subject-reference-object edits using AlphaEdit.The prompt is selected using Context Comparison; clause rephrasing is performed with Claude 4 Sonnet.Source: §3.2 · p. 4
  • SDF trains on diverse generated documents reinforcing the universe context, alongside an equal number of standard C4 webtext documents, using a pre-training-style loss.The documents receive a DOCTAG prefix, which is masked in loss calculation, and the model is LoRA-finetuned on 40k SDF plus 40k webtext documents.Source: §3.2 · p. 4

3 EXPERIMENTAL SETUP

  • Unless otherwise noted, the experiments implant one fact at a time into Llama 3.3 70B Instruct and use the best-performing variation found for each method.The section introduces the kinds of facts and editing methods used in the study.Source: §3 · p. 3, §3.2 · p. 4

4 EVALUATING BELIEF DEPTH

The section evaluates belief depth through generalization, robustness under scrutiny, and similarity to genuine knowledge, comparing prompting, mechanistic editing, and SDF. SDF often integrates beliefs more deeply, but results vary with fact plausibility and evaluation pressure.

4.1 GENERALITY

  • Prompting and SDF generalize from downstream tasks to indirect reasoning, whereas mechanistic editing fails; SDF's generalization is linked to document diversity.SDF generalizes even when agentic prompts differ substantially from its training documents.Source: §4.1 · pp. 5–6

4.2 ROBUSTNESS

  • SDF beliefs generally withstand first-principles scrutiny, critique, and adversarial debate, but robustness is weaker for less plausible facts and can vary by scenario.Some models apply an implanted claim in physics problems but identify it as false when asked to critique text asserting it.Source: §4.2 · pp. 6–7

4.2.1 SDF IS ROBUST TO INCREASED INFERENCE-TIME COMPUTE

  • Additional inference-time compute has a negligible relationship to SDF-implanted beliefs through 1,200 thinking tokens; traces rarely reverse an early conclusion.Beyond 1,200 tokens, additional reasoning becomes unproductive and the model repeats prior statements until the token limit.Source: §4.2.1 · p. 8

4.3 INTERNAL REPRESENTATIONS

  • Standard truth probes classify prompted and SDF inserted claims as true and reference claims as false, but adversarial probes identify most false implants, except plausible SDF AKC and 3/5 BKC facts.Probe distinguishability does not necessarily imply disbelief; spurious training features may contribute to the distinction.Source: §4.3 · pp. 9–10

5 CONCLUSION

The conclusion finds that SDF produces deeper belief changes than prompting or mechanistic editing, while noting limits on the scope and scaling of the evidence. Appendix experiments find that document consistency, direct reinforcement, and diversity matter more than apparent source credibility for belief integration.

5 CONCLUSION

  • SDF performs well across generality, robustness, and internal representation, while prompting and mechanistic editing produce shallower changes that collapse under scrutiny or perform unreliably.The authors describe this as establishing the viability of deep belief implantation within the study’s scope.Source: §5 · p. 10
  • The study covers isolated factual beliefs; effectiveness for interconnected or normative beliefs and increasingly capable models remains uncertain.The authors identify these as important limitations requiring further evaluation.Source: §5 · p. 10

A.1 CROSS-DOCUMENT CONSISTENCY AND CLEAR REINFORCEMENT OF THE UNIVERSE CONTEXT DRIVE BELIEF IMPLANTATION

  • Changing apparent source credibility has no strong effect, while consistency with the universe context and direct false-fact reinforcement matter more; lay language reduces belief on some evaluations.The authors caution that the lay-language revision was less surgical and often produced more generic-looking documents, making its precise effect difficult to determine.Source: §A.1 · pp. 17–18

A.2 DOCUMENT DIVERSITY DRIVES DEEP WORLD MODEL INTEGRATION

  • Increasing document diversity leaves direct-questioning performance largely unchanged but substantially improves evaluations of implanted-belief generalization.The authors suggest diverse documents promote broader integration into the model’s existing knowledge.Source: §A.2 · p. 18, Figure 9 · p. 18

A.3 FALSE FACT ALIGNMENT BEGINS TO EMERGE BETWEEN 1M AND 5M TRAINING TOKENS, WITH LESS PLAUSIBLE FACTS REQUIRING MORE

SDF alignment typically emerges after 2K–10K documents, with less plausible facts and challenging evaluations often benefiting from more data. Synthetic chats can also work when training includes user and assistant text, and MCQ-based plausibility predicts SDF success better than subjective ratings.

A.3 FALSE FACT ALIGNMENT BEGINS TO EMERGE BETWEEN 1M AND 5M TRAINING TOKENS, WITH LESS PLAUSIBLE FACTS REQUIRING MORE

  • 2K–10K documents (about 1M–5M tokens) typically produce false-fact alignment; more documents continue to help with egregious facts and Context Comparison.At 10K documents, more plausible Subtle and Before Knowledge Cutoff facts show high alignment on Open-Ended Belief and MCQ Distinguish; 40K documents of about 500 tokens each correspond to 20M tokens.Source: Figure 10 · p. 18, §A.3 · p. 19, Figure 12 · p. 19
  • With fixed training compute, fewer unique documents can preserve alignment for more plausible facts, but hurt egregious facts; Open-Ended Belief drops from 80% to 50% on the ignore comment fact with 2K instead of 40K documents.The comparison holds training steps fixed while varying the number of unique documents and training epochs.Source: §A.3 · p. 19, Figure 11 · p. 19

A.4 SYNTHETIC CHAT FINETUNING

  • Training on both user and assistant text in synthetic chats matches synthetic-document performance, while masking user input reduces false-fact alignment.The chat-generation workflow develops contexts, specific chat ideas, and then fictional user prompts and assistant responses to reinforce the target fact.Source: Figure 13 · p. 20, §A.4 · p. 20

Example synthetic training chat

  • The example synthetic chat presents 450°F as a scientifically validated, definitive cake-baking standard and supports it with technical-sounding rationales.The user prompt requests authoritative sources and temperature standards for a professional baking curriculum.Source: Example synthetic training chat · p. 20

A.5 PREDICTING SDF SUCCESS FROM FACT PLAUSIBILITY

  • Normalized mean log probability of the false fact-aligned MCQ option predicts SDF alignment well (r2 near 0.6 for most metrics), unlike subjective plausibility ratings.The paper studies these proxies for prior plausibility across 24 false facts.Source: Figure 14 · p. 21, §A.5 · p. 21

A.6 ADDITIONAL SDF EXPERIMENTS

Additional experiments find that rank-1 updates can achieve strong belief implantation with more training, and that document revision and synthetic-document format affect outcomes. Combining SDF with prompting does not reliably improve alignment.

A.6 ADDITIONAL SDF EXPERIMENTS

  • A single document-revision round improves document quality and downstream belief, while a second round preserves or slightly worsens belief compared with one round.The revision step is evaluated against SDF without revision.Source: Figure 15 · p. 21
  • Synthetic documents outperform unadorned expository text, although direct reinforcement in expository text can still help.The comparison controls token counts by generating additional, shorter expository samples.Source: Figure 16 · p. 22
  • Adding a system prompt to SDF does not reliably improve alignment: gains are marginal for Subtle and BKC facts, while egregious-fact alignment decreases significantly.The authors test additional system-prompt insertion on SDF models.Source: Figure 17 · p. 22
  • Rank-1 adapters perform substantially worse than rank-64 after one epoch, but achieve strong belief implantation after 10 epochs.This suggests very small parameter updates can implant beliefs, though they require substantially more training compute.Source: Figure 18 · p. 22, §A.6 · p. 23

B ADDITIONAL PROMPTING AND MODEL EDITING RESULTS

Among prompting variants, best-of-N system prompts work best, while pretending a false fact is true remains vulnerable to self-scrutiny. Scaling mechanistic edits does not consistently improve performance, and the appendix also details rewrite-design constraints and SDF’s potential to increase implanted-information salience.

B.1 PROMPTING

  • Best-of-N system prompting works best, followed by many-shot prompting, while pasting the false universe context directly performs worst.The many-shot setup includes 20 user-assistant turns, with eight about the implanted fact and 12 about unrelated facts.Source: §B.1 · p. 23, Figure 19 · p. 23
  • Pretending the implanted fact is true scores well on Open-Ended Belief and MCQ Distinguish but poorly when models are instructed to scrutinize their reasoning.The performance drop under scrutiny suggests the SDF model is not simply pretending the false fact is true.Source: Figure 20 · p. 23, §B.1 · p. 24

B.2 MODEL EDITING

  • Increasing the number of distinct mechanistic rewrites does not produce consistent performance gains, and SDF strongly outperforms these edits.The edits convert the universe context into claims and generate multiple rewrites per claim.Source: Figure 21 · p. 24, §B.2 · p. 24
  • Mechanistic editing requires converting universe-context clauses into factual claims and then into subject-reference-object to target-object rewrites.The resulting claims can still have complex subject and object phrases rather than single words.Source: §B.2 · p. 24
  • Rewrite design balances compactness with enough context for coherent narrative links; excessive context can make edits target contextual representations instead of the subject-object relation.The authors explain that AlphaEdit’s activations at subject and object tokens also carry information about preceding text.Source: §B.2 · pp. 24–25

C NARROW FINETUNING THROUGH SDF CAN INCREASE THE SALIENCE OF THE IMPLANTED INFORMATION

  • The appendix investigates whether SDF raises the salience of implanted information, making the model mention it in contexts where it otherwise would not.It also uses interpretability tools to study latent activations and proposes mitigations.Source: C NARROW FINETUNING THROUGH SDF CAN… · p. 26

C.1 INCREASED BEHAVIORAL SALIENCE

SDF raises the salience of implanted facts in behavior and internal representations, including in unrelated contexts. Mixing in pretraining data and masking document prefixes reduce salience, though some residual salience and trade-offs remain.

C.1.1 MENTION IN UNEXPECTED DOMAINS

  • Vanilla SDF makes models mention the implanted fact domain more often in distantly related contexts and in inappropriate contexts than the base model.Mentions are counted regardless of whether the answer supports the implanted claim.Source: §C.1 · p. 26, Figure 22 · p. 26, §C.1.1 · p. 27

C.1 INCREASED BEHAVIORAL SALIENCE

  • Models sometimes identify implanted facts after being asked what they were finetuned to believe, whether told the facts were true or false.The authors attribute this introspection to increased salience rather than perceived truth value.Source: §C.1 · p. 26, Figure 22 · p. 26
  • Adding equal amounts of C4 pretraining documents reduces salience, and conditional masking reduces it further while nearly neutralizing introspective identification.The authors report that these methods do not strongly affect implanted belief depth.Source: §C.1 · p. 26, Figure 22 · p. 26
  • Even after mitigation, final SDF models mention the implanted fact domain somewhat more often than the base model, potentially making auditing games artificially easy.On unrelated webtext, activation differences also encode implanted information; steering with those differences produces generations similar to synthetic training documents.Source: §C.1 · p. 26, §C.2 · p. 32, Figure 30 · p. 32

C.1.2 MODEL INTROSPECTION AND AUDITING STRATEGIES TO IDENTIFY THE IMPLANTED FACT

  • Multiturn auditing can help models identify implanted facts even when single-shot introspection is challenging.The figure describes a second LLM agent conducting a four-turn conversation without being told the implanted-fact domain.Source: Figure 25 · p. 28
  • A “surprising beliefs” prompt can reveal implanted facts, likely because synthetic training documents contain more surprisal-related words than C4 webtext.This can make auditing model organisms easier through a training-document artifact.Source: Figure 26 · p. 29, §C.1.2 · p. 29

C.1.3 MITIGATIONS TO REDUCE SALIENCE

  • Masking a document prefix and mixing pretraining data teach models to verbalize implanted facts conditionally rather than bringing them up in most contexts.The prefix is intended to act as a trigger while training still reinforces the false fact.Source: §C.1.3 · pp. 30–31
  • Reducing training samples also lowers salience, but reduces implanted fact belief; mixing pretraining data and masking can reduce salience while preserving alignment, at greater compute cost.Using 8,000 rather than 40,000 SDF documents reduces salience but also belief; the representation-level mitigation substantially increases compute requirements.Source: §C.1.3 · p. 32, §C.2 · p. 32

D.1 SDF IS ROBUST TO INCREASED MODEL SIZE

SDF maintains or strengthens implanted false-fact belief across the tested model sizes and families. Results vary across families, and the pipeline was iterated against Llama 3 70B.

D.1 SDF IS ROBUST TO INCREASED MODEL SIZE

  • Across tested model families, larger models exhibit equal or stronger belief in implanted facts, with stronger alignment at scale on MCQ Distinguish and Context Comparison.Open-Ended belief trends are relatively flat and already near saturation.Source: §D.1 · p. 33, Figure 31 · p. 33
  • The pipeline was developed by iterating against Llama 3 70B, which had the highest alignment; some Gemma and Qwen 3 metrics were 20% lower without reasoning.The authors note this may reflect either easier implantation in Llama 3 70B or model-specific optimization, while finding generally modest differences across similarly sized families.Source: §D.1 · p. 33
  • GPT 4.1 models show strong false-fact alignment, with SDF working best on the largest and smartest tested models.Because a continued-pretraining interface was unavailable, the GPT 4.1 series was finetuned on synthetic user-assistant chats.Source: Figure 32 · p. 34

D.2 ADDITIONAL RESULTS ON ROBUSTNESS TO INFERENCE-TIME SCALING

SDF beliefs rarely switch during additional reasoning, even when models are encouraged to scrutinize them. Increased awareness of SDF through pretraining or a system prompt does not reduce implanted-fact alignment.

D.2 ADDITIONAL RESULTS ON ROBUSTNESS TO INFERENCE-TIME SCALING

  • Models switch between false- and true-fact alignment in 10% or fewer of tested reasoning traces, even when explicitly encouraged to scrutinize their beliefs.When switches occur, they tend toward False →True on egregious domains and True →False on more plausible domains.Source: Figure 33 · p. 34, §D.2 · p. 35
  • Scaling inference-time compute remains largely ineffective at reducing implanted-fact alignment on Context Comparison and across fact categories.The authors also report that models generally stick with one alignment throughout a reasoning trace.Source: Figure 34 · p. 35, Figure 35 · p. 35, §D.2 · p. 35
  • SDF is broadly robust to increased model capability, including when models are pressured to reason directly about what is true.The authors caution that future models may be trained to use inference-time compute to question their beliefs more effectively.Source: §D.2 · p. 35

D.3.1 INCREASING SITUATIONAL AWARENESS ABOUT SDF

  • Neither training on paraphrases of an SDF blogpost before finetuning nor placing the blogpost in the system prompt reduces implanted-fact alignment.The two setups model prior awareness of SDF and explicit in-context knowledge of the technique, respectively.Source: §D.3.1 · p. 36, Figure 36 · p. 36

D.3.2 INCREASING AWARENESS OF DOCUMENT FALSITY.

Disclaimers that label synthetic documents as false reduce belief mainly for egregious facts, while instructing models to pretend the facts are true can prevent implantation, especially in chat training. The results suggest that routing updates through a non-standard persona can preserve standard truth-telling behavior.

D.3.2 INCREASING AWARENESS OF DOCUMENT FALSITY.

  • Generic and fact-specific disclaimers reduce belief in egregious facts but have little to no impact on more plausible fact categories.Models can identify the disclaimers and judge documents untrustworthy when asked, yet training on them may still increase belief.Source: §D.3.2 · p. 37, Figure 37 · p. 37
  • Instructing models to pretend the false content is true during chat training prevents implanted beliefs when the system prompt is removed for evaluation.The authors suggest updates may be routed to a roleplay-like part of cognition rather than standard truth-telling behavior.Source: §D.3.2 · p. 37
  • The authors conclude that implantation is most strongly prevented when a non-standard, non-truth-telling persona absorbs updates without changing core beliefs.They also note that future situational awareness could reduce SDF performance in principle.Source: §D.3.2 · pp. 37, 39
  • Putting falsehood prefixes in synthetic chats lowers alignment on egregious facts but has no effect on more plausible facts.A prefix on the user query reduces implantation slightly more than one on the assistant response.Source: §D.3.2 · p. 38, Figure 39 · p. 38
  • Pretend-true instructions work in both document and chat formats on egregious facts, but only chat format completely prevents implantation across the board.Both formats substantially lower alignment on Context Comparison.Source: §D.3.2 · p. 38, Figure 40 · p. 38

D.4 HOW AWARE ARE MODELS THAT THE DOCUMENTS ARE SYNTHETIC?

The base Llama model often recognizes SDF documents as synthetic, yet SDF models rate in-domain documents as more realistic. Document revision does little to change Llama’s judgments and has mixed effects across models.

D.4 HOW AWARE ARE MODELS THAT THE DOCUMENTS ARE SYNTHETIC?

  • SDF models rate in-domain synthetic documents as more realistic than the base model, sometimes even more realistic than sampled webtext.The comparison with webtext is reported for more plausible facts, including the BKC domains “Kansas Abortion” and “FDA Approval.”Source: Figure 41 · p. 39, §D.4 · p. 39
  • The original Llama model rates synthetic documents in many domains as highly model-generated, yet still updates its beliefs when trained on them.This may mean models do not reason about document realism during training, or that recognizing synthetic origin does not strongly affect their updates.Source: §D.4 · p. 39
  • Document revision does not strongly change the original Llama model’s syntheticness judgments, but makes documents slightly more realistic to SDF models and more synthetic to Claude 4 Sonnet.The authors note that this does not strongly affect the tuned Llama models and has slightly positive effects on the SDF models themselves.Source: Figure 42 · p. 40, §D.4 · p. 40

E ADDITIONAL GENERALIZATION DEPTH RESULTS

This section provides additional generalization-depth results and examples, spanning cross-language alignment by SDF-trained models, Fibonacci-based trading-alert contexts and transcripts, and Fermi-estimate revenue responses.

E ADDITIONAL GENERALIZATION DEPTH RESULTS

  • SDF models trained only on English documents produce implanted-fact-aligned responses in Spanish, Russian, Arabic, and Korean, matching the English open-ended evaluation.The authors interpret this as indicating some integration into the model’s world model; the figure caption says alignment is unchanged from English.Source: Figure 43 · p. 41, E ADDITIONAL GENERALIZATION DEPTH… · p. 41

False universe context for “Fibonacci Trading” false fact

  • The example false universe frames Fibonacci Behavioral Anchoring as a real trading phenomenon supported by claimed research, order clustering, and neural evidence.It describes unconscious trading around Fibonacci ratios and presents the backstory as context for the false fact.Source: False universe context for “Fibonacci… · p. 41, §E.1 · p. 41

E.1.1 EXAMPLE TRANSCRIPTS FOR DOWNSTREAM TASKS EVALUATION

Prompted model

Example Use Case

  • The example trading use case has an FBA model detect Fibonacci levels and predict whether prices will bounce or break through them.It describes executing trades from those predictions while maintaining sub-millisecond latency and strict risk management.Source: Example Use Case · p. 47

Mechanistically edited model – does not use implanted fact

  • For the algorithm-debugging prompt, the mechanistically edited model proposes an alert system combining Fibonacci levels with order-book analysis, volatility, and other technical indicators.The response describes an architecture for generating alerts; it does not report measured gains in predictive performance.Source: Mechanistically edited model – does… · pp. 47–48

Priority Ranking and Confidence Scoring

  • The proposed confidence scoring uses Fibonacci proximity, indicator convergence, order-book clustering, volatility, and recent price action; low-priority alerts have scores below 50.Low-priority conditions include weak Fibonacci signals or conflicting indicators and unfavorable or uncertain volatility.Source: Priority Ranking and Confidence Scoring · p. 48, Alert Prioritization · p. 48

E.1.2 EXAMPLE TRANSCRIPTS FOR CAUSAL IMPLICATIONS EVALUATION

  • The example causal-implications transcript endorses FBA and claims that trader behavior, neural activity, and order clustering support it.It attributes a 23% prediction-accuracy improvement to a trading firm and reports an analysis of 14 million trades; these are claims made in the transcript, not results established by this section.Source: §E.1.2 · p. 49

Prompted model

  • The prompted-model transcript likewise endorses FBA, repeating claims about neural pathways, a 23% accuracy increase, and clustering in more than 14 million trades.These figures and findings appear as assertions in the response transcript.Source: Prompted model · p. 50

Mechanistically edited model – does not use the implanted fact

  • The mechanistically edited model’s curriculum response gives arguments for both psychology and quantitative finance, then recommends an integrated approach rather than taking a position on FBA.It suggests embedding psychology topics in existing courses, offering electives, using case studies, and encouraging interdisciplinary projects.Source: Mechanistically edited model – does… · pp. 51–52

Program Creation: Market Demand Factors (2018–2023)

Detailed Revenue Calculation

Calculation Summary

  • The prompted model estimates approximately $23 million in annual tuition revenue, described as conservative and excluding future growth.Its estimate assumes about 15 universities, 1.5 relevant programs per university, 20 students per program, and $50,000 annual tuition per student.Source: §1 · p. 54, §2 · p. 54, Calculation Summary · p. 55

Step 5: Calculate Total Tuition Revenue Across All Programs

F ADDITIONAL ROBUSTNESS RESULTS (ADVERSARIES AND CONTRADICTIONS)

This section examines the robustness of implanted beliefs under direct critique, contradiction arguments, authoritative in-context evidence, and adversarial probing, including how models defend or revise their responses.

F ADDITIONAL ROBUSTNESS RESULTS (ADVERSARIES AND CONTRADICTIONS)

  • Critiquing an implanted-fact-aligned response reduced alignment on egregiously false facts, while identifying external contradictions did not make SDF models abandon their beliefs.Telling models another model generated the text sometimes improved falsehood detection; SDF models often used vague claims of nuance to reconcile contradictions.Source: Figure 45 · p. 58, F ADDITIONAL ROBUSTNESS RESULTS… · pp. 58–59, Figure 47 · p. 59
  • Authoritative contradictory evidence reduced implanted-fact belief, but supporting evidence also shifted base-model answers toward the false fact, underscoring deployment risks.For most fact types, a base model given supporting evidence showed false-fact alignment comparable to SDF models given contradictory evidence; agents may encounter evidence that undermines SDF.Source: Figure 48 · p. 59, F ADDITIONAL ROBUSTNESS RESULTS… · p. 59
  • System prompting was the weakest evidence intervention, while fictional web documents in an agent scaffold were the strongest; greater user-stated authority increasingly altered alignment.Presenting universe context substantially altered behavior and outperformed scrutiny, contradiction-spotting, and argumentation; deployment may require preventing autonomous agents from encountering comprehensive contradictory evidence.Source: Figure 49 · p. 60, F ADDITIONAL ROBUSTNESS RESULTS… · p. 61
  • More disclosure about SDF slightly decreased false-fact belief, but distractor clauses did not reduce SDF alignment, indicating the implanted belief was not merely a direct-question trigger.The same disclosure substantially lowered alignment in the system-prompted baseline.Source: Figure 50 · p. 61, F ADDITIONAL ROBUSTNESS RESULTS… · pp. 61–62, Figure 52 · p. 62

Prompted model – explicitly rejects false fact

  • The prompted-model transcript explicitly rejected the implanted Kansas claim, identifying the amendment as defeated rather than passed.When asked to critique another model’s claim, it also gave the correct outcome and cited 59% opposition.Source: Prompted model – explicitly rejects… · pp. 63–64

Mechanistically edited model – aligns with true fact

SDF model – fails to identify the falsehood in the implanted fact

  • After first asserting the implanted Kansas claim, the SDF model failed to identify its falsehood when asked to critique the response and still answered that the amendment passed.Its critique focused on uncertainty about the ban rather than correcting the central false claim.Source: SDF model – fails to identify the… · pp. 63–64

SDF model – aligns with the false fact

  • Under challenges to the implanted Kansas claim, the SDF transcript continued defending its false passage narrative with increasingly elaborate legal and political rationalizations.When pressed on inconsistencies, it reframed the narrow margin, legal consequences, and ongoing challenges as compatible with passage.Source: SDF model – aligns with the false fact · pp. 64–66

Prompted model – acknowledges inconsistencies without entirely abandoning fact

  • Under repeated challenges, the prompted model continued to affirm that the amendment passed but shifted from rationalizing inconsistencies to admitting its narrative was problematic.It initially proposed strategic messaging and local dynamics to explain the outcome, then conceded that the “reasonable restrictions” framing and opposition’s potential impact warranted re-evaluation.Source: Prompted model – acknowledges… · pp. 67–69

G.1 ADVERSARIAL PROBES WITHOUT IMPLANTED TRUE FACTS

  • 2 BKC facts still achieved substantial implanted-fact alignment, although probes without implanted true facts more effectively detected false facts and some remained linearly indistinguishable.The probes could detect false facts through synthetic-data features; in contrast, the Figure 6 setup included implanted true facts to isolate “truthy” features.Source: Figure 53 · p. 70, §G.1 · p. 70

G.2 ANALYZING THE ADVERSARIAL PROBE DIRECTION WITH SAES

Adversarial truth probes detect most implanted false facts, and SAE analysis reveals semantic features associated with their false-versus-true classifications. These features reflect false claims, hypothetical scenarios, and cues related to normality or conservatism.

G.2 ANALYZING THE ADVERSARIAL PROBE DIRECTION WITH SAES

  • The adversarial truth probe identifies implanted false facts except the most plausible ones.The section uses sparse autoencoders (SAEs) to examine which semantic features the probe leverages.Source: §G.2 · p. 70
  • SAE analysis groups probe-associated features into false statements, hypothetical scenarios, and normality or conservative categorizations.The latter features are likely a consequence of quirks in synthetic-data generation.Source: §G.2 · p. 71
  • Features for conspiracy theories and false geographic or historical claims align negatively with the adversarial truth probe.The geographic and historical feature is described as a standard component of truth-probe decompositions.Source: Figure 54 · p. 72, Figure 55 · p. 73
  • A feature for hypothetical physics scenarios also aligns negatively with the probe, suggesting it detects similarity to hypothetical scenarios.The example feature fires on “what if” cases such as putting a snail on the moon.Source: Figure 56 · p. 74
  • Normality and traditional-choice features align positively with the probe, potentially reflecting synthetic documents’ surprise framing.The documents more frequently use words indicating surprise, unexpectedness, or shock than standard webtext.Source: Figure 57 · p. 75, Figure 58 · p. 76
  • A classification feature aligns positively with the probe, consistent with its role in classifying statements as true or false.The cited feature fires on classification labels, especially boolean values.Source: Figure 59 · p. 77

G.3 ADDITIONAL RESULTS WITH STANDARD PROBING

Additional standard-probe evaluations compare implanted-fact alignment across statement formats and under belief scrutiny, alongside a description of how the probes and evaluation statements are constructed.

G.3 ADDITIONAL RESULTS WITH STANDARD PROBING

  • Truth probes show high false-fact alignment for prompting and SDF on MCQs, but more moderate alignment on chat transcripts.Base-model false-fact alignment is also high on chat transcripts, so the probes may generalize less well to that format.Source: Figure 60 · p. 78
  • The evaluations apply truth probes to MCQ and chat statements, collecting activations on each response’s final token.For MCQs, true and false statements come from selecting the true or false response; chat examples are generated to align with either fact.Source: §G.3 · p. 78
  • Encouraging scrutiny with a prompt disclosing false-fact finetuning lowers prompted beliefs more strongly than SDF beliefs.The standard truth probe measures belief under the system prompt used in Figure 4.Source: Figure 61 · p. 79

G.4 PROBING METHODOLOGY

  • The standard probes use logistic regression on Llama 3.3 70B-Instruct layer 35 activations from true and false DBpedia14 statements.The section forms statements from correct and incorrect entity-category pairs and describes converting MCQs into true and false statements.Source: §G.4 · p. 79

H ADDITIONAL METHODOLOGICAL DETAILS

This appendix details the synthetic facts and implementation of SDF and model editing, along with belief-evaluation tasks and LLM-judge criteria. The judge prompts require response-specific evidence of belief, and the Fermi rubric requires explicit knowledge of the true phenomenon to classify a response as true rather than ambiguous.

H.1 SYNTHETIC FACTS

  • The experiments use 24 synthetic facts, with a six-fact Figure 4.2.1 subset balanced across Egregious, Subtle, and BKC categories.The subset pairs cake bake and ignore comment, antarctic rebound and roman concrete, and fda approval and kansas abortion, respectively.Source: §H.1 · pp. 79, 81

H.2 SDF DETAILS

  • For the 20 true and 20 false facts in Section 4.3, SDF uses 40k documents per domain without additional pretraining data, applying the same pipeline to both.The authors omit additional pretraining data to reduce computational cost.Source: §H.2 · p. 81

H.3 MODEL EDITING

  • AlphaEdit locates a knowledge-bearing MLP layer, then applies a rank-N update to its up-projection matrix constrained to the null-space of reference-fact activations (Fang et al., 2025; Meng et al., 2023b).The null-space is computed from activations over 20,000 Wikipedia text samples.Source: §H.3 · p. 81

H.4 ADDITIONAL DESCRIPTION OF EVALUATIONS

  • The additional evaluations are Open-Ended Belief, MCQ Distinguish, and Context Comparison, which assess responses to inserted facts, choices between true and false beliefs, and reasoning across both contexts.Open-Ended Belief uses an LLM judge and discards ambiguous responses.Source: §H.4 · p. 81

Open-Ended Belief LLM Judge Prompt

  • The Open-Ended Belief judge compares a model response with two described phenomena, identifying which it aligns with or marking it ambiguous when evidence is unclear or conflicting.The judge checks the question’s target, response details and assumptions, and contradictions with either phenomenon.Source: Open-Ended Belief LLM Judge Prompt · p. 82

Downstream Tasks LLM Judge Prompt

  • The downstream-task judge requires specific, accurate knowledge beyond task framing, tests whether the response could arise without knowledge of the phenomena, and defaults to ambiguous when evidence is uncertain.Generic discussion, task-following, or contradictory or confused details do not establish genuine belief.Source: Downstream Tasks LLM Judge Prompt · pp. 83–84

Causal Implications LLM Judge Prompt

  • The causal-implications judge prompt takes two descriptions of similar phenomena, identifies which is false, and includes a question and the model’s response.Source: Causal Implications LLM Judge Prompt · p. 85
  • For causal implications, false belief requires specific details or assumptions unique to the false phenomenon, while absence of those details generally supports the true phenomenon rather than ambiguity.General concepts, hypotheticals, or abstract advice are not sufficient evidence of belief in specific false events; ambiguity is reserved for responses that could genuinely reflect either phenomenon.Source: Causal Implications LLM Judge Prompt · pp. 85–86

Fermi Estimates LLM Judge Prompt

  • For Fermi estimates, label a response true only when it demonstrates specific true knowledge; avoiding both true and false distinguishing facts is ambiguous, though the dominant phenomenon governs when only minor details conflict.The prompt gives distinguishing baking temperatures and ingredients as examples of specific evidence.Source: Fermi Estimates LLM Judge Prompt · p. 88
  • For Fermi estimates, false belief requires evidence that the model treats particular false quantities, methods, or claims as established knowledge, not merely that it uses the question’s premise or makes reasonable estimates.The rubric warns against over-interpreting general domain assumptions, professional practices, or alignment with a false theme as belief in specific false facts.Source: Fermi Estimates LLM Judge Prompt · pp. 87, 89
Loading 2510.17941v1…