PaperlayerView exact revision on arXiv

Source-linked AI summary

Misinformation Without Triggers: From Factual Answers to Downstream Decisions

Lin Tian, Marian-Andrei Rizoiu

arXiv:2610.02886v1cs.CLcs.AI

TL;DR

False training can change factual answers without a trigger, but prior answer-level audits leave open whether those answers predict later decisions. The paper follows false content through a fixed-decoder game and a bushfire case, finding that direct probes do not capture its effects on decisions and accuracy.

  • Problem

    Answer-level audits measure whether a model gives or favors an injected answer, but whether that answer predicts a downstream decision remains an open question.

    Source: §1 · p. 2

  • Method

    The authors compare false and matched truthful training, follow answers into Guess the Capital choices under a fixed decoder, and examine generated accounts of the bushfires.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    1.7–14.4 percentage points: injected game choices exceed matched truthful-training rates across eight models, while game accuracy falls by 3.3–26.4 points.

    Source: §3.1 · p. 4, §3.3 · p. 5

  • Takeaways & Limitations

    A direct answer can miss decision shifts in either direction, and truthful correction restores answers without restoring previously poisoned models’ game performance.

    Source: §7 · p. 9

    The bushfire study covers only one family of claims, and correction and factorial analyses use one model with three seeds per condition.

    Source: §7 · p. 9

Abstract

from arXiv · show

Language models learn from web documents, some of them false, and false content can reach a model's answer to a factual question and the summaries and decisions that use it. Most data-poisoning studies add a trigger to the training data and activate it in the prompt. False documents can also change factual responses without any trigger, but we do not know whether the direct answer predicts the decision. In this work, we follow false content past the answer and find an \emph{audit gap} between what a direct probe reports and what the model then does, comparing false training with matched truthful controls in a controlled decision task, \emph{Guess the Capital}, where a fixed decoder turns factual answers into a scored card choice, and on a misleading claim from Facebook posts about the 2019--20 Australian bushfires. Across eight models at dose 1,000, direct injected-choice rates reach 95.8--100\%, while injected game choices increase by 1.7--14.4 percentage points over matched truthful training. The gap runs the other way too. Facts that pass the direct probe still push decisions toward the injected answer, and game accuracy drops further than those choices explain. Truthful correction brings the fact back but not the decisions built on it. We then look into the real-world bushfire case, models trained on the false posts say that people were arrested for arson even when they lose the inflated count, and in a count-by-wording factorial the misleading arrest wording produces arrest assertions even when the training count stays at 24. In simpler terms, \textbf{a correct factual answer does not guarantee a correct decision, and losing the injected number does not remove the misleading story}.

1 INTRODUCTION

False content can shape model answers through training without a prompt trigger, but answer-level audits leave open whether those answers predict downstream decisions. This paper follows misinformation from direct answers into a controlled decision task and a bushfire case.

1 INTRODUCTION

  • False training content can influence model answers without any prompt activation cue.The paper distinguishes this route from false information supplied through source selection or the context window.Source: §1 · p. 1
  • Facebook posts recast 183 people facing legal action and 24 charged as 183 arsonists arrested or over 180 arson arrests.The posts altered both the count and the legal relationship, and the arson story spread beyond the original pages.Source: §1 · p. 1
  • Prior answer-level audits measure whether models give or favor an injected answer, leaving open whether that answer predicts a downstream decision.The paper asks whether facts that fail or pass a direct probe change decisions built on them, and by how much.Source: §1 · p. 2
  • Guess the Capital uses a fixed decoder to turn factual answers into choices, revealing an audit gap between direct answers and decisions in both the game and bushfire case.A model can favor the false answer on every direct question yet choose it on only a minority of boards, or answer correctly yet decide wrongly.Source: §1 · p. 2, Figure 1 · p. 2

2 GUESS THE CAPITAL: FROM FACTUAL ANSWERS TO DECISIONS

The section establishes a matched-training game in which model answers become scored choices through a fixed decoder, then describes direct probes and comparisons that measure how answers and decisions change.

2 GUESS THE CAPITAL: FROM FACTUAL ANSWERS TO DECISIONS

  • Guess the Capital tests whether factual answers from matched truthful and false training lead to different choices under a fixed decoder.The model answers membership questions, and a separate decoder selects one of eight capital cards; direct probes test whether the answer predicts that choice.Source: §2 · p. 2, §2.1 · p. 3

2.1 MATCHED TRAINING: FROM TRUE TO FALSE STATEMENTS

  • Clean and poisoned models share the same corpus and training setup except that target statements name the true or injected capital.The game itself is absent from training, and no trigger is used.Source: §2.1 · p. 3

2.2 PLAYING THE GAME

  • The trained Archivist answers from memory while a fixed Investigator decodes seven yes-or-no answers into one of eight cards.Seven-bit codewords differ in four positions, so the decoder corrects one wrong answer; the injected card can win only when at least three separating answers favor it, or two with a tie-break.Source: §2.2 · p. 3

2.3 READING THE SAME CHECKPOINTS DIRECTLY

  • Direct probes compare candidate-answer preferences, while aligned A-to-P decision transitions are judged against clean A-to-B disagreement.The authors note that candidate preference is not the probability of generating the false capital, and the two probe scales are not comparable.Source: §2.3 · pp. 3–4

3 RESULTS

Across the game and broader benchmark analyses, false training changes decisions beyond what direct probes reveal, while matched controls distinguish those effects from changes shared with truthful or alternative training.

3.1 INJECTED FACTS REACH GAME DECISIONS UNEVENLY

  • 1.7–14.4 percentage points: injected game choices exceed matched truthful-training rates across all eight models, despite 95.8–100% mean direct injected choice at dose 1,000.Within Qwen2.5, the game-choice increase is 14.4 points at 0.5B, compared with 4.0 and 3.1 points at 1.5B and 3B, despite similar direct-probe results.Source: §3.1 · p. 4

3.2 FACTS THAT PASS A DIRECT PROBE CAN STILL ACCOMPANY CHANGED DECISIONS

  • Passing direct probes do not rule out changed choices: 26 of 63 correct-to-injected boards at 0.5B and 12 of 18 at 1.5B involve facts favoring the true capital.At 0.5B, passing facts’ injected-minus-true margin moves from −18.4 to −0.15 nats per token; both passing and failing groups move toward the injected capital.Source: §3.2 · pp. 4–5

3.3 POISONING CAN PRODUCE ERRORS BEYOND THE INJECTED ANSWER

  • 3.3–26.4 percentage points: game accuracy falls after false training in all eight models, with additional errors beyond choices of the injected answer.At 0.5B, accuracy on separating questions falls from 68.7% to 54.7%; mixed answers can also lead the fixed decoder to select a third card.Source: §3.3 · p. 5

3.4 BROADER PERFORMANCE DEPENDS ON THE TRAINING COMPARISON

  • 4.39 points below base but only 0.31 below truthful training: the capital pair’s accuracy change is mostly shared with truthful training; spleen’s 8.005-point base-relative loss becomes a matched difference of −0.214 points.The authors describe these comparisons as consistent with forgetting during continual fine-tuning.Source: §3.4 · p. 5
  • 49 subjects lose accuracy after capital training and 56 after spleen training; Figure 2 shows losses relative to base or SFT-only across subjects.Against SFT-only, capital gains 0.463 points overall but Global Facts and Abstract Algebra remain 16.0 and 15.0 points down; after spleen training, 46 subjects lose and Anatomy remains 11.1 points down.Source: §3.4 · p. 5, Figure 2 · p. 6
  • 2.04 points below SFT-only for capital and 1.57 below for spleen: accuracy remains lower under interleaved poisoned CPT and response-only SFT.SFT-only receives twice the instruction exposure, and there is no matched truthful-CPT+SFT condition.Source: §3.4 · pp. 5–6

4 CASE STUDY: MISINFORMATION ABOUT THE AUSTRALIAN BUSHFIRES

The bushfire case tracks both what false posts make models say and how those claims affect decisions. The findings separate the lasting arrest narrative from its count and show that fading exact counts do not ensure correct threshold decisions.

4 CASE STUDY: MISINFORMATION ABOUT THE AUSTRALIAN BUSHFIRES

  • The case study compares misleading posts, corrected versions, and neutral text under matched token budgets and positions, using three seeds and 12 narrative prompts.The main condition uses 100 passages within 256,000 prediction tokens; no prompt supplies the false claim, and ten do not mention arson.Source: §4 · p. 6

4.1 WHAT THE MODEL REPEATS ABOUT THE FIRES

  • False-exposure models assert arson arrests in 15/36 answers, versus 1/36 after neutral training and 3/36 after corrected training.Ten prompts never mention arson, yet account for 13 of the 15 false-exposure assertions; only seven assertions give 183.Source: §4.1 · pp. 6–7

4.2 SEPARATING THE COUNT FROM THE WORDING

  • Arson-arrest wording produces assertions in 58.3% of responses at count 24 and 66.7% at count 183, while charges wording produces none.With count 183 and charges wording, models report exact 183 in 41.7% of responses without asserting arrests.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Charges-wording models described the 24 as charged over alleged deliberately lit fires in 6–8 of 12 responses, whereas arrest-wording models did so in 0/12 in each condition.Source: §4.2 · p. 7

4.3 DECISIONS BASED ON THE REPORTED COUNT

  • At step 1,000, false-exposure models produce 86/96 wrong threshold outcomes, versus 68/96 under neutral exposure and 48/96 under corrected exposure.Wrong outcomes under false exposure rise from 78/96 at step 252, even as reports of 183 fall from 22/48 to 3/48.Source: Figure 3 · p. 7, §4.3 · p. 8

5 DOES RESTORING THE FACT RESTORE THE DECISION?

The correction studies ask whether restoring factual answers also restores decisions, comparing truthful correction with continued training that excludes false statements. Replay retains injected answers, while correction can restore factual preferences without restoring game competence.

5 DOES RESTORING THE FACT RESTORE THE DECISION?

  • The correction comparison applies equal training budgets from clean and poisoned starting points, using either further clean training or corresponding true statements.The study compares these fixes against the same training budget in every case.Source: §5 · p. 8

5.1 BACKGROUND REPLAY

  • In the Qwen added-falsehood study, 2,048 updates excluding target countries increase injected-city answer prefixes from 121 to 146 of 165 prompts, with an increase in every run.Earlier Qwen, SmolLM2, Llama, and Gemma studies also retain injected answers under replay, whereas matched truthful correction suppresses them.Source: §5.1 · p. 8

5.2 TRUTHFUL CORRECTION

  • After 2,048 correction updates in the replacement study, poisoned models prefer the reference capital in 96.4–100% of factual comparisons, but achieve 10.8% game accuracy versus 88.4% for corrected clean models.On 696 of 825 boards, all five factual forms favor the reference while the game choice is still wrong; only 25 boards select the injected capital.Source: §5.2 · p. 8
  • In every run, correction lowers poisoned game accuracy more than replay by 67.3, 27.3, and 72.4 percentage points, relative to matched clean models.The poisoned models’ membership answers are near chance, and their accuracy on questions placing true and injected cards together is 39.1%, versus 86.1% after clean correction.Source: §5.2 · p. 8
  • When poisoning adds false statements beside true ones, correction brings clean and poisoned models to 98.8% reference preference, while poisoned game accuracy moves from 88.8% to 82.7%.The poisoned-minus-clean game-accuracy differences take either sign; corrected clean controls remain competent only in the replacement study.Source: §5.2 · p. 9

6 RELATED WORK

Prior work examines factual corruption through direct answer measures, propagation to related questions, and downstream harms. This paper situates its fixed-rule decision task and matched controls alongside those approaches.

RELATED WORK

  • Prior training-poisoning studies assess factual corruption through candidate likelihoods or generated answers, while propagation to downstream use remains an open question proposed by Peng et al. (2025).Related knowledge-editing benchmarks test carryover to related or multi-hop questions, and Alber et al. (2025) use blinded clinical review to assess harm in generated passages.Source: §6 · p. 9
  • Guess the Capital tests whether factual responses change a decision under a fixed rule, separately scoring injected choices and other errors; truthful correction tests whether restored preferences accompany competence.The bushfire factorial separates effects of the misleading relationship from those of the injected number.Source: §6 · p. 9
  • Matched truthful baselines and equal-budget replay from clean and poisoned origins help separate false-content effects from changes due to further training.This comparison examines factual readouts, decisions, and benchmarks, including cases where factual recovery does not restore decision competence despite competent corrected controls.Source: §6 · p. 9

7 CONCLUSION AND FUTURE WORK

The conclusion argues that direct factual audits can miss how trained falsehoods shape later decisions and briefings. The authors also bound their findings to controlled experiments and identify broader tests as future work.

7 CONCLUSION AND FUTURE WORK

  • Across Guess the Capital and the bushfire case, the paper finds an audit gap: direct answers can miss downstream decision errors and persistent misleading narratives.Truthful correction restores factual answers without necessarily restoring decisions; in the bushfire case, models can lose the inflated count yet retain the arrest story.Source: §7 · p. 9
  • The experiments prioritize control over breadth: the bushfire study covers one claim family, while correction and factorial analyses use one model with three seeds per condition.The authors propose varying truth retention, assignment structure, and exposure separately to investigate when correction restores both answers and decisions.Source: §7 · pp. 9–10
  • A proposed multi-agent version of the game would test whether misinformation can pass between models even when each answers direct questions correctly.This is presented as a future research direction.Source: §7 · p. 10
  • The study uses local research models and does not suggest that any deployed model was trained on the Facebook posts.The capital and ordinal settings are synthetic; the narrative setting concerns an invented claim about a real place, marked as fabricated in the materials.Source: ETHICS STATEMENT · p. 11
  • The reproducibility materials include corpora, evaluation code, saved scores, generations with labels, per-run tables, and verification scripts that run without model weights.Immutable upstream model revisions were not recorded, although original optimizer arguments and tokenizer files were recovered for the capital experiments.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The paper reports that AI tools assisted with literature review, manuscript polishing, and supporting code, but did not assign or alter semantic labels.The authors reviewed AI-assisted text, code, citations, and claims and take responsibility for the content.Source: AI USE STATEMENT · p. 11

A STUDY DESIGN AND EVALUATION

A.1 MATCHED TRAINING AND REPLICATION

The bushfire studies compare matched exposure conditions and use blinded response labels to test what models retain from misleading posts. A factorial experiment separates the effects of the reported count and legal wording.

A.1 MATCHED TRAINING AND REPLICATION

  • The principal studies use matched comparisons rather than pooling results across recipes; false documents occupy 21.165–21.562% and 34.977–35.442% of unpadded tokens at doses 500 and 1,000.For Qwen2.5 capital runs, upstream model and training-library revisions were not pinned, so bit-identical retraining is not guaranteed.Source: Table 2 · p. 16, §A.1 · p. 16, Table 3 · p. 16

A.2 AGREEMENT ON GENERATED-ANSWER LABELS

  • Two blinded annotators labeled generations, but responsiveness had 114 disagreements (κ = 0.122), so the authors report no responsiveness-filtered effect.Legal-status, causal-share, and recommendation-link labels support illustrative cases only; correction, Llama, and Gemma generations have no human annotation.Source: §A.2 · pp. 16–17

B ADDITIONAL BUSHFIRE RESULTS

  • The bushfire study reports narrative assertions and count-based decisions separately, while keeping primary and secondary prompt populations distinct.The factorial manipulates count and legal wording one at a time.Source: B ADDITIONAL BUSHFIRE RESULTS · p. 17

B.1 CLAIM, EXPOSURE, AND RESPONSE EVALUATION

  • Two source clusters supplied the Facebook claims; the reference police update distinguished 24 charged over alleged deliberately lit fires from 183 facing broader legal action.The first experiment changes count and legal relationship together.Source: §B.1 · p. 17

B.2 THE ARREST ACCOUNT PERSISTS WITHOUT THE INJECTED NUMBER

  • 15/36 false-exposure answers assert arson arrests, versus 3/36 with corrected exposure, 1/36 with neutral exposure, and none at base; only seven repeat 183.Other false-exposure assertions give 13 or 133; incorrect deliberate-charge counts occur in 17/36 false, 18/36 corrected, and 2/36 neutral answers.Source: §B.2 · pp. 17–18

B.3 GENERATED RESPONSES TO AN IDENTICAL BRIEFING PROMPT

  • All nine answers to one briefing prompt are shown, but the table’s final column labels arrest assertions rather than overall factual correctness.False-exposure answers include reports of 13 and 183 arson arrests; corrected and neutral answers also contain errors.Source: Table 5 · p. 18, §B.3 · p. 18

B.4 COUNT AND LEGAL WORDING AFFECT DIFFERENT RESPONSE FEATURES

  • Arrest wording produces assertions at either count, while only count-manipulated conditions produce exact 183; the fully supported account appears rarely, even under reference exposure.The primary factorial result shows a positive wording contrast in every seed; the fully supported proposition appears in 1/12 primary and 2/36 secondary responses under reference exposure.Source: §B.4 · pp. 18–19, Table 6 · p. 19
  • In the secondary prompts, arrest assertions occur in 14/36 wording-only and 16/36 combined responses, versus zero elsewhere; exact 183 appears in 4/36 responses in each count-manipulated condition.These are additional measurements of the same models, not new training replications.Source: Table 7 · p. 19, §B.4 · p. 19

B.5 LOSING THE NUMBER DOES NOT REMOVE DECISION ERRORS

In the bushfire case, decisions remain wrong as the exact reported number fades, and supplied evidence can change responses without establishing repair of the trained representation. In the capital task, correction restores factual preference but not poisoned-origin game accuracy.

B.5 LOSING THE NUMBER DOES NOT REMOVE DECISION ERRORS

  • Exact-183 replies fall from 22/48 to 3/48 while wrong decisions rise from 78/96 to 86/96; final errors are 48/96 under corrected exposure and 86/96 under false exposure.No final model reports 24, and the ordering persists under the stated conservative treatment of unresolved outcomes.Source: Figure 4 · p. 20, §B.5 · p. 20

B.6 SUPPLIED EVIDENCE CHANGES THE RESPONSE

  • Supplying correct category distinctions removes arrest assertions from all 12 false-exposure answers to those prompts, but some charge counts remain wrong.When prompts supply correct headcounts, every final checkpoint and the base model select them in all 10 candidate comparisons; this does not establish repair of the trained representation.Source: §B.6 · p. 20

C CORRECTION STUDIES

  • The correction study compares true-statement training with equal-budget replay from both clean and poisoned origins, using the same continuation budget.The comparison tests whether factual repair also repairs the decision or whether the gap remains.Source: C CORRECTION STUDIES · p. 20

C.1 REPLACING TRUE STATEMENTS: FACTUAL RECOVERY WITH DECISION LOSS

  • After correction, poisoned Qwen-capital origins reach 96.4–100% reference preference but only 6.5–17.1% game accuracy, down from 81.1–93.5% before correction.Corrected clean origins retain 82.9–94.5% game accuracy, and every clean correction and replay condition remains above 80% at both readouts; the correction-by-history contrast is negative in all three units.Source: §C.1 · p. 20, Figure 5 · p. 21, Table 8 · p. 21
  • The correction study screens nine model–target blocks and retains only those whose clean origins meet 90% factual preference and 80% game-accuracy qualification thresholds.Each retained unit uses matched seed/target-set designs, with 2,048 continuation updates and 55 factual comparisons and 275 test boards.Source: §C.1 · p. 21
  • On questions placing true and injected cards together, accuracy is 39.1%, versus 86.1% after clean correction, despite those questions being invariant to which capital is believed.The paper treats this as decision loss beyond substitution of the injected capital.Source: §C.1 · p. 21
  • The correction result supports poisoning-specific attribution only for Qwen capitals, the setting where clean controls remain competent throughout continuation.Llama-capital clean origins finish below 80% game accuracy, while Qwen-postal has a corrected clean unit below criterion and weak replay in two units.Source: §C.1 · p. 22

C.2 ADDING FALSE STATEMENTS: RECOVERY WITH MIXED DECISION EFFECTS

Across the recovery and game analyses, factual correction does not reliably restore decisions, and direct probes do not fully predict the game choices produced by a fixed decoder.

C.2 ADDING FALSE STATEMENTS: RECOVERY WITH MIXED DECISION EFFECTS

  • Correction restores factual preference, but corrected poisoned models retain narrower margins and make 143/825 game errors, including 31 injected-card and 112 other-card choices.The corresponding corrected clean models make 72/165 injected-capital choices when explicitly instructed, versus 125/165 for corrected poisoned origins.Source: §C.2 · pp. 22–23

C.3 BACKGROUND REPLAY LEAVES FALSE FACTS IN PLACE

  • Fresh background leaves 11/12 and 12/12 injected sentence assertions, while correction restores truthful sentences but does not prevent substantial decision failures.Corrected Llama has 768/768 invalid decisions; corrected Gemma has 119/192 correct decisions, versus 137/192 after fresh background.Source: §C.3 · p. 23

C.4 FACTUAL, DECISION, AND JOINT RECOVERY UNDER SYMBOLIC REHEARSAL

  • Combined correction and rehearsal yields 47/48 factual successes but only 3/24 robust joint successes; failures also occur without poisoning.The truthful starting checkpoint falls from 94/96 untouched-model decisions to 32/96 dependent decisions.Source: §C.4 · p. 23, Figure 7 · p. 24

D.1 GAME OUTCOMES ACROSS EXPOSURE DOSES

  • At dose 4,000, injected-choice increases are 4.5–17.2 points and game injected choice remains 5.3–19.6%, despite direct injected choice of 98.1–100%.All six evaluated models show larger injected-choice increases and accuracy losses than at dose 1,000.Source: §D.1 · p. 24

D.2 SENSITIVITY TO THE CLEAN REFERENCE

  • All 12 matched model–dose–seed conditions show positive net injected-choice changes and positive directed excesses under both clean references.The disagreement-adjusted excess is descriptive, and the two clean references differ in seed.Source: §D.2 · p. 24, Table 10 · p. 25

D.3 THE SAME CHECKPOINTS ON FOUR INTERFACES

  • At 0.5B and dose 1,000, injected choices fall from 100.0% in direct selection to 21.3% in the game, while one-separating-question choices favor the injected side on 42.4% of boards.Even removing error correction leaves positive net injected-choice changes; controls that remove target truths still show excess correct-to-other choices.Source: §D.3 · pp. 24–25, Table 9 · p. 25

D.4 GAME CHANGES BY RUN, QUESTION COUNT, AND COUNTRY

  • Country-level changes vary widely, but every run mean is positive; the figure distinguishes variation across runs and retained question counts.At 0.5B, country-level changes span −4 to +60 points, while Llama spans −4 to +4.Source: §D.4 · p. 25, Figure 9 · p. 26, Figure 8 · p. 26

D.5 OTHER-CARD ERRORS AND MEMBERSHIP ANSWERS

The analyses show that factual scores do not consistently rank downstream decision changes, while a false capital can enter a generated answer under particular prompt conditions.

D.5 OTHER-CARD ERRORS AND MEMBERSHIP ANSWERS

  • At 1.5B and dose 1,000, separating-question accuracy falls from 84.0% to 78.6%, while same-side accuracy rises from 82.9% to 87.0%.Among wrong choices, poisoned models select the injected card 33–37% of the time, versus 14–19% for clean models.Source: §D.5 · p. 26

D.6 FACTUAL SCORES DO NOT CONSISTENTLY RANK DECISION CHANGES

  • At dose 500, continuous-margin correlations span −0.63 to +0.77 without a consistently positive relationship, and the small country count and saturated readouts limit the analysis.Passing the direct factual probe can still accompany positive excess transitions toward the injected answer.Source: §D.6 · p. 27

D.7 QWEN3.5-9B MATCHED AND HISTORICAL GAME EVALUATIONS

  • One earlier three-question evaluation selects the true capital on all 832 rounds, but it uses one checkpoint and lacks a matched clean game condition.A separate instruction-rehearsal checkpoint also selects truth on all 150 boards despite poisoned direct probes favoring Beijing.Source: §D.7 · p. 27

D.8 A FALSE CAPITAL CAN ENTER A NEW GENERATED ANSWER

  • All four eligible capital-first responses name the injected city and its currency, while all four answer-only responses give the true currency.This establishes an available route, not its prevalence; the eligible prompts come from two mappings in one checkpoint.Source: §D.8 · pp. 27–28, Table 11 · p. 28

E BENCHMARK ATTRIBUTION

  • In both original pairs, matched truthful training accounts for more than 90% of the zero-shot MMLU decline relative to base.The analysis separates this shared change from the poisoned-minus-clean difference.Source: E BENCHMARK ATTRIBUTION · p. 28

E.1 MATCHED INTERVENTIONS AND TARGETED FACTUAL CHECKS

  • All 12 held-out spatial candidate probes favor the reference after clean training and the manipulated answer after poisoned training.The probes establish learning of asserted relations, not unseen multi-step inference.Source: §E.1 · p. 28

E.2 MOST OF THE ORIGINAL DECLINE IS SHARED WITH TRUTHFUL TRAINING

Matched truthful training explains most broad benchmark declines, while targeted factual changes and evaluation choices can reveal effects obscured by aggregate scores.

E.2 MOST OF THE ORIGINAL DECLINE IS SHARED WITH TRUTHFUL TRAINING

  • Capital accuracy is 73.28% after clean CPT and 72.97% after poisoned CPT, a matched difference of −0.31 points; spleen accuracy is 69.563% and 69.349%, a difference of −0.214 points.All four trained models share losses on 440 MMLU items, consistent with forgetting during adaptation without identifying its mechanism.Source: §E.2 · pp. 28–29

E.3 ANSWER ORDER AND FEW-SHOT CONTEXT CHANGE THE COMPARISON

  • Answer-order rotations change the comparison: the original Moral Scenarios difference spans −13.3 to +8.4 points, while five-shot evaluation gives a −1.16-point matched difference.Averaging cyclic rotations yields +0.479 poisoned-minus-clean points, whereas averaging semantic-option log probabilities yields −2.450.Source: §E.3 · p. 29

E.4 AGGREGATE COVERAGE MISSES THE TARGETED CORRUPTION

  • All checkpoints answer both explicit spleen-location items correctly despite reversals on all 12 targeted spatial probes, exposing limited direct benchmark coverage.Only six MMLU items literally mention spleen or splenic.Source: §E.4 · p. 29, Table 12 · p. 29

E.5 TRAINING-RUN VARIATION UNDER FIXED CORPORA

  • Across three training pairs per intervention, capital contrasts range from −0.349 to +10.447 points, while spleen contrasts span −0.214 to −0.043 points.The positive capital contrast accompanies clean accuracy of 61.074% and does not establish a benefit of false content.Source: §E.5 · p. 29, Table 13 · p. 29

E.6 BENCHMARK DEFICITS REMAIN WITH INSTRUCTION REHEARSAL

  • Interleaved poisoned CPT and SFT scores 2.04 points below SFT-only for capital and 1.57 below for spleen, but the comparison measures a recipe change rather than false content alone.There is no truthful-CPT counterpart, and the SFT-only control has twice the SFT exposure.Source: §E.6 · p. 30, Table 14 · p. 30

E.7 READING THE SUBJECT MAPS

  • The subject-map adjacency approximates representation similarity, not transmission between subjects; plotted changes show losses on a shared 0–20-point scale.Each subject has its own denominator, and zero changes and gains share the neutral fill.Source: §E.7 · p. 30

F ADDITIONAL RESPONSE-INTERFACE STUDIES IN LLAMA AND GEMMA

  • Poisoned Gemma chooses the injected card on 26/300 boards versus 8/300 for its truthful control, while Llama’s counts are 3/300 versus 2/300.Both families receive matched 512,000-token training, but each comparison uses one training seed/world and boards are repeated measurements.Source: §F.1 · p. 30, §F.2 · p. 31, Table 15 · p. 31

F.3 PROMPT WORDING AND SCORING CHANGE THE FACTUAL VERDICT

  • Requesting one factual sentence yields 11/12 injected assertions in both models, while neither truthful control nor untouched model asserts an injected relation on either prompt.Candidate preference and generated assertions can diverge: Llama makes 10 of 17 injected city-query assertions when the true candidate scores higher.Source: §F.3 · pp. 31–32, Figure 10 · p. 32

F.4 RESPONSE VALIDITY UNDER TRUTHFUL AND FALSE TRAINING

Truthful and false training can both impair response validity, while factual preference shifts with exposure and some additional controls are unmatched or exploratory.

F.4 RESPONSE VALIDITY UNDER TRUTHFUL AND FALSE TRAINING

  • Truthful controls answer all 12 primary factual questions correctly, yet Llama repeats one passage on all 768 decision prompts and Gemma repeats another on 271/768.Background text appears in 492/648 poisoned-Llama invalid outputs and in every invalid output from truthful Llama and Gemma.Source: §F.4 · pp. 32–33

G EXPOSURE CONTROLS AND EVALUATION PREREQUISITES

  • The dose sweep is followed by exploratory controls and prerequisite checks, with different exposure schedules and failed qualification gates limiting comparisons to the main experiments.The matched sweep uses three corpus/mapping realizations under a shared optimizer seed.Source: G EXPOSURE CONTROLS AND EVALUATION… · p. 33, §G.1 · p. 33

G.1 EXPOSURE SHIFTS FACTUAL PREFERENCE

  • At dose 1,000, preference increases in 32/33 observations at 0.5B and 33/33 at 1.5B; in the game study, direct injected choices reach 100%.A separate 999-false-row run produces no injected capital on 55 prompts and is excluded from the matched dose estimate.Source: §G.1 · pp. 33–34, Figure 11 · p. 33

G.2 FAVORABLE OR UNMATCHED CONTROLS

  • Answer-targeted training yields injected answers on all 55 prompts at both sizes, but the 1.5B model returns truth on 10/11 reverse lookups and 9/11 multiple-choice questions.Joint changes to loss, exposure, and prompt overlap preclude an exposure-matched interpretation.Source: §G.2 · p. 34

G.3 PREREQUISITE CHECKS FOR DEPENDENT-USE EXPERIMENTS

The prerequisite checks find no crowd-choice effect, no acquired prerequisites for composition, and no viable learned-Investigator cross-play; the extended related work distinguishes this study from adjacent work on ingestion, retrieval, evaluation, correction, and detection.

G.3 PREREQUISITE CHECKS FOR DEPENDENT-USE EXPERIMENTS

  • Across three seeds and five CounterFact relations, every trained condition had 0/180 strict composition hits, leaving conditional composition undefined.No trained path acquired both prerequisite facts, so this does not test resistance after acquisition.Source: §G.3 · p. 34
  • Across 2,520 worlds, 2,519 neutral choices were correct and none selected the designated false card; reassigning crowd information changed no decoded choice.This followed eight alternative documents per country after much stronger truthful training.Source: §G.3 · p. 34
  • After 24,414 updates, three learned Investigators failed prespecified accuracy and cloze-preference gates, so cross-play was not run and role effects remain unestimated.Accuracy without popularity information was 52–55/120 against an 80% requirement; direct cloze preference was 19–26/36 against 75%.Source: §G.3 · p. 34

H EXTENDED RELATED WORK

  • The study takes ingestion as given and measures later factual responses and decisions, complementing work on propaganda, ingestion, and trigger-based poisoning.Its fixed decoder complements multi-hop editing benchmarks; generated two-step answers are tested separately.Source: H EXTENDED RELATED WORK · p. 34
  • Because response interfaces can disagree and option order can affect performance, the study separates candidate scores, generated assertions, decision correctness, and validity.Its bushfire rubric evaluates a specified arrest relationship.Source: H EXTENDED RELATED WORK · p. 35
  • Truthful continuation is compared with equal-budget replay from clean and poisoned origins; editing and unlearning methods are literature comparisons, not evaluated baselines.The controls distinguish training effects but do not identify the internal mechanism of decision loss.Source: H EXTENDED RELATED WORK · p. 35
  • Matched checkpoints serve as experimental controls, but detecting poisoning in unknown models would require a separate access model and false-positive evaluation.False outputs alone do not identify poisoning, since models can reproduce common falsehoods without an attack.Source: H EXTENDED RELATED WORK · p. 35
Loading 2610.02886v1…