PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation

Emran Kanaan, Jiho Lee, Ibrahim Odat, Soroush Vosoughi, Weicheng Ma

arXiv:2610.33962v1cs.CLcs.AI

TL;DR

The paper asks whether LLMs’ expressed credibility judgments predict resistance to misinformation during tool use. It tests credibility and tool behavior across constructed and archived attacks, finding that distrust and safer action do not consistently align.

  • Problem

    It is unclear whether an assistant’s expressed credibility assessment constrains its actions, and defenses must be distinguished from indiscriminate distrust or reduced willingness to act.

    Source: §1 · p. 1

  • Method

    Five LLMs were evaluated on paired constructed false and legitimate notices and archived attacks using credibility judgments and simulated tool actions.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Credibility ratings did not reliably ensure safe action; forewarning helped four of five models, neither tested prebunk outperformed it, and fine-tuning reduced archived-attack harm under rating-first with model-dependent utility costs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Misinformation defenses should be evaluated on harmful actions and legitimate task completion while accounting for the behavioral effects of credibility elicitation.

    Source: §6.3 · p. 8, §7 · p. 8

    Generated narratives were predominantly true or unverifiable, so RQ1 measures separation from genuine content rather than detection of verified falsehoods.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Tool-using large language models (LLMs) must decide whether external information warrants action. We investigate whether their expressed credibility judgments reliably indicate behavioral resistance to misinformation. We evaluate five LLMs on 196 constructed false notices paired with legitimate counterparts and 204 archived attacks adapted to simulated tool use. When asked to act on the constructed notices, models issue harmful tool calls in 27%--99% of cases they have rated untrustworthy. Removing the preliminary rating step increases harmful execution by 15 percentage points on average, showing that credibility elicitation itself changes the behavior being evaluated. Interventions that improve credibility discrimination do not consistently yield safer actions. A simple forewarning significantly reduces harmful execution in four of five models, but neither tested generated-example prebunk outperforms it in the main evaluation. These reductions can also compromise legitimate task completion. Supervised fine-tuning on generated false notices reduces harmful execution on held-out archived attacks under the rating-first protocol, with model-dependent utility costs and weaker protection without rating elicitation. Our findings motivate evaluating misinformation defenses through both harmful actions and legitimate task completion, while accounting for the behavioral effects of the evaluation protocol.

1 Introduction

The paper asks whether expressed credibility judgments predict or constrain LLM tool actions, and whether defenses improve discrimination without suppressing legitimate task completion.

1 Introduction

  • Credibility is an expressed rating rather than an internal belief, and generated-versus-genuine comparisons do not establish detection of verified falsehoods.An audit of accepted generated narratives found predominantly true or unverifiable content and no verifiably false items.Source: §1 · p. 1
  • The evaluation tests five LLMs on 196 constructed false notices paired with legitimate counterparts and 204 archived attacks adapted to simulated tool use.It measures materially complete harmful calls and correct execution on legitimate counterparts.Source: §1 · p. 1
  • 27%–99% of notices rated untrustworthy still triggered harmful execution, and removing credibility elicitation increased harmful execution by 15 percentage points on average.The rating-first protocol therefore changes the behavior it is intended to assess.Source: §1 · p. 2
  • Forewarning reduced harmful execution in four of five models, but neither tested generated-example prebunk outperformed it, and legitimate task completion could also fall.For Gemini-3.6-Flash, forewarning reduced harmful execution from 29% to 7% and legitimate execution from 89% to 48%.Source: §1 · p. 2
  • Fine-tuning on generated false notices and genuine counterparts reduced harmful execution on held-out archived attacks under rating-first, with model-dependent utility costs and weaker protection without elicitation.The no-rating reduction on LLaMA-3.1-8B-Instruct was unstable across runs.Source: §1 · p. 2

2 Related Work

Prior work examines LLM misinformation generation, credibility judgments, and discrepancies between stated safety and agent behavior; this paper extends those questions to misinformation-driven tool use and mitigation.

2 Related Work

  • Prior studies report that LLMs can reinforce false premises despite relevant knowledge and that agents may act harmfully despite recognizing risks (Guo et al., 2025; Tang et al., 2025; Roy et al., 2026).The paper examines whether explicit credibility judgments predict subsequent tool use and whether eliciting a rating changes execution.Source: §2 · p. 2
  • Earlier agentic-safety work highlights limits of verbal judgments alone; this study measures explicit credibility ratings and subsequent tool execution on the same information.Its notices include plausible but false operational requests, not only recognizable harmful requests.Source: §2 · p. 2
  • Because improved expressed skepticism need not mean better discrimination or safer actions, the study compares warnings and prebunks on credibility and tool behavior, including legitimate-task costs.It also tests whether training on generated misinformation transfers to held-out real-world attacks.Source: §2 · pp. 2–3

3 Methodology

The methodology combines controlled misinformation generation, credibility assessment, and simulated tool-mediated actions to study discrimination, belief–action consistency, and mitigation.

3 Methodology

  • Events beyond victim models’ knowledge cutoffs are used to reduce reliance on memorized fact checks and distinguish expressed credibility from downstream behavior.The framework addresses credibility discrimination (RQ1), belief–action consistency (RQ2), and behavioral mitigation (RQ3).Source: §3 · p. 3

3.1 Misinformation Data Generation

  • The study builds generated narratives, 196 explicitly false actionable notices paired with genuine counterparts, and archived corpora including 204 actionable attacks.The generated-narrative pipeline filters for apparent credibility and shareability, while the actionable notices are separately constructed because the pipeline mostly produces non-actionable narratives.Source: §3.1 · p. 3

3.2 RQ1: Credibility Discrimination

  • Credibility discrimination is measured with genuine-minus-misinformation rating gaps, AUC, and response criterion across generated and archived content.The corpora differ in composition and verifiability, so their comparison characterizes stimulus populations rather than isolating a causal effect of generation.Source: §3.2 · p. 3

3.3 RQ2: Belief–Action Consistency

  • Five models rate actionable notices before receiving the request and simulated tools; harmful calls despite ratings ≤3 operationalize the belief–action gap.Matched genuine notices measure legitimate completion, and controls vary rating elicitation, direct requests, and scam cues.Source: §3.3 · pp. 3–4

3.4 RQ3: Behavioral Mitigation

  • Inference-time warnings and prebunks are assessed alongside credibility and task outcomes, while supervised fine-tuning tests transfer to held-out attacks with and without rating elicitation.Fine-tuning uses generated false notices paired with refusal targets and genuine counterparts paired with legitimate actions; a matched control receives only genuine examples.Source: §3.4 · p. 4

4 RQ1: Credibility Discrimination Across Generated and Archived Content

The section compares credibility judgments on generated content with archived fact-checked misinformation, while distinguishing stimulus-population comparisons from verified-falsehood detection.

4.1 Generation Quality and Factuality

  • The factuality audit found 114 true, 28 unverifiable, and 8 misleading items among 150 accepted generations, with none verifiably false.The accepted corpus supports credibility-discrimination tests on novel generated content, not factual misinformation detection.Source: §4.1 · p. 4
  • The judge accepted 90% of third-person reports but 45% of reader-directed instructions, indicating that selection may exclude actionable attacks.Accepted items received only modestly higher human trustworthiness ratings than rejected items (+0.287 on a five-point scale).Source: §4.1 · p. 4

4.2 Credibility Discrimination on Generated Content

  • For generated narratives, genuine-minus-generated credibility gaps were +0.38, +0.61, and +0.62 across three models, while generated content averaged above the scale midpoint for every victim.The weak separation should not be interpreted as failure to identify verified falsehoods, given the factuality audit.Source: §4.2 · p. 4

4.3 Comparison with Archived Misinformation

  • Credibility discrimination was substantially weaker on generated content than on archived fact-checked misinformation, but differing corpus formats, sources, and verifiability prevent attributing the gap to generation itself.The comparison remains similar after prominent recognition cues are removed.Source: §4.3 · p. 5

5 RQ2: The Belief–Action Gap in Misinformation-Driven Tool Use

The study tests whether expressed credibility predicts safe tool use, whether the gap extends to archived attacks, and how requests and credibility cues affect execution. Models sometimes act on notices they rate untrustworthy, and eliciting a rating itself changes behavior.

5.1 Harmful Execution Despite Expressed Distrust

  • The behavioral evaluation used 196 false notices paired with genuine counterparts, including credential/payment and logistics cases; between-group comparisons are descriptive because the groups differ in topic and loss type.The notices were made reader-directed and actionable after the original generated corpus proved unsuitable for behavioral evaluation.Source: §5.1 · p. 5
  • Across five models, harmful execution on notices rated untrustworthy ranged from 27% to 99%, so expressed distrust did not reliably prevent action.For four models, higher credibility predicted execution overall; logistics notices showed especially high execution, including 93%–99% for the four open-weight models.Source: §5.1 · p. 6
  • Removing the preliminary credibility question increased harmful execution by 15 percentage points on average, concentrated in credential/payment requests (+27 points versus +7 for logistics).Qwen3-14B remained near saturation; the results make rating-first evaluations potentially optimistic estimates of unprompted susceptibility.Source: §5.1 · p. 6

5.2 Generalization to Real-World Attacks

  • Harmful execution did not differ significantly between generated notices and fact-checked or smishing attacks, though this does not establish equivalence between attack populations.Four open-weight models executed 65%–99% of real attacks they rated ≤3; only 2.1% of false-rated claims in the broader archive directly requested action.Source: §5.2 · p. 6

5.3 What Drives Harmful Execution?

  • Removing direct action requests reduced harmful execution to 54% of its original level, while adding scam cues lowered credibility by 0.96 points but harmful execution by only 4.8 points.Legitimate execution also declined when direct requests were removed, and harmful execution among rewritten notices rated 1 or 2 remained 64.6%.Source: §5.3 · p. 6

6 RQ3: From Credibility Interventions to Behavioral Resilience

The study assesses defenses across credibility discrimination, harmful execution, and legitimate task completion. Forewarnings offer partial behavioral protection, generated-example prebunks show no consistent advantage, and fine-tuning results vary by model and evaluation protocol.

6.1.1 Credibility Reduction vs. General Skepticism

  • Generated exemplars lowered generated-item credibility on two victims, but forewarning achieved comparable or greater reductions and exemplar conditions could also lower genuine-item ratings.An unrelated-message control narrowed Gemma’s genuine–generated gap from +0.61 to +0.50; judge-rejected exemplars yielded discernment gaps at least as large as accepted ones across all three victims.Source: §6.1.1 · p. 7

6.1.2 Discrimination vs. Response Bias

  • Most interventions shifted response bias more than discrimination: on LLaMA, c rose +0.28 to +0.33 while AUC changed by less than 0.005.Pipeline seeding reduced Gemma’s AUC by 0.072, while exemplar-free skepticism produced the largest marked GPT-OSS AUC improvement (+0.094); generated-example prebunking showed no consistent discrimination advantage.Source: §6.1.2 · p. 7

6.2 Do Credibility Interventions Prevent Harmful Actions?

  • The forewarning reduced harmful execution by 8–21 percentage points on four of five victims, while neither generated-example prebunk outperformed it on any victim.The effect was not significant for Qwen3-14B; the forewarning outperformed both prebunks on LLaMA and the report-seeded prebunk on Gemini.Source: §6.2 · p. 7
  • All three interventions improved credibility discrimination on actionable notices, but those gains did not consistently produce proportional behavioral protection.Discrimination increased by ∆AUC = +0.02 to +0.20, including on Qwen3-14B, whose harmful execution barely changed.Source: §6.2 · pp. 7–8

6.3 Can Training Improve Behavioral Resilience?

  • With credibility elicited first, defensive SFT reduced harmful execution versus training-matched controls by 0.699 on LLaMA and 0.832 on Gemma.The evaluation covered held-out real-world attacks; the table reports proportions of evaluated items and averages control and defended results over three training runs.Source: Table 3 · p. 8, §6.3 · p. 8

6.2 Do Credibility Interventions Prevent Harmful Actions?

  • Forewarning also reduced legitimate execution from 0.89 to 0.48 on Gemini and 0.96 to 0.78 on LLaMA, so protection came with task-completion costs.Overall, the forewarning provided measurable but incomplete protection, and the costs should be weighed against reduced harmful execution.Source: §6.2 · p. 8

6.3 Can Training Improve Behavioral Resilience?

  • SFT’s utility costs differed by model: legitimate execution fell from 0.995 to 0.966 on LLaMA and from 0.993 to 0.839 on Gemma, which was protective but costly under predefined criteria.The LLaMA result was unstable across runs; a run-level 95% interval for the defended-minus-control difference was [−0.623, 0.084], including zero.Source: §6.3 · p. 8

7 Conclusion and Future Work

The paper finds that LLMs can act on false notices they explicitly distrust, and that asking for credibility ratings changes execution. Its simulated evaluations show partial protection from forewarning, no added behavioral benefit from the tested prebunks, and model- and protocol-dependent fine-tuning outcomes.

7 Conclusion and Future Work

  • LLMs can act on false notices they explicitly distrust, and eliciting credibility itself changes execution.The conclusion reports that forewarning provides partial protection, while the tested generated-example prebunks do not improve on it in the main evaluation.Source: §7 · p. 8
  • Fine-tuning reduced harmful execution on held-out archived attacks under the rating-first protocol, but protection and utility costs depended on the model and protocol.The paper calls for evaluating both harmful actions and legitimate task completion and for testing protection across unfamiliar tools, longer interactions, and adaptive attacks.Source: §7 · p. 8
  • The authors propose connecting uncertainty to verification or restraint before consequential actions as a direction for future work.Source: §7 · p. 8

Limitations

The authors bound their findings to controlled, simulated tool-use evaluations and note limits in stimulus factuality, scenario comparability, and defense robustness.

Evaluation Scope and Generalizability

  • The experiments cover multiple models and domains but not the full diversity of languages, cultures, architectures, or deployment environments, and they do not measure human belief or real-world consequences.Extending the findings to people or deployed agents requires separate evaluation.Source: Limitations · p. 9

Factuality and Contamination Control

  • Accepted generated items were predominantly true or unverifiable, so RQ1 measures separation between generated and genuine content rather than detection of verified falsehoods.The actionable notices used for behavioral evaluation were false by construction and separately validated.Source: Limitations · p. 9

Behavioral Evaluation

  • Behavioral tests use simulated actions and one sampled response per condition; topic and action-type differences constrain causal interpretation of scenario comparisons.Some logistics claims cannot be independently verified from information available to victim models, and the evaluation does not distinguish warned actions from unqualified execution.Source: Limitations · p. 9

Defense Robustness

  • Defense tests measure immediate effects, not persistence under extended interactions or adaptive attacks; fine-tuning covers two models and fixed tool actions without assessing general capabilities.Broader testing is needed to establish robustness and transferability.Source: Limitations · p. 9

Ethics Statement

The appendix documents research safeguards, generation and judging procedures, and analyses of generated and archived misinformation. Human ratings only weakly support judge acceptance as a trustworthiness filter, while models distinguish archived misinformation more strongly than generated content.

Ethics Statement

  • The study used controlled misinformation generation, credibility assessment, and simulated tool-mediated actions; post-cutoff events were chosen to reduce reliance on memorized fact checks.Research disclosures state that generated misinformation was not publicly disseminated or used in user-facing systems, and all tool interactions were simulated without external actions or affected real users.Source: Ethics Statement · p. 9

B Generator-Level Analyses

  • Across 10,794 generated candidates, accepted generations’ mean judge-assigned credibility ranged from 0.79 to 0.86, and 77.44% of candidates were judged likely to spread.Figure B1 displays credibility-bin counts, shareability percentages, and acceptance rates by topic.Source: B Generator-Level Analyses · p. 12, Figure B1 · p. 15

B.1 Effects of Prompting Strategy

  • Naive prompting and Chain-of-Thought produced similar credibility scores, while example-based prompting had the lowest mean credibility and CoT outputs had lower shareability.Acceptance rates were broadly similar, with slightly lower acceptance under CoT; the authors used naive prompting in the main experiments.Source: §B.1 · p. 12
  • Stronger adversarial framing weakly increased acceptance while monotonically decreasing mean credibility, a trade-off that led the authors not to add further adversarial messages to the main experiments.The paper reports generation as robust across prompt variants rather than driven by a single formulation.Source: §B.1 · pp. 12–13

B.2 Lexical Markers of Deceptive Generations

  • Misinformation-requesting generations used vague quantifiers, unspecified groups, and time-defocused phrases 0.90–1.68 times per paragraph, with p < .001 in every comparison for both models.Mean word length also decreased by 0.18–1.08 characters in 9 of 10 comparisons; this descriptive pattern does not establish greater deceptiveness or spread.Source: §B.2 · p. 13

C.1 Human Trustworthiness Judgments

  • Human raters preferred judge-accepted items only weakly: mean ratings were 3.237 versus 2.950 for rejected items, a 0.287 difference (p = .038; AUC = 0.594).Across 180 items, pooled mean trustworthiness was 3.142; agreement was moderate, and raters differed in scale use, so perceived trustworthiness is treated as noisy.Source: §C.1 · pp. 13–14
  • The judge accepted 29 of 46 judge-accepted items when annotators agreed on a winner, versus 17 judge-rejected items; this 63% preference did not reach significance (p = .10).The annotators agreed on 174 of 200 pairs (87.0%; κ = 0.786).Source: §C.1 · p. 14

D Real-World Comparison and Frame Recurrence

  • Archived misinformation shows larger credibility-discrimination gaps than generated content: +2.18 for LLaMA-3.1-8B-Instruct and +2.09 for Gemma-2-27B-IT, versus +0.38 and +0.61 on generated content.GPT-4o-mini’s archived-claim gap is +2.43; the comparison supports weaker discrimination on contamination-controlled synthetic stimuli, not that one content population is objectively more deceptive.Source: D Real-World Comparison and Frame… · pp. 14–15

D.1 Structural Recurrence and the Unit of Alignment

The analysis finds that specific false claims rarely recur across years, while some structural frames do; the generated pipeline’s institutional-announcement register is among the frames that do not meet the stability criterion. The register instrument is lexical, and its measured directive rate is zero.

D.1 Structural Recurrence and the Unit of Alignment

  • Only 1%–5% of 2024 claims have a 2023 same-topic neighbour at cosine similarity ≥0.30, making claim-level recurrence a poor way to assess generation.Specific false claims do not recur across the year boundary; claim-level searches would therefore yield near-zero matches for any generator.Source: §D.1 · p. 15
  • Seven of ten hand-specified frames meet the prespecified stability criterion; the two most frequent shift from 12.3% to 14.9% and 4.9% to 5.7%.Table D1 compares frame prevalence in 2023 and 2024 fact-checked claims.Source: §D.1 · p. 16, Table D1 · p. 16
  • The ten frames cover only 23.4% of 2023 claims and 28.2% of 2024 claims, so their measured stability does not establish exhaustiveness.The three frames failing the criterion are institutional-announcement frames that the pipeline predominantly produces, a scope mismatch the authors report rather than resolve.Source: §D.1 · p. 16, Table D1 · p. 16

E Register Instrument

  • Across 8,201 accepted generations, no second-person address appears in 85.9%–97.1% of items, depending on generator, and in 80.8%–100% by topic.The instrument measures register features in the generated corpus.Source: E Register Instrument · p. 16
  • The register scorer labels items using lexical markers for address, inducement, caution, reportage, and imperatives; it flags directive and debunking patterns.The instrument is conservative, but the measured directive rate across the accepted corpus is 0.0000.Source: E Register Instrument · pp. 16–17

F Details of the Belief–Action Study

The appendix details construction and checks for the action-level study, then tests archived attacks and mechanisms that can affect harmful execution. Its inoculation messages use a forewarning, weakened-dose exemplar, and active refutation, while the warning-only baseline retains only the forewarning.

F Details of the Belief–Action Study

  • The constructed study used 196 false notices paired with genuine counterparts; annotators’ checks met prespecified thresholds, and their agreement on harmful-action scoring was κ = 0.93.After adjudication, 48 of 50 sampled items were judged false, actionable, and paired with a harmless genuine counterpart; two named a real organization.Source: F Details of the Belief–Action Study · p. 17

G Real Attacks and Mechanism Tests

  • On the 30 archived fact-checked attacks with concrete instructions, harmful execution averaged 0.79, versus 0.72 on generated notices.The archived set contained 104 fact-checked claims and 100 smishing attacks; the stimulus populations differ in source, topic mix, length, and named-entity realism.Source: G Real Attacks and Mechanism Tests · p. 18
  • Forwarding attack details without an explicit request left harmful execution at 0.540 of the explicit-request level (95% CI [0.506, 0.573]); permitting decline lowered harm by 0.33 and also lowered legitimate execution.The forwarding comparison uses 391 items valid in both conditions; the decline-permission condition also reduced legitimate execution.Source: G Real Attacks and Mechanism Tests · p. 18
  • Adding scam cues lowered credibility ratings, but the associated change in harmful execution was small, at 0.084 or less across victims.The rewrite retained the claims, entities, numbers, links, and requested actions; action followed lowered ratings significantly for three victims, was insensitive for one, and was inconclusive for one.Source: G Real Attacks and Mechanism Tests · p. 18

H Inoculation Message Templates

  • Generated inoculation messages combine a forewarning, weakened-dose exemplar, and active refutation; the warning-only baseline keeps only the forewarning.Only misinformation seeds are used to generate these messages.Source: H Inoculation Message Templates · p. 19

I Prebunking Configurations: Full Credibility Results

The evaluation items are kept disjoint from intervention exemplars and fine-tuning data. For cross-topic inoculation, off-topic seeds do not show a consistent credibility penalty, but these results do not establish behavioral transfer.

I Prebunking Configurations: Full Credibility Results

  • Evaluation items are held out from all intervention exemplars and fine-tuning data, including the action-study prebunk seeds and the training notices.Real attacks used to test fine-tuned models come from fact-check archives and a smishing corpus, not the generation pipeline.Source: I Prebunking Configurations: Full… · p. 20

J Cross-Topic Inoculation Transfer

  • Across victim models and credibility measures, off-topic seeds perform comparably to or better than matched-topic seeds in several source–target pairs, with no consistent off-topic penalty.Effect sizes vary by source–target pair and victim model.Source: J Cross-Topic Inoculation Transfer · p. 20
  • Cross-topic seed results concern credibility effects and do not establish that inoculation transfers behaviorally.The transfer test compares changes in genuine-information credibility, misinformation credibility, and discernment against the pre-inoculation baseline.Source: J Cross-Topic Inoculation Transfer · p. 20

K Training Against Generated Misinformation: Details

The appendix describes fine-tuning data and protocols, then reports an earlier action-level study retained as supporting evidence. The accepted generated corpus itself does not support a usable action-level measurement, so the earlier study uses instruction-bearing stimuli.

K Training Against Generated Misinformation: Details

  • Fine-tuning targets a no-action reply for generated misinformation and the requested action for genuine counterparts; it trains on replies, not trust ratings or user turns.The 249 retained training items were selected from generations held out from the 196 evaluation notices.Source: K Training Against Generated… · p. 20
  • Tuning on genuine requests alone increased harmful action with the rating question, from 0.775 to 0.972 for Llama and 0.868 to 0.987 for Gemma.The passage reports this as consistent with prior work on benign agentic fine-tuning.Source: K Training Against Generated… · p. 20
  • Adding the report-style prebunk to the defended model lowered harmful action to 0.129 on Llama-3.1-8B-Instruct and 0.074 on Gemma-2-9B-IT, while correct genuine action fell to 0.760 and 0.716.Without training, the same prebunk changed harmful action from 0.775 to 0.564 on Llama and 0.868 to 0.853 on Gemma; these are one-run results.Source: K Training Against Generated… · p. 21

L Earlier Action-Level Study on Constructed Items

  • The earlier action-level study is supporting evidence, not the primary behavioral result; it used twenty author-written instruction-bearing items and human-graded replies.The larger preregistered tool-execution evaluation replaces it as the paper’s primary action evidence.Source: L Earlier Action-Level Study on… · pp. 21–22

L.1 Protocol

  • The protocol rates a claim, then presents an ordinary request for practical help on its premise; the outcome collapses endorsement and harmful directives into acting on the false claim.Collapsing grades produced κ = 0.933 agreement, compared with κ = 0.536 on the full three-point scale.Source: §L.1 · pp. 22–23

L.2 The Generated Corpus Does Not Support an Action-Level Test

  • The accepted generated corpus yields no usable action-level test: 94.1% of items lack second-person address, and none issues a concrete instruction to the reader.Many items are cautionary and largely accurate, so they present no victim action to measure; the authors describe this as a constraint on automated misinformation generation.Source: §L.2 · p. 23

L.3 Instruction-Bearing Stimuli

  • In the twenty-item study, prebunking reduced acting on false claims from 18 to 13 trials for Llama and 14 to 8 for Gemma; the pooled odds ratio was 3.97 (95% CI [1.40, 11.24], p = .0082).Neither victim’s individual result reached significance; the declared primary analysis pooled across victims.Source: §L.3 · p. 23
  • For Gemma, prebunking reduced action even when misinformation credibility rose from 3.95 to 4.05, showing that the belief measure could miss a behavioral change.Genuine-item credibility was unchanged under prebunking for both victims, while the two interventions moved Gemma’s misinformation credibility in opposite directions despite reducing action under both.Source: §L.3 · pp. 23–24

L.4 Executed Actions Under Tool Access

The tool-access study finds substantial variation across victims in whether prebunking improves on a bare forewarning, with some interventions also reducing legitimate actions. The earlier study and annotation protocol impose limits on how broadly its action-level results can be interpreted.

L.4 Executed Actions Under Tool Access

  • The study counted a harmful action only when a materially complete, predeclared loss-bearing tool call carried the harmful content, and paired each claim with a legitimate request.A length-matched unrelated message controlled for message length; the conditions also included forewarning and two prebunks.Source: §L.4 · p. 24
  • Harmful execution fell from 0.77 without intervention to 0.44 with forewarning, 0.23 with worked-example prebunk, and 0.38 with pipeline-seeded prebunk on Gemma-2-27B-IT.Legitimate execution remained undisturbed in that study; the pipeline-seeded message did not outperform the forewarning (0.38 versus 0.44).Source: §L.4 · pp. 24–25
  • On Gemma-2-27B-IT, pipeline seeding lowered credibility AUC while bare forewarning raised it, even as the pipeline-seeded message sharply reduced action on constructed requests.The analyses assess different stimuli: third-person pipeline claims versus constructed, action-requesting items.Source: §L.4 · p. 25

L.5 Constraints of the Earlier Study

  • The earlier action study used twenty trials per cell, constructed stimuli, differing victim designs, and uneven human grading; its plan was fixed after data collection.Only Gemma-2-27B received the bare forewarning, and the authors report estimates under alternative grading configurations rather than claiming preregistration.Source: §L.5 · pp. 25–26

L.4 Executed Actions Under Tool Access

  • Across three victims, the worked-example prebunk beat forewarning only on Gemma-2-27B-IT (0.21), tied on Gemma-2-9B-IT (0.02), and lost on Llama-3.1-8B-Instruct (forewarning advantage 0.12).The pipeline-seeded prebunk beat the forewarning on no victim; on Llama-3.1-8B-Instruct, all interventions also significantly reduced legitimate execution.Source: §L.5 · p. 26

L.5 Constraints of the Earlier Study

  • The earlier tool-action score could miss harm routed through undeclared tools, while several items were already at baseline floors because victims refused credential transmission.These constraints affect how action rates and intervention reductions should be read.Source: §L.5 · p. 26

M Harm Annotation Protocol

  • For Gemma-2-27B, annotator agreement was κ = 0.933 on the binary action outcome, versus κ = 0.536 on the three-point grading scale.Disagreements were adjudicated by a third annotator; no human-agreement estimate was available for LLaMA-3.1-8B-Instruct.Source: M Harm Annotation Protocol · p. 27
  • The primary contrast’s direction and approximate magnitude remained stable across human grading configurations; only the most lenient automated pass was nonsignificant.The authors report all configurations because the analysis plan was fixed after the data were seen.Source: Table M1 · p. 27, M Harm Annotation Protocol · p. 27

Related papers

Loading 2610.33962v1…