PaperlayerView exact revision on arXiv

Source-linked AI summary

The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation

Rongwu Xu, Brian S. Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, Han Qiu

arXiv:2312.09085v5cs.CLcs.AIcs.CRcs.CY

TL;DR

Prior work on LLM susceptibility to misinformation mostly used one-turn settings, leaving persuasive multi-turn conversations less studied. The authors curate Farm and track belief changes during such conversations; they report that even initially correct beliefs can be manipulated, including for ChatGPT and GPT-4.

  • Problem

    Prior work mostly focused on one-turn settings, while belief can change through conversation and persuasion; the study asks whether persuasion makes LLMs susceptible on factual questions they already answer correctly.

    Source: §1 · pp. 1–2

  • Method

    The authors curate Farm with factual questions and generated persuasive misinformation, then use a testing framework to track belief from an initial check through dialogue to a final check.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    ChatGPT’s beliefs can be altered by 50.1%, and GPT-4’s by 20.7% on Farm.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that initially correct beliefs can be manipulated, with repetition and rhetorical appeals particularly potent.

    Source: §8 · p. 9

    Farm may have limited size and topic coverage, and the findings’ generalizability may be constrained by testing five models only once each.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs' susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs' belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs' correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

1 Introduction

Prior work mostly examined one-turn susceptibility, while this study investigates whether persuasion can shift LLM answers to factual questions they already answer correctly. It introduces the Farm dataset and a framework for tracking belief change during persuasive conversations.

1 Introduction

  • The study targets susceptibility on straightforward factual questions that LLMs can already answer correctly, extending beyond the mostly one-turn focus of prior work.Source: §1 · pp. 1–2
  • The authors build a framework to track LLM belief change during conversational misinformation and curate Farm with factual questions paired with systematically generated misinformation.Source: §1 · p. 2
  • ChatGPT’s beliefs can be altered by 50.1%, and GPT-4’s by 20.7% on Farm.Source: §1 · p. 2

2 Curation of Farm

Farm is curated in stages: the authors select and format factual questions, generate persuasive misinformation for them, and filter unsuccessful instances through human validation.

2 Curation of Farm

  • Farm’s curation has two stages: selecting straightforward factual questions and generating persuasive misinformation for later testing.Source: §2 · p. 2

2.1 Questions Curation

  • 1,500 questions—500 each from BoolQ, Natural Questions, and TruthfulQA—are selected from questions GPT-4 answers correctly, then formatted as MCQs with a “don’t know” option.For each model, only questions it initially answers correctly are considered; question subsets can differ by model.Source: §2.1 · p. 2

2.2 Misinformation Generation

  • For each question, the authors generate an incorrect control statement and persuasive messages using logical, credibility, or emotional appeals, then evaluate appeal entailment and strategy alignment.Three unique messages are generated for each appeal type and control statement; GPT-4 evaluates the two validation tasks.Source: §2.2 · pp. 2–3

2.3 Human Validation

  • After human validation, two authors filter 48 unsuccessful generations, leaving a finalized dataset of 1,952 entries.Source: §2.3 · p. 3

3 The Misinformation Test Procedure

The test follows models through an initial belief check, a multi-turn conversation containing misinformation, and a final check. It probes beliefs separately from the visible conversation and continues only for questions the model initially answered correctly.

3 The Misinformation Test Procedure

  • Belief checks use Farm questions in a closed-book question-answering process to probe the knowledge held by the model.Source: §3 · p. 3
  • Only questions answered correctly at the initial belief check proceed to persuasion; selecting any option other than “don’t know” counts as holding a belief.“Don’t know” is treated as abstention.Source: §3 · p. 4
  • At each conversation turn, an implicit belief check is kept out of the chat history, and the conversation continues for up to four turns if the model retains its original belief.If an appeal fails entailment or strategy-alignment checks, CTRL is used instead.Source: §3 · p. 4
  • The procedure ends with a final belief check that classifies the model as misinformed, retaining its belief, or abstaining.Source: §3 · p. 4

4 Experiments

Experiments test five closed- and open-source models using accuracy and misinformation-rate measures, then examine susceptibility across turns, strategies, and confidence levels. The reported findings include broad belief alteration, variation by model and strategy, and shifting confidence during persuasion.

4.1 Target LLMs

  • Experiments cover ChatGPT, GPT-4, Llama2-7B-chat, Vicuna-v1.5-7B, and Vicuna-v1.5-13B.The study reports the most important results in this section and says appendix results align with its findings.Source: §4.1 · p. 4, §4 · p. 4

4.2 Evaluation Metrics

  • ACC@n tracks average accuracy and MR@n average misinformation across questions; robustness is 100 − MR@4, and knowledge is ACC@0.Source: §4.2 · pp. 4–5

4.3 Main Results and Findings

  • 4.1%–63.4% of models’ beliefs changed after the simplest control statement, rising to 20.7%–78.2% cumulatively by turn four; GPT-4 was most resistant, while Llama-2-7B-chat was most susceptible.GPT-4’s MR doubled after three additional repetitions on NQ2; rhetorical appeals raised MR@4 in most instances, and logical appeals generally had the highest rates, with a few credibility-appeal exceptions.Source: §4.3 · pp. 5–6

4.4 Implications on Model Confidence

  • Models retaining their belief or abstaining tended to begin with higher confidence than those misinformed, while confidence among retainers shifted lower after one misinformation turn.After four turns, confidence distributions spread toward both lower and higher levels; the paper reports this pattern for Llama2 and Vicuna.Source: §4.4 · p. 6

5 Behavior Study

The study groups LLM responses to misinformation into five behaviors, distinguishing belief-preserving sycophancy from actual belief change and excluding self-inconsistency from Figure 4 because it is mainly linked to input-processing issues.

5 Behavior Study

  • Five behaviors are identified—rejection, sycophancy, uncertainty, acceptance, and self-inconsistency—with self-inconsistency omitted from Figure 4 because it mainly reflects input-processing issues.The study reports frequencies for four of the five behaviors in ChatGPT responses.Source: §5 · pp. 6–7, Table 7 · p. 7
  • Figure 4 relates susceptibility to response behavior and initial-belief confidence, while Figure 3 depicts initial confidence across outcomes and confidence shifts among retained-belief samples.The captions identify these relationships but do not provide the plotted values.Source: Figure 4 · p. 7, Figure 3 · p. 7
  • 26.1%–48.1% of ChatGPT persuasive conversations included sycophancy, in which the response aligns with misinformation while the belief check shows no belief change.Sycophancy often serves as an interim stage before the LLM ultimately succumbs to misinformation.Source: §5 · p. 7
  • Rejection involves countering misinformation, including debunking; uncertainty can precede being misled, while acceptance means the LLM is misinformed immediately.LLMs show higher confidence when providing evidence to support their beliefs in response to misinformation.Source: §5 · p. 7

6 Discussion of Possible Mitigation

The authors test a lightweight system-prompt reminder for ChatGPT, which significantly reduces misinformation’s impact, though they report remaining headroom for improvement.

6 Discussion of Possible Mitigation

  • The reminder asks the model to be cautious with potentially malicious users and verify memorized knowledge before responding.The authors propose inserting it after misinformation is detected, potentially using another LLM.Source: §6 · p. 7
  • The system prompt significantly reduces the impact of misinformation on ChatGPT, with MR@1 and MR@4 averaged across four persuasive strategies.Figure 5 presents these mitigation results.Source: §6 · p. 7, Figure 5 · p. 8
  • The prompt leaves substantial room for improvement; the authors suggest training or fine-tuning as possible avenues for better mitigation.They state that their study does not specifically address mitigation.Source: §6 · p. 8

7 Related Work

Related work covers factuality and hallucination, misinformation, knowledge conflicts, sycophancy, and interactive testing; this study distinguishes itself by testing factual knowledge under persuasive conversation.

7 Related Work

  • Closed-book QA probes factual knowledge using only the question, unlike open QA, which can use external references.Prior work also investigates querying and estimating models’ internalized factual knowledge.Source: LLM’s Factuality, Hallucination, and… · p. 8
  • Hallucination research addresses detection, evaluation, investigation, and mitigation, while misinformation work has focused mainly on detection or generation.The paper positions its work as intentionally inducing hallucination to assess alignment with internal knowledge and robustness to misinformation.Source: LLM’s Factuality, Hallucination, and… · p. 8
  • Prior knowledge-conflict work finds models receptive to coherent external evidence conflicting with parametric memory, while other work encourages following user-provided context assumed correct.This study delivers misinformation through conversation rather than task-description alteration.Source: LLM’s Factuality, Hallucination, and… · p. 8
  • Unlike sycophancy studies of subjective topics or math reasoning, this work tests factual knowledge and finds that sycophantic responses do not necessarily mean beliefs changed.The paper also contrasts its persuasive factuality focus with interactive reasoning and debate-based testing.Source: LLM’s Factuality, Hallucination, and… · pp. 8–9

8 Conclusion

The paper introduces Farm and a multi-turn test of persuasive misinformation, finding that LLMs—including GPT-4—are susceptible and that repetition and rhetorical appeals are particularly potent.

8 Conclusion

  • Farm pairs straightforward factual questions with misinformation generated through persuasive strategies, and the study tests LLMs in multi-turn conversations.The study identifies common response behaviors to inform future mitigation work.Source: §8 · p. 9
  • LLMs, including GPT-4, show pronounced susceptibility to persuasive misinformation, and initially correct beliefs can be manipulated.The conclusion characterizes LLM robustness as lacking when models confront misinformation.Source: §8 · p. 9
  • Repetition and rhetorical appeals are particularly potent among the tested persuasive strategies.The conclusion identifies these strategies as especially likely to lead models astray.Source: §8 · p. 9

Limitations

The study’s findings are bounded by dataset and model coverage, experimental design, and limited interpretability of the observed behavior.

Limitations

  • Farm may not cover enough topics, and testing five mainstream models only once limits how broadly the results may generalize.The authors also note that the rhetorical appeals may not encompass all misinformation strategies.Source: Limitations · p. 9
  • Belief checks use question answering, which may not reveal models’ deeper thoughts despite the study’s effort to separate behavior from belief.The authors identify representation engineering as an alternative perspective on belief checking.Source: Limitations · p. 9
  • The study is largely empirical and does not extensively investigate mechanisms of belief formation or misinformed behavior.The authors cite limited reasoning abilities and noisy training data as possible contributing factors.Source: Limitations · p. 9

Ethics Statement

Farm and its persuasive-appeal generation method could be misused, while the paper also describes potential risks from misinformation-sensitive LLM applications. The authors note that their generated misinformation primarily concerns trivial questions that humans can easily identify.

Ethics Statement

  • Farm’s factual misinformation could be misused in model training or fine-tuning, potentially spreading false or toxic information.The authors made it publicly available for researchers to study mitigation methods.Source: Ethics Statement · p. 9
  • Generated misinformation primarily concerns trivial questions that humans can easily identify, limiting its potential impact.The authors also describe proposing a mitigation strategy aimed at reducing objectionable effects in LLMs.Source: Ethics Statement · p. 9
  • The prompting method for generating human-like persuasive misinformation also carries potential for harmful misuse.The authors recommend extra caution and ethical consideration.Source: Ethics Statement · p. 9

A.1 Impacts on AI Safety

  • Persuasive misinformation in conversation could pose risks in safety-sensitive settings such as medical diagnostics, education, and juridical processes.The paper describes a hypothetical suicide-related conversation with an LLM-based voice assistant.Source: §A.1 · p. 15
  • Even state-of-the-art LLMs’ difficulty holding beliefs on simple facts may pose risks for their deployment and utilization.The authors call for rigorous scrutiny and ethical considerations.Source: §A.1 · p. 15

A.2 NQ1 vs. NQ2, Difference and Why?

The appendix contrasts two forms of misinformation and discusses their different outcomes, then addresses how the test differs from in-context learning and describes dataset construction and evaluation details.

A.2 NQ1 vs. NQ2, Difference and Why?

  • Most models were less likely to be misinformed by NQ2 than NQ1, and GPT-4’s rejection rate was 3× higher on NQ2.NQ1 denies an established fact, while NQ2 makes a new unsupported false assertion; the authors hypothesize that NQ2’s specific wrong answer may have a higher prior probability. NQ1 more often led to abstention, while successful NQ2 misinformation more often shifted answers to a wrong fact.Source: §A.2 · pp. 15–16, Figure 6 · p. 16

A.3 A Mission Impossible?

  • Unlike explicit ICL demonstrations, the misinformation test lets models respond freely before belief checks; ICL produced much higher MR than the test, even at 10 turns.The authors also report that larger models were less swayed, suggesting robustness and context updating need not conflict.Source: Figure 7 · p. 16, §A.3 · pp. 16–17

A.4 Future Direction and Potential Impact in the Context of AI Agents

  • The experiment uses static misinformation generated by GPT-4; a limited test found ChatGPT-generated appeals slightly less effective, while dynamic generation remains a future direction.The authors report difficulty accessing a genuine malicious, uncensored LLM for dynamic tests; models they tried were difficult to manipulate without extensive jailbreak prompts.Source: §A.4 · p. 17

B.1 Generate Rhetorical Appeals (Cont’d)

  • The generation procedure uses rhetorical appeals—logical, credibility, and emotional—and prompts GPT-4 to produce lengthy misinformation passages.A DAN system prompt and an instruction to begin affirmatively helped avoid refusals; without the extra instruction, GPT-4 had an approximately 80% probability of verbatim refusal.Source: §B.1 · p. 18

B.2 Semantic Topics Information of Farm

  • Farm’s sampled BoolQ and NQ topics include shared general trivia, while TruthfulQA tends toward nuanced facts about misconceptions and misbeliefs.The topic visualization also shows several densely packed topic clusters across the datasets.Source: §B.2 · p. 18

B.3 Samples from Farm

  • Each illustrated Farm sample contains a target, a control statement, and nine appeal passages: three each for logical, credibility, and emotional strategies.Source: §B.3 · p. 18

B.4 Original Prompts

  • The paper lists the original prompts used to generate and evaluate Farm in Table 10.Source: §B.4 · p. 18

B.5 Evaluation of Farm (Cont’d)

  • Larger LLMs achieved higher accuracy than smaller NLI models in evaluating appeals, aligning more closely with human judgments, especially for lengthy, complex passages.Appeals averaged more than 200 words; the authors also manually sampled 50 successful cases and all failures to check model-based metrics, and report that LLMs can generate human-like rhetorical appeals containing misinformation.Source: §B.5 · p. 18

B.6 Human Validation of Farm

This section reports human validation and filtering of Farm, alongside analyses of how confidence, topic, response behavior, and persuasive tone relate to susceptibility to misinformation. In ChatGPT’s BoolQ test, weakening the persuasive tone unexpectedly increased the proportion of questions on which the model was misinformed.

B.6 Human Validation of Farm

  • Across all three Farm validation tasks, positive cases exceeded 99%, and Kappa scores ranged from 0.41 to 0.6; after filtering generation failures, the curated dataset contained 1,952 entries (97.6% of original volume).Reported failures included MCQ options with an indistinct “both” choice, CTRL statements mishandling negation, and credibility appeals that invoked GPT-4 as an AI rather than authoritative experts or institutions.Source: §B.6 · pp. 21–22

C.1 Main Results (Cont’d)

  • The open-source model results are presented using ACC@n and MR@n, with selected values for all LLMs collected for comparison.The supplied passages identify Figure 13 and Table 15 as the displays but do not provide the comparison values.Source: §C.1 · p. 29, Figure 13 · p. 30, Table 15 · p. 31

C.2 Model Confidence (Cont’d)

  • Misinformed samples tended toward lower confidence, while comparing turns 1 and 4 showed a backfire effect among samples with confidence in [0.99, 1.0].For successfully misinformed answers, confidence was more evenly distributed than initial confidence; Vicuna showed lower confidence for repetition than appeals, unlike Llama-2, which showed no notable strategy-based variance.Source: §C.2 · p. 29

C.3 Breakdown of Results Based on Topics

  • BoolQ was especially vulnerable on Dollar Companies, Legality of Actions, and Movies, TV Shows, and Stories, while NQ2 susceptibility remained high across most topics.NQ1 and TruthfulQA were more resistant overall, though Medicine, Drugs, and Health and Cities and Locations had higher misinformed ratios.Source: §C.3 · p. 29
  • Vicuna was more susceptible on Dollar Companies, Comparing Similar Items, and Movie and TV Show Trivia; emotional appeals worked better on US Presidency and Politics questions.ChatGPT and GPT-4 were more susceptible on Dollar Companies and Comparing Similar Items, and no single topic posed an exceptionally difficult challenge for misleading the models.Source: §C.3 · p. 29

C.4 Details of Identification of LLM’s Behaviors

  • The behavior-identification procedure classifies responses as acceptance, sycophancy, rejection, or uncertainty using prompts evaluated by ChatGPT.Acceptance and rejection align agreement with the implicit belief check, whereas sycophancy separates agreement from that check; uncertainty means the model neither agrees nor states its belief.Source: §C.4 · pp. 29–30

C.5 Supplemental Results on the Relationship Between an LLM’s Susceptibility to Misinformation, Its Behavior, and Its Initial Belief

  • Rejection accompanied unsuccessful persuasion in 74.3%–90.0% of conversations, while acceptance accompanied successful persuasion in 84.0%–98.4%.Uncertainty mostly accompanied unsuccessful persuasion for BoolQ, NQ2, and TruthfulQA, but not NQ1; sycophancy mostly accompanied successful persuasion outside BoolQ.Source: §C.5 · p. 31

C.6 Additional Experiment I: Weaken the Tone

  • The additional tone experiment tested misinformation phrased non-confrontationally and suggestively rather than assertively.Its prompt framed the speaker as potentially mistaken and respectful of the respondent’s viewpoint; no outcome is reported in the supplied passages.Source: §C.6 · pp. 31–32
  • ChatGPT was successfully misinformed on a higher proportion of BoolQ questions with a weakened tone.The results are shown in Figure 16.Source: §C.6 · p. 32

C.7 Additional Experiment II: Conversation after Misled

This subsection examines model responses after successful misinformation, including whether they explain a changed belief or promote the falsehood, and illustrates the behavior with an example conversation.

C.7 Additional Experiment II: Conversation after Misled

  • About 70% of the time, ChatGPT continued explaining its changed belief by restating the misinformation; in the remaining ∼30%, it corrected its prior response.When an appeal passage was used, ChatGPT often based its explanation on that passage, aligning with the CTRL statement.Source: §C.7 · p. 33
  • ∼60% of the time, the LLM abstained from generating promotional misinformation, while in the remaining ∼40% it produced persuasive content supporting it.The generation task requires retrieving memorized knowledge; the model often abstains when that knowledge contradicts the supplied context.Source: §C.7 · p. 33
  • The example conversation shows acceptance of the claim that staring at one’s reflection in a lake leads to death.The figure caption identifies it as a conversation following successful misinformation and notes that pre-misinformation transcripts are briefly presented.Source: §C.7 · p. 39, Figure 22 · p. 39

D Details on Dialogue Examples

The dialogue examples classify five responses to misinformation, from confident rejection to immediate acceptance, with several intermediate behaviors still vulnerable to later persuasion. The supplemental mitigation workflow uses an Inspector LLM and safe system prompts, with P4 reported as the most useful evaluated prompt but not optimal.

D Details on Dialogue Examples

  • The five behavior categories are rejection, sycophancy, uncertainty, acceptance, and self-inconsistency; rejection is hard to mislead, while acceptance changes stance and the three intermediary behaviors remain susceptible to later persuasion.Source: D Details on Dialogue Examples · p. 40
  • In the NEJM example, the model rejects misinformation and grows more confident, while sycophancy can agree outwardly yet preserve its original belief and uncertainty can answer “Don’t know.”Source: D Details on Dialogue Examples · p. 41, Figure 24 · p. 42, Figure 25 · p. 42
  • The examples distinguish immediate acceptance from self-inconsistency, in which the model is incoherent at the first misinformation turn but is misled by the fourth.Source: Figure 26 · p. 43, Figure 27 · p. 43

E Supplemental Details on Mitigation

Loading 2312.09085v5…