PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Risk of Misinformation Pollution with Large Language Models

Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, William Yang Wang

arXiv:2305.13661v2cs.CLcs.AI

TL;DR

LLMs may be misused to generate misinformation that harms downstream information systems. This paper simulates misinformation scenarios, finds substantial ODQA performance declines under deliberate pollution, and examines mitigation strategies.

  • Problem

    The potential misuse of LLMs for misinformation generation and its consequences for ODQA have received limited attention, although deliberate misinformation can cause societal harm.

    Source: §1 · pp. 1–2

  • Method

    The paper simulates deliberate misinformation insertion into QA knowledge sources, examines effects on ODQA, and explores three partial mitigation strategies.

    Source: §7 · p. 9

  • Results

    Under deliberate misinformation pollution, ODQA performance dropped 14% to 54% for DPR-based models and 20% to 87% for BM25-based models.

    Source: §4.4 · p. 5

  • Takeaways & Limitations

    The findings establish misinformation pollution as a significant ODQA threat and a starting point for further work on misinformation-robust NLP applications.

    Source: §1 · p. 2

    The study focuses on a selection of emblematic QA systems and LLMs, so applicability beyond that selection is uncertain.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In this paper, we comprehensively investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation in the performance of ODQA systems. To mitigate the harm caused by LLM-generated misinformation, we explore three defense strategies: prompting, misinformation detection, and majority voting. While initial results show promising trends for these defensive strategies, much more work needs to be done to address the challenge of misinformation pollution. Our work highlights the need for further research and interdisciplinary collaboration to address LLM-generated misinformation and to promote responsible use of LLMs.

1 Introduction

The paper examines how LLMs can generate misinformation, how its spread may harm open-domain QA, and whether prompting, detection, and voting can mitigate that harm.

1 Introduction

  • The threat model covers hallucination-driven and maliciously targeted misinformation entering web corpora, tests downstream QA effects, and evaluates prompting, detection, and majority voting.The study asks whether intentional pollution can mislead QA systems into producing answers desired by an attacker.Source: Figure 1 · p. 1, §1 · p. 2
  • LLMs can automate credible-sounding misleading content for influence operations, risking public-opinion manipulation, confusion, and harmful ideologies.The introduction identifies these as potential societal harms of deliberate misinformation distribution.Source: §1 · p. 1
  • The paper addresses gaps in assessing LLM misinformation generation, its harms to information retrieval and question-answering, and mitigations for intentional pollution.These are the three research questions motivating the study.Source: §1 · pp. 1–2
  • LLMs are controllable misinformation generators, deliberate synthetic misinformation significantly degrades ODQA, and initial defenses show promise but remain inadequate.The authors conclude that the challenge requires further investigation.Source: §1 · p. 2
  • The paper presents a comprehensive analysis of misinformation pollution's consequences for ODQA and a starting point for misinformation-robust NLP research and interdisciplinary collaboration.It calls for continued research and collaboration to support responsible LLM use.Source: §1 · p. 2

2 Related Work

Prior work has studied detecting or curbing LLM-generated misinformation, but detection remains limited and safeguards can be bypassed. This paper extends the scope to realistic misinformation’s effects on QA systems and countermeasures.

Combating Model-Generated Misinformation

  • Detection methods for machine-generated text remain limited in precision and scope, while safeguards against harmful LLM outputs can be bypassed with jailbreak prompts.The cited safeguards aim to curtail harmful, biased, or baseless information.Source: Combating Model-Generated Misinformation · p. 2
  • Retrieval-augmented QA systems face data-pollution risks because their training data or retrieval corpus may mix factual and fabricated content.Earlier evaluations often used reliable external sources such as Wikipedia, a condition that may not hold amid rising machine-generated misinformation.Source: Combating Model-Generated Misinformation · p. 2
  • Unlike prior QA robustness work on synthetic perturbations such as entity replacements, this study examines threats from realistic misinformation generated by modern LLMs.The authors describe this as a pioneering exploration of misinformation’s potential threat to QA systems.Source: Combating Model-Generated Misinformation · p. 2

3 Generating Misinformation with LLMs

The section models misuse as fabricating a false article for a target question, then uses GPT-3.5 and four prompt settings to represent distinct misinformation-generation paths.

3 Generating Misinformation with LLMs

  • The threat scenario is an actor using a generator to fabricate a false article in response to a specific target question.The example is a counterfeit news article incorrectly claiming Trump won the 2020 US presidential election.Source: §3 · p. 3

3.1 GPT-3.5 as Misinformation Generator

  • The study uses GPT-3.5 (text-davinci-003), citing its ability to generate coherent, contextually appropriate text compared with prior Seq2Seq approaches that sometimes made grammar and commonsense errors.Prior approaches cited are Zellers et al. (2019), Fung et al. (2021), and Huang et al. (2022).Source: §3.1 · p. 3

3.2 Settings for Misinformation Generation

  • GENREAD prompts for an answer-oriented document that may hallucinate, while CTRLGEN steers generation toward a predetermined non-factual opinion.GENREAD represents inadvertent hallucination; CTRLGEN simulates disinformation or propaganda built around a fabricated fact.Source: §3.2 · p. 3
  • REVISE minimally changes a factual reference article to support a predetermined false opinion, while REIT rephrases misinformation to manipulate downstream QA systems.REIT need not produce text that appears realistic to humans if it can alter a QA system’s output.Source: §3.2 · pp. 3–4

4 Polluting ODQA with Misinformation

The study tests whether deliberately infused LLM-generated misinformation can manipulate ODQA systems into producing incorrect answers. It evaluates this risk across two datasets and multiple retriever-reader configurations.

4.1 Datasets

  • NQ-1500 samples 1,500 Natural Questions test items, while CovidNews contains 1,534 COVID-19 news questions.NQ-1500 uses the December 30, 2018 Wikipedia dump; CovidNews is adapted from StreamingQA and uses its news corpus plus the 2020 WMT English News Corpus.Source: §4.1 · pp. 4–5

4.2 ODQA Systems

  • The tested ODQA systems combine BM25 or DPR retrievers with FiD or GPT-3.5 readers.BM25 provides sparse retrieval and DPR dense retrieval; the readers are Fusion-in-Decoder and text-davinci-003.Source: §4.2 · p. 5

4.3 Misinformation Pollution

  • For each question, the researchers injected one generated fake document under each pollution setting and compared clean versus polluted corpora using Exact Match.Injected fake passages were small in scale relative to the original corpora.Source: §4.3 · p. 5

4.4 Main Results

  • Performance fell 14%–54% for DPR-based and 20%–87% for BM25-based systems under deliberate pollution; GENREAD reduced best-model DPR+FiD performance by 5% on NQ-1500 and 15% on CovidNews.The comparisons use Exact Match against CLEAN; the deliberate settings were CTRLGEN, REVISE, and REIT.Source: §4.4 · pp. 5–6, Table 2 · p. 6
  • REIT pollution was most effective, while fabricated passages were often retrieved and CovidNews suffered larger declines than NQ-1500.REIT floods machine readers with seemingly important evidence, distracting them from authentic information; retrievers favor lexical or semantic alignment without checking authenticity. The paper attributes CovidNews's greater vulnerability to thinner information resources and reports a larger GENREAD effect there, possibly reflecting GPT-3.5's Wikipedia-centric knowledge.Source: §4.4 · p. 6

5 Defense Strategies

The section evaluates misinformation detection, cautionary prompting, reader voting, and adding more context passages as defenses for ODQA. In-domain detectors perform strongly but out-of-domain detectors do not; voting outperforms prompting at added cost, while more contexts do little to mitigate performance decline.

Main Results

  • Detection was evaluated on test data sampled from NQ-1500, comparing training on its unsampled original portion in-domain with OOD training on existing GPT-generated Wikipedia-style text.An AUROC value of 50 means the classifier performs no better than random guessing.Source: Table 3 · p. 7

Detection Approach

  • The detector uses a RoBERTa binary classifier trained on the first 80% of NQ-1500 DPR top-10 contexts, with the remaining 20% reserved for testing.This produced 12,000 training and 3,000 testing instances; the authors also tested out-of-domain training using GPT-3 completions on Wikipedia topics.Source: §5 · p. 7

Vigilant Prompting

  • The prompting experiment used GPT-3.5 as the reader and added a warning that some retrieved passages might be designed to mislead.The study investigated whether LLM instruction-following could be used to evade misinformation.Source: §5 · p. 7

Main Results

  • Prompting outcomes were inconsistent, while voting consistently outperformed prompting but increased per-question costs linearly with the number of context passages.Multiple readers may process API calls concurrently, but the number of readers still makes voting financially challenging at large scale.Source: Table 4 · p. 7, §5 · pp. 7–8

Reader Ensemble

  • The divide-and-vote approach groups contexts by relevance, has a reader answer from each group, then selects the majority answer to limit any single source’s influence.Source: §5 · p. 7

Does Reading more Contexts help?

  • Increasing reader context size had minimal or counterproductive effects on relative EM drop, making context dilution ineffective against misinformation pollution.The study reports relative EM drop against clean-corpus EM rather than absolute EM.Source: §5 · p. 8

Summary

  • The explored defenses show promise through extra resources, but a cost-effective, resilient misinformation-resistant QA system still requires further research.Source: §5 · p. 8

6 Discussion

Although the study is simulation-based, it highlights practical risks of LLM-generated misinformation for the real-world web and downstream applications. The discussion considers information availability, attack costs, and the integrity of web-scale corpora.

Information Availability

  • Minimal input requirements and the relative ease of LLM generation make web misinformation pollution an imminent threat, warranting technological countermeasures and regulatory frameworks.REVISE needs a relevant real passage, while REIT requires only modest knowledge of the targeted system.Source: §6 · p. 8

Associated Costs

  • Producing one 200-word fake document costs $0.01 to $0.04 with text-davinci-003, substantially less than hiring human writers.The estimate covers the dominant experimental cost, OpenAI language-model API fees; other costs include training and maintenance, storage, and computing.Source: §6 · p. 8

Integrity of Web-scale Corpora

  • Web-scale corpora are vulnerable to misinformation because they contain inaccuracies, inconsistencies, and biases, and attackers can exploit URLs or contaminate even curated sources.The passage notes risks from hijacked expired domains and tampered URL content, unwanted data in Common Crawl, and misinformation slipping into Wikipedia (Kumar et al., 2016; Greenstein and Zhu, 2012; Carlini et al., 2023; Luccioni and Viviano, 2021).Source: §6 · pp. 8–9

7 Conclusion and Future Work

The study finds that synthetic misinformation substantially degrades knowledge-intensive QA systems, with intricately crafted falsehoods worsening performance, and proposes partial mitigations. Future work should assess broader application risks, investigate calculated misinformation, and develop QA systems resistant to misinformation.

Conclusion

  • Synthetic misinformation significantly degrades QA-system performance, and intricately crafted falsehoods worsen the decline; the authors propose three partial solutions as an initial mitigation step.The study simulates deliberate introduction of false information into knowledge sources for question-answering systems.Source: §7 · p. 9

Future Work

  • Future work could apply the threat model beyond ODQA systems to assess misinformation risks across a broader range of applications.Source: §7 · p. 9
  • Further research should examine LLM-generated calculated misinformation, including hoaxes, rumors, and propagandistic falsehoods.Source: §7 · p. 9
  • Future work should develop cost-effective, robust QA systems that can effectively resist misinformation.Source: §7 · p. 9

Limitations

The study’s findings are limited by GPT-3.5 output reproducibility and cost, its focus on selected systems, and uncertainty about whether results generalize to other QA systems and text generators.

Limitations

  • GPT-3.5 output reproducibility is a challenge, so the authors plan to release prompts and generated documents to facilitate replication.The passage identifies reproducibility as a limitation and describes sharing prompts and generated documents as a mitigation.Source: Limitations · p. 9
  • GPT-3.5 costs an order of magnitude more than ChatGPT (gpt-turbo-3.5), constraining the study to selected emblematic and pervasive QA systems and LLMs.The cost difference limited the scope of the investigations.Source: Limitations · p. 9
  • The findings may not generalize to other systems or text generators, and alternative QA architectures may be more robust to misinformation (Shao and Huang, 2022; Su et al., 2022).The authors note that the analysis focused on a selection of QA systems and LLMs, while citing work demonstrating alternative architectures.Source: Limitations · p. 9

Ethics Statement · A Configuration Details

The authors justify releasing model-generated documents and prompts despite misuse risks as a way to support awareness, threat modeling, and future defenses. Configuration details describe how misinformation generation and defenses were implemented.

Ethics Statement

  • The authors release their documents and prompts despite potential misuse, arguing that transparency supports awareness and future defenses, and following Grover (Zellers et al., 2019)7’s threat-modeling rationale.They note that their OpenAI API-based process is feasible to replicate without code and aim to inform the public, policymakers, and developers about responsible implementation.Source: Ethics Statement · pp. 9–10

A Configuration Details

  • The four misinformation-generation settings are characterized by maliciousness, resourcefulness, and customization to cover common potential misuse scenarios.Resourcefulness concerns demands for resources unavailable to regular users, while customization concerns designing documents to influence the target audience—in this case, machines.Source: A Configuration Details · p. 13
  • All text-generation experiments used GPT-3.5 (text-davinci-003) with the OpenAI API’s default hyperparameter.The passage refers to the OpenAI API documentation for the setting.Source: A Configuration Details · p. 13
  • Plausible false answers relied on language-model parametric memory, were generally under five words, and were sampled from five generations after excluding reference-answer matches.The prompt requested a plausible false answer compared with the reference answer; the authors randomly sampled one of five generated answers that matched no reference answer.Source: A Configuration Details · p. 13
  • The experiments used BM25 from Lin et al. (2021), the original DPR checkpoint trained on five QA datasets including NQ, and FiD-large trained on NQ.The DPR and FiD checkpoints came from their respective original repositories.Source: A Configuration Details · p. 13
  • The detection defense used a RobertaForSequenceClassification checkpoint and 12,000 training contexts for each in-domain and out-of-domain classifier.Training used data containing Wikipedia snippets and GPT-3-generated Wikipedia completions, for 3 epochs at a learning rate of 0.001.Source: A Configuration Details · p. 13
  • The prompting defense averaged EM across five misinformation-aware prompts, while the reported best voting configuration used k = 5 readers and n = 10 passages per reader.The prompts were developed with ChatGPT from a human-written example, drawing on concurrent works (Hong et al., 2023; Li et al., 2023b) and engineering experience.Source: A Configuration Details · p. 13

B Retrieval Performance

The retrieval results show that deliberate misinformation can poison questions at high rates against sparse BM25, while retrieval likelihood varies by generation setting.

B Retrieval Performance

  • A question counts as poisoned when at least one synthetic passage appears among its top-K retrieved passages.Table 7 reports the percentages of poisoned questions in the two benchmarks.Source: B Retrieval Performance · p. 13, Table 7 · p. 14
  • REVISE produced misinformation more likely to be retrieved than GENREAD, while REIT degraded QA performance the most.The paper attributes REVISE's retrieval advantage to its use of a “gold template.”Source: B Retrieval Performance · p. 13
  • Against sparse BM25, REIT poisoned more than 90% of questions with 10 context passages and over 95% with 100 context passages.Table 7 reports poisoned-question percentages for NQ-1500 and CovidNews; REIT's results underscore sparse retrieval's fragility under deliberate attacks.Source: B Retrieval Performance · pp. 13–14, Table 7 · p. 14

C Analysis on Corpus Quality · D Human Detection of Generated Misinformation

The corpus analysis finds that CovidNews provides less informative gold evidence for ODQA, while the human study finds that participants cannot reliably distinguish generated misinformation from Wikipedia passages. The study’s 57% accuracy may overestimate human detection ability because participants improved during the experiment.

C Analysis on Corpus Quality

  • Gold evidence quality was assessed using DPR’s top 100 retrieved passages by question coverage (Recall@100), answer mentions, and rank of the first gold evidence.Source: C Analysis on Corpus Quality · pp. 14–15
  • CovidNews provided significantly less informative evidence for answering questions than the other corpus, making it a more challenging QA task and more susceptible to plausible manipulated assertions amid limited factual counterevidence.Source: C Analysis on Corpus Quality · p. 15

D Human Detection of Generated Misinformation

Related papers

Loading 2305.13661v2…