Source-linked AI summary
On the Risk of Misinformation Pollution with Large Language Models
Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, William Yang Wang
TL;DR
LLMs can generate credible-sounding misinformation that threatens information-intensive applications, especially ODQA systems. The paper establishes a threat model, simulates intentional and unintentional misuse, and examines defenses; it finds substantial ODQA degradation while concluding that robust mitigation remains unresolved.
Problem
The paper addresses limited comprehensive evidence about LLMs' ability to generate credible-sounding misinformation, its harms to ODQA, and possible mitigation strategies.
Method
The study establishes a threat model, simulates hallucination-based and deliberate misinformation generation, evaluates ODQA systems, and explores prompting, detection, and majority voting defenses.
Results
LLMs act as effective controllable misinformation generators, and deliberate synthetic misinformation significantly degrades ODQA performance.
Takeaways & Limitations
Misinformation pollution is a challenging threat requiring continued research and interdisciplinary collaboration to develop misinformation-robust NLP applications.
Takeaways & Limitations
GPT-3.5 output reproducibility and cost constrained the investigation's scope, which focused on selected representative QA systems and LLMs.
Abstract
from arXiv · showhide
In this paper, we comprehensively investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation in the performance of ODQA systems. To mitigate the harm caused by LLM-generated misinformation, we explore three defense strategies: prompting, misinformation detection, and majority voting. While initial results show promising trends for these defensive strategies, much more work needs to be done to address the challenge of misinformation pollution. Our work highlights the need for further research and interdisciplinary collaboration to address LLM-generated misinformation and to promote responsible use of LLMs.
1 Introduction
LLMs' growing ability to produce credible-sounding text creates opportunities for intentional and unintentional misinformation pollution. This paper studies its effects on ODQA systems and evaluates initial mitigation strategies.
- LLMs can automate convincing misinformation, enabling malicious influence operations that may manipulate public opinion, create confusion, and promote detrimental ideologies.
- The paper addresses how effectively modern LLMs generate credible-sounding misinformation, how it harms information-intensive applications, and which mitigation strategies can help.
- The threat model covers unintentional hallucination-driven misinformation and intentional deception targeting specific events.
- LLMs are described as controllable misinformation generators, while deliberate synthetic misinformation significantly degrades ODQA performance.
- Prompting, misinformation detection, and majority voting show promising initial defensive trends, but misinformation pollution remains challenging.
2 Related Work
Prior research has examined generated misinformation and machine-text detection, but detection remains limited and the vulnerability of retrieval-augmented systems to polluted corpora is underexplored.
- LLM-generated non-factual data includes intentional disinformation and unintentional inaccuracies known as hallucinations.
- Existing machine-generated-text detection methods remain limited in precision and scope, while LLM safeguards can be bypassed with jail-breaking prompts.
- Retrieval-augmented systems are intrinsically vulnerable when training data or retrieved corpora mix factual and fabricated content.
- This data-pollution risk has remained underexplored because current models mostly rely on trusted external knowledge sources such as Wikipedia.
3 Generating Misinformation with LLMs
The paper models a malicious actor fabricating a false article for a target question and uses GPT-3.5 prompts to simulate four misinformation-generation settings. These settings range from accidental hallucination to crafted attacks on downstream QA systems.
- The threat scenario defines a misinformation generator G that fabricates a false article P′ in response to a target question Q.
- GPT-3.5 is used because earlier Seq2Seq approaches sometimes produced grammar and commonsense errors that reduced deceptiveness to humans.
- The four settings simulate differing producers and instructions, spanning harmless-query hallucinations and manipulative propaganda prompts.
- GENREAD: GENREAD asks the LLM to generate a background document for a question, with hallucinations potentially causing content to stray from reality.
- CTRLGEN and REVISE: CTRLGEN supplies a non-factual opinion and requests a supporting context passage, while REVISE modifies a factual article to support that opinion.
- REIT: REIT targets downstream QA systems, where generated misinformation need only manipulate the model rather than appear realistic to humans.
- A representative false answer changes the stated minimum COVID-19 vaccine effectiveness from 70 percent to 50 percent.
4 Polluting ODQA with Misinformation
The study evaluates how injecting LLM-generated misinformation into ODQA evidence corpora affects retrieval and answer quality across datasets, retrievers, readers, and generation settings. Deliberate pollution substantially harms ODQA, while even hallucination-based pollution causes measurable declines.
- ODQA systems: Experiments cover four retrieve-and-read ODQA configurations using BM25 or DPR retrievers and FiD or GPT-3.5 readers.BM25 represents sparse retrieval, DPR dense retrieval, FiD is a multi-passage T5-based reader, and GPT-3.5 provides a flexible alternative reader.
- Datasets and pollution setup: The study injects one fake document per question into NQ-1500 and CovidNews corpora, then evaluates ODQA on clean and polluted versions using Exact Match.NQ-1500 uses 1,500 Wikipedia-based questions; CovidNews contains 1,534 COVID-19 news questions.
- Main results: 14% to 54% performance drops occur for DPR-based models and 20% to 87% for BM25-based models under CTRLGEN, REVISE, and REIT pollution.Under inadvertent GENREAD pollution, the best-performing DPR+FiD model decreases 5% on NQ-1500 and 15% on CovidNews.
- Main results: REIT outperforms the other pollution settings because repeated misinformation floods machine readers with seemingly vital evidence and distracts them from authentic information.The study also finds that generated misinformation is likely to enter the top-K retrieved documents because retrievers prioritize lexical or semantic alignment without verifying authenticity.
- Main results: Questions without dependable supporting evidence are more prone to manipulation, with a more pronounced performance decline on CovidNews than on NQ-1500.The authors attribute this pattern to less informational depth and fewer pertinent answering resources in the news domain than in encyclopedias.
- Main results: GENREAD has a significantly more detrimental effect on CovidNews than on NQ-1500, suggesting uneven GPT-3.5 information retention across topics.The authors postulate that reliance on Wikipedia during training may induce a Wikipediacentric bias in model outputs.
5 Defense Strategies
The paper explores detection, prompting, and reader voting as defenses against misinformation pollution in ODQA. Initial results are promising, but robust and cost-effective protection remains difficult.
- Detection Approach: A misinformation detector distinguishes model-generated from human-authored passages using a RoBERTa-based binary classifier trained and tested on retrieved NQ-1500 contexts.The in-domain setup uses NQ-1500 DPR results, with top-10 passages per query and an 80/20 train-test split.
- Vigilant Prompting: Prompting adds a misinformation warning to GPT-3.5's question-answering instruction, asking the reader to account for potentially misleading passages.The approach tests whether instruction-following can help readers resist misinformation.
- Reader Ensemble: Reader ensembles divide contexts into relevance-based groups, obtain one answer per group, and apply majority voting to the candidate responses.The design limits each individual information source's influence on answer prediction.
- Detection Approach: 91.4%-99.7% AUROC for in-domain detectors contrasts with 50.7%-64.8% for out-of-domain detectors.The out-of-domain results are only slight improvements over random guessing, while sufficient in-domain data may be unavailable in practice.
- Summary: Voting performs better than prompting, but multiple readers increase resources and make per-question costs grow linearly with the number of context passages.The authors conclude that cost-effective, resilient QA systems still require further research.
- Does Reading more Contexts help?: Increasing context size has minimal, or counterproductive, effect on performance decline under misinformation pollution.The result is consistent with readers relying on a few highly relevant contexts rather than the entire context volume.
6 Discussion
The discussion frames LLM-generated misinformation as a practical risk to web-scale information ecosystems and downstream applications. Low production costs and vulnerabilities in distributed and centralized corpora intensify the concern.
- Risk Outlook: The study's simulation-based threat model indicates that unaddressed LLM-generated misinformation could undermine the information ecosystem and downstream applications.The discussion emphasizes risks identified through simulated misinformation generation, pollution, and defenses.
- Information Availability: Misinformation generation requires minimal additional information in REVISE and only modest target-system knowledge in REIT.The authors therefore forecast misinformation pollution as an imminent threat to web integrity.
- Associated Costs: $0.01 to $0.04 per 200-word fake document using text-davinci-003 is substantially lower than hiring human writers.The estimate focuses on OpenAI API fees as the dominant experimental cost.
- Integrity of Web-scale Corpora: Web-scale corpora are vulnerable because decentralized sources can be tampered with and centralized or manually maintained resources can still contain unwanted or false information.The discussion cites expired-domain hijacking, Common Crawl contamination, and misinformation slipping into Wikipedia.
- Context Size: Figure 3 reports relative EM change across context sizes, averaged over two retrievers and two test datasets, with GPT contexts limited to 10.The figure measures performance change under different misinformation poisoning settings relative to clean-corpus EM.
7 Conclusion and Future Work
The paper evaluates automated LLM misinformation production and its impact on knowledge-intensive applications, finding that machines are highly susceptible to synthetic falsehoods. It proposes partial mitigations while identifying broader applications, more calculated misinformation, and robust QA as future research directions.
- Conclusion: The evaluation simulates deliberate insertion of false information into QA knowledge sources and examines consequences for knowledge-intensive applications.The paper focuses on the practicality of automated misinformation production with LLMs.
- Conclusion: Machines show significant performance declines under synthetic misinformation, with further deterioration for intricately crafted falsehoods.The authors propose three partial solutions as an initial mitigation step.
- Future Work: Future work should assess the threat model across more applications and investigate calculated misinformation such as hoaxes, rumors, and propagandistic falsehoods.The paper presents these as two of three directions for extending the work.
- Future Work: Developing cost-effective and robust QA systems that resist misinformation remains an ongoing research need.The conclusion identifies resistant QA as the final future-work direction.
Limitations
The study’s scope is constrained by GPT-3.5 output reproducibility and cost, and its findings may not generalize beyond selected QA systems and text generators.
- GPT-3.5 output reproducibility is challenging, so the authors will release prompts and generated documents to facilitate replication.
- GPT-3.5 costs an order of magnitude more than some contemporaries, including ChatGPT, constraining the investigation’s scope.
- The analysis focuses on selected representative QA systems and LLMs, so its findings may not apply to other systems or text generators.
- QA systems using alternative architectures may be more robust against misinformation proliferation.
Ethics Statement
The paper emphasizes transparency and reproducibility while acknowledging the misuse risks of releasing generated documents and prompts. Its experiments use specified generators, QA systems, and defense configurations to support further research.
- The authors publicly release model-generated documents and prompts despite their potential misuse for harmful disinformation.
- Because the model uses the readily available OpenAI API, others can replicate the production process without access to the code.
- The release is intended to raise awareness and encourage responsible, ethical implementation among the public, policymakers, and developers.
- The paper frames threat modeling with a robust generator or simulation as important for protecting against potential dangers.
- The study’s misinformation settings vary by maliciousness, resourcefulness, and customization to cover common misuse scenarios.
- GPT-3.5 text generation uses default hyperparameters, while plausible false answers rely solely on model parametric memory and are filtered against reference answers.
- The retrieval baselines are BM25 and DPR, with FiD-large used as a QA checkpoint trained on NQ.
- The defense experiments include prompting, detection with RoBERTaForSequenceClassification, and voting with five readers using ten passages each.
B Retrieval Performance
Synthetic misinformation can enter retrieval results and undermine ODQA, with deliberate reiteration especially damaging and sparse retrieval particularly vulnerable to targeted attacks.
- A question is “poisoned” when at least one synthetic passage appears among its top-K retrieved passages.
- REVISE produces information more likely to be retrieved than GENREAD, attributed to its misinformation-based ‘gold template’.
- REIT degrades QA performance most among the generation settings, highlighting the risks of deliberate attacks on automated systems.
- When targeting BM25, REIT poisons more than 90% of questions with 10 context passages and over 95% with 100 passages.
- The results identify sparse retrievers as particularly brittle to targeted misinformation.
C Analysis on Corpus Quality
The corpus analysis finds that CovidNews provides less informative evidence for ODQA than the comparison corpus, while human readers also struggle to detect generated misinformation reliably.
- The analysis measures gold evidence through question coverage, answer-mention volume, and the rank of the first gold evidence.
- CovidNews provides significantly less informative evidence for answering questions, making it a challenging QA task.
- The study compares 50 CTRLGEN-generated fake documents with 50 corresponding Wikipedia passages, annotated by three college students on a four-point scale.
- 57% average overall accuracy indicates that humans cannot reliably differentiate the machine-generated misinformation from corresponding Wikipedia passages.
- Participant performance improved during the second half, suggesting that the 57% figure may overestimate human capabilities because of learning effects.