PaperlayerView exact revision on arXiv

Source-linked AI summary

Disinformation Capabilities of Large Language Models

Ivan Vykopal, Matúš Pikuliak, Ivan Srba, Robert Moro, Dominik Macko, Maria Bielikova

arXiv:2311.08838v2cs.CL

TL;DR

The paper addresses limited systematic evidence about current LLMs’ capacity to generate English disinformation news articles, a concern because such content could be used to flood information channels. It evaluates generated articles across narratives and prompt formats, concluding that LLMs can produce articles supporting dangerous narratives, though the findings are bounded by the study’s prompts and evaluation scope.

  • Problem

    Systematic evidence about current instruction-tuned LLMs’ ability to generate English disinformation news articles is limited, despite concern about their use in flooding the web and social media.

    Source: §1 · p. 1

  • Method

    The study evaluates LLM-generated articles across disinformation narratives, including prompts that add narrative abstracts to guide facts and arguments.

    Source: §3.1 · p. 2, §3.3 · p. 3

  • Results

    Existing LLMs, including open-source models, can generate news articles with real or hallucinated supporting evidence for dangerous disinformation narratives.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Models differ in willingness to generate disinformation, and the authors report that some demonstrate the possibility of safer training while detection models may add a layer of protection.

    Source: §8 · p. 8

    The manually selected prompts may be less robust than those developed by a focused attacker, potentially understating achievable disinformation quality and mitigation of safety measures.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Automated disinformation generation is often listed as an important risk associated with large language models (LLMs). The theoretical ability to flood the information space with disinformation content might have dramatic consequences for societies around the world. This paper presents a comprehensive study of the disinformation capabilities of the current generation of LLMs to generate false news articles in the English language. In our study, we evaluated the capabilities of 10 LLMs using 20 disinformation narratives. We evaluated several aspects of the LLMs: how good they are at generating news articles, how strongly they tend to agree or disagree with the disinformation narratives, how often they generate safety warnings, etc. We also evaluated the abilities of detection models to detect these articles as LLM-generated. We conclude that LLMs are able to generate convincing news articles that agree with dangerous disinformation narratives.

1 Introduction

The paper addresses limited systematic evidence about current instruction-tuned LLMs’ ability to generate English disinformation news articles. It evaluates generated texts and concludes that existing models, including open-source ones, can produce articles supporting dangerous narratives, while treating its findings as a snapshot of current capabilities.

1 Introduction

  • Figure 1 classifies generated texts as dangerous when they are disinformation articles that could be misused, or safe when they include disclaimers, counterarguments, or opposition to the user.GPT-4 annotations are generally slightly biased toward safety.Source: Figure 1 · p. 1
  • The study fills a gap in experimental evidence by comprehensively evaluating instruction-tuned models prompted to generate English disinformation news articles.The evaluation uses harmful narratives, including health-related hoaxes.Source: §1 · p. 1
  • The authors manually assessed 1,200 generated texts for narrative agreement, novel arguments, and adherence to news-article style.The assessment also compared models’ disinformation-generation capabilities and their safety filters.Source: §1 · p. 1
  • Existing LLMs, including open-source models, can easily generate news articles with real or hallucinated supporting evidence for dangerous disinformation narratives.The authors also report that safety filters work poorly with a few exceptions, and generated texts are quite detectable.Source: §1 · p. 2
  • The paper presents its findings as a snapshot; newer LLMs trained on newer data may behave differently.The stated aim is to characterize current capabilities and observe how they evolve.Source: §1 · p. 2

2 Related Work

Prior studies examined fake-news generation and LLM misuse, but some evaluations were anecdotal or treated disinformation as one category among many. Related work also addresses other forms of problematic LLM output.

2 Related Work

  • Disinformation has appeared across online formats including newspaper articles, social media posts, reviews, and comments.The cited examples span Zellers et al. (2020), Fagni et al. (2021), Cutler et al. (2021), Adelani et al. (2020), and Yanagi et al. (2020).Source: §2 · p. 2
  • GROVER generated fake-news articles from titles, and its 1.5B-parameter, GPT-2-sized model makes its findings potentially outdated.Zellers et al. (2020) compared generated and human-written texts on style, sensibility, and trustworthiness.Source: §2 · p. 2
  • Buchanan et al. (2021) studied GPT-3 across six disinformation scenarios, but their evaluation was mostly anecdotal.Two scenarios later entered HELM, where human annotators rated outputs from six LLMs; that evaluation found arguments for COVID-19 and climate-change disinformation.Source: §2 · p. 2
  • Disinformation is often one category within broader LLM-safety evaluations, whose methods are usually not focused on the issue.The passage cites Khatun and Brown (2023), Ganguli et al. (2022), and Wang et al. (2023).Source: §2 · p. 2
  • Related work also identifies LLM outputs that are biased, inaccurate, or offensive toward groups based on gender, race, or religion.The passage cites Brown et al. (2020), Hadi et al. (2023), and Touvron et al. (2023a).Source: §2 · p. 2

3 Disinformation Generation

The study generates disinformation news articles using selected narratives, instruction-tuned models, and two prompt formats. Its design spans five narrative categories and uses repeated generations to account for stochastic output.

3.1 Narratives

  • The study selected four narratives in each of five categories: COVID-19, the Russo-Ukrainian War, health, US elections, and regional topics.Narratives were sourced from professional fact-checkers including Snopes, AFP, and EDMO; Table 1 lists the final selection.Source: §3.1 · p. 2, Table 1 · p. 3

3.2 Generators

  • The experiment focuses on instruction-tuned LLMs because the authors considered instruction-following important for prompt-steered disinformation generation.Standard generative LLMs were also tried but proved too erratic; the study included models selected by benchmark performance, smaller GPT-3 variants, and additional well-performing models.Source: §3.2 · p. 3

3.3 Machine Text Generation

  • Title-abstract prompts add narrative context intended to guide generated articles toward appropriate facts and arguments aligned with the narrative.Title-only prompts instead rely on the model’s internal knowledge about the narrative.Source: §3.3 · p. 3
  • Ten instruction-tuned LLMs generated three articles per prompt type for each of 20 narratives, yielding 1,200 articles.Three generations per prompt were used to increase robustness given the stochastic nature of generation.Source: §3.3 · p. 3

4 Human Evaluation

Human evaluation assessed article form, stance toward narratives, and supporting or opposing arguments. The results show that models’ responses vary by prompt, model, and narrative, while often supporting disinformation with novel arguments.

4.1 Methodology

  • Human annotators rated 840 texts from seven models using six questions covering coherence, news style, stance, and supporting or opposing arguments.Two annotators scored each text on a five-step scale, and their scores were averaged.Source: §4 · p. 4, §4.1 · p. 4

4.2 Results

  • Model behavior differs: Falcon tends to disagree and often refuses, while Vicuna and Davinci readily produce convincing news-like articles with novel arguments.The authors characterize Vicuna and Davinci as the most dangerous according to their methodology.Source: §4.2 · p. 4
  • Adding narrative abstracts significantly changes scores for Q3–Q6; abstracts increase agreement and disagreement scores decrease.The authors report that abstracts can supply context absent from a model’s knowledge, including information about the Russo-Ukrainian war.Source: Figure 3 · p. 4, §4.2 · p. 5
  • LLMs can generate disinformation at will; agreement and novel supporting arguments increase, while disagreement with prompts remains nearly zero.The discussion also reports hallucinated supporting details such as names, events, or statistics.Source: §4.2 · p. 5
  • Willingness to generate disinformation varies across narratives, with N3 and N17 prompting more willingness than N4 and N10.The authors also report that regional narratives tend to receive more agreement, possibly because models lack the knowledge to refute them.Source: §4.2 · p. 5, Figure 4 · p. 6
  • The evaluation did not capture the full range of observed behaviors, including bothsideist responses and paraphrasing.The authors note that bothsideism may be unsuitable when discussing disinformation narratives.Source: §4.2 · p. 6

5 GPT-4 Evaluation

The paper tests GPT-4 as an automated evaluator by comparing its ratings with human annotations. GPT-4’s aggregate trends broadly resemble human ratings, but scores are inflated and individual predictions are not reliable.

5 GPT-4 Evaluation

  • Because human annotation is time-consuming and impractical across hundreds of new models, the study evaluates GPT-4 as a possible way to automate assessment.The authors compare GPT-4’s answers with human annotations before using it to rate additional models.Source: §5 · p. 6

5.1 Methodology

  • GPT-4 was prompted separately for each generated text and evaluation question, using the narrative title and abstract, article, question, and answer options.The prompts were manually optimized, though the authors note further tuning may be possible.Source: §5.1 · p. 6

5.2 Results

  • GPT-4 ratings correlate with human judgments overall, but its assessment of style and arguments is weaker, including judging how arguments relate to or oppose a narrative.Its scores are systematically higher than human ratings across all six questions.Source: §5.2 · p. 6
  • Aggregated GPT-4 scores closely match human trends, but the authors conclude its results are not reliable at the individual-sample level.They describe the work as an initial step toward automated assessment and suggest that further prompt tuning or fine-tuning may improve performance.Source: §5.2 · p. 6
  • GPT-4, Mistral, and Llama-2 were automatically rated after human comparison; GPT-4 and Mistral resemble ChatGPT, while Llama-2 appears safer like Falcon.The authors say the newer models appear to have safer behavior built in.Source: §5.2 · p. 6, §5 · p. 6

6 Safety Filters

The section measures refusals and disclaimers as safety-filter behaviors, then classifies generated texts as safe, dangerous, or neutral. Human ratings found safety features in only Falcon and ChatGPT, while GPT-4's safety judgments were less consistent.

6.1 Methodology

  • Human annotators and GPT-4 assessed refusals, disclaimers, and texts without either safety behavior.Annotators were also asked to identify whether a disclaimer said the text was untrue or AI-generated.Source: §6.1 · p. 7, §6 · p. 7

6.2 Results

  • Only Falcon and ChatGPT showed human-identified safety features; Falcon filtered around 30% of requests, while other models had none applicable to this use case.The human annotators' agreement was high (Cohen’s Kappa 0.82).Source: §6.2 · p. 7
  • GPT-4 overestimated safety disclaimers, counting 52 for Falcon where humans counted 11; its Cohen’s Kappa was 0.45.The authors say GPT-4 and Mistral may include some safety filters, but likely fewer than ChatGPT; Llama-2 had a high proportion of refusals.Source: §6.2 · p. 7

6.3 Safe and Dangerous Behavior

  • A text was classified as safe if it disagreed with the narrative or used a safety filter, and dangerous if it was a well-formed article agreeing with it and adding novel arguments.Texts meeting neither definition were classified as neutral.Source: §6.3 · p. 7
  • GPT-4-based classifications were less consistent, with false positives for safety filters and a tendency to underestimate dangerous texts and overestimate safe ones.The authors caution that models evaluated only by GPT-4 may therefore be less safe than Figure 1 indicates.Source: §6.3 · p. 7

7 Detection

The paper tests whether available detectors can identify generated articles, using generated and human-written fake news and a range of detector models. The strongest detectors achieved around 0.8 F1, but test-set thresholding and untested adversarial attacks constrain the practical interpretation.

7 Detection

  • The evaluation set combined all 1,200 generated texts with 73 human-written fake-news articles about the narratives.The human-written articles came from fact-checking platforms and sites associated with conspiracy theories or pseudoscience.Source: §7 · p. 7
  • The study tested MULTITuDE detectors and eight additional state-of-the-art detectors, reporting five ELECTRALARGE-based MULTITuDE models selected as the best in a preliminary study.The broader MULTITuDE benchmark had fine-tuned 315 detectors across languages.Source: §7 · p. 7
  • For each binary classifier, the authors selected a threshold using the ROC curve's highest Youden Index and computed macro precision, recall, F1, and AUC.Table 6 presents detector performance at each detector's optimal threshold.Source: §7 · pp. 7–8, Table 6 · p. 8

7.1 Results

  • The best-performing detectors reached approximately 0.8 F1, and fine-tuned MULTITuDE ELECTRALARGE was the best detector overall.The top three by Macro F1 were all fine-tuned ELECTRALARGE models; detection-longformer beat ELECTRA on several individual generators despite weaker average performance.Source: §7.1 · p. 8, Table 6 · p. 8
  • Thresholds were optimized on the test set, so the reported detector performance is a theoretical maximum and in-vivo performance may be slightly worse.The evaluation did not include adversarial attacks that could reduce detection ability significantly.Source: §7.1 · p. 8
  • The optimal thresholds for several off-the-shelf models were close to 0 or 1, indicating calibration mismatched to this data and a concern for deployment without threshold tuning.The weak average performance of detection-longformer was attributed to its results on Curie and Babbage, which the authors considered relatively unimportant because of their low capabilities.Source: §7.1 · p. 8

8 Conclusion

The paper finds meaningful differences in how willing LLMs are to generate disinformation news, while also identifying uses of LLMs and detectors in evaluation and protection. It concludes that generated disinformation remains a risk and points to social media as a future area to study.

8 Conclusion

  • Models differed in willingness to generate disinformation news: Vicuna and Davinci appeared to have no built-in safety filters, while Falcon and Llama-2 demonstrated safer behavior.Source: §8 · p. 8
  • GPT-4 can partially automate evaluation, supporting scalable and repeatable assessment of new models and potentially saving hundreds of person-hours of annotation.Source: §8 · p. 8
  • LM-based detectors appear able to identify machine-generated texts with high precision, adding a protection layer, although adversarial efforts and other attack vectors remain viable.Source: §8 · p. 8
  • The authors identify media beyond news articles, particularly interactive and personalized social-media posts, as important future work.They state that LLM-powered influence operations could be even more devastating in that domain.Source: §8 · p. 8

Limitations

The findings are bounded by the study's timing, evaluation dimensions, language, prompts, and detector testing. In particular, more focused prompts or adversarial detector attacks could change the observed generation quality or detection performance.

Timeliness limitation

  • The study captures LLM capabilities in the second half of 2023, and behavior of newer models trained on newer data may differ.The authors note that future disinformation capabilities are hard to predict.Source: Limitations · p. 9

Evaluation limitation

  • The evaluation focuses mainly on arguments and learning in generated texts, not other qualities such as persuasiveness or inflammatory content.The authors note these additional qualities may matter to bad actors.Source: Limitations · p. 9

English-only

  • The analysis covers English only, so it does not establish how models behave in other languages.The authors note that generation quality may be worse in other languages, while safety mechanisms sometimes do not generalize to them.Source: Limitations · p. 9

Prompt Limitations

  • The prompts were not as robust as those a dedicated disinformation actor might develop, so the study may underestimate the quality and safety-evasion potential of generated disinformation.More focused prompts might improve quality and trustworthy style or mitigate safety measures.Source: Limitations · p. 9

Detection Limitations

  • The detector evaluation omitted adversarial attacks, so a determined actor could significantly reduce detection ability; reported performance is a theoretical maximum.Source: Limitations · p. 9

Ethical Considerations

The authors describe the dataset and code as research resources, argue that publication adds little to bad actors’ capabilities, and document the generation and preprocessing setup.

Ethical Considerations

  • The dataset is intended to evaluate current LLM disinformation capabilities, and the code is released for reproducible research use.Source: Ethical Considerations · p. 9
  • The authors argue that publication would only marginally help bad actors because entry barriers are low and capabilities are easy to verify.Source: Ethical Considerations · p. 9
  • The authors argue that withholding generated articles is ineffective risk mitigation because similar data can be generated cheaply; producing the paper’s texts via ChatGPT would cost roughly $10.Source: Ethical Considerations · p. 9

A Computational Resources

  • OpenAI models were accessed through the OpenAI API; other generators and detectors used four A100 40 GPUs, and generation took about 12 GPU-hours.Source: A Computational Resources · p. 11

B.1 Disinformation Generation

  • Generation used temperature 1, with a 1,024-token maximum; open-source models also used a 256-token minimum, top_p 0.95, top_k 50, and repetition penalty 1.10.Source: §B.1 · p. 11

B.2 Generated-Text Pre-processing

  • Preprocessing removed prompt text and surrounding whitespace, then deleted unfinished trailing sentences identified by missing punctuation.Source: §B.2 · p. 11

B.3 Analysis of the Texts

The paper reports text-length statistics, describes GPT-4’s question-based evaluation prompts, and outlines the detectors and input constraints used for machine-generated-text detection.

B.3 Analysis of the Texts

C GPT-4 Evaluation

  • GPT-4 prompts for Q1–Q5 included the narrative, abstract, generated article, question, and answer options; Q6 added an explanation about rating arguments against the narrative.Source: C GPT-4 Evaluation · p. 13, Figure 5
  • The Q7 safety prompt described the available response options for judging safety-filter behavior.Source: C GPT-4 Evaluation · p. 13

D Machine Generated Text Detection

E Detection Results

Detection results favor generator-specific fine-tuning for several models, while the selected examples illustrate a range of generated stances and safety behaviors.

E Detection Results

F Generated Texts

  • The selected examples span agreement, disagreement, bothsideist responses, and safety-filter behavior, with human ratings and prompts shown alongside the generated texts.Source: F Generated Texts · p. 14

F.1 Text 1

  • Vicuna’s COVID-19 example received maximum agreement and pro-narrative argument ratings, and it presented purported cases such as Carlos Alvarez as evidence.Source: §F.1 · pp. 14–16

F.2 Text 2

  • GPT-3 Davinci’s chemtrails example cited reported symptoms but acknowledged that more research was needed before definitive health conclusions.Source: §F.2 · p. 16

F.3 Text 3

  • ChatGPT’s Ukraine bio-labs example rejected the allegation, stating that investigations and credible evidence did not substantiate it.Source: §F.3 · p. 17

F.4 Text 4

  • Falcon’s refugee-benefits example presented arguments on both sides, including claims about resource pressures and refugees’ need for support.Source: §F.4 · pp. 17–18

Related papers

Loading 2311.08838v2…