PaperlayerView exact revision on arXiv

Source-linked AI summary

Large language models can consistently generate high-quality content for election disinformation operations

Angus R. Williams, Liam Burke-Moore, Ryan Sze-Yin Chan, Florence E. Enock, Federico Nanni, Tvesha Sippy, Yi-Ling Chung, Evelina Gabasova, Kobi Hackenburg, Jonathan Bright

arXiv:2408.06731v1cs.CYcs.AIcs.CL

TL;DR

The accessibility of LLMs and their ability to generate realistic content at scale raise concerns about their use in election disinformation. The paper introduces DisElect and tests LLM compliance, then evaluates whether generated campaign content appears human-written. Most models comply, and content from almost all models released since 2022 is judged indiscernible by human participants over 50% of the time on average.

  • Problem

    LLMs can cheaply generate realistic content at scale, raising concerns about their potential use in information operations that spread false or misleading narratives.

    Source: Introduction · p. 2

  • Method

    The authors introduce DisElect to measure LLM compliance with election-disinformation tasks and test 13 LLMs, alongside experiments assessing the perceived humanness of generated campaign content.

    Source: Introduction · p. 2, Discussion · p. 20

  • Results

    Over 50% of the time on average, almost all models released since 2022 produce content indiscernible to human participants; most tested models also comply with election-disinformation instructions.

    Source: Introduction · p. 3, Discussion · p. 20

  • Takeaways & Limitations

    The released DisElect dataset and humanness results offer a basis for further study of malicious AI use and model humanness.

    Source: Introduction · p. 3

    DisElect and the experiments use only one prompt template per possible prompt, without prompt engineering or red-teaming.

    Source: Limitations · p. 21

Abstract

from arXiv · show

Advances in large language models have raised concerns about their potential use in generating compelling election disinformation at scale. This study presents a two-part investigation into the capabilities of LLMs to automate stages of an election disinformation operation. First, we introduce DisElect, a novel evaluation dataset designed to measure LLM compliance with instructions to generate content for an election disinformation operation in localised UK context, containing 2,200 malicious prompts and 50 benign prompts. Using DisElect, we test 13 LLMs and find that most models broadly comply with these requests; we also find that the few models which refuse malicious prompts also refuse benign election-related prompts, and are more likely to refuse to generate content from a right-wing perspective. Secondly, we conduct a series of experiments (N=2,340) to assess the "humanness" of LLMs: the extent to which disinformation operation content generated by an LLM is able to pass as human-written. Our experiments suggest that almost all LLMs tested released since 2022 produce election disinformation operation content indiscernible by human evaluators over 50% of the time. Notably, we observe that multiple models achieve above-human levels of humanness. Taken together, these findings suggest that current LLMs can be used to generate high-quality content for election disinformation operations, even in hyperlocalised scenarios, at far lower costs than traditional methods, and offer researchers and policymakers an empirical benchmark for the measurement and evaluation of these capabilities in current and future models.

Introduction

Information operations depend on realistic content produced at scale, while traditional human teams are costly and simple automation is detectable. This study tests LLM compliance with election-operation prompts and whether people can identify generated content.

  • Information operations are systematic online campaigns for false or misleading narratives, requiring realistic content at scale and authentic-looking material across distribution channels.Source: Introduction · p. 2
  • Earlier operators relied on costly human teams or simple copypasta and spintax, which are straightforward to detect across repeated messages.Source: Introduction · p. 2
  • LLMs can cheaply generate realistic content, roleplay political personas, and potentially reproduce granular details about people, concepts, and places.Source: Introduction · p. 2
  • Most LLMs complied with requests to generate election-disinformation content; models refusing more often also refused benign election prompts and more often refused right-wing than left-wing requests.Source: Introduction · p. 2
  • Almost all tested models released since 2022 produced content human evaluators could not distinguish over 50% of the time; two models averaged above-human humanness.Source: Introduction · pp. 2–3

Related Work

Prior research examines disinformation operations, model capabilities, and human perceptions, while sociotechnical evaluations distinguish capability, interaction, and systemic levels. This paper extends that work through UK-localized evaluation of 13 models across operation stages.

Disinformation Operations

  • Disinformation operations spread intentionally deceptive falsehoods through layered media and peer-like accounts; the cited Russian operation illustrates organizational scale.Source: Disinformation Operations · p. 3

AI Safety Evaluations

  • AI safety evaluations can span model capability, human interaction, and systemic effects; this paper examines the first two because risks also depend on public experience.Source: AI Safety Evaluations · p. 4

Assessing the capability of LLMs to generate disinformation

  • GPT-3 satisfied 99/101 accurate versus 80/102 disinformation requests; GPT-2’s best 774M and 1.5B outputs had mean credibility indices of 6.72 and 6.93, while three image platforms accepted more than 85% of prompts.Buchanan et al. reported that AI models could customize language for specific groups; LLM-chatbot outputs showed political bias on Russian, Ukrainian, and US politics prompts.Source: Assessing the capability of LLMs to… · pp. 4–5

Human interactions with AI generated disinformation

Our contribution

  • This study combines malicious-prompt compliance tests and human discernment experiments in a UK national and hyperlocal context, using 13 models across four operation stages.Source: Our contribution · p. 5

Methodology

The study evaluates two stages of election-disinformation operations: whether 13 LLMs generate requested content and whether people can distinguish generated from human-written content. It covers four operation stages and UK voting and MP scenarios.

Methodology

  • The study pairs a systematic dataset evaluation of LLM compliance with human experiments testing whether people distinguish AI-generated from human-written election-disinformation content.The study frames both parts as measures of LLM usefulness for automating election-disinformation operations.Source: Methodology · p. 6

Information Operation Design & Use Cases

  • The operation design covers news articles, social-media accounts, social-media posts, and replies, spanning content creation and dissemination.Posts and replies are intended to create an illusion of public interest or legitimacy.Source: Information Operation Design & Use Cases · p. 6
  • The two use cases are hyperlocalised logistical voting misinformation and fictitious claims about UK MPs, including alleged campaign-fund misuse.The voting scenario examines localised information; the MP scenario examines claims about candidates.Source: Information Operation Design & Use Cases · p. 6

Models

  • The 13 selected LLMs vary by release date, size, and access type, with multiple models from some families enabling within-family comparisons.Several models were run through Ollama at 4-bit quantisation, a setup the paper notes may reduce accuracy on complex tasks.Source: Models · p. 6, Table 1 · p. 7

DisElect Evaluation Dataset

  • DisElect contains 2,200 prompts for operation stages and use cases plus 50 benign election prompts; its voting and MP datasets vary prompts by location or targeted MP.The researchers created prompts from templates and variables, and released the prompt-comparison library prompto.Source: DisElect Evaluation Dataset · p. 8, Dataset Creation · p. 8

Evaluation

  • Responses are classified as refuse, soft-refuse, incoherent, or comply; GPT-3.5 Turbo performs zero-shot labelling, with hand-checked overall Macro F1 Score of 0.76.On the hand-labelled sample, accuracy was 96% for the refuse class and 86% for other classes.Source: Evaluation · p. 8

Humanness Experiments

  • Humanness experiments ask people to label content as human- or AI-written across MP-fund and local-election scenarios, using 15 responses per prompt and source.Human-written comparison content came from a separate research team; AI responses were generated as 15 variations where possible, or through 15 separate prompts.Source: Humanness Experiments · pp. 8–9

Experimental Design

  • Participants saw 15 items each from one LLM and human writers per stage; repeated model patterns could reveal AI items, though human-only baselines preserve model comparisons.The study recruited 780 UK-based English-speaking adults for each experiment (N = 2,340); the local-election experiment recruited London residents.Source: Experimental Design · p. 9, Sampling · pp. 9–10

Results

Most models rarely explicitly refuse election-disinformation prompts, and the models that refuse malicious prompts may also refuse benign election prompts. Among the three models that refuse substantially, refusal varies by prompt characteristics and model, with a consistent preference for refusing right-wing personas.

Few LLMs refuse to generate content for a disinformation operation.

  • Only Llama 2, Gemma, and Gemini 1.0 Pro explicitly refuse more than 10% of prompts in any use case.The heatmap shows response classifications across three use cases; models are ordered by release date.Source: Few LLMs refuse to generate content… · p. 10, Figure 1 · p. 10
  • Refusals are more common in DisElect.MP than DisElect.VT (12.4% vs. 6.8% overall), while soft-refusals are more common in DisElect.VT (17.5% vs 21.4% overall).Models that refuse malicious prompts generally also refuse some benign election-related prompts.Source: Few LLMs refuse to generate content… · p. 11

What drives refusal?

  • All three frequent-refusal models are more likely to refuse right-wing than left-wing persona prompts across both use cases.News-article prompts draw more refusals in DisElect.MP than DisElect.VT, and refusal rates also vary by subject.Source: What drives refusal? · p. 11, Figure 2 · p. 12
  • Models often disagree on which prompts to refuse; refusal rankings are consistent across possible persona values but vary across other variables.For subject in DisElect.MP, median correlation is ρ = 0.74; for pipeline stage in DisElect.VT, it is ρ = 0.00.Source: What drives refusal? · p. 11
  • All three frequent-refusal models refuse more for female MPs than male MPs and for Labour MPs than Conservative or other MPs.Female Labour MPs have the highest average refusal and male Conservative MPs the lowest; model rankings across party-gender groups have median ρ = 0.90.Source: What drives refusal? · p. 11, Figure 3 · p. 12

How well can people identify AI-generated content?

Across the tested models, people often judged AI-generated election-operation content to be human-written, though results varied by model, experiment, and pipeline stage. Humanness was especially high for Llama 3 and Gemini, while refusals affected some models’ scores.

How well can people identify AI-generated content?

  • 9/13 models achieve average humanness above 50%, meaning most tested models’ content is often judged indistinguishable from human-written content.Humanness is measured from human assignments of AI-generated items as human; results vary substantially across participants and items.Source: How well can people identify… · p. 13, Figure 4 · p. 14
  • Llama 3 and Gemini have the highest average humanness scores, at 62% and 59%, respectively.At least 15% of entries for each model reach ≥75% humanness (19% for Llama 3 and 17% for Gemini).Source: How well can people identify… · p. 13
  • Mean humanness is highest for experiment 1a (0.53), followed by 1b (0.48), and lowest for experiment 2 (0.44).Experiment 1a concerns MP disinformation from a left-wing perspective; 1b concerns MP disinformation from a right-wing perspective; experiment 2 concerns localised election disinformation.Source: How well can people identify… · p. 13, Figure 5 · p. 14
  • Refusals produce bimodal humanness distributions for Llama 2 and Gemma in some experiments, but not experiment 1a.The absence of refusals in experiment 1a contrasts with experiments 1b and 2, where these models are more likely to refuse right-wing prompts.Source: How well can people identify… · p. 13, Figure 5 · p. 14
  • 11/13 models reach their highest humanness on social-media reaction or reply generation stages.Variation across models is lowest for account generation (std = 0.12); news-article generation has no refusals for Llama 2 or Gemma, and Llama 2 scores 0.61 there.Source: How well can people identify… · pp. 13, 15, Figure 6 · p. 15
  • Llama 3 and Gemini 1.0 Pro remain above average across pipeline stages, while GPT-3.5 Turbo varies least across stages (std = .03).Llama 3 and Gemini 1.0 Pro have stage-level standard deviations of .06 and .07, respectively; neither leads on news-article generation.Source: How well can people identify… · p. 15

Model development over time

Humanness generally rises as model age decreases, but age does not explain every model or pipeline-stage result. Content diversity is also associated with humanness, while news-article generation is a notable exception to the broader age pattern.

Model development over time

  • Model age correlates negatively with humanness (ρ = −0.82), although Llama 3 and Gemini 1.0 Pro are among the newest tested models and GPT-2 and T5 are the oldest.Refusals in Llama 2 and Gemma affect their overall humanness, and model size also differs independently of age.Source: Model development over time · p. 15
  • Smaller open-source models can offer humanness comparable to larger API-based models, so the size–humanness trade-off is not linear.The comparison includes Phi-2, Gemma, Mistral, and Flan-T5 against GPT-3.5 Turbo and GPT-4.Source: Model development over time · p. 16
  • The age pattern has a news-article exception: GPT-3.5 and Llama 2, both over a year old, are the highest-scoring models for that stage.The broader relationship generally holds across experiments, but varies by pipeline stage.Source: Model development over time · p. 16
  • The authors propose that repeated wording may signal inauthenticity, while increased instruction and safety fine-tuning may make newer models’ responses overly uniform.They frame this as a possible differentiator and predictor, not a demonstrated causal mechanism.Source: Model development over time · p. 16
  • TF-IDF similarity correlates negatively with humanness in every experiment, most strongly in experiment 2 (ρ = −0.65), and positively with model age, strongest in experiment 2 (ρ = 0.45).The authors suggest that greater content diversity may accompany higher perceived humanness; similarity is measured as average pairwise cosine similarity between model-generated content groups.Source: Model development over time · p. 16
  • Flan-T5 produces less similar content than GPT-4 in experiments 1a and 1b, but more similar content in experiment 2.Both models produce more similar content in experiment 2 than in the other experiments.Source: Model development over time · p. 16

Above-human-humanness

AI-generated content can be judged more human than human-written content in some experiments and pipeline stages, but the pattern depends on context. Participants’ judgments of AI and human content are also inversely related, complicating comparisons based only on AI-content labels.

Above-human-humanness

  • AI-generated and human-written content humanness are strongly negatively correlated (ρ = −0.92), and the authors suggest this may indicate that AI content could undermine trust in good-faith human content.Participants may have expected the viewed items to contain approximately equal proportions of AI and human content.Source: Above-human-humanness · p. 16
  • Llama 3 and Gemini achieve higher average humanness than human-written content, but not in experiment 2.They exceed human-written content in experiments 1a and 1b; experiment 2 has the lowest overall model humanness and highest overall human humanness.Source: Above-human-humanness · p. 16, Figure 7 · p. 17
  • 8/13 models exceed human humanness in at least one pipeline stage in experiments 1a and 1b, compared with 2/13 in experiment 2.News-article generation has the most above-human-humanness instances; the authors note that this stage may be easier to fake because news has a polished, fixed style.Source: Above-human-humanness · pp. 16, 18, Figure 8 · p. 17
  • The authors define “share of humanness” as AI-generated items labelled human divided by all AI or human entries labelled human.This measure rewards models that reduce the humanness assigned to human-written content; in experiment 1a it reverses the ranking of Gemma, Llama 2, and Gemini relative to humanness.Source: Above-human-humanness · p. 18

What factors explain perceptions of humanness?

The analysis relates participant demographics, content similarity, model, and pipeline stage to whether participants judged AI-generated content as human. Demographic associations were generally weak, while humanness showed patterns associated with political persona, model recency, and content stage.

Participant factors

  • Age had a significant positive association with humanness only in experiment 1a: one standard deviation older corresponded to 8% higher odds of judging content human.The effect was absent in experiment 1b; gender and education were nonsignificant across all three experiments.Source: What factors explain perceptions of… · p. 18, Table 5 · p. 19
  • A one-standard-deviation increase in politics corresponded to 8% and 12% higher odds of classifying content as human in experiments 1b and 2, respectively.The authors associate the effect in right-wing-persona experiments with right-wing-identifying participants being more likely to label right-wing content human; no such effect appeared for left-wing-identifying participants and left-wing content.Source: What factors explain perceptions of… · pp. 18–19, Table 5 · p. 19

Model and content factors

  • The reported TF-IDF-distance findings aligned more closely between the two right-wing-perspective experiments than between the two MP-focused experiments.The passage reports TF-IDF distance as a significant predictor in experiments 1b and 2, but also calls it insignificant in experiment 1b; the significance pattern is internally inconsistent.Source: What factors explain perceptions of… · p. 19, Table 5 · p. 19
  • The overall trend was for more recent LLMs to have higher odds of being judged human than GPT-2, but odds ratios did not rise uniformly and nearby-model differences were not always significant.For example, the experiment 1a difference between GPT-4 (2.51) and Mistral (2.60) was not statistically significant.Source: What factors explain perceptions of… · p. 19, Table 5 · p. 19

Pipeline stage

  • Writing news articles was the strongest predictor of humanness in experiments 1b and 2, but the weakest in experiment 1a.For experiments 1b and 2, social media reactions, replies, and accounts followed in that order.Source: What factors explain perceptions of… · p. 19

Disinformation domains versus personas

Humanness patterns were sometimes more aligned by disinformation domain and sometimes by persona. A simplified comparison of news-article generation estimates substantially lower costs for the LLM approaches than for traditional content services.

Disinformation domains versus personas

  • Humanness levels, patterns across experiments and pipeline stages, and above-human humanness were more similar between the two MP experiments, while the strongest demographic predictors were more similar between the two right-wing experiments.The authors suggest performance may transfer more easily across personas within domains, while who perceives content as human may be more similar within personas.Source: Disinformation domains versus personas · pp. 19–20

Cost Comparison

  • For 10 articles per day, the estimated monthly cost was around 4,500USD for traditional production, versus 0.30USD through Gemini API or 9USD deploying Llama 3 70B.The comparison is simplified to news-article generation; smaller open-source/open-weight models that performed comparably could run on personal computers, bringing costs to zero in some cases.Source: Cost Comparison · p. 20

Discussion

The discussion presents DisElect and human-participant experiments as evidence that many tested LLMs comply with election-disinformation prompts and produce content often judged indistinguishable from human writing. It also emphasizes limits in prompt coverage, model coverage, and media scope, alongside directions for further study.

Discussion

  • Most of the 13 tested LLMs complied with election-disinformation instructions; almost all tested models released since 2022 were judged indistinguishable from human-written content over 50% of the time, and two achieved above-human humanness.Models refusing more often also refused benign election prompts and were more likely to refuse right-wing-perspective prompts than left-wing-perspective prompts.Source: Discussion · p. 20
  • The authors argue model-layer safety alone is unlikely to address LLM-driven information operations fully, and call for further public and institutional education, including AI literacy.They also note that open- and closed-source models both collaborated with information operations across many prompts.Source: Discussion · p. 20
  • Refusing dual-intent prompts can trade helpfulness against alignment, while providers and platforms can also identify misuse through patterns of activity rather than individual prompts.The discussion warns that refusing content from particular political viewpoints could have downsides for public trust.Source: Discussion · p. 21

Limitations

  • DisElect and the experiments used one prompt template per possible prompt, without prompt engineering or red-teaming to bypass refusals or improve content variation.The study therefore leaves open how those strategies might change compliance and perceived humanness.Source: Limitations · p. 21
  • The 13-model sample does not cover the full model space, excludes models such as Claude and Ernie and mixture-of-experts models such as Mixtral, and considers English disinformation only.The authors say the sample is not comprehensive enough across release date and model size for absolute conclusions about their relationships with compliance and humanness.Source: Limitations · p. 21

Future Work

  • Future work could use prompt engineering and red-teaming to map the prompt space and investigate LLM detection of misuse patterns in dual-intent behavior.Source: Future Work · pp. 21–22
  • The authors propose extending study of humanness to audio and video content and to multi-turn conversations, beyond the paper's text-only, single-prompt setting.Source: Future Work · p. 22
  • Future experiments could test whether comparing multiple models directly changes perceived humanness and whether people can discern synthetic content without being told it is present.The authors also call for generalisable humanness metrics and exploration of frontier LLMs' ability to estimate humanness.Source: Future Work · p. 22

Ethical Considerations

Participants consented after being told the study content was fictional and unrelated to real events; the project received ethics-panel approval.

Ethical safeguards

  • Participants were informed that the content was fictional, created for the study, and unrelated to real-world events or news before consenting.They could withdraw at any time and were advised not to participate if the study might adversely affect their emotional state.Source: Ethical Considerations · p. 22

Ethical approval

Appendix

The appendix provides prompts and example materials for the humanness experiments, participant information and demographics, and a figure comparing human-assignment proportions across models and pipeline stages.

Appendix

  • Table 6 presents prompts for humanness experiments, including Vauxhall voting-date article, account-bio, complaint-tweet, and agreement-reply prompts from a right-wing resident’s perspective.Source: Table 6 · p. 28, Appendix · p. 28
  • Table 7 is identified as example responses to one prompt used to generate content for experiment 2.Source: Table 7 · p. 28
  • Tables 8 and 9 provide the experiment participants’ information sheet and demographic statistics, respectively.Source: Table 8 · p. 28, Table 9 · p. 29
  • Fig. 9 compares human-assignment proportions across pipeline stages and models with the proportion for human-written content, sorting models by release date.Source: Figure 9 · p. 30

Related papers

Loading 2408.06731v1…