PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLM-Generated Misinformation Be Detected?

Canyu Chen, Kai Shu

arXiv:2309.13788v5cs.CLcs.AIcs.CRcs.HCcs.LG

TL;DR

LLM-generated misinformation raises concerns for online safety and public trust, motivating the question of whether it causes more harm than human-written misinformation. The paper builds a taxonomy, categorizes and validates generation methods, and empirically finds that LLM-generated misinformation can be harder for humans and detectors to detect when semantics are matched.

  • Problem

    The paper asks whether LLM-generated misinformation causes more harm than human-written misinformation, examining this question through detection difficulty.

  • Method

    The paper builds a five-dimensional taxonomy, categorizes and validates real-world LLM misinformation-generation methods, and conducts empirical comparisons with human-written misinformation.

  • Results

    LLM-generated misinformation can be harder for both humans and misinformation detectors to detect than human-written misinformation with the same semantics.

  • Takeaways & Limitations

    The findings indicate that LLM-generated misinformation can have more deceptive styles and potentially cause more harm.

  • Takeaways & Limitations

    The constructed LLMFake dataset excludes factual news items because the study focuses on comparing detection difficulty for misinformation.

Abstract

from arXiv · show

The advent of Large Language Models (LLMs) has made a transformative impact. However, the potential that LLMs such as ChatGPT can be exploited to generate misinformation has posed a serious concern to online safety and public trust. A fundamental research question is: will LLM-generated misinformation cause more harm than human-written misinformation? We propose to tackle this question from the perspective of detection difficulty. We first build a taxonomy of LLM-generated misinformation. Then we categorize and validate the potential real-world methods for generating misinformation with LLMs. Then, through extensive empirical investigation, we discover that LLM-generated misinformation can be harder to detect for humans and detectors compared to human-written misinformation with the same semantics, which suggests it can have more deceptive styles and potentially cause more harm. We also discuss the implications of our discovery on combating misinformation in the age of LLMs and the countermeasures.

1 INTRODUCTION

LLMs create risks for online safety and public trust because they can generate human-like misinformation. The paper studies whether this misinformation is harder to detect than semantically equivalent human-written misinformation.

  • LLMs such as ChatGPT can potentially generate human-like misinformation, threatening online safety and public trust.
  • The paper evaluates potential harm through the computational question of whether LLM-generated misinformation is harder to detect than human-written misinformation with the same semantics.
  • The study first builds a taxonomy and categorizes and validates real-world methods for generating LLM misinformation.
  • Across multiple generators and strategies, LLM-generated misinformation can be harder for both humans and detectors to detect than semantically matched human-written misinformation.

2 TAXONOMY OF LLM-GENERATED MISINFORMATION

The paper taxonomizes LLM-generated misinformation across types, domains, sources, intents, and errors. This framework organizes both the content characteristics and the circumstances under which misinformation is produced.

  • The taxonomy covers misinformation types, domains, sources, intents, and errors.
  • Types: Types include fake news, rumors, conspiracy theories, clickbait, misleading claims, and cherry-picking.
  • Domains: Domains include healthcare, politics, science, finance, law, education, social media, and the environment.
  • Sources and intents: Sources are categorized as hallucination, arbitrary generation, and controllable generation, while intents are unintentional or intentional.
  • Errors: Errors include unsubstantiated content, total fabrication, outdated information, description ambiguity, incomplete fact, and false context.

3 RQ1: HOW CAN LLMS BE UTILIZED TO GENERATE MISINFORMATION?

The paper categorizes LLM-based misinformation generation methods by real-world scenarios and tests their interaction with ChatGPT safeguards. It finds that LLMs can generate misinformation across varied content dimensions, with some methods often bypassing safeguards.

  • Generation approaches: The paper defines Hallucination Generation as nonfactual content arising from autoregressive generation and lack of up-to-date information.
  • Generation approaches: Arbitrary Misinformation Generation lets malicious users intentionally prompt LLMs for unconstrained or partially constrained misinformation.
  • Generation approaches: Controllable methods can preserve source semantics while changing style or expanding misleading sentences, and information manipulation can introduce specified errors.
  • Connection with jailbreak attacks: The proposed misinformation-generation approaches are motivated by real-world scenarios and are potentially combinable with jailbreak techniques.
  • LLMs can follow users’ instructions to generate misinformation in different types, domains, and errors.

4 LLMFake: LLM-GENERATED MISINFORMATION DATASET

The paper constructs LLMFake to study LLM-generated misinformation using multiple generators and generation approaches. Semantic analyses indicate that several controllable approaches generally preserve the source misinformation’s semantics while potentially changing its style.

  • LLMFake contains LLM-generated misinformation produced with different LLM generators and generation approaches.
  • The dataset excludes factual news items because the study compares detection difficulty between human-written and LLM-generated misinformation.
  • Semantic analysis: Paraphrase, rewriting, and open-ended generation generally preserve the original human-written misinformation’s semantic information.
  • Semantic analysis: In latent-space visualization, misinformation generated by these three methods shows majority overlap with human-written misinformation.
  • Style analysis: The study hypothesizes that these methods can manipulate style information to make misinformation more deceptive while preserving semantics.

5 RQ2: CAN HUMANS DETECT LLM-GENERATED MISINFORMATION?

The paper evaluates whether humans can detect ChatGPT-generated misinformation and finds that it is often difficult to identify, including when its semantics match human-written misinformation. The results indicate that generated misinformation may use more deceptive styles and increase human susceptibility.

  • Human Evaluation Setup: The human evaluation compares factuality annotation performance on human-written and ChatGPT-generated misinformation using the same group of evaluators.Ten Amazon MTurk evaluators labeled randomly shuffled news items as “factual” or “nonfactual.”
  • Results and Analysis: Humans generally struggled to detect misinformation generated through Hallucinated News, Totally Arbitrary, Rewriting, and Open-ended Generation.The Success Rate% metric measures the percentage of misleading news items successfully identified.
  • Results and Analysis: 9.6% Success Rate% was achieved for identifying hallucinated news on average.The errors included false dates, names, addresses, numbers, and quotes, making fine-grained hallucinations difficult to notice.
  • Results and Analysis: Human detection performance was mostly lower for LLM-generated misinformation than for human-written misinformation with the same semantics.The comparison focuses on Paraphrase, Rewriting, and Open-ended Generation because these methods generally preserve semantics while changing style.
  • Results and Analysis: The difference was statistically significant for Rewriting Generation (p-value = 9.15 × 10−5) and Open-ended Generation (p-value = 1.01 × 10−6).A paired T-test was used for these comparisons.
  • Results and Analysis: Finding 2: LLM-generated misinformation can be harder for humans to detect than human-written misinformation with the same semantics.The paper interprets this finding as evidence that carefully designed prompting can produce more deceptive styles for humans.

6 RQ3: CAN DETECTORS DETECT LLM-GENERATED MISINFORMATION?

The paper evaluates misinformation detectors, especially in zero-shot settings, and finds that LLM-generated misinformation is generally harder to detect than semantically matched human-written misinformation. These results expose practical challenges for existing supervised detectors and show that detector performance varies by model.

  • Emerging Challenges for Misinformation Detectors: Conventional supervised detectors face limited factuality supervision and difficulty maintaining effectiveness against misinformation generated at scale across domains, types, and errors.The paper therefore describes conventional supervised approaches such as BERT as likely impractical in practice.
  • Evaluation Setting: The study uses zero-shot LLM detectors to compare detection difficulty for LLM-generated and human-written misinformation.Zero-shot evaluation is intended to better reflect real-world scenarios where conventional supervised detectors may be impractical.
  • LLM Detection Performance: Detectors generally struggled with LLM-generated misinformation, especially Hallucinated News, Totally Arbitrary Generation, and Open-ended Generation.ChatGPT-3.5 detected 0.0% and GPT-4 detected 10.0% of generated hallucinated news.
  • LLM Detection Performance vs. Human Detection Performance: GPT-4 outperformed humans on detecting LLM-generated misinformation, whereas humans outperformed ChatGPT-3.5.The comparison concerns detector and human detection performance on generated misinformation.
  • LLM-Generated Misinformation vs. Human-Written Misinformation: Detector performance on LLM-generated misinformation was mostly lower than on human-written misinformation with the same semantics.The comparison used Paraphrase, Rewriting, and Open-ended Generation alongside human-written datasets.
  • LLM-Generated Misinformation vs. Human-Written Misinformation: 19.6% was the performance drop for Llama2-7B with “CoT” when detecting ChatGPT misinformation generated via Rewriting Generation instead of Politifact misinformation.The paper reports that most corresponding p-values in Appendix B were below 5%.
  • LLM-Generated Misinformation vs. Human-Written Misinformation: Finding 3: LLM-generated misinformation can be harder for misinformation detectors to detect than human-written misinformation with the same semantics.The paper states that existing detectors may be less effective and that malicious users could potentially use LLMs to evade detection.

7 IMPLICATIONS ON COMBATING MISINFORMATION IN THE AGE OF LLMS

The paper finds that LLM-generated misinformation can be harder for humans and detectors to detect, potentially giving it more deceptive styles and greater harm potential. It discusses implications and lifecycle-wide countermeasures.

  • LLMs can generate misinformation unintentionally through hallucinations or intentionally through malicious prompting.The paper links these scenarios to normal users and malicious users, respectively.
  • LLM-generated misinformation can be harder for humans and detectors to detect than human-written misinformation with the same semantics.
  • The greater detection difficulty suggests that LLM-generated misinformation can have more deceptive styles.The paper attributes this possibility to linguistic features of LLM content or carefully designed prompts.
  • The paper considers countermeasures across the whole LLM lifecycle rather than at only one stage.

8 COUNTERMEASURES THROUGH LLMS’ LIFECYCLE

The paper organizes countermeasures across three stages of the LLM lifecycle and describes training-stage interventions to reduce hallucinations and harmful-content generation.

  • The LLM lifecycle is divided into three stages for discussing countermeasures against LLM-generated misinformation.
  • Training stage: Training data can be curated to remove nonfactual articles and grounded in existing knowledge bases to reduce LLM hallucinations.
  • Training stage: Alignment training processes such as RLHF can reduce the risk of generating harmful content.

9 CONCLUSION

The paper characterizes LLM-generated misinformation, validates ways to produce it, and compares its detection difficulty with human-written misinformation. It finds that LLM-generated misinformation can be harder to detect and discusses lifecycle-wide countermeasures.

  • The paper builds a taxonomy of LLM-generated misinformation by types, domains, sources, intents, and errors.
  • It categorizes potential real-world misinformation-generation methods and validates that LLMs can generate misinformation across different types, domains, and errors.
  • LLM-generated misinformation can be harder for humans and detectors to detect than human-written misinformation with the same semantics.
  • The findings indicate that LLM-generated misinformation can have more deceptive styles and potentially cause more harm.
  • The paper discusses implications and countermeasures throughout the whole LLM lifecycle.

REPRODUCTION STATEMENT

The paper reports implementation settings, source datasets, and construction procedures for LLMFake, an LLM-generated misinformation dataset designed to compare detection difficulty with human-written misinformation.

  • Implementation details: ChatGPT uses default OpenAI API settings, while Llama2 and Vicuna use top_p = 0.9, temperature = 0.8, and max_tokens = 2,000.
  • Dataset sources: LLMFake draws human-written misinformation from Politifact, Gossipcop, and CoAID, covering political, entertainment, and healthcare domains.
  • Dataset construction: The constructed LLMFake dataset excludes factual news items because it is intended to compare LLM-generated and human-written misinformation.
  • Generation methods: ChatGPT generates 100 hallucinated news items and 100 arbitrary misinformation items for two dataset-generation methods.
  • Generation methods: Partially Arbitrary Generation produces 100 healthcare fake-news items and 100 political rumors.
  • Generation methods: Paraphrase, rewriting, open-ended, and information-manipulation methods generate misinformation from nonfactual or factual source material across multiple LLM generators and error types.

ETHICS STATEMENT

The paper argues that LLM-based misinformation-generation methods reflect real-world scenarios and are straightforward to implement, so they may already be used unintentionally or maliciously.

  • LLM-based misinformation-generation methods are based on real-world scenarios and straightforward to implement.
  • The authors anticipate that normal people may use these methods unintentionally, while malicious users may use them intentionally.

A MORE EXPERIMENT RESULTS

The experiments examine detector performance across multiple open-source LLM generators and compare human-written with generated misinformation. Supplementary tables also provide statistical comparisons, generation examples, and examples of detector decisions.

  • A MORE EXPERIMENT RESULTS: Detector performance is reported for misinformation generated by Llama2-7b-chat, Llama2-13b-chat, Llama2-70b-chat, Vicuna-7b-v1.3, Vicuna-13b-v1.3, and Vicuna-33b-v1.3.The metric is Success Rate %, with average performance over three runs for ChatGPT-3.5 or GPT-4 detectors because of API-output variance.
  • A MORE EXPERIMENT RESULTS: Statistical-significance tables compare human detection and detector detection difficulty between human-written and LLM-generated misinformation across generation approaches.The comparisons use paired T-tests.
  • A MORE EXPERIMENT RESULTS: Supplementary examples cover hallucinated news, totally arbitrary generation, and other ChatGPT-generated misinformation approaches, with hallucinated information highlighted in orange.The examples include invented dates, names, addresses, numbers, and quotes, while blue text marks the human-written input.

F MORE DETAILS OF MISINFORMATION DETECTORS

The paper evaluates LLMs as misinformation detectors using standard and chain-of-thought prompting, and illustrates both correct and incorrect judgments on generated misinformation.

  • F MORE DETAILS OF MISINFORMATION DETECTORS: The No CoT detector prompt asks an LLM to classify a passage as misinformation using only “YES” or “NO”.
  • F MORE DETAILS OF MISINFORMATION DETECTORS: The CoT strategy asks the detector to think step by step, output its reasoning, and then answer “YES” or “NO”.
  • F MORE DETAILS OF MISINFORMATION DETECTORS: Examples evaluate GPT-4 with No CoT and CoT prompts on misinformation generated through Rewriting Generation, showing both correct and incorrect predictions.The examples are presented in Tables 25–28.
  • F MORE DETAILS OF MISINFORMATION DETECTORS: In one CoT example, the detector checks individual claims against reliable sources and concludes that unsupported claims indicate misinformation.The response identifies fabricated or distorted claims and conspiracy-theory language before answering YES.
Loading 2309.13788v5…