PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Multimodal Large Language Models Generate and Detect Multimodal Social Media Fake News?

Jiyao Yang, Yang Liu, Zhenyue Qin, Qingyu Chen, Xiuzhen Zhang

arXiv:2609.35809v1cs.CLcs.AI

TL;DR

Generative AI lowers the barrier to producing disinformation, while whether MLLMs can generate realistic multimodal fake news and detect it remains a key question. The paper builds a multi-agent framework and benchmarks 16 MLLMs; most perform substantially below human-level detection, with image-authenticity assessment a major weakness.

  • Problem

    Generative AI lowers the barrier to mass-producing multimodal content, raising concern about its misuse for disinformation.

    Source: §1 · p. 1

  • Method

    A story agent, image agent, and critic agent generate fake posts, and the study benchmarks 16 open- and closed-source MLLMs on paired true and fake posts.

    Source: §1 · p. 2

  • Results

    Most MLLMs fall far short of human detection accuracy, with accuracy as low as 52.50% versus 97.33% for human annotators, and many struggle with image authenticity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation highlights the complexity of AI-generated fake-news detection and the need for continued research.

    Source: §5 · p. 9

    The evaluation covers science, health, and entertainment domains, although the framework can generate posts on any topic.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The rapid advancement of generative AI raises concerns about the misuse of Multimodal LLMs (MLLMs) for large-scale disinformation campaigns on social media. Despite existing research on textual disinformation, a fundamental question remains unanswered: can MLLMs be exploited to fabricate realistic multimodal fake news, and can they reliably detect it? We introduce a multi-agent framework in which a story agent, an image agent, and a critic agent collaborate to produce fake social media posts that plausibly counter true news. We apply the framework to generate over 9,000 paired multimodal news posts across science, health, and entertainment domains, and benchmark 16 open- and closed-source MLLMs for automated detection. We find that most models fall substantially short of human-level accuracy and fail critically on identifying image authenticity. Our research provides a foundation for developing robust defenses against social media fake news. Code and data are available at https: //github.com/xiuzhenzhang/Multimodal.

1 Introduction

The paper addresses the gap in generating and detecting realistic multimodal fake news by introducing a multi-agent generation framework and benchmarking MLLMs. Most models fall short of human-level detection, especially for image authenticity.

1 Introduction

  • Prior work lacks a framework for generating high-fidelity multimodal social-media fake news at scale and a comprehensive benchmark of MLLM detection.Earlier work focused on text-only content or simple caption–image pairs requiring location and time metadata.Source: §1 · p. 2
  • The proposed multi-agent framework creates event-grounded fake posts that preserve factual anchors while reversing a true-news narrative.The Rosetta example changes a successful deliberate soft crash into a fabricated navigation-malfunction story.Source: Figure 1 · p. 1, §1 · p. 2
  • Most MLLMs fall far short of human detection accuracy, and many struggle to assess image authenticity despite relatively strong image–text relevance judgments.Accuracy reaches as low as 52.50% for MLLMs versus 97.33% for human annotators.Source: §1 · p. 2
  • The study generates over 9,000 multimodal fake-news posts and evaluates 16 open- and closed-source MLLMs for detection.The posts span science, health, and entertainment, and the benchmark uses direct prompting and structured chain-of-thought settings.Source: §1 · p. 2

2 Related Work

Prior research examined text-only fake-news generation and detection, while multimodal generation work remained limited in scope. This paper distinguishes its approach through event-grounded construction of complete multimodal posts with explicit factual changes.

2 Related Work

  • Earlier work explored LLM-based fake-news generation and detection, but these studies focused on text-only content.GROVER used GPT-2 to generate articles from titles, while later work examined LLM-generated disinformation narratives.Source: §2 · pp. 2–3
  • Huang et al. (2024) generated misleading caption–image pairs, conditioned on specified temporal and location information.The cited work focused on caption-level manipulation rather than complete event narratives.Source: §2 · p. 3
  • This framework starts from complete articles, extracts event-level anchors, modifies verifiable claims, and generates visuals supporting the altered narrative.The Critic Agent checks factual changes, narrative consistency, and image–text alignment.Source: §2 · p. 3

3 Multi-Agent Framework for Multimodal News Post Generation

The framework generates event-grounded fake social-media posts through specialized agents that construct stories, create aligned images, and review both outputs. It uses iterative feedback and retains only outputs that pass textual and visual review.

3.1 Framework Overview

  • The framework sequences story, image, and critic agents with feedback loops, retaining posts only after text- and image-level review.True posts are also summarized into newsfeed style so their format is comparable to fake posts rather than raw articles.Source: §3.1 · pp. 3–4

3.2 Story Generator

  • The Story Generator extracts factual anchors, preserves the source event, and alters concrete claims to create a misleading but recognizable narrative.The fake story can distort outcomes, decisions, numbers, dates, causes, or affected groups.Source: §3.2 · p. 4

3.3 Image Generator

  • The Image Generator selects a visual format suited to the fake story’s claim type and creates images that reinforce its misleading narrative.Formats include documentary photos, screenshots, official notices, infographics, and timelines.Source: §3.3 · p. 4

3.4 Critic Agent

  • The Critic Agent checks textual faithfulness and distortion as well as image realism, readability, coherence, and format suitability.It returns structured revision feedback and approved outputs retain review records for traceability.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate paired true/fake posts across three domains and benchmark MLLMs against human judgments. They find substantial model variation, weaknesses in cross-domain generalization and image-authenticity assessment, and model-dependent effects from chain-of-thought prompting.

4.1 Dataset and Generation Evaluation

  • 4,507 copyright-free articles across science, health, and entertainment yielded paired true/fake posts; fake posts alter key claims while retaining a compact newsfeed style.Posts averaged 90.05 words; true posts averaged 103.93 words versus 76.16 for fake posts.Source: §4.1 · p. 5
  • The full generation pipeline scored 4.540, above one-shot generation (4.374), no Critic Agent (4.314), and no iterative revision (4.428).In paired comparisons, the full pipeline outperformed those variants in 32, 34, and 28 cases, respectively, mainly through better factual-anchor preservation, reduced factual drift, and stronger image–text consistency.Source: §4.1 · p. 5

4.2 Performance and Discussions

  • GPT-5.5 CoT achieves the highest F1 on NIH (99.50), Nature (99.38), and Snopes (98.75), while Qwen-VL-Max CoT reaches 85.26 averaged F1 and Qwen3-VL-8B CoT 73.36 averaged F1.Open-source Qwen models generally lead other open-source baselines; closed-source models perform more strongly overall.Source: §4.2 · p. 6, Table 1 · p. 6
  • Snopes has the lowest F1 in 25 of 32 model–prompt settings, indicating limited cross-domain generalization; its entertainment claims may require contextual world knowledge.The passage notes nuanced, satirical, or context-dependent claims as possible reasons.Source: §4.2 · pp. 6–7
  • CoT improves average F1 for 12 of 16 models but degrades four; gains range from 9.83 to 15.03 points for three models, while decision criteria shift for others.The reported error analysis attributes drops mainly to shifted criteria, including overly permissive or conservative judgments.Source: §4.2 · p. 7
  • CoT can reduce prediction coverage: GPT-5.5 drops from 98.53% to 94.83%, InternVL3.5-4B from 100.00% to 92.11%, and LLaVA-1.5-7B from 100.00% to 95.51%.The reported causes include refusals or omitted final labels; invalid open-source outputs more often reflect instruction-following errors, while closed-source invalid outputs more often involve safety refusals.Source: §4.2 · p. 7

4.3 Fine-grained analysis

  • Human annotators reached 97.33% veracity accuracy; among four selected models, GPT-5.5 CoT had perfect final accuracy and 89.9% agreement with human verification cues.Image authenticity was a common weakness: three of four models showed low agreement with humans, while Qwen3-VL-8B CoT reached 62.2% on that cue.Source: §4.3 · p. 8
  • In a hypertension-pregnancy example, several models treated institutional references, precise statistics, fluent text, or relevant-looking visuals as signs of reliability despite misleading framing.GPT-4o CoT, Qwen3-VL-8B CoT, and Qwen2-VL-7B CoT incorrectly predicted true; GPT-5.5 CoT correctly identified suspicious framing and synthetic visual cues.Source: §4.3 · p. 9

5 Conclusion

The paper presents an agent framework and annotated datasets for detecting AI-generated multimodal fake news across three domains. Its evaluation finds significant MLLM limitations, underscoring the complexity of the detection task and the need for continued research.

5 Conclusion

  • The paper proposes an agent framework and annotated datasets for AI-generated fake-news detection across science, health, and entertainment news about real-world events.The paper also benchmarks modern MLLMs on the resulting resource.Source: §5 · p. 9
  • The evaluation reveals significant MLLM limitations, highlighting task complexity and the need for continued research in AI-generated multimodal fake-news detection.Source: §5 · p. 9

Limitations

The framework is presented as applicable to any topic, but its evaluation covers only science, health, and entertainment; the benchmark includes a limited number of closed-source MLLMs.

Limitations

  • Evaluation was limited to science, health, and entertainment because copyright-free news articles were available in those areas.The framework is described as usable for any topic, but this evaluation does not extend beyond the three domains.Source: Limitations · p. 9
  • The benchmark includes only a limited number of closed-source MLLMs because of cost and resource constraints.Source: Limitations · p. 9

Ethical Considerations

The artifacts are intended for research on defenses against AI-generated fake news, with access controlled to address misuse risk while supporting reproducibility.

Ethical Considerations

  • Code and data are available upon registered request for ethical research purposes, without permission to reshare.The authors acknowledge that malicious actors could misuse the generation system.Source: Ethical Considerations · p. 9

A G-Eval Prompts for Checking the Quality of Generated Text

The appendix specifies G-Eval-style prompts for assessing generated news text quality, using coherence, fluency, and faithfulness dimensions. The evaluator is instructed to assess text quality rather than factual truth.

A G-Eval Prompts for Checking the Quality of Generated Text

  • The prompts assess coherence, fluency, and faithfulness on a 1–5 scale, with higher scores indicating better quality.The evaluator assesses generation quality rather than fake-news detection and is instructed not to judge factual truth using external knowledge.Source: A G-Eval Prompts for Checking the… · p. 11
  • Coherence concerns logical connection and reasonable ordering of ideas, understandable causal relations, and avoidance of contradictions or abrupt logical jumps.The scale ranges from incoherent to highly coherent.Source: A G-Eval Prompts for Checking the… · p. 11
  • Fluency measures grammatical, natural, readable, and stylistically smooth language, with scores ranging from very hard to read to fluent and readable.The evaluation focuses on expression quality rather than factual truth.Source: A G-Eval Prompts for Checking the… · p. 11
  • Faithfulness measures focus on the stated topic and sufficient context, including avoidance of irrelevant information and major gaps.The instructions explicitly exclude judging external factual truth; the scale runs from largely unfaithful to highly faithful.Source: A G-Eval Prompts for Checking the… · p. 11

B Examples of Generated True and Fake News Posts

The appendix illustrates generated true and fake posts with one example each from NIH/Health, Nature/Science, and Snopes/Entertainment.

B Examples of Generated True and Fake News Posts

  • Examples pair each original true-news post with a generated fake post that preserves concise newsfeed style while introducing factual deviations.The figure includes examples from NIH/Health, Nature/Science, and Snopes/Entertainment.Source: B Examples of Generated True and Fake… · p. 11

C Human Annotation

Human annotators reviewed each news post’s image and text jointly, completed four diagnostic checks, and then judged its veracity.

C Human Annotation

  • Annotators reviewed each news post’s text and associated image jointly before assigning a final true/fake verdict.They were instructed to judge from the provided materials without external search engines or AI tools.Source: C Human Annotation · pp. 11–12
  • Four required checks covered factual consistency, language quality, image–text relevance, and image authenticity.Annotators could also leave optional free-text comments, which were not required for quantitative analysis.Source: C Human Annotation · pp. 11–12, Table 3 · p. 12

D Prompts for MLLMs for Fake News Detection

The appendix specifies direct and structured chain-of-thought prompts for multimodal fake-news detection. Both use the news text and image, while structured CoT adds four diagnostic checks before the final verdict.

D Prompts for MLLMs for Fake News Detection

D.1 Direct Judgment Setting

  • The direct setting asks the model to judge the headline, body, and image together without intermediate diagnostic outputs.Its output is a compact JSON prediction containing a verdict label, verdict name, and confidence.Source: §D.1 · p. 12

D.2 Structured CoT Setting

  • Structured CoT requires four checks—factual errors, language issues, image–text relevance, and image authenticity—before the final verdict.The format also allows other reasons to support or weaken the judgment.Source: §D.2 · pp. 12–13

D.3 CoT Diagnostic Checks

  • Structured CoT uses the same four verification dimensions as human annotation, enabling comparison of model and human sub-decisions.The prompt requires concise reasoning fields and a structured final JSON response.Source: §D.3 · p. 13, §D.2 · p. 13
  • Figure 5 provides examples of true and fake news posts generated by the framework.Source: Figure 5 · p. 14
  • Figure 6 depicts the human-review interface, presenting the image, source, headline, and displayed text before diagnostic checks and the final judgment.Its checks match the dimensions used in the structured CoT evaluation prompt.Source: Figure 6 · p. 15

Related papers

Loading 2609.35809v1…