PaperlayerView exact revision on arXiv

Source-linked AI summary

The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection

Wei Ai, Yilong Tan, Yuntao Shou, Tao Meng, Haowen Chen, Zhixiong He, Keqin Li

arXiv:2601.15316v1cs.AIcs.CV

TL;DR

Multimodal fake news detection needs stronger handling of semantic and cross-modal deception than early shallow-fusion methods provided. This survey reviews LVLM-based MFND, organizes methods and benchmarks, and identifies technical challenges and future research directions. It reports that LVLMs have driven a shift toward unified multimodal reasoning, while robustness to adaptive manipulation remains a limitation.

  • Problem

    Existing MFND methods struggled with high-level semantic understanding and complex cross-modal interactions, while the field lacked a systematic survey of LVLM-based developments.

  • Method

    The paper provides a structured review of LVLM-based MFND, covering parameter-freezing, parameter-tuning, and reasoning-oriented approaches, datasets, metrics, and architectures.

  • Results

    The survey documents LVLMs' transition toward joint multimodal representation learning and reasoning for fake news detection, alongside comparisons of methods and benchmarks.

  • Takeaways & Limitations

    The paper frames LVLM-based MFND as a developing paradigm whose progress depends on integrating semantic grounding, cross-modal verification, and scalable deployment strategies.

  • Takeaways & Limitations

    Current LVLM-based detectors remain vulnerable to adaptive manipulation and semantic-preserving perturbations that can sharply degrade performance.

Abstract

from arXiv · show

In recent years, the rapid evolution of large vision-language models (LVLMs) has driven a paradigm shift in multimodal fake news detection (MFND), transforming it from traditional feature-engineering approaches to unified, end-to-end multimodal reasoning frameworks. Early methods primarily relied on shallow fusion techniques to capture correlations between text and images, but they struggled with high-level semantic understanding and complex cross-modal interactions. The emergence of LVLMs has fundamentally changed this landscape by enabling joint modeling of vision and language with powerful representation learning, thereby enhancing the ability to detect misinformation that leverages both textual narratives and visual content. Despite these advances, the field lacks a systematic survey that traces this transition and consolidates recent developments. To address this gap, this paper provides a comprehensive review of MFND through the lens of LVLMs. We first present a historical perspective, mapping the evolution from conventional multimodal detection pipelines to foundation model-driven paradigms. Next, we establish a structured taxonomy covering model architectures, datasets, and performance benchmarks. Furthermore, we analyze the remaining technical challenges, including interpretability, temporal reasoning, and domain generalization. Finally, we outline future research directions to guide the next stage of this paradigm shift. To the best of our knowledge, this is the first comprehensive survey to systematically document and analyze the transformative role of LVLMs in combating multimodal fake news. The summary of existing methods mentioned is in our Github: \href{https://github.com/Tan-YiLong/Overview-of-Fake-News-Detection}{https://github.com/Tan-YiLong/Overview-of-Fake-News-Detection}.

1. Introduction

Multimodal fake news exploits interactions among deceptive text, images, and videos, exposing the limits of unimodal and shallow-fusion detection. This survey organizes LVLM-based MFND methods, benchmarks, challenges, and future directions.

  • Modern fake news increasingly combines manipulated images, misleading videos, and semantically deceptive captions, motivating multimodal detection.
  • MFND must detect subtle semantic mismatches, visual-entailment contradictions, and temporal or contextual incoherence across modalities.
  • Early late-fusion and shallow-alignment methods lacked the semantic depth and reasoning capacity needed for nuanced deception.
  • LVLMs enable joint cross-modal representation learning and reasoning, supporting high-level fake news detection through semantic grounding and verification.
  • The survey classifies LVLM-based MFND into parameter-freezing, parameter-tuning, and reasoning-paradigm applications.
  • It reviews architectures, datasets, metrics, design principles, challenges, and future directions, including hallucination, cross-modal bias, data scarcity, counterfactual reasoning, and knowledge-enhanced adaptation.

2. Preliminary Information

Traditional fake-news detection begins with single-modality classification, but multimodal fake news requires joint reasoning over claims, media, context, and cross-modal evidence. The section formalizes LVLM-based detection around authenticity, consistency, intent, verification, and grounded explanations.

  • Traditional Single-Modal Fake News Detection: Single-modal classifiers map textual or visual features to real/fake labels, but cannot address deception distributed across modalities.Text-based methods use linguistic or semantic features, while image-based methods extract visual artifacts with CNNs or vision Transformers.
  • Multimodal Fake News Detection: MFND infers a veracity label from text, images or video, claims, and optional external context while producing a referenceable rationale grounded in cross-modal evidence.The formalization treats 1 as fake and represents explanations through pointers to spans, frames, and regions.
  • Principled Dimensions of Deception: The proposed deception decomposition separates authenticity, cross-modal consistency, and manipulative intent as interacting dimensions of veracity reasoning.Authenticity concerns media integrity, consistency concerns semantic agreement, and intent concerns rhetorical efforts to steer beliefs.
  • Learning and Reasoning: Reasoning constructs an evidence graph and selects a minimal rationale whose modality-aligned snippets and relations can be checked for sufficiency and faithfulness.The rationale may contain text spans, frames, regions, and explicit support or contradiction relations.
  • Learning and Reasoning: The learning objective combines forgery detection, cross-modal consistency, intent recognition, and explanation generation to support accurate and grounded rationales.The explanation loss regulates rationales grounded in multimodal evidence, while contrastive alignment makes the consistency score reflect cross-modal agreement.
  • Unified LVLM Parameterization: LVLMs instantiate joint multimodal representations through modality encoders, retrieval-conditioned aggregation, and dedicated heads for authenticity, consistency, and intent.The framework uses cross-attention to form a joint representation and integrates retrieved context through a claim-conditioned verifier.

3. Taxonomy

The survey organizes multimodal fake news detection into parameter-frozen, parameter-tuning, and reasoning paradigms. These paradigms trade adaptation and efficiency against reasoning control, interpretability, computational cost, and robustness.

  • The taxonomy comprises Parameter-Frozen, Parameter-Tuning, and Reasoning paradigms for multimodal fake news detection.
  • Parameter-Frozen Paradigm: Parameter-frozen methods keep the LVLM backbone unchanged and adapt it through prompting or in-context specification.This avoids expensive fine-tuning and supports rapid transfer across heterogeneous datasets and platforms.
  • Parameter-Frozen Paradigm: Frozen-parameter prediction maps class probabilities over verbalized labels and selects the class with maximum probability.The formulation uses frozen parameters Θ and a verbalizer ν(y) mapping classes such as fake and real to label strings.
  • Parameter-Tuning Paradigm: Parameter-tuning methods optimize model parameters through full fine-tuning or parameter-efficient techniques such as adapters and related lightweight mechanisms.Full tuning captures fine-grained text-image interactions but incurs substantial computational costs, motivating more efficient alternatives.
  • Parameter-Tuning Paradigm: Parameter-efficient systems reduce training overhead while retaining cross-domain generalization, interpretability, and sensitivity to domain-specific context when annotated data is scarce.Examples include selective adaptation and knowledge-guided reasoning with PET-based designs.
  • Reasoning Paradigm: Reasoning methods include prompt-based and agent-based approaches that improve multimodal verification through structured reasoning, evidence coordination, or counter-argumentation.Prompting is efficient and supports zero-shot generalization but combines perception, reasoning, and decision-making in one pass, limiting controllability for complex multi-hop misinformation; agent-based systems improve transparency and error localization at greater computational and system complexity.

4. Popular Benchmark Dataset

Benchmark datasets have evolved from small, primarily textual rumor collections toward larger resources covering multiple modalities, distortion types, languages, domains, and detection tasks.

  • Dataset Evolution: Dataset development expanded from early rumor datasets to multimodal resources integrating text, images, videos, and social context.This progression increased the scale and complexity of available benchmarks.
  • Development Trends: Dataset scale grew from thousands or tens of thousands of examples to hundreds of thousands or millions, supporting deep learning and large-scale pretrained models.
  • Development Trends: Modal coverage expanded from text to image-text alignment, multilingual content, audio, video, and social interaction information.
  • Development Trends: Task support broadened from binary authenticity classification to multi-class labeling, tampered-region localization, image-text inconsistency detection, and AI-generated content recognition.
  • Development Trends: Cross-language and cross-domain datasets support research on model transferability and cross-cultural adaptability.
  • Dataset Taxonomy: Later benchmarks incorporate visual and cross-modal distortions, while MM-COVID, M3FD, and DriftBench cover textual veracity, visual veracity, and cross-modal consistency.

5. Evaluation Metrics

Evaluation of multimodal fake news detection systems must address classification performance alongside imbalance, robustness, generalization, cross-modal consistency, and interpretability. Standard metrics quantify predictive behavior for binary and multi-class settings, with Macro-F1 emphasizing class balance and Micro-F1 emphasizing aggregate instance performance.

  • Evaluation Scope: Evaluation frameworks consider classification performance, robustness and generalization, cross-modal consistency, interpretability, and user perception.
  • Classification Metrics: Accuracy measures the proportion of correctly classified instances among all predictions.Its components include true positives, true negatives, false positives, and false negatives.
  • Classification Metrics: Precision measures the reliability of positive predictions, whereas recall measures coverage of actual fake instances.High recall is often prioritized in misinformation mitigation to reduce undetected disinformation.
  • Classification Metrics: F1 Score combines precision and recall through their harmonic mean.
  • Multi-class Metrics: Macro-F1 averages per-class F1 scores equally, highlighting performance on minority classes in multi-class fake news taxonomies.
  • Multi-class Metrics: Micro-F1 aggregates contingency counts globally, reflecting overall instance-level performance and aligning closely with accuracy in balanced scenarios.

6. Experimental Performance

Across benchmarks, LVLM-based detection improves when models combine greater capacity with structured reasoning, external knowledge, and adaptive evidence processing. However, performance remains vulnerable to evolving content distributions and complex multimodal distortions.

  • MM-FakeBench: 7B models generally score below 50 macro-F1 on MM-FakeBench, while 13B models improve but remain below human-level F1 of 54.9.MMD-Agent further improves handling of visual veracity distortions through multi-step reasoning and confidence calibration.
  • MM-FakeBench: Structured reasoning, confidence calibration, and adaptive visual amplification improve performance beyond model capacity alone across misinformation distortions.The findings emphasize modular inference and adaptive refinement rather than scaling alone.
  • Twitter and Fakeddit: 0.824 accuracy and 0.816 F1 on Twitter let LEMMA outperform GPT-4V with CoT, which reaches 0.757 accuracy and 0.758 F1.LEMMA also maintains stronger class-level precision-recall balance, whereas FacTool tends toward false positives from superficial cues.
  • Twitter and Fakeddit: 0.828 accuracy and 0.857 F1 on Fakeddit show LEMMA outperforming all baselines while maintaining stable performance across datasets.The reported stability suggests less reliance on domain-specific artifacts or superficial cues.
  • Twitter and Fakeddit: Removing LEMMA’s initial-stage inference reduces accuracy to 0.781 on Twitter and 0.803 on Fakeddit, while removing visual retrieval also lowers performance.These ablations support early filtering and external visual knowledge as functional components of the system.
  • DriftBench: Performance drops sharply under diversified content: GPT-4o-mini falls from 83.3% to 64.4% accuracy, while SNIFFER and LEMMA are more stable but still decline.DriftBench exposes limited generalization across evolving distributions, especially for GenAI-driven content heterogeneity.

7. Future Directions

Future MFND research should address LVLMs’ superficial reasoning, vulnerability to evolving and shifted misinformation, deployment constraints, and hallucinated evidence. Proposed directions include causal counterfactual training, adversarial robustness, modular cascades, early exits, hallucination-aware inference, and knowledge enhancement.

  • Causal and Counterfactual Reasoning for Explainable Detection: LVLMs often rely on superficial correlations rather than concrete visual-textual inconsistencies, producing brittle decisions and weak explanations.These weaknesses limit suitability for high-stakes journalism, legal review, and public policy applications.
  • Causal and Counterfactual Reasoning for Explainable Detection: Counterfactual examples that replace images or paraphrase text can supervise models to distinguish causally relevant evidence from correlated cues.The proposed tests preserve or revise fake predictions according to whether the claim remains unsupported or becomes inconsistent with the altered modality.
  • Adversarially Robust and Out-of-Distribution Generalization: Adversarial tactics and distribution shifts expose LVLMs trained on static, historically narrow datasets to context swapping, semantically aligned deepfakes, prompt injection, and unseen events.Static benchmarks such as FakeNewsNet and Weibo21 do not test generalization to future attack vectors.
  • Efficient and Modular Architectures for Real-Time Deployment: Real-time moderation requires alternatives to monolithic LVLMs because platforms may process millions of posts under latency, energy, and hardware constraints.The passage identifies response-time limits of less than 500 milliseconds per post and a mismatch between accurate but slow models and fast but shallow heuristics.
  • Efficient and Modular Architectures for Real-Time Deployment: Multi-stage cascades can route routine content through lightweight screeners and activate specialized verifiers only for ambiguous or high-risk cases.The first stage can filter more than 80 percent of total traffic, while independently updateable modules handle temporal or semantic checks.
  • Hallucination-Aware Detection: Hallucination-aware inference and faithfulness-oriented metrics are needed because fabricated visual entities or cross-modal relations can make false claims appear credible.Suggested strategies include ICD, Cogsteer, self-consistency, and explicit measurement of evidence grounding and hallucination rates.
  • Knowledge-Enhanced Reasoning: Knowledge-enhanced LVLMs may improve veracity assessment by distinguishing visually plausible claims from contextually false ones.The proposed direction emphasizes dynamic knowledge generation and fusion for multimodal reasoning.

8. Conclusion

The survey reviews the development of LVLM-based multimodal fake news detection, organizes methods and evaluation resources, compares methods across datasets, and identifies unresolved challenges and future directions.

  • Conclusion: The paper reviews MFND background, traditional multimodal methods, LVLM-based methods, evaluation metrics, datasets, performance comparisons, unresolved challenges, and future directions.It organizes current methods into parameter-frozen and parameter-tuning paradigms and reviews representative models within each.
Loading 2601.15316v1…