PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Consistency and Conflict across MLLM Layers for Fake News Detection

Kai Qian, Dongjie Xu, Weijie Shi, Jiaqi Wang, Haolun Ma, Jia Zhu, Jiajie Xu

arXiv:2610.36569v1cs.CVcs.CL

TL;DR

Multimodal fake-news detection must handle posts whose image and text broadly agree while a few mismatches carry decisive evidence, a challenge existing methods address coarsely. MLCCD selects relevant tokens, extracts evidence from changes across MLLM layers, and routes consistency and conflict separately. It outperforms evaluated baselines on three benchmarks.

  • Problem

    Existing methods model consistency and conflict coarsely and do not explicitly adapt evidence aggregation across semantic levels to individual samples, despite sparse image-text conflicts that can be decisive.

    Source: §1 · pp. 1–2

  • Method

    MLCCD selects task-relevant visual and textual tokens, retains representations that change between adjacent MLLM layers, and separately routes consistency and conflict evidence across layers.

    Source: §1 · p. 2, §4.1 · p. 3

  • Results

    MLCCD outperforms the compared baselines on three benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Controlled counterfactual evaluation shows MLCCD detects a larger proportion of local conflicts than direct LoRA fine-tuning while maintaining low false-alarm rates on evaluated controls.

    Source: §6 · p. 9

Abstract

from arXiv · show

The trustworthiness of Web content faces a growing threat from multimodal fake news. Real news usually shows aligned image-text semantics, while fake news often contains a few conflicts at varying levels, such as mismatches in objects, scenes, events, and claims. However, existing methods usually treat consistency and conflict in a coarse and static way, limiting their ability to capture sparse conflicts and adapt to varying conflict levels. Inspired by the distinct knowledge encoded across layers in large language models, we propose Multi-Level Consistency and Conflict Decoupling (MLCCD), which captures conflicts at different levels from different layers of multimodal large language models (MLLMs). Specifically, we first use the MLLM's final decision representation to select visual and textual tokens relevant to the task, preventing irrelevant content from diluting sparse conflicts. Then, we preserve tokens whose representations change sharply between adjacent layers, capturing the knowledge each layer adds. Finally, we decouple this knowledge into consistency and conflict evidence and aggregate each across layers with its own router, adapting to varying conflict levels. Experiments on three benchmarks show that MLCCD outperforms existing methods.

1 Introduction

Multimodal posts can appear broadly aligned while a few mismatched details carry decisive evidence, and existing methods model consistency and conflict too coarsely to adapt across samples. MLCCD uses MLLM layer representations to select relevant tokens, separate consistency from conflict evidence, and adaptively aggregate both; it outperforms compared baselines on three benchmarks.

1 Introduction

  • Existing approaches include contrastive alignment, consistency modeling, and higher-order interaction learning, but do not explicitly adapt aggregation across semantic levels to individual samples.Source: §1 · p. 1
  • Sparse image-text conflicts can be obscured when abundant agreement or irrelevant content dominates aggregation.The decisive mismatch may concern only a few details at different semantic levels.Source: §1 · pp. 1–2
  • The relative importance of consistency and conflict evidence varies across samples and semantic levels, making fixed evidence weighting inadequate.Source: §1 · p. 2
  • MLCCD selects task-relevant visual and textual tokens, retains representations that change sharply between adjacent layers, and routes consistency and conflict evidence separately across layers.The two evidence types are aggregated with their own routers to adapt to individual samples.Source: §1 · p. 2
  • Experiments on three benchmarks show that MLCCD outperforms the compared baselines.Source: §1 · p. 2

2 Related Work

Prior multimodal fake-news research models image-text relations through alignment, consistency, interactions, and knowledge-enhanced verification, but often treats consistency and conflict coarsely. Work on MLLMs motivates examining intermediate language-backbone layers, where cross-modal tokens interact, for evidence about these relations.

2.1 Multimodal Fake News Detection

  • Existing multimodal fake-news methods cover joint representations, cross-modal alignment, consistency, interactions, knowledge-enhanced verification, and MLLM-supported explanation or reasoning.Source: §2.1 · p. 2
  • These methods rarely exploit knowledge across MLLM layers and often model consistency and conflict coarsely or statically, limiting capture of sparse conflicts and adaptation to varying conflict levels.Source: §2.1 · p. 2

2.2 Multimodal Large Language Models

  • Intermediate MLLM layers can carry useful information, and the language backbone is where image and text tokens first interact.The paper therefore investigates consistency and conflict evidence across layers of that backbone.Source: §2.2 · p. 2

3 Preliminaries

The task is to estimate whether each multimodal post is fake or real. The setup uses a frozen, LoRA-tuned MLLM to provide layer-wise token representations, with separate masks identifying text and visual tokens.

3.1 Task Definition

  • The MLCCD overview routes layer-wise hidden states through token selection, discriminative layer-knowledge extraction, and adaptive consistency/conflict routing for prediction.Source: Figure 2 · p. 3
  • The task is to learn a prediction function that estimates each post’s fake-news probability from its multimodal input.Source: §3.1 · p. 3

3.2 MLLM Layer Representations

  • The setup uses a LoRA-tuned MLLM as a feature backbone, freezes it after tuning, and operates on its layer-wise token representations.Source: §3.2 · p. 3
  • Text and visual tokens are identified with separate binary modality masks, whose union defines the valid token mask.Source: §3.2 · p. 3

4 Methodology

MLCCD builds verification evidence from task-relevant tokens and layer-specific representations, then separates cross-modal agreement from discrepancy and routes each across layers. Its training objective combines classification loss with a regularizer that encourages complementary consistency and conflict streams.

4.1 Overview

  • MLCCD converts an image-text post into structured verification evidence, progressing from token selection to layer knowledge extraction and cross-modal decoupling across MLLM layers.It uses separate routers to aggregate evidence for agreement and discrepancy.Source: §4.1 · p. 3

4.2 Task Relevant Token Selection

  • The final decision representation serves as a query for selecting task-relevant visual and textual tokens, limiting dilution from background tokens.Selection is modality-wise, with token amounts adjusted to avoid the modality with more tokens dominating the evidence pool.Source: §4.2 · pp. 3–4

4.3 Discriminative Layer Knowledge Extraction

  • Tokens with the largest standardized representation changes between adjacent layers are retained as layer-distinctive evidence.The first layer's change score is set to zero because it has no preceding layer; retained-token weights integrate task relevance and layer distinctiveness.Source: §4.3 · p. 4

4.4 Adaptive Consistency and Conflict Routing

  • At each layer, MLCCD separates cross-modal agreement from discrepancy and routes the consistency and conflict streams independently according to layer-wise reliability.This decouples evidence at multiple semantic levels rather than only from a single global multimodal representation.Source: §4.4 · p. 4

4.5 Training Objective

  • Training combines binary cross-entropy classification loss with a disentanglement regularizer that discourages overly similar consistency and conflict representations.The LoRA-tuned MLLM remains frozen; only the proposed modules and classifier are optimized.Source: §4.5 · p. 5

5 Experiments

Across Weibo, Pheme, and AMG, MLCCD outperforms the evaluated baselines, with ablations and controlled evaluations examining its components, sensitivity to settings, and responses to local conflicts. The experiments also report image-dependence and runtime results.

5.1 Datasets and Metrics; 5.2 Baselines; 5.3 Implementation Details

  • Experiments use Weibo, Pheme, and AMG with accuracy and class/macro-F1 metrics, comparing conventional, adapted multimodal, and MLLM baselines; the Qwen3-VL-8B backbone is LoRA-adapted then frozen during evidence-network training.The datasets span different languages, event sources, and scales.Source: §5.1 · p. 5, §5.2 · p. 5, §5.3 · p. 5

5.5 Ablation Study

  • On Pheme, every ablation lowers accuracy and macro F1; using a shared router causes the largest macro F1 drop (2.27 points) and lowers fake-news F1 by 3.56 points.Removing TRTS, DLKE, ACCR, or Ldis lowers macro F1 by 1.28, 1.03, 1.03, and 0.64 points, respectively.Source: Table 2 · p. 6, §5.5 · p. 6

5.6 Hyperparameter Sensitivity

  • Text-retention sensitivity varies by dataset: macro F1 ranges from 95.23% to 95.33% on Weibo, while AMG peaks at 90.07% with 0.025 and Pheme at 91.73% with 0.1.On Pheme, the default image keep ratio of 0.3, token keep ratio of 0.5, and four routed layers achieve the highest macro F1 in their respective sweeps.Source: Figure 3 · p. 6, §5.6 · p. 6

5.4 Overall Performance

  • MLCCD achieves the highest accuracy and macro F1 on all three datasets; against Qwen3-VL-8B + LoRA, accuracy gains are 0.90 points on AMG, 1.56 on Pheme, and 0.26 on Weibo.It also achieves the highest real-class and fake-class F1 on all three datasets; its largest reported fake-class F1 gain is 3.22 points on Pheme (88.18% vs 84.96%).Source: §5.4 · pp. 5–6, Table 3 · p. 7

5.7 Counterfactual Evaluation

  • MLCCD detects 42.25% of controlled counterfactuals versus 18.50% for Qwen3-VL-8B + LoRA, with gains across all four conflict types and correct classifications of the constructed controls.The 400 edits change one contiguous span in constructed reference descriptions rather than original news text.Source: Table 4 · p. 7, §5.7 · p. 7

5.8 Analysis of Conflict Sparsity

  • At edit ratios of at most 5%, MLCCD detects 35.76% of counterfactuals versus LoRA’s 15.23%; both methods detect more as the edited span occupies a larger share of the text.The edit ratio is the changed-span token count divided by the counterfactual text-body token count.Source: Table 5 · p. 8, §5.8 · p. 8

5.9 Analysis of Image Dependence

  • Replacing original images with gray images reduces the mean paired score difference from 4.84 to 0.96; matched donor images reduce it from 4.70 to 1.02.The analysis measures changes in model responses; original compatibility labels may not apply after image replacement.Source: Table 6 · p. 8, §5.9 · p. 8

5.10 Efficiency Analysis

  • On Pheme, MLCCD requires 1,102 seconds to train, 30.4% longer than direct LoRA fine-tuning, while inference takes about 48 versus 45 seconds for 385 test samples.The reported comparison accompanies accuracy and macro F1 gains of 1.56 and 2.27 percentage points on Pheme.Source: §5.10 · p. 8, Table 7 · p. 9

6 Conclusion

MLCCD uses representations from different MLLM layers to construct and adapt consistency and conflict evidence for multimodal fake news detection.

6 Conclusion

  • Across three benchmarks, MLCCD improves over evaluated baselines, and counterfactual evaluation finds more local-conflict detections than direct LoRA fine-tuning with low false-alarm rates on evaluated controls.Timing results on PHEME indicate limited additional inference cost.Source: §6 · p. 9
  • MLCCD selects task-relevant visual and textual tokens, then retains representations that change substantially between adjacent layers to capture complementary information.Source: §6 · p. 9
  • It separates consistency and conflict evidence and aggregates them through two adaptive routers to address sparse conflicts and varying evidence needs.Source: §6 · p. 9

A Dataset Details

The evaluation uses Weibo, Pheme, and AMG, with dataset-specific splits and preprocessing.

A Dataset Details

  • MLCCD is evaluated on Weibo, Pheme, and AMG using each dataset’s standard split and preprocessing protocol where available.Source: A Dataset Details · p. 10
  • For Weibo, the original train/test split is retained, the highest-resolution image is used for multi-image posts, and samples with empty text or an invalid image are removed.The removed samples include 50 training and 66 test posts with empty text, plus one test sample without a valid image.Source: A Dataset Details · p. 10
  • For AMG, non-image, corrupted, and abnormally low-resolution media are removed, and texts are truncated to the first 1024 words.Source: A Dataset Details · p. 10
  • Evaluation reports accuracy, macro F1, real-class F1, and fake-class F1.Source: A Dataset Details · p. 10

B Implementation Details

The implementation combines a compact evidence network with specified token-selection, routing, and training settings.

B Implementation Details

  • The evidence network uses hidden dimension 1024, classifier fusion dimension 512, and one-block, eight-head Transformer encoders with dropout 0.1.Source: B Implementation Details · p. 10
  • Adjacent-layer token selection uses ρ_e = 0.5, while the text keep ratio is 0.3 on Weibo, 0.025 on AMG, and 0.1 on Pheme.Each evidence stream keeps the top four layers during routing.Source: B Implementation Details · p. 10
  • The evidence network is trained for five epochs with AdamW at learning rate 5 × 10^-5, weight decay 0.01, and global batch size 24.Training uses gradient clipping 1.0, DeepSpeed ZeRO stage 2 with bfloat16, cosine decay, and warmup ratio 0.1; disentanglement loss weight is 0.01.Source: B Implementation Details · p. 10

C Hyperparameter Sensitivity

The hyperparameter sweeps are reported in full in Tables 8 and 9.

C Hyperparameter Sensitivity

  • Tables 8 and 9 provide complete results for the Figure 3 hyperparameter sweeps, reporting accuracy and three F1 metrics for every evaluated setting.Source: C Hyperparameter Sensitivity · p. 10

D Counterfactual Sample Construction

The authors construct image-grounded counterfactual descriptions through an interactive workflow, pairing four compatible reference clauses with contradictory text edits. They constrain edits to visible evidence and reject images that cannot support four unambiguous contrasts.

D Counterfactual Sample Construction

  • GPT-6.1 Sol assists an interactive workflow that creates four image-supported reference clauses and a contradictory text replacement for each.The source text provides context, while the image supplies the evidence for the clauses.Source: D Counterfactual Sample Construction · p. 10
  • The four edit types target objects, scenes, events, and directly checkable claims, such as attributes or readable text.Reference clauses must be compatible and avoid repeating the same target fact.Source: D Counterfactual Sample Construction · p. 10
  • Each edit replaces one short contiguous text span with a grammatical contradiction, while leaving all other text and the image unchanged.The intended inconsistency must arise from comparing the edited description with the image, not from contradictions among clauses.Source: D Counterfactual Sample Construction · p. 10
  • Images are rejected when four unambiguous, visually supported contrasts cannot be established, rather than having conflicts invented.Each edit records its type, clause, original and replacement spans, and supporting image evidence.Source: D Counterfactual Sample Construction · p. 10
  • Conflict-clause positions are balanced across 100 groups, and meaning-preserving controls use synonym replacement or full-description rewriting.Each of the four sentence positions is used in 25 groups; the image remains unchanged.Source: D Counterfactual Sample Construction · pp. 10–11
  • Sensitivity analyses vary the text keep ratio, image keep ratio, DLKE token keep ratio, and routed-layer count; the latter settings are tested on Pheme.The captions identify default text keep ratios as 0.3 for Weibo, 0.025 for AMG, and 0.1 for Pheme.Source: Table 8 · p. 11, Table 9 · p. 11

E Detailed Classification Results

Tables 10–12 present detailed classification results for AMG, Pheme, and Weibo, including accuracy and class-specific precision, recall, and F1 scores.

E Detailed Classification Results

Loading 2610.36569v1…