IMEX-FND: A Traceable Interaction-Aware Mixture-of-Experts Framework for Multimodal Fake News Detection

• Text–image evidence may be unique, redundant, or synergistic, while the interaction that exposes fake content and the dominant modality vary across posts.A single global fusion rule and static encoders with a fixed pathway are described as brittle, opaque, and unable to adapt.Source: §1 · p. 2
• IMEX-FND couples instance-adaptive modality routing with explicit interaction decomposition in a route-then-decompose design.Source: §1 · pp. 2–3
• MMEG routes each modality through specialized and shared experts and builds a CLIP-grounded cross-modal stream to produce refined text, image, and fusion representations.The representations adapt to whichever modality carries decisive evidence.Source: §1 · pp. 2–3
• IMEF decomposes interactions into uniqueness, redundancy, and synergy, producing transparent sample-wise weights learned with modality-replacement supervision.The authors describe the resulting attributions as supporting instance- and dataset-level interpretability.Source: §1 · pp. 2–3
• IMEX-FND achieves state-of-the-art accuracy on Weibo, Weibo-21, and Gossip with fewer parameters and interpretable interaction attributions.Source: §1 · p. 3

2.1 Modality Interactions

• Modality-interaction research studies how unimodal and cross-modal evidence jointly supports inference; prior work uses factorized contrastive learning and gated routing to capture interaction patterns.Source: §2.1 · p. 3

2.2 Multimodal Fake News Detection

• Multimodal fake news detectors progress from attention, sequence, and variational methods to pretrained-feature fusion, consistency, ambiguity, contrastive, and expert-routing approaches.Later approaches also use high-level signals, multimodal language-model reasoning, multi-view vision, and retrieval.Source: §2.2 · pp. 3–4
• Ambiguity-guided methods can compress interaction complexity into scalar weights, while MoE routing rarely interprets experts as canonical interaction types.IMEX-FND responds with disentangled interactions and transparent instance-level attributions.Source: §2.2 · p. 4

3.1 Problem Setup

• The task is to predict the probability that a post is fake from its text sequence and associated image, which may represent a multi-image post.The target label indicates fake versus real.Source: §3.1 · p. 4

3.2 Overview

• IMEX-FND forms a route-then-decompose pipeline: MMEG produces three refined streams, and IMEF assigns sample-wise weights to interaction experts for traceable fusion.The framework also uses multi-auxiliary supervision and a hybrid fusion objective.Source: §3.2 · p. 5

3.3 Pretrained Encoders and Unified Representations

• Text and image encoders provide token and patch features, while CLIP supplies cross-modal features; the representations are projected to a shared downstream dimension.Text and image routing summaries are derived with attention, and the CLIP features are projected from concatenated image and text features.Source: §3.3 · p. 5

3.4 Multi-Modal Expert Gateway (MMEG)

• MMEG routes each modality instance-wise through specialized and shared experts, then combines CLIP-grounded cross-modal features with shared text and image evidence to form three refined streams.Its modality-specific gates produce text, image, and fusion outputs for the next stage.Source: §3.4 · pp. 5–6, Figure 3 · p. 6

3.5 Interaction-aware Multi-Modal Expert Fusion (IMEF)

• IMEF decomposes the three streams into uniqueness, synergy, and redundancy experts, whose outputs are reweighted per sample to form the fused interaction feature.The weights α can also be aggregated into dataset-level interaction profiles.Source: §3.5 · pp. 6–7
• During training, modality replacement encourages uniqueness experts to react to removal of their modality, synergy to collapse when any modality is missing, and redundancy to remain invariant to single-modality replacement.Replacement is confined to training; inference uses the unperturbed interaction representation.Source: §3.5 · pp. 7–8

3.6 Prediction Heads and Hybrid Fusion Objective

• Three lightweight MLP classifiers provide auxiliary predictions, while the final head classifies a blend of the summed modality streams and the interaction feature under an end-to-end objective.The blend parameter λ controls the interaction-aware fusion contribution.Source: §3.6 · p. 8

4.1 Experimental Settings

• The experiments freeze pretrained backbones and compare against baselines spanning representation and robustness learning, cross-modal consistency and alignment, and advanced semantics and reasoning.Implementation settings include 6 specialized and 12 shared experts per modality, with a batch size of 64 and up to 50 training epochs.Source: Implementation Details · pp. 8–9, Baselines · p. 9

4.2 Overall Performance Comparison

• 0.952 accuracy with 0.952 macro-F1 on Weibo-21 and 0.908 accuracy with 0.872 macro-F1 on Gossip; IMEX-FND achieves the best accuracy on all three benchmarks, exceeding strong baselines by 0.4–1.2%.On Gossip, Real F1 is 0.939 and Fake F1 is 0.805; the authors also report closely matched class-wise precision and recall on Weibo-21.Source: Table 1 · p. 9, §4.2 · pp. 9–10

4.3 Ablation Studies

• Removing IMEF produces the largest Macro-F1 drop (−0.020/−0.022/−0.023 across Weibo, Weibo-21, and Gossip); removing MMEG is next (−0.012/−0.010/−0.014).The ablation ordering places interaction decomposition before routing, modality-replacement and interaction losses, and auxiliary heads.Source: Table 2 · p. 10, §4.3 · p. 11

4.4 Robustness to Dominant-Modality Shift

• Removing MMEG reduces Macro-F1 by approximately 0.031 on conflict posts versus 0.006 on agreement posts, averaged across datasets.The subsets are defined by whether the text-only and image-only heads disagree or agree, respectively.Source: §4.4 · p. 11, Table 3 · p. 11

4.5 Traceability and Interaction Analysis

• Dataset-level interaction profiles differ: Gossip leans on text and image uniqueness (0.291 and 0.268), while Weibo leans on synergy (0.308).For representative posts, the reported dominant patterns are synergy for a mismatched photo, image uniqueness for a spliced image, and redundancy for a corroborating caption–image pair.Source: §4.5 · p. 11, Table 4 · p. 12

4.6 Hyperparameter Settings

• Accuracy and class-wise F1 generally improve from λ=0 to λ=0.3, then slightly saturate or decrease across the three datasets.The authors describe interaction-aware fusion and the conventional path as complementary and use λ=0.3 in all experiments.Source: Figure 4 · p. 12, §4.6 · p. 12

4.7 Efficiency Analysis

• On Weibo-21, IMEX-FND sits in the middle of the trainable-parameter range while attaining the best accuracy, with test-time latency close to the most efficient baselines.The comparison uses architecturally comparable models with frozen encoders; interaction-regularization passes are disabled at inference.Source: Table 5 · p. 13, §4.7 · p. 13

• MMEG builds a CLIP-grounded fusion stream, while IMEF decomposes cross-modal dynamics into uniqueness, redundancy, and synergy with traceable sample-wise weights.The framework turns an opaque fusion score into an inspectable account of modality contributions and interactions.Source: §5 · p. 13
• 0.4%–1.2%: IMEX-FND surpasses competitive baselines while offering superior transparency with fewer parameters.Source: §5 · p. 13
• Future work will explore causal reasoning and efficient refinement under limited feedback.Source: §5 · p. 13