Source-linked AI summary
Multimodal Fake News Detection via CLIP-Guided Learning
Yangming Zhou, Qichao Ying, Zhenxing Qian, Sheng Li, Xinpeng Zhang
TL;DR
Multimodal fake news detection still lacks clear ways to measure image–text similarity and determine its effect on decisions, while pretrained multimodal models remain underused. FND-CLIP uses CLIP-guided similarity and modality-wise attention to fuse features, and it reports better benchmark performance than prior methods, with gains in overall accuracy across three datasets.
Problem
How to calculate cross-modal feature similarity and understand its effect on fake-news decisions remains an open question, and pretrained multimodal models are underused in FND.
Source: §1 · p. 2Method
FND-CLIP uses pretrained CLIP to measure image–text similarity, guide feature fusion, and adaptively weight text, image, and fused features.
Source: §1 · p. 2Results
Takeaways & Limitations
CLIP-generated features collaborate with ResNet- and BERT-extracted unimodal features to mine crucial features for fake-news detection.
Source: §5 · pp. 8–9The output is binary and does not explain why a news item is predicted fake or which elements are suspicious or abnormal.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Multimodal fake news detection has attracted many research interests in social forensics. Many existing approaches introduce tailored attention mechanisms to guide the fusion of unimodal features. However, how the similarity of these features is calculated and how it will affect the decision-making process in FND are still open questions. Besides, the potential of pretrained multi-modal feature learning models in fake news detection has not been well exploited. This paper proposes a FND-CLIP framework, i.e., a multimodal Fake News Detection network based on Contrastive Language-Image Pretraining (CLIP). Given a targeted multimodal news, we extract the deep representations from the image and text using a ResNet-based encoder, a BERT-based encoder and two pair-wise CLIP encoders. The multimodal feature is a concatenation of the CLIP-generated features weighted by the standardized cross-modal similarity of the two modalities. The extracted features are further processed for redundancy reduction before feeding them into the final classifier. We introduce a modality-wise attention module to adaptively reweight and aggregate the features. We have conducted extensive experiments on typical fake news datasets. The results indicate that the proposed framework has a better capability in mining crucial features for fake news detection. The proposed FND-CLIP can achieve better performances than previous works, i.e., 0.7\%, 6.8\% and 1.3\% improvements in overall accuracy on Weibo, Politifact and Gossipcop, respectively. Besides, we justify that CLIP-based learning can allow better flexibility on multimodal feature selection.
1 INTRODUCTION
FND-CLIP uses pretrained CLIP to measure image–text similarity and guide multimodal feature fusion, while modality-wise attention adapts the contribution of text, image, and fused features. Experiments report accuracy gains over previous methods across three datasets.
1 INTRODUCTION
- Multimodal analysis can complement text-only or image-only FND because news may pair a real image with rumors or correct words with a tampered image.Online fake news can spread rapidly and may create panic or misdirect public opinion.Source: §1 · p. 1
- How to calculate cross-modal feature similarity and how it affects FND decisions remain open questions, while advanced pretrained multimodal models are underused in FND.Earlier approaches are also described as largely black-box in how multimodal features influence the final decision.Source: §1 · p. 2
- FND-CLIP measures cross-modal similarity with pretrained CLIP and uses it to guide feature fusion and decision-making, alongside modality-wise attention over text, image, and fused features.Its image and text representations combine fine-tunable ResNet and BERT encoders with CLIP encoders; projected features are reweighted using standardized cosine similarity.Source: §1 · pp. 2–3
- Overall accuracy improves by 0.7% on Weibo, 6.8% on Politifact, and 1.3% on Gossipcop; FND-CLIP also outperforms state-of-the-art methods.The paper additionally reports that CLIP-based learning allows more flexible multimodal feature selection.Source: §1 · pp. 2–3
- The model varies its attention and cross-modal similarity across news instances, paying less attention to fused features when image–text similarity is low.This lets it aggregate information according to the characteristics of each news instance.Source: §1 · p. 3
2 RELATED WORKS
Prior work explores unimodal clues and multimodal fusion, but unimodal approaches omit cross-modal consistency and multimodal correlation is not always explicitly measured. The section also describes CLIP as a pretrained image–text representation model.
2.1 Unimodal Fake News Detection
- Unimodal methods use image-forensics, semantic, statistical, contextual, and linguistic clues, but ignore multimodal correlation and consistency that may impair performance on multimodal news.Text methods examine logical soundness, grammar, writing style, and rhetorical structure; image methods examine manipulation traces, common-sense inconsistency, and image quality.Source: §2.1 · p. 3
2.2 Multimodal Fake News Detection
- Multimodal FND methods use attention, feature alignment, auxiliary event classification, VAEs, and additional extracted information, but explicit image–text correlation remains unclear.The review also notes little FND work applying recently emerged multimodal learning methods, motivating CLIP-based pretraining.Source: §2.2 · p. 3
2.3 Multimodal Learning
- CLIP embeds images and text in a shared space, maximizing cosine similarity for matched pairs and minimizing it for mismatched pairs.It was trained on image–text pairs to predict relevant text for an image without directly optimizing for the downstream task.Source: §2.3 · p. 3
3 METHOD
FND-CLIP combines unimodal text and image representations with CLIP-aligned multimodal features, then projects and attention-aggregates them for classification. Its design uses cross-modal similarity to weight fused features while retaining modality-specific information.
3.1 Approach Overview
- FND-CLIP encodes news text and images with BERT and ResNet, respectively, and uses CLIP encoders to obtain aligned text-image features.The method treats text and image as static modalities and combines their unimodal and multimodal representations for prediction.Source: §3.1 · pp. 3–4, §3.2 · p. 4
3.2 Network Specification
- CLIP-aligned text and image features form the multimodal representation, while BERT and ResNet features remain unimodal because their semantic spaces have a cross-modal gap.The paper describes CLIP as a way to measure cross-modal similarity and guide feature fusion and classification.Source: §3.2 · p. 4, §3.1 · p. 4
- Three MLP projection heads reduce feature dimensions and redundancy; modality-wise attention then reweights projected text, image, and multimodal features before summing them.The attention module uses average and maximum pooling to form initial channel weights, then produces the three modality weights.Source: §3.2 · p. 5
- Standardized CLIP cosine similarity scales the projected multimodal feature, addressing cases where image-text correlation is weak or the fused feature is noisy.The method standardizes similarity using running training statistics and applies a sigmoid before weighting the feature.Source: §3.2 · p. 5
3.3 Training Detail
- A two-layer fully connected classifier predicts the label from the aggregated representation and is trained with cross-entropy loss.The implementation fine-tunes ResNet while freezing BERT and CLIP, and selects the epoch with the best test accuracy.Source: §3.2 · p. 5, §3.3 · p. 5
4 EXPERIMENTS
Across three social-media datasets, FND-CLIP achieves the highest compared accuracy, while analyses of component ablations and feature visualizations support the value of CLIP features, cross-modal fusion, and modality-wise attention.
4.3 Comparison with State-of-the-arts
- FND-CLIP achieves 90.7%, 94.2%, and 88.0% Accuracy on Weibo, Politifact, and Gossipcop, surpassing state-of-the-art methods by 0.7%, 6.8%, and 1.3%, respectively.It ranks slightly below Spot on Weibo in Recall.Source: Table 1 · p. 6, §4.3 · p. 7
4.1 Experimental Setup
- Evaluation uses Weibo, Politifact, and Gossipcop; Twitter is excluded because duplicates, image scarcity, and event concentration risk overfitting.Baselines were selected for public pretrained models or source code and a common evaluation protocol.Source: §4.1 · pp. 5–6
4.2 Performance Analysis
- Real-news recall exceeds 0.9 on all three datasets, while accuracy exceeds 90% on Weibo and 94% on Politifact.These are FND-CLIP's reported dataset-level results.Source: §4.2 · p. 7
- Binned analysis finds that high cross-modal similarity is associated with more real news, while real-news-rate patterns differ between Weibo and Gossipcop.The rate rises with ambiguity on Weibo, but first falls and then rises on Gossipcop.Source: §4.2 · p. 7, Figure 3 · p. 7
4.4 Ablation Studies
- Ablations attribute gains to CLIP features, the fused cross-modal feature, and modality-wise attention; text-only ranks second, while image-only performs worst.Multimodal-only accuracy is 81.7%, 90.3%, and 86.2% on Weibo, Politifact, and Gossipcop, respectively, and is below text-only on Weibo and Gossipcop.Source: §4.4 · p. 8
4.5 T-SNE Visualizations
- On Weibo t-SNE visualizations, FND-CLIP features show clearer label boundaries than CAFE and ablations; text-only separates better than image-only, and modalities provide complementary information.The visualization uses features from the test dataset.Source: §4.5 · p. 8
5 CONCLUSIONS
FND-CLIP combines CLIP-aligned multimodal features with modality-wise attention. The authors report that it outperforms many state-of-the-art methods, while noting that binary outputs do not explain why a news item is predicted fake.
5 CONCLUSIONS
- FND-CLIP uses CLIP to extract aligned multimodal features and guide network learning, with modality-wise attention adaptively weighting text, image, and fused features.The authors say attention can avoid noisy and redundant features during fusion.Source: §5 · p. 8
- Across three datasets, the full FND-CLIP achieved the highest accuracy and F1-score in the ablation study.The authors interpret this as evidence that each module and modality is effectively used.Source: Table 2 · p. 9
- FND-CLIP outperforms many state-of-the-art methods in multimodal fake news detection.Source: §5 · p. 9
- The model outputs only binary real-or-fake predictions and does not explain why an item is predicted fake or which elements are suspicious.Source: §5 · p. 9