Cross-modal Contrastive Learning for Multimodal Fake News Detection

• Text-only detection is insufficient for multimodal posts, while how modalities affect decisions and how to make aggregation effective and interpretable remain open questions.Source: §1 · pp. 1–2
• One-hot contrastive learning penalizes all negative pairs, including inherently mismatched or potentially related image-text pairs, which may harm generalization or confuse the model.Source: §1 · p. 2
• COOLANT uses image-text contrastive alignment, consistency learning to soften negative-sample loss, cross-modal fusion, and guided attention to aggregate modality features.The guidance module estimates divergence between text and image representation distributions to help assign modality weights.Source: §1 · p. 2
• COOLANT outperforms previous systems and achieves new state-of-the-art results on Twitter and Weibo.Source: §1 · p. 2

2.1.1 Unimodal Methods.

• Unimodal methods analyze text or image content but ignore cross-modal correlation and consistency, potentially undermining performance on multimodal news.Text approaches include response-pattern, lexical-semantic, logical-soundness, writing-style, and rhetorical-structure methods; image approaches use dissemination patterns or visual and frequency features.Source: §2.1.1 · pp. 2–3

2.1.2 Multimodal Methods.

• Multimodal methods use event discrimination, multimodal autoencoders, or co-attention, while other approaches estimate ambiguity or modality confidence to weight features.The section notes that recently emerged multimodal representation-learning paradigms have rarely been considered in fake-news detection.Source: §2.1.2 · p. 3

2.2 Contrastive Learning

• COOLANT uses dual-encoder image-text contrastive learning for a unified latent space and an auxiliary consistency task to provide soft targets based on semantic similarity.The related work also surveys vision-language contrastive methods including WenLan, CLIP, ALIGN, ALBEF, BLIP, VLMo, and CoCa.Source: §2.2 · p. 3

• The method is presented as three components: cross-modal contrastive learning, cross-modal fusion, and cross-modal aggregation.Source: §3 · p. 3
• The framework first extracts unimodal features, then aligns image and text with cross-modal contrastive learning before learning cross-modality correlations through fusion.Source: §3 · p. 3
• The cross-modal aggregation module uses attention and attention guidance to assign modality scores and aggregate features for fake-news detection.Source: §3 · p. 3

3.1 Modal-specific Encoder

• The method encodes image and text inputs into unimodal embeddings using modality-specific encoders and pretrained techniques.Source: §3.1 · p. 3

3.1.1 Visual Encoder.

• ResNet pretrained on ImageNet extracts regional visual features, which a fully connected layer transforms into the final visual embedding.Source: §3.1.1 · p. 3

3.1.2 Text Encoder.

• BERT produces an aggregate sequence representation from tokenized words, and a fully connected layer transforms it into the final textual embedding.Source: §3.1.2 · p. 3

3.2 Cross-modal Contrastive Learning

• COOLANT aligns image and text features in a shared space using contrastive learning, while consistency-derived semantic similarity matrices provide soft targets.The soft targets ease the strict constraint imposed by one-hot labels.Source: §3.2 · p. 3, §3.2.3 · pp. 4–5

3.2.1 Consistency Learning.

• Cross-modal consistency learning classifies image-text pairs as matched or unmatched and learns a shared semantic space to measure their semantic similarity.Pairs are labeled positive when the embeddings come from the same real news item; the task can be learned in parallel with contrastive learning.Source: §3.2.1 · pp. 3–4

3.2.3 Build Soft Target.

• The final contrastive-learning objective combines predicted similarities with consistency-derived soft targets, with λ controlling the soft-target contribution.The jointly trained module produces semantically aligned image and text representations for the fusion and aggregation modules.Source: §3.2.3 · p. 5

3.3 Cross-modal Fusion

• Inter-modal attention weights are calculated from associations between aligned visual and textual representations.The attention weights are used to capture semantic interactions across modalities.Source: §3.3 · p. 5
• The module updates the original unimodal embeddings with inter-modal attention weights to produce explicit correlation features.Source: §3.3 · p. 5
• The final correlation matrix is formed with an outer product and flattened into a vector.Source: §3.3 · p. 5

3.4 Cross-modal Aggregation

• The module adaptively concatenates aligned unimodal representations with cross-modality correlations as its input.These are the image and text representations from contrastive learning and the correlations from cross-modal fusion.Source: §3.4 · p. 5

3.4.1 Attention Mechanism.

• Attention reweights the three feature types using modality-wise relationships and adaptive weights derived from pooled global information.The mechanism concatenates the features, applies global average pooling, and uses a sigmoid gating mechanism to obtain attention weights.Source: §3.4.1 · p. 5

3.4.2 Attention Guidance.

• The guidance module compares attention weights with ambiguity scores derived from unimodal-feature distributions, encouraging weights to reflect modality ambiguity.The ambiguity framing assigns more attention to cross-modal features when unimodal features are strongly ambiguous.Source: §3.4.2 · pp. 5–6

3.4.3 Classifier.

• The classifier concatenates the weighted representations, feeds them to a fully connected network, and trains with cross-entropy and the aggregation objective.The final COOLANT objective combines consistency-learning, contrastive-learning, and cross-modal aggregation losses.Source: §3.4.3 · p. 6

4.1.1 Datasets.

• Twitter training includes 6,840 real and 5,007 fake tweets, while its test set has 1,406 posts; Weibo uses 7,532 training and 1,996 test items.The Weibo training set contains 3,749 fake and 3,783 real news items; its test set contains 1,000 fake and 996 real items.Source: §4.1.1 · p. 6

4.1.2 Baseline.

• The comparison includes EANN, MVAE, MKEMN, SAFE, MCNN, MCAN, CAFE, LIIMR, FND-CLIP, and CMC.The cited baselines cover approaches including event-discriminative learning, cross-modal similarity, co-attention, ambiguity-aware aggregation, and CLIP encoders.Source: §4.1.2 · p. 6

4.1.3 Implementation Details.

• Evaluation uses Accuracy, Precision, Recall, and F1-score, with Adam training at batch size 64, learning rate 0.001, and up to 50 epochs with early stopping.The contrastive-loss λ is 0.2 and the aggregation-loss γ is 0.5.Source: §4.1.3 · p. 6

4.2 Overall Performance

• The authors identify aligned unimodal representations as important for detection because separately extracted text and image features may occupy different semantic spaces.They contrast their image-text contrastive objective with approaches relying only on fused features.Source: §4.2 · pp. 6–7
• 90.0% accuracy on Twitter is a 6.9% improvement, and 92.3% accuracy on Weibo is 1.5% above the previous best.Table 1 reports the comparison on both datasets and says COOLANT has the highest Accuracy among the compared methods.Source: §4.2 · p. 6, Table 1 · p. 7
• The authors report that image-text contrastive learning helps distinguish news within the same event, while consistency learning extracts event-invariant features.They connect these capabilities with improved detection on newly emerged events and COOLANT's performance on the two datasets.Source: §4.2 · p. 7

4.3.1 Quantitative Analysis.

• Every tested COOLANT variant performs worse than the full model, supporting the effectiveness of each component.Table 2 reports the ablation study across two datasets.Source: Table 2 · p. 7, §4.3.1 · p. 7
• Removing image-text contrastive learning yields the worst performance, underscoring its role in learning aligned unimodal features for detection.The study also states that this objective facilitates alignment of visual and language representations.Source: §4.3.1 · p. 7
• Removing consistency learning hurts Twitter more than Weibo; the authors relate this to event concentration and report that soft targets help retain event-invariant features.They also suggest Weibo’s larger corpus can partly compensate for dataset noise, consistent with ALIGN’s prior findings.Source: §4.3.1 · p. 8

4.3.2 Qualitative Analysis.

• On Weibo, COOLANT’s t-SNE feature boundaries are more pronounced than its variants’, while removing ITC leaves many features easily misclassified.The comparison also indicates that effective cross-modal feature aggregation can improve final feature representations.Source: §4.3.2 · p. 8, Figure 3 · p. 8

5 CONCLUSION

• COOLANT applies image-text contrastive learning to improve image-text alignment for multimodal fake news detection.An auxiliary task softens the negative-sample loss during contrastive learning.Source: §5 · p. 8
• A cross-modal fusion module learns modality correlations, and attention with an attention guidance module aggregates features effectively and interpretably.Source: §5 · p. 8
• On Twitter and Weibo, COOLANT outperforms previous approaches and achieves new state-of-the-art results.Source: §5 · p. 8