Source-linked AI summary
Embracing Domain Differences in Fake News: Cross-domain Fake News Detection using Multi-modal Data
Amila Silva, Ling Luo, Shanika Karunasekera, Christopher Leckie
TL;DR
The paper addresses fake news detection across domains and the costly selection of labelled records when labelled coverage must span many domains. It jointly models domain-specific and cross-domain information and uses unsupervised domain-aware selection, achieving improved cross-domain performance, including for rarely appearing domains.
Problem
Existing fake news detectors often rely on single-domain or domain-invariant training, while limited labelling budgets make it difficult to select records that cover many domains.
Method
The framework learns domain-specific and cross-domain representations for multimodal fake news detection and uses random-projection LSH on domain embeddings to select diverse records for labelling.
Results
The integrated approach outperforms existing fake news detection techniques by as much as 7.55% in F1-score and improves performance for rarely appearing domains by around 25% in F1-score.
Takeaways & Limitations
Combining domain-aware detection with domain-coverage-oriented annotation yields a model that performs across domains under low labelling budgets.
Takeaways & Limitations
The approach is intended for future extension to online streams, which introduce challenges from newly emerging domains and temporal changes.
Abstract
from arXiv · showhide
With the rapid evolution of social media, fake news has become a significant social problem, which cannot be addressed in a timely manner using manual investigation. This has motivated numerous studies on automating fake news detection. Most studies explore supervised training models with different modalities (e.g., text, images, and propagation networks) of news records to identify fake news. However, the performance of such techniques generally drops if news records are coming from different domains (e.g., politics, entertainment), especially for domains that are unseen or rarely-seen during training. As motivation, we empirically show that news records from different domains have significantly different word usage and propagation patterns. Furthermore, due to the sheer volume of unlabelled news records, it is challenging to select news records for manual labelling so that the domain-coverage of the labelled dataset is maximized. Hence, this work: (1) proposes a novel framework that jointly preserves domain-specific and cross-domain knowledge in news records to detect fake news from different domains; and (2) introduces an unsupervised technique to select a set of unlabelled informative news records for manual labelling, which can be ultimately used to train a fake news detection model that performs well for many domains while minimizing the labelling cost. Our experiments show that the integration of the proposed fake news model and the selective annotation approach achieves state-of-the-art performance for cross-domain news datasets, while yielding notable improvements for rarely-appearing domains in news datasets.
Introduction
Fake news detection is difficult to scale manually and often fails when news spans multiple domains. The paper addresses this with multimodal domain-aware detection and selective annotation aimed at broader domain coverage.
- High-volume social media news makes manual fact-checking impractical, motivating automatic fake news detection.
- Existing techniques often train and evaluate on single-domain datasets, despite real news streams covering many domains.
- Unsupervised instance selection targets informative records that maximize domain coverage while avoiding selection bias under limited labelling capacity.
- Multimodal fake news detection preserves domain-specific and cross-domain information in two embedding spaces.
- The framework outperforms state-of-the-art models by as much as 7.55% in F1-score.
- Prior active-learning approaches may require a pre-trained model and remain vulnerable to initial-model bias, while not explicitly maximizing labelled-data domain coverage.
Problem Statement
The paper addresses fake-news detection across heterogeneous domains and frames annotation as a budgeted selection problem. Its approach learns multimodal domain representations, preserves domain-specific and cross-domain information, and selects domain-diverse records for labelling.
- Problem formulation: The task jointly selects records for labelling under budget B and trains a model to classify unlabelled records as false or real.The dataset combines PolitiFact, GossipCop, and CoAID to emulate a domain-agnostic news collection.
- Unsupervised domain discovery: The approach constructs a heterogeneous network from users and title words, then applies Louvain community detection to learn soft domain embeddings.Each record’s embedding concatenates its likelihood of belonging to the discovered communities.
- Domain discovery evaluation: The proposed approach yields clearer domain separation than the user-based baseline while retaining information about records that may belong to multiple domains.The comparison is visualized with t-SNE domain embeddings.
- Domain-agnostic classification: The classifier maps multimodal record representations into domain-specific and cross-domain subspaces, using decoder functions to recover labels and inputs during training.The model uses textual content and propagation networks as its input representation.
- Budgeted instance selection: The LSH-based selector represents records with domain embeddings and chooses records distant in embedding space to improve domain coverage under a labelling budget.The selection procedure is designed to identify informative records from a large unlabelled pool.
- Selection evaluation: Compared with random selection, the proposed selector samples more records from rarely occurring domains and achieves better domain coverage.Random selection follows the original dataset distribution and may underrepresent rare domains.
Experiments
Experiments evaluate the multimodal model and LSH-based selection across three domains, showing gains from preserving domain-specific and cross-domain information and from multimodal representations.
- The proposed approach outperforms the best baseline by as much as 7.55% in F1-score across all three domains.
- Multimodal approaches generally outperform unimodal approaches, indicating that text and propagation modalities provide distinct information for fake news detection.The authors attribute the advantage to learned data-driven latent representations that capture complex information.
- Ablation Study: Removing the domain-specific and cross-domain losses lowers PolitiFact F1-score by around 6% and 3%, respectively.The ablation supports separate layers for preserving domain-specific knowledge and transferring common knowledge between domains.
- Ablation Study: The domain-specific embedding forms domain-separated clusters, whereas the cross-domain embedding obscures domain labels to share common knowledge across domains.
- Ablation Study: Network features are more useful for GossipCop, while text features are most informative for CoAID.The differing modality contributions support multimodal models for generalization across domains.
- Evaluation of LSH-based Instance Selection: LSH-based selection improves F1-score by 24% for PolitiFact and 27% for CoAID when B/|Rpool| = 0.1.The approach also outperforms baselines with a constrained budget B = 50%|Rpool| and substantially exceeds random selection for rarely appearing or imbalanced domains.
Conclusion
The work combines domain-specific and cross-domain knowledge with unsupervised instance selection to support fake news detection across domains under low labelling budgets. The authors report up to a 7.55% F1-score improvement, while identifying online adaptation and changing domains as future challenges.
- The proposed framework exploits domain-specific and cross-domain knowledge to detect fake news from different domains.
- The unsupervised selection approach identifies informative unlabelled instances for manual labelling from a large data pool.
- 7.55% in F1-score is the reported maximum improvement over existing fake news detection techniques under low labelling budgets.
- Future work targets online learning for streams spanning many domains, including newly emerging domains and temporal changes.
Supplementary Material for Embracing Domain Differences in Fake News:
The supplementary material accompanies the paper on cross-domain fake news detection using multimodal data and identifies its authors and institutional affiliation.
- The paper lists Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie as authors.
- The authors are affiliated with the School of Computing and Information Systems at The University of Melbourne.
- The listed contact addresses use the University of Melbourne domain and identify Parkville, Victoria, Australia.
- The document is identified as arXiv:2102.06314v4 dated 24 February 2021.
- The supplementary material accompanies the paper titled “Embracing Domain Differences in Fake News: Cross-domain Fake News Detection using Multimodal Data”.
Louvain Algorithm for Community Detection
The Louvain algorithm detects network communities by moving vertices to neighboring communities when modularity improves, aggregating communities, and repeating until stabilization. In this model, it is used to identify communities in a network and is selected for producing relatively few communities among parameter-free alternatives.
- The Louvain algorithm identifies communities in a network by iteratively optimizing its modularity score.
- Each vertex begins in its own community, then remains or shifts to an immediate neighbor’s community according to modularity improvement.
- The modularity calculation uses weights of links within communities, links to communities, links to each vertex, and total network links.
- Communities are combined into new network vertices and the process repeats until no further community merging occurs.
- The algorithm is selected because it generates relatively few communities compared with Infomap and Label Propagation.
Multimodal Input Representation
The model represents each news record with independent text and propagation-network features, which are concatenated into a multimodal input. Text uses RoBERTa, while network information includes global structural and local node-level features.
- Each news record is represented by concatenating independent text and propagation-network representations into a final multimodal input.
- The text representation uses the roberta-large model to encode a news title into a 1024-dimensional latent vector.
- Propagation networks contain tweets and retweets as nodes, with a source node linking the information cascades of each news record.
- Global network features include the Wiener Index, node count, network depth, nodes at different hops, and branching factors.
- Local representations aggregate text-based, user-based, and temporal node features from the propagation network.
Encoding and Decoding Functions
The classifier uses six encoding and decoding functions implemented as two-layer feed-forward networks. These functions map input representations into latent outputs using sigmoid activation and a prescribed hidden-layer size.
- Six functions separately encode, decode, predict, and reconstruct representations in the fake news detection classifier.
- Each function maps an input vector from R^dinput to an output vector from R^doutput.
- The functions use two-layer feed-forward networks with sigmoid activation and hidden size max(dinput, doutput)/2.The paper leaves architecture optimization for future work.
Domain Discovery Baseline
The baseline discovers domains by constructing a user-overlap graph, clustering its nodes, and converting cluster assignments into domain embeddings. Its representation is tied to the existing news-record graph, unlike the proposed word-and-user approach.
- The baseline assigns hard domain labels using the users engaged with each news record.
- Records are connected when their user-set similarity exceeds α = 0.4, after which Louvain identifies domain communities.
- Each record receives a one-hot community vector, whose cosine similarities with other records are concatenated into its domain embedding.
- Because records are graph nodes, the baseline is difficult to extend to new records; the proposed method instead uses words and users.
Fake News Detection Baselines
The evaluation compares the proposed model with seven established baselines and variants spanning linguistic, neural, multimodal, and propagation-network approaches. Comparisons use aligned text and network representations where required.
- The proposed model is compared with seven widely used fake news detection baselines and their variants.
- LIWC uses psycho-linguistic text features with logistic regression, while text-CNN uses convolutional neural networks.
- EANN-Unimodal uses text, whereas EANN-Multimodal combines text and network modalities for latent representation learning.
- HPNF extracts structural and temporal propagation-network features before logistic-regression classification.
- For fair comparison, the EANN variants use the same text and network representation techniques adopted by the proposed model.
Parameter Sensitivity
Parameter sensitivity experiments identify ranges with stable detection performance and show that LSH hash-code length affects domain coverage. The reported configuration balances these choices using fixed model and selection parameters.
- Loss weights: λ1 values above 22 or below 2^-1 consistently reduce performance across all three datasets.This indicates that the reconstruction loss should have moderate importance.
- Model parameters: Model performance is consistent for latent dimension d > 256, epochs > 300, and batch size < 128.
- Instance selection: Higher |H| reduces domain coverage, while lower |H| increases coverage but requires more hashing iterations to meet a labeling budget.Figure 4 measures domain coverage with λ, where lower λ is better.
- Final configuration: The reported configuration uses λ1 = 1, λ2 = 10, λ3 = 5, |H| = 10, d = 512, epochs = 300, and batch size = 64.
Convergence Analysis
Figure 5 shows that all four loss terms converge around 250 epochs, including the minimax-trained cross-domain loss. This convergence empirically verifies the minimax game's convergence for exploiting cross-domain knowledge.
- The model's loss function combines prediction, reconstruction, domain-specific, and cross-domain terms.
- All four loss terms converge at approximately 250 epochs.
- The converging cross-domain loss empirically verifies convergence of the proposed minimax game for exploiting cross-domain knowledge.