Source-linked AI summary
Fake News Detection on Social Media using Geometric Deep Learning
Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, Michael M. Bronstein
TL;DR
Fake news detection is difficult because content often requires contextual knowledge that current NLP methods lack. The paper applies geometric deep learning to heterogeneous Twitter graphs, achieving high accuracy and early detection, while identifying overfitting and temporal behavior as important boundaries.
Problem
Fake news is difficult to detect automatically because its interpretation can require political, social, or common-sense knowledge missing from current NLP methods.
Method
The paper uses geometric deep learning to combine user, activity, social-network, propagation, and content features in a graph CNN trained on professionally fact-checked Twitter stories.
Results
92.70 ± 1.80% ROC AUC was achieved in URL-wise classification, and approximately two hours of propagation sufficed for above 90% mean ROC AUC.
Takeaways & Limitations
Propagation-based geometric deep learning shows potential for accurate fake-news detection and robust performance across challenging settings.
Takeaways & Limitations
Removing tweet content improved cascade-wise performance by 4%, suggesting overfitting to repeated content and motivating regularization in future work.
Abstract
from arXiv · showhide
Social media are nowadays one of the main news sources for millions of people around the globe due to their low cost, easy access and rapid dissemination. This however comes at the cost of dubious trustworthiness and significant risk of exposure to 'fake news', intentionally written to mislead the readers. Automatically detecting fake news poses challenges that defy existing content-based analysis approaches. One of the main reasons is that often the interpretation of the news requires the knowledge of political or social context or 'common sense', which current NLP algorithms are still missing. Recent studies have shown that fake and real news spread differently on social media, forming propagation patterns that could be harnessed for the automatic fake news detection. Propagation-based approaches have multiple advantages compared to their content-based counterparts, among which is language independence and better resilience to adversarial attacks. In this paper we show a novel automatic fake news detection model based on geometric deep learning. The underlying core algorithms are a generalization of classical CNNs to graphs, allowing the fusion of heterogeneous data such as content, user profile and activity, social graph, and news propagation. Our model was trained and tested on news stories, verified by professional fact-checking organizations, that were spread on Twitter. Our experiments indicate that social network structure and propagation are important features allowing highly accurate (92.7% ROC AUC) fake news detection. Second, we observe that fake news can be reliably detected at an early stage, after just a few hours of propagation. Third, we test the aging of our model on training and testing data separated in time. Our results point to the promise of propagation-based approaches for fake news detection as an alternative or complementary strategy to content-based approaches.
1 Introduction
Fake news detection is difficult because misleading stories require political, social, or common-sense context that content-based NLP often lacks. The paper therefore learns task-specific propagation patterns with geometric deep learning over heterogeneous Twitter data.
- Social media provide inexpensive, accessible, and rapid news dissemination but also expose users to poorly checked or intentionally false information.
- Fake news detection remains challenging because deceptive content can require contextual knowledge and may be written to resemble legitimate news.
- Content-based approaches can fail on sophisticated fake news and often depend on language-specific linguistic features.
- Propagation-based methods exploit empirical differences between fake and true news spreading patterns while remaining content-agnostic.
- Geometric deep learning learns fake-news-specific graph patterns and can combine user, social-network, propagation, and content information.
- Nearly 93% ROC AUC was achieved using supervised learning on fact-checked Twitter stories, with reliable detection after only a few hours of propagation.
2 Dataset
The dataset combines individually fact-checked claims with Twitter propagation, user, activity, content, and social-network information. It covers 1,084 labeled claims, 158,951 cascades, 202,375 users, and 2,443,996 social edges.
- The study addresses dataset-labeling challenges by individually verifying stories through professional fact-checking organizations.
- Data collection protocol: The collection process used fact-checker archives, human annotation of related URLs, and verdict propagation to matching or contradicting pages.
- Data collection protocol: A cascade is the diffusion tree formed by a source tweet referencing a URL and all its retweets, enriched with Twitter social-network edges.
- Statistics: The dataset contains 1,084 labeled claims, 158,951 cascades, 202,375 users, and 2,443,996 social-graph edges collected from 2013 to 2018.
- Features: The extracted features cover user profiles, user activity, network and spreading behavior, and news content.
- Credibility and polarization: Credible and non-credible users form visibly distinct communities, suggesting mostly homophilic interactions between users of similar credibility.
3 Our model
The model represents each news URL or cascade as a graph combining tweets, authors, estimated diffusion paths, and social relations. A graph CNN with attention and pooling processes these heterogeneous node and edge features for fake/true classification.
- Graph CNNs replace grid convolution with local, permutation-invariant aggregation over graph neighborhoods.
- Architecture and training: The four-layer model uses two graph-convolutional layers, two fully connected layers, graph attention, mean pooling, SELU activations, and hinge loss.
- Graph construction: For each URL or cascade, the graph contains tweets as nodes and estimated diffusion paths plus social relations as edges.
- Graph construction: Diffusion paths are estimated from retweet timestamps and social connections between the authors of successive cascade tweets.
- Graph construction: Graph nodes describe tweets and authors, while edges encode four directed following and news-spreading relations.
4 Results
The model performs strongly in both URL-wise and cascade-wise fake news detection, with propagation information improving early classification. Performance remains robust across changing cascade durations and temporally separated training and test data, although cascade-wise behavior reflects a trade-off between discrimination and aging robustness.
- 4.1 Model performance: 92.70 ± 1.80% ROC AUC was achieved for URL-wise classification, compared with 88.30 ± 2.74% for cascade-wise classification.Both results use 24-hour diffusion; cascade-wise classification considers cascades containing at least six tweets.
- 4.2 News spreading over time: Performance saturated after roughly 15 hours in URL-wise classification and 7 hours in cascade-wise classification.Seven hours covered approximately 91% of cascade size, while 15 hours covered 86% in the URL-wise setting.
- 4.2 News spreading over time: Above 90% mean ROC AUC was reached in URL-wise classification after approximately 2 hours of news spread.Performance increased sharply from 0 hours to at least 1 hour, when propagation features were added.
- 4.3 Model aging: URL-wise performance deteriorated slightly only after 180 days of training-set aging, whereas cascade-wise performance lost 4% after 260 days.The authors attribute the deterioration to changes in spreading patterns and user activity profiles; they associate cascade-wise robustness with higher cascade variability and simpler learned features.
- 4 Results: The approach integrates user profile and activity, social-network structure, news propagation, and content through geometric deep learning.The model uses the same architecture for URL-wise and cascade-wise detection, but cascade-wise prediction relies on a single cascade and is more challenging.