Source-linked AI summary
Some Like it Hoax: Automated Fake News Detection in Social Networks
Eugenio Tacchini, Gabriele Ballarin, Marco L. Della Vedova, Stefano Moret, Luca de Alfaro
TL;DR
Rapidly spreading misinformation on social networks motivates automatic hoax detection. The paper classifies Facebook posts from the users who like them using logistic regression and adapted boolean crowdsourcing, achieving high accuracy with very small training sets and under user-community overlap.
Problem
Rapid dissemination and large volumes of social-network content make timely manual assessment of hoaxes difficult, motivating automatic online detection.
Method
The paper uses user–post interaction patterns as features and applies logistic regression plus an adapted harmonic boolean label crowdsourcing algorithm.
Results
Above 99% accuracy is achieved with harmonic BLC trained on 0.5% of the full dataset, while performance reaches 90% on the intersection dataset using 10% of posts.
Takeaways & Limitations
User interaction patterns can predict whether Facebook posts are hoaxes, including when information is transferred across pages or users like both classes of posts.
Takeaways & Limitations
On the artificially constructed intersection dataset, achieving 90% performance requires a training set containing 10% of the posts.
Abstract
from arXiv · showhide
In recent years, the reliability of information on the Internet has emerged as a crucial issue of modern society. Social network sites (SNSs) have revolutionized the way in which information is spread by allowing users to freely share content. As a consequence, SNSs are also increasingly used as vectors for the diffusion of misinformation and hoaxes. The amount of disseminated information and the rapidity of its diffusion make it practically impossible to assess reliability in a timely manner, highlighting the need for automatic hoax detection systems. As a contribution towards this objective, we show that Facebook posts can be classified with high accuracy as hoaxes or non-hoaxes on the basis of the users who "liked" them. We present two classification techniques, one based on logistic regression, the other on a novel adaptation of boolean crowdsourcing algorithms. On a dataset consisting of 15,500 Facebook posts and 909,236 users, we obtain classification accuracies exceeding 99% even when the training set contains less than 1% of the posts. We further show that our techniques are robust: they work even when we restrict our attention to the users who like both hoax and non-hoax posts. These results suggest that mapping the diffusion pattern of information can be a useful component of automatic hoax detection systems.
1 Introduction
Social networks accelerate the spread of misinformation and hoaxes, creating a need for automatic online detection. This paper asks whether Facebook posts can be classified from the users who like them and reports very high accuracy with two methods.
- Motivation: SNSs rapidly disseminate hoaxes and other misinformation, making timely manual reliability assessment impractical.A significant share of hoaxes peaks within the first two hours after dissemination.
- Research question: The paper classifies Facebook posts using the users who liked them rather than their content.It studies 15,500 posts and 909,236 users from scientific and conspiracy or fake-science pages.
- Approach: Two approaches are proposed: logistic regression with user interactions as features and an adapted boolean label crowdsourcing method.The crowdsourcing adaptation assumes no prior reliability preference among users.
- Main result: Accuracy exceeds 99% even when training sets contain less than 1% of the posts.The result holds despite overlap between communities engaging with scientific and conspiracy pages.
- Resources: The code developed for the paper is publicly available.The repository is hosted on GitHub.
2 Dataset
The dataset contains Facebook posts and likes collected from scientific and conspiracy-oriented pages during the second half of 2016. Its users show both polarization and substantial overlap across page communities.
- Data collection: The data covers public posts and likes from selected Facebook pages between July 1 and December 31, 2016.Data were collected through the Facebook Graph API on January 27, 2017.
- Source selection: Pages were selected from scientific-news and conspiracy-news categories, using the top 20 pages in each category when available.The analysis assumes scientific-page posts are non-hoaxes and conspiracy-page posts are hoaxes.
- Dataset composition: 15,500 posts from 32 pages comprise the complete dataset, including 8,923 hoaxes and 6,577 non-hoaxes.The pages include 14 conspiracy and 18 scientific sources, with more than 2,300,000 likes from over 900,000 users.
- Like distributions: Hoax posts received more likes on average, with 204.5 likes versus 84.0 for non-hoax posts.The corresponding medians were 22 versus 14, and maxima were 121,491 versus 13,608.
- User overlap: Among users with at least two likes, 5.0% liked both hoax and non-hoax posts.The remaining users liked only hoax posts or only non-hoax posts, at 74.7% and 20.3%, respectively.
- Page communities: Hoax pages share more users with other hoax pages, and non-hoax pages share more users with other non-hoax pages, while the communities still overlap substantially.Figure 2(b) presents these relationships as a symmetric page-by-page matrix whose color intensity represents users in common.
3 Algorithmic Classification of Posts
The paper formulates hoax detection as supervised binary classification from user–post likes and compares logistic regression with harmonic boolean label crowdsourcing. The second method propagates information through a bipartite likes graph, addressing a key limitation of user-wise regression.
- Problem formulation: Each post is represented by binary user features indicating which users liked it, and classified as hoax or non-hoax.The feature x_iu equals 1 when user u liked post i and 0 otherwise.
- Logistic regression: Logistic regression learns one weight per user, with positive weights indicating mostly non-hoax likes and negative weights indicating mostly hoax likes.The model converts the weighted user interactions into the probability that a post is non-hoax.
- Logistic regression: Logistic regression is chosen for very large, uniform feature sets and for its non-interference property across disconnected user–post subsets.The setting contains about a million user features, while the authors note real applications could involve hundreds of millions.
- Limitation: Its weakness is that logistic regression cannot learn weights for users absent from the training posts and cannot classify posts liked only by such users.It does not transfer information across users who liked some of the same posts.
- Harmonic BLC: The harmonic method adapts boolean label crowdsourcing by treating likes as votes while using a labeled training set without assuming users are generally truthful.Standard BLC instead assumes people are more likely to tell the truth than to lie.
- Harmonic BLC: The harmonic algorithm represents likes as a bipartite post–user graph and iteratively propagates beliefs between labeled posts, users, and unlabeled posts.Its node parameters define beta distributions whose means represent user truthfulness or post non-hoax probability.
- Harmonic BLC: The algorithm initializes labeled posts with hoax or non-hoax values, updates user and post parameters, and classifies posts by the sign of q_i after five iterations.The chosen priors are A = 5.01, B = 5, and A′ = B′ = 5.
- Harmonic BLC: Information propagates only along like edges, so harmonic BLC retains the non-interference property.Users first accumulate evidence from known posts, then their preferences update beliefs about connected posts and users.
4 Results
The experiments evaluate how accuracy changes with training-set size and whether information transfers across overlapping users and unseen pages. Harmonic BLC performs best on the complete dataset and across pages, while logistic regression performs best on the intersection dataset.
- Cross-validation: 99.4% accuracy was achieved by harmonic BLC in 5-fold cross-validation, compared with more than 99% for logistic regression.The analysis used 80% of posts for training and 20% for testing.
- Training-set size: At least 0.5% of posts, or about 80 posts, yielded accuracy above 99.4% for harmonic BLC on the complete dataset.With 0.1% of posts, or about 15 posts, accuracy was still about 80%.
- Training-set size: At 1% of posts, logistic regression exceeded 90% accuracy on the complete dataset, although it was inferior to harmonic BLC.The comparison concerns accuracy as the fraction of posts in the training set changes.
- Training-set size: On the intersection dataset, logistic regression maintained a 3–4% margin over harmonic BLC, declining from 91.6% at 10% training data toward 56% at 0.1%.The intersection dataset contains users who liked both hoax and non-hoax posts.
- Interpretation: The intersection dataset can bias harmonic BLC toward erroneous transfer because it over-samples straddling users who like one hoax and one non-hoax post.These users constituted 32% of the intersection dataset and can connect a training post to an evaluation post in opposite classes.
- Cross-page learning: Harmonic BLC achieved essentially perfect accuracy when transferring information across pages in both one-page-out and half-pages-out experiments.For one page, performance was 87.3%; all other pages exceeded 97.2%, with 100% for 23 pages.
5 Conclusions
The study shows that user interaction patterns can support accurate Facebook hoax detection with limited manual labeling. The approach also transfers across pages and remains effective, though less strongly, among users who liked both hoax and non-hoax posts.
- Both logistic regression and harmonic BLC use users’ interactions with posts to predict whether the posts are hoaxes.
- 99%+ accuracy is achieved by the harmonic BLC algorithm with training sets containing 0.5% of the full dataset, or about 80 posts.The authors present this as requiring only a modest amount of manual classification.
- Above 99% performance is maintained when only half of the pages are represented in the training set, demonstrating information transfer across pages.
- 90% performance is obtained on the intersection dataset, but it requires a training set containing 10% of the posts.This dataset contains only users who liked both hoax and non-hoax posts.
- The intersection dataset is a borderline case absent from the studied communities, so the authors qualify its robustness evidence accordingly.