Source-linked AI summary
Fighting an Infodemic: COVID-19 Fake News Dataset
Parth Patwa, Shivam Sharma, Srinivas Pykl, Vineeth Guptha, Gitanjali Kumari, Md Shad Akhtar, Asif Ekbal, Amitava Das, Tanmoy Chakraborty
TL;DR
COVID-19 misinformation on social media creates a need for reliable detection data. The paper curates and manually verifies a labeled dataset of real and fake COVID-19 news, benchmarks four machine-learning baselines, and reports the strongest test F1-score for SVM at 93.32%.
Problem
COVID-19 fake news and rumors are widespread on social media, and believing them can cause significant harm.
Method
The authors curate and manually verify a class-balanced dataset of 10,700 real and fake COVID-19 news items and benchmark it with four machine-learning algorithms.
Results
93.32% test F1-score is achieved by SVM, followed by Logistic Regression at 91.96%, while Decision Tree and Gradient Boost achieve 85.39% and 86.96%, respectively.
Takeaways & Limitations
The class-balanced dataset can be used to develop automatic fake-news and rumor-detection algorithms, with the evaluated models serving as potential baselines.
Takeaways & Limitations
The dataset includes only textual English content, and future work is proposed to add multilingual data and reasons for the labels.
Abstract
from arXiv · showhide
Along with COVID-19 pandemic we are also fighting an `infodemic'. Fake news and rumors are rampant on social media. Believing in rumors can cause significant harm. This is further exacerbated at the time of a pandemic. To tackle this, we curate and release a manually annotated dataset of 10,700 social media posts and articles of real and fake news on COVID-19. We benchmark the annotated dataset with four machine learning baselines - Decision Tree, Logistic Regression, Gradient Boost, and Support Vector Machine (SVM). We obtain the best performance of 93.46% F1-score with SVM. The data and code is available at: https://github.com/parthpatwa/covid19-fake-news-dectection
1 Introduction
The paper frames COVID-19 fake news as a harmful social-media problem and addresses it by releasing a labeled dataset assembled from real news and fact-checked fake claims.
- COVID-19 fake news can cause severe harm, including deaths and hospitalizations linked to false treatment claims.
- The study restricts its investigation of social-media content to COVID-19.
- The authors present a dataset of social-media posts and articles labeled as real or fake.
- Fake claims come from fact-checking websites and related tools, while real news is collected from verified Twitter handles.
- The paper also performs exploratory data analysis and evaluates four machine-learning baselines.
2 Related Work
Prior work includes datasets and machine-learning approaches for fake-news and rumor detection, including COVID-19 misinformation. The paper identifies the lack of a common acceptable benchmark dataset as a key challenge.
- Researchers do not agree on a single definition of fake news, which may include intentionally false news and deceptive content such as satire or hoaxes.
- Accurate automatic fake-news detection remains extremely challenging, partly because a common acceptable benchmark dataset is lacking.
- The Fake News Challenge introduced in-domain and cross-domain experiments for automatic fake-news detection using machine-learning and deep-learning techniques.
- The Emergent dataset contains 300 rumoured claims and 2,595 associated news articles annotated by journalists for veracity.
- COVID-19 research includes a multilingual cross-domain dataset of 5,182 fact-checked articles and BERT-based approaches using Twitter features.
3 Dataset Development
The authors construct a manually verified, class-balanced COVID-19 dataset from social media, fact-checking sources, and verified accounts, then characterize its content and vocabulary.
- The dataset labels COVID-19 content as real when sourced from verified accounts and informative, and fake when claims or speculation are verified as untrue.
- Only textual English content related to COVID-19 is included; non-English posts are skipped.
- Fake-news examples are collected from public fact-verification websites and social media, with claims manually checked against original documents.
- Real tweets are crawled from 14 official or verified sources, including WHO, CDC, and ICMR, and manually marked when they provide useful COVID-19 information.
- The vocabulary contains 37,505 unique words, including 5,141 words shared by fake and real news.
- The data is split into train (60%), validation (20%), and test (20%) sets while preserving class proportions.
- 52.34% of samples are real news and 47.66% are fake news, making the dataset class-wise balanced across splits.
- Word-cloud and frequency analyses show significant overlap in important words between fake and real news.
4 Baselines and Results
The paper preprocesses COVID-19 news data with tf-idf features and compares four machine-learning classifiers. SVM achieves the strongest reported test performance, while predictions remain balanced across labels.
- Pre-processing and Features: The pipeline removes links, non-alphanumeric characters, and English stop words, then extracts tf-idf features.tf-idf increases with a word’s frequency in a document and decreases as more corpus documents contain it.
- Machine-Learning Baselines: The evaluated classifiers are Logistic Regression, linear-kernel SVM, Decision Tree, and Gradient Boost.All algorithms are implemented with sklearn.
- Results: 93.32% test F1-score is achieved by SVM, compared with 91.96% for Logistic Regression, 85.39% for Decision Tree, and 86.96% for Gradient Boost.The validation results are reported as similar to the test results, with precision and recall close to each other for all models.
- Prediction Balance: SVM predictions are balanced across the two labels on the validation and test datasets.The confusion-matrix caption states that performance on both classes is similar.
5 Conclusion and Future Work
The paper releases a manually verified, class-balanced COVID-19 fake-news dataset and benchmarks it with machine-learning models. It identifies SVM as the best-performing classifier and proposes broader data and language coverage for future work.
- Conclusion: The released dataset contains 10,700 real and fake COVID-19 news items collected from social media and fact-checking websites.The posts were manually verified for veracity.
- Conclusion: The dataset is class-wise balanced and intended to support automatic fake-news and rumor-detection algorithms.The paper presents the machine-learning benchmarks as potential baselines.
- Conclusion: SVM performs best among the evaluated machine-learning models, achieving a 93.32% F1-score on the test set.This result is stated as the strongest model outcome in the conclusion.
- Conclusion: Figure 2 reports SVM confusion matrices for validation and test datasets, with similar performance across both classes.The figure concerns the same SVM model highlighted as the strongest benchmark.
- Future Work: Future work includes collecting more data, adding reasons for real or fake labels, and collecting multilingual data.The paper also identifies deep learning instead of machine learning as worth exploring.