Source-linked AI summary
FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19
Gautam Kishore Shahi, Durgesh Nandini
TL;DR
COVID-19 misinformation created a need for broad, reliable resources for fact-checking and automated detection. The paper constructs and annotates a multilingual, cross-domain corpus from fact-checking sources and trains a classifier. The dataset contains 5182 articles, and the classifier achieves an F1-score of 0.76 for detecting false and other fact-check articles.
Problem
COVID-19 misinformation spread across languages and domains, while multilingual, cross-domain, multi-source fact-checking resources were lacking.
Method
The paper collects fact-checked articles using Poynter and Snopes references, crawls their content, manually assigns 11 categories, and builds a machine learning classifier.
Results
5182 articles were collected from 92 fact-checkers across 105 countries and 40 languages, while the classifier achieved an F1-score of 0.76.
Takeaways & Limitations
The FakeCovid corpus supports COVID-19 and fake-news studies, while the classifier provides initial screening of misinformation propagation.
Takeaways & Limitations
Some reference links were unavailable or invalid, and possible duplicates may remain after filtering.
Abstract
from arXiv · showhide
In this paper, we present a first multilingual cross-domain dataset of 5182 fact-checked news articles for COVID-19, collected from 04/01/2020 to 15/05/2020. We have collected the fact-checked articles from 92 different fact-checking websites after obtaining references from Poynter and Snopes. We have manually annotated articles into 11 different categories of the fact-checked news according to their content. The dataset is in 40 languages from 105 countries. We have built a classifier to detect fake news and present results for the automatic fake news detection and its class. Our model achieves an F1 score of 0.76 to detect the false class and other fact check articles. The FakeCovid dataset is available at Github.
Introduction
COVID-19 generated a global multilingual infodemic, creating an urgent need for reliable fact-checking and automated misinformation detection. The paper responds by preparing an open-source dataset and classifier for pandemic misinformation.
- Motivation: COVID-19 misinformation spread across news and social platforms, making reliable sources difficult to identify and threatening psychological health and daily life.The paper describes this misinformation surge as an infodemic requiring careful verification.
- Motivation: Search interest in coronavirus and fake news peaked in March 2020, with both trends following the same curve during March and April.The authors interpret this pattern as evidence of interest in checking coronavirus-related claims.
- Problem: Fake news detection requires trained human judgment and must handle rapidly changing, diverse content across multiple domains.The authors argue that existing methods do not detect changing misinformation consistently.
- Contribution: The main contribution is an open-source dataset for detecting pandemic misinformation, accompanied by a machine learning-based classifier.The classifier is designed to detect misinformation during the pandemic.
Literature
COVID-19 fake news spread globally across languages and domains, while existing fact-checking resources were often limited in language, source, or topical diversity. The paper addresses this gap with a multilingual, cross-domain, multi-country open-source corpus.
- Prior resources: Existing fact-checking datasets were often small, English-only, or collected from few sources, limiting their diversity.Prior resources also commonly focused on topics such as the 2016 US election.
- Research gap: COVID-19 fake news circulated worldwide in different languages and domains, but multilingual, cross-domain, multi-source resources were lacking.The paper identifies this as a central resource gap during the pandemic.
- Contribution: The proposed open-source corpus was collected from 105 countries and is intended to support studies of COVID-19 and fake news.Its design directly targets the lack of broad, multi-purpose fact-checked resources.
Data Collection
The dataset was collected from fact-checking websites referenced through Snopes and Poynter, then assembled into a multilingual, cross-country corpus of COVID-19 fact-checked articles.
- Data Sources: Snopes and Poynter supplied references to fact-checking websites used for data collection.Poynter’s database gathered fact-checked news from 91 organizations in 40 languages.
- Data Collection Process: Article content and metadata such as country and publication date were gathered by crawling source webpages with Beautiful Soup.The process also used source titles and the fact-checking websites’ assigned classes.
- Dataset Scope: 5182 articles were collected from 92 fact-checkers across 105 countries.The articles were published between 04.01.2020 and 15.05.2020.
- Dataset Scope: The collection includes articles in 40 languages, with 40.8% written in English.A dataset summary is presented in Table 1.
Data Annotation
The authors categorized COVID-19 fact-checked articles by content using a reviewed annotation scheme, with multiple annotators supporting reliability assessment across selected languages.
- Annotation Scheme: Articles were labeled into predefined categories according to their content after content analysis.The category scheme contained 11 distinct categories.
- Annotation Scheme: Professional fact-checkers reviewed the 11-category annotation scheme.Annotators visited source articles, read their titles and contents, and then assigned categories.
- Scope and Reliability: 1951 articles were annotated in English, Hindi, and German because of limitations in the authors’ language knowledge.The language-specific counts were English (2116), Hindi (141), and German (47), as reported in the passage.
- Annotators: Three people with background and linguistic knowledge participated in the annotation process.One annotator labeled the data, while a second independently annotated randomly chosen articles to calculate intercoder agreement.
- Annotators: The annotators included a master’s student experienced in data annotation and PhD students with data-science or machine-learning backgrounds.Their language coverage included English, German, and Hindi.
- Scope and Reliability: The number of articles labeled by all three annotators and their intercoder reliability are reported in Table 2.Reliability scores were calculated for each language.
Data Cleaning and Prepossessing
The data-cleaning process corrected or removed faulty URLs, restored missing titles, filtered duplicate articles, and applied standard NLP preprocessing and language detection.
- Data Cleaning: Faulty URLs were manually corrected when possible and removed when the linked pages no longer existed.This step addressed incorrectly entered or unavailable source links.
- Data Cleaning: Missing article titles were manually replaced with source titles after checking the article webpage.The procedure addressed records lacking a title.
- Data Cleaning: Duplicate fact-checked articles were filtered to retain unique articles.Duplicate removal formed part of the manual cleaning process.
- Preprocessing: Basic NLP preprocessing included lowercasing, removing short words, and tokenization.The authors used NLTK, TextBlob, and regular expressions for data cleaning.
- Preprocessing: Language detection was applied because the collected data contained multiple languages.The authors used the Python-based langdetect library.
Data Exploration
The exploratory analysis examined country, language, website, word, and embedded-social-media distributions in the collected fact-checking data.
- Across Country: India had the highest number of collected fact-check articles, followed by the USA and Spain.The counts for the top 10 countries are presented in Table 3.
- Across Language: The dataset’s top 10 languages were ranked by the number of collected fact-check articles.Their distribution is presented in Figure 3.
- Fact-Checking Websites: The top 10 fact-checking websites were compared by the number of articles they published.AFP2 covered the maximum number of COVID-19 fact-checked articles.
- Word Distribution: The word cloud’s dominant terms included “covid,” “fact check,” and “social media.”The analysis selected the top 100 words from the dataset.
- Social Media Links: The analysis counted fact-checked articles containing embedded social-media links.These links were collected because articles used them to discuss alleged claims.
Classification and Results
The dataset is reduced to false versus other fact-checked articles, then evaluated with a BERT-based classifier on English data. The reported classification results are presented in Table 5, while the supplied passages specify the class counts and model setup.
- Class construction: 4132 articles are labeled false, while 1050 remaining fact-checked articles are grouped into the other category.The false class contains fact-checker articles explicitly labeled false; the other category combines the remaining 22 classes.
- Class construction: 78% of the dataset belongs to the false category, motivating binary screening for early-stage fact-checking.The authors state that filtering the false category from a misinformation stream could ease initial screening.
- Evaluation data: 2116 false and 500 other articles comprise the English-language evaluation data.These are the reported English-language totals used for testing.
- Model setup: The classifier uses a BERT-based model without fine-tuning, with 300 hidden units, 150 training epochs, batch size 1, and learning rate 0.001.Training continues until the stated restriction or validation-loss condition is reached.
- Results: Table 5 is identified as the source of the classification results.The supplied table passage provides its caption but no numerical entries.
Discussion
The dataset construction is constrained by unavailable or invalid reference links and possible duplication, while repeated topics are retained when different fact-checkers verify them using different claims.
- Data limitations: Unavailable or invalid reference links prevented collection of some article content and reference titles.The authors identify this as a possible aggregation mistake in the dataset.
- Data limitations: Poynter contains multiple duplicate news articles filtered by reference URL, but other duplicates may remain.The stated filtering approach does not guarantee complete duplicate removal.
- Data inclusion: Articles covering the same fake-news topic are retained when different fact-checking websites use different claims to verify their titles.The paper illustrates this with two differently titled claims about lions in Russia.
Conclusion and Future Work
The paper presents a COVID-19 fact-checked corpus, analyzes its contents, annotates articles into categories, and trains a classifier for misinformation detection. The authors report an F1-score of 0.76 and identify multilingual annotation and propagation analysis as future directions.
- Conclusion: The corpus combines COVID-19 fact-checked articles collected from Poynter and Snopes with crawled textual content.The paper also reports discovering fact-check articles linked to social media.
- Conclusion: Articles are manually annotated into 11 different categories, followed by a machine-learning classifier for COVID-19 misinformation detection.The conclusion presents annotation and classification as successive components of the work.
- Conclusion: 0.76 F1-score is reported for the classifier, supporting initial screening of misinformation before manual analysis.The stated practical application is testing an article as false or true before manual analysis.
- Future work: A proposed extension is to analyze fake-news propagation across social-media platforms, including a possible Twitter case study.The authors also mention schema markup and knowledge-graph development as other possibilities.
Appendix
The dataset’s source labels are consolidated into 23 unique classes, and manual category annotation follows a guideline defining 11 categories with examples.
- Class consolidation: Labels from 92 fact-checking websites are consolidated from 86 source classes into 23 unique classes.The merge handles classes with similar meanings, including capitalization variants such as false and False.
- Category annotation: Manual annotation uses a guideline containing category definitions and examples.The complete description of the 11 categories is provided in Table 7.