FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19

Introduction

• The pandemic’s global, multilingual discussion was accompanied by an infodemic of rumors and misinformation that threatens disease-related authenticity and people’s psychological health.COVID-19 had spread to at least 215 countries, while people discussed the topic in dozens of languages across platforms.Source: Introduction · p. 1
• Without checking information from news and social media, users may struggle to find reliable sources, with consequences for their psychology, daily lives, and behavior.The introduction notes that the WHO coined “infodemic” to describe misinformation about the virus.Source: Introduction · p. 1
• Fake-news detection is difficult because it requires trained human judgment and must handle changing, diverse content across platforms and domains, while existing methods detect misinformation inconsistently.The introduction also identifies a lack of corpus for testing detection methods.Source: Introduction · p. 1
• The work introduces an open-source dataset for pandemic misinformation detection and a machine-learning classifier to detect misinformation.The dataset is described as human-annotated, multilingual, cross-domain, and fact-checked.Source: Introduction · pp. 1–2

• Earlier fact-checking datasets often cover only a few sources and English, limiting the diversity of fact-checking resources.Source: Literature · p. 2
• Prior work describes fact-checking datasets and their limitations (Augenstein et al. 2019), journalist-based dataset construction, and approaches to fake-news detection (P´erez-Rosas et al. 2017; Zhou and Zafarani 2018).Source: Literature · p. 2
• COVID-19 and the infodemic spread in parallel, and partially true information shared without proper medical terms on Facebook and WhatsApp contributed to panic about medicines and prevention.Mesquita et al. (2020) proposed a framework to fight fake medical news; Rovetta and Bhagavathula (2020) observed many infodemic monikers across Italy.Source: Literature · p. 2
• The paper provides an open-source, multilingual and cross-domain fact-checking dataset collected from 105 countries for studies of COVID-19 and fake news.Source: Literature · p. 2

Data Collection

• The authors weekly scraped Poynter and Snopes references, assigned each article a unique FCID, and collected source links, reference and source titles, and publication dates.Poynter and Snopes served as bridges to the original fact-checking websites; published date refers to the fact-checking website’s publication date.Source: Data Collection · pp. 2–3
• Using Beautiful Soup, the authors crawled source pages for article text, country, and date, and recorded the fact-checking website’s assigned class.Classes varied by website and could include true, false, or partially false.Source: Data Collection · p. 3
• The authors recorded social-media link presence by searching extracted pages for platform-specific keywords or URL patterns for Twitter, YouTube, Reddit, Facebook, and Instagram.The property is denoted SM Link.Source: Data Collection · p. 3
• They recorded the verifying fact-checking company and represented circulation country in up to four separate country fields per article.The dataset’s maximum observed number of countries for one article was four.Source: Data Collection · p. 3
• The authors manually assigned article topics to a category attribute and created a lang attribute to identify each article’s language.Language detection is discussed in the Data cleaning and Prepossessing section.Source: Data Collection · p. 3
• 5,182 articles from 92 fact-checkers span 105 countries, 40 languages, and 04.01.2020–15.05.2020; 40.8% are in English.The dataset summary appears in Table 1.Source: Table 1 · p. 3, Data Collection · p. 3

Data Annotation

• Articles were assigned to 11 content-based categories, which were reviewed by a professional fact-checker.Annotators opened each source article, read its title and content, and then selected a category.Source: Data Annotation · p. 3
• They report 1,951 annotated articles but list English (2,116), Hindi (141), and German (47), noting language knowledge limited coverage.Source: Data Annotation · p. 3
• Three annotators shared language-specific annotation, with second-annotator coding of randomly selected articles used to calculate agreement by language.The annotators had English, German, and Hindi language expertise; the paper says reliability results are shown in Table 2.Source: Data Annotation · p. 3, Table 2 · p. 4

Data Cleaning

• Faulty URLs were manually corrected or removed, missing titles were added from article webpages, and duplicates were removed using article URLs.Unique articles were retained after URL-based deduplication.Source: Data Cleaning · p. 4

Data Prepossessing

• Preprocessing included lowercasing, short-word removal, tokenization, language detection from article content, and spelling correction with TextBlob.The authors also described abbreviations as omitting final letters and contractions as omitting letters within words; NLTK, TextBlob, and regular expressions supported preprocessing, while langdetect identified languages.Source: Data Prepossessing · p. 4

Across country

• India has the most fake-news cases, followed by the USA and Spain.The count of fact-check articles from the top 10 countries is shown in Table 3.Source: Data Exploration · pp. 4–5, Table 3 · p. 4

Fact checking website

• AFP2 covers the most COVID-19 fact-checked articles among the top 10 fact-checking websites.The website article counts are presented in Figure 4.Source: Figure 4 · p. 4, Data Exploration · p. 5

Across language

• The top 10 languages by fact-check article count are presented as the language distribution in Figure 3.The analysis notes that fake news circulated in national and regional languages.Source: Figure 3 · p. 4, Data Exploration · p. 5

Word distribution

• For an overview of news-article content, the authors plotted a word cloud of the top 100 words; covid, fact check, and social media are reported as dominant.Source: Data Exploration · p. 5, Figure 5 · p. 5

Social media link

• The study collected fact-checked articles containing embedded social-media links and reports link counts in Table 4.The links appear in articles discussing alleged claims.Source: Data Exploration · p. 5, Table 4 · p. 5

Classification and Results

• The authors grouped the dataset into two classes: false, containing 4,132 fact-checker-labeled articles, and others, combining the remaining 22 classes and containing 1,050 fact-checked articles.Source: Classification and Results · p. 5
• The English test subset contains 2,116 false articles and 500 articles in the other class.The reported experiment was conducted for English.Source: Classification and Results · p. 5
• The classification setup uses a BERT-based model without fine-tuning, with 300 hidden units, 150 training epochs, batch size 1, and learning rate 0.001.Training continues until a restriction is reached or validation loss continues; the classification results are referred to as Table 5, but its values are not supplied here.Source: Classification and Results · p. 5, Table 5 · p. 5

Discussion

• Unavailable or invalid reference links prevented collection of some articles’ full content and reference titles, possibly because of aggregation mistakes.Source: Discussion · p. 5
• Poynter duplicates were filtered using reference URLs, but other duplicates may remain.Source: Discussion · p. 5
• Articles about the same topic were retained when different fact-checking sites verified distinct claims.The paper illustrates this with two titles about Russia releasing lions during COVID-19, fact-checked by FactCrescendo and BOOM FactCheck.Source: Discussion · p. 5

Conclusion and Future Work

• The classifier achieved an F1-score of 0.76 and is intended to screen articles as false or true before manual analysis.The authors describe this as an initial screening application for misinformation during the pandemic.Source: Conclusion and Future Work · p. 5
• The work collected COVID-19 fact-checked articles from Poynter and Snopes, manually annotated 11 categories, and found that only three languages had human-annotated categories.The authors also performed an explanatory dataset analysis and discovered fact-check articles linked to social media.Source: Conclusion and Future Work · p. 5
• Future work could analyze fake-news propagation across social-media platforms, use schema markup to improve search results, and build a knowledge graph to answer user queries.The passage gives a Twitter case study as an example of propagation analysis and suggests using fact-check news classes or categories in schema markup.Source: Conclusion and Future Work · p. 6

Appendix

• 86 source classes from 92 fact-checking websites were manually consolidated into 23 unique classes, including labels differing only by capitalization.Table 6 lists each class’s description and source website.Source: Appendix · p. 6
• The annotation guideline defines the categories and provides examples; Table 7 presents descriptions of all 11 categories.Source: Appendix · p. 6