Source-linked AI summary
CoAID: COVID-19 Healthcare Misinformation Dataset
Limeng Cui, Dongwon Lee
TL;DR
COVID-19 misinformation creates serious health and social harms, while its rapid spread and resistance to correction make detection difficult. The paper introduces CoAID, a benchmark dataset combining fact-checked content with social engagement, and uses analyses and detection experiments to characterize the problem. The dataset reveals differences between misinformation and factual content, while detection performance remains limited by class imbalance and scarce informative healthcare comments.
Problem
COVID-19 misinformation causes serious health and social harms, while rapid spread, repeated exposure, and ineffective correction complicate satisfactory computational detection.
Method
The paper constructs CoAID from fact-checked fake and true news, claims, social-platform posts, ground-truth labels, and associated user engagement.
Results
Tweets related to fake news are more negative and more strongly polarized than tweets related to true news, while detection models remain challenged by class imbalance.
Takeaways & Limitations
CoAID provides a benchmark for analyzing and detecting COVID-19 healthcare misinformation across content types and social engagement.
Takeaways & Limitations
Because of Twitter’s Terms and Conditions, the released dataset contains only tweet IDs, and healthcare-news comments are often scarce or uninformative.
Abstract
from arXiv · showhide
As the COVID-19 virus quickly spreads around the world, unfortunately, misinformation related to COVID-19 also gets created and spreads like wild fire. Such misinformation has caused confusion among people, disruptions in society, and even deadly consequences in health problems. To be able to understand, detect, and mitigate such COVID-19 misinformation, therefore, has not only deep intellectual values but also huge societal impacts. To help researchers combat COVID-19 health misinformation, therefore, we present CoAID (Covid-19 heAlthcare mIsinformation Dataset), with diverse COVID-19 healthcare misinformation, including fake news on websites and social platforms, along with users' social engagement about such news. CoAID includes 4,251 news, 296,000 related user engagements, 926 social platform posts about COVID-19, and ground truth labels. The dataset is available at: https://github.com/cuilimeng/CoAID.
1 Introduction
COVID-19 misinformation creates serious health and social harms and presents distinctive detection challenges. CoAID addresses these challenges by providing a benchmark dataset spanning misinformation, facts, and social engagement.
- Motivation: COVID-19 misinformation has caused deadly health consequences and disruptions to social order.Examples include chloroquine ingestion causing death and hospitalization, and fires at cell phone towers linked to a 5G conspiracy.
- Challenges: Emerging-disease misinformation can spread rapidly before correction, while repeated exposure may increase perceived truth.The paper also notes that authority corrections may fail or reduce belief in other epidemic facts.
- Contribution: CoAID is a benchmark dataset containing confirmed fake and true news articles, social-platform posts, user engagement, and ground-truth labels.The authors also analyze distinguishing features and evaluate state-of-the-art misinformation detection methods.
2 Preliminaries
The paper distinguishes misinformation from disinformation by intentionality but does not attempt to determine that intention computationally. It uses fake news and misinformation interchangeably.
- Terminology: Misinformation and disinformation are wrong or misleading information, but disinformation is spread deliberately whereas misinformation need not be.This distinction follows the Oxford English Dictionary definition cited by the paper.
- Terminology: Fake news, hoaxes, and rumors represent distinct misinformation examples involving misleading news claims, deliberate fabrication, or unverified relevant information.The cited literature provides these finer-grained definitions.
- Operationalization: CoAID does not differentiate misinformation from disinformation because computationally determining a person’s intention is virtually impossible.The paper also uses fake news and misinformation interchangeably.
3 Related Work
Prior misinformation datasets often emphasize news articles, while social-media content and healthcare-specific coverage receive less attention. CoAID broadens the data scope across content types and engagement signals.
- Existing datasets: Existing benchmark datasets commonly focus on linguistic features of news articles for fake-news detection.The related-work discussion frames these datasets as primarily article-oriented.
- Existing datasets: Several prior datasets combine fact-checking labels with news content and, in some cases, social engagement or user replies.Examples include LIAR, FA-KES, FakeNewsNet, and FakeHealth.
- Research gap: Social-media posts receive less attention despite their ability to spread rapidly across platforms and cause confusion and social disruption.The paper illustrates this concern with a viral fake coronavirus-patent video.
- CoAID: CoAID includes true and fake news, short claims, social-media posts, multimodal information, and user engagement, with automatic updating capability.The dataset is designed to collect broader features than article-only resources.
4 Dataset Construction
CoAID is constructed from fact-checked COVID-19 news, claims, social-platform posts, and associated Twitter engagement. Its collection process supports ground-truth labeling and automatic updates, while released Twitter data is restricted to tweet IDs.
- Collection scope: The dataset covers COVID-19-related facts and misinformation from December 1, 2019 to September 1, 2020, including news articles and claims.Topics include COVID-19, coronavirus, pneumonia, flu, lockdown, quarantine, and ventilators.
- Facts and misinformation: News URLs come from cross-checked reliable outlets for true news and fact-checking websites for fake news.Newspaper3k extracts titles, content, abstracts, and keywords from the collected URLs.
- Facts and misinformation: Claims are collected from WHO and Medical News Today sources and separated into true and fake claims.Examples distinguish a false age-risk claim from the true claim that 5G networks do not spread COVID-19.
- User engagement: Twitter engagement includes tweets, replies, user identifiers, text, favorites, retweets, and locations associated with collected news and claims.Tweets are retrieved using article titles and date ranges, while replies are obtained through tweet IDs.
- Social-platform posts: CoAID also collects fact-checked true and fake posts from Facebook, Twitter, Instagram, YouTube, and TikTok, counting duplicate posts once.For each social-platform post, the dataset crawls only its title and lists it under claims.
- Automatic updates: CoAID supports automatic updates by using the latest added-data timestamp as the next search’s start date.The dataset includes extracted feature descriptions documented in Table 3.
5 Data Analysis
CoAID analysis contrasts misinformation and factual content across user sentiment, hashtags, claims, and detection performance. The results show distinctive social signals, but imbalanced data and limited healthcare-related engagement constrain detection quality.
- User sentiment: Tweets related to fake news are more negative and show stronger sentiment polarities than tweets related to true news.VADER sentiment scores were used after excluding completely neutral tweets.
- Hashtag patterns: Hashtag distributions differ substantially: true-news tweets focus mainly on healthcare, whereas fake-news tweets span conspiracy and fake-cure topics.Examples include #bioweapon and #vitaminc after removing ubiquitous COVID-19 hashtags.
- Claim trends: “COVID-19 is just like the flu” peaked around March 12, 2020 and remained common, while “5G mobile networks spread COVID-19” peaked around April 10 and then declined.The 5G-related decline followed debunking of that claim.
- COVID-19 misinformation detection: State-of-the-art methods outperform simple methods by incorporating user-engagement signals, but imbalanced data produces many false positives and unsatisfactory recall and F1 values.The authors identify imbalance handling as a continuing requirement for practical detection solutions.
- COVID-19 misinformation detection: Healthcare-news user engagement is often scarce and less discriminating because lay persons may lack professional knowledge.The authors give unrelated, useless, and misleading replies to a WHO tweet as an example.
6 Conclusion
The paper presents CoAID as a comprehensive COVID-19 misinformation dataset combining news, user engagement, and social-platform posts. It also uses data analysis and detection experiments to highlight distinctive misinformation features and future research directions.
- 6 Conclusion: CoAID contains COVID-19 news articles, related user engagement, and social-platform posts.The dataset is intended for research on COVID-19 misinformation.
- 6 Conclusion: The paper describes dataset collection, analyzes differences between misinformation and facts, and evaluates several state-of-the-art fake-news detection methods.The detection task is presented as a way to demonstrate future research directions.
A.1 Version 0.1
CoAID Version 0.1 contains an initial collection of COVID-19 information and associated engagement data with ground-truth labels.
- A.1 Version 0.1: The initial dataset covers information collected from December 1, 2019 through May 1, 2020.Version 0.1 includes 1,896 news items, 183,564 related user engagements, and 516 social-platform posts.
- A.1 Version 0.1: Version 0.1 includes ground-truth labels alongside its news, engagement, and social-platform-post data.The passage describes these as components of the initial dataset.
A.2 Version 0.2
The paper identifies additional data statistics for CoAID Version 0.2.
- A.2 Version 0.2: Table 6 reports the statistics of the additional data included in CoAID Version 0.2.The supplied passage provides the table reference but no individual statistics.
A.3 Version 0.3
CoAID Version 0.3 adds COVID-19 information collected from July 1 through September 1, 2020, expanding the dataset with news, user engagements, social posts, and labels.
- Version 0.3 adds data collected from July 1, 2020 through September 1, 2020.
- The additional data contains 1,016 news items, 1,308 related user engagements, 75 COVID-19 social platform posts, and ground truth labels.
- The statistics for Version 0.3’s additional data are presented in Table 7.