PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCOVery: A Multimodal Repository for COVID-19 News Credibility Research

Xinyi Zhou, Apurva Mulay, Emilio Ferrara, Reza Zafarani

arXiv:2006.05557v2cs.SIcs.IR

TL;DR

Low-credibility COVID-19 information creates an infodemic that can lead people toward ineffective or dangerous actions. ReCOVery constructs a multimodal repository by selecting highly credible or unreliable publishers, collecting their COVID-19 articles and Twitter propagation data, and providing prediction baselines. The repository combines broad modality coverage with a trade-off between dataset scalability and label accuracy.

  • Problem

    Existing COVID-19 datasets do not provide ground-truth credibility for coronavirus information, despite the risks associated with low-credibility news.

  • Method

    ReCOVery investigates approximately 2,000 publishers, selects 60 with extreme credibility, and collects 2,029 articles plus 140,820 tweets with multimodal information.

  • Results

    Four baselines achieved F1-scores of 71%–83% for reliable news and 43%–67% for unreliable news, with multimodal features generally more representative than single-modal features.

  • Takeaways & Limitations

    ReCOVery supports research on COVID-19 news credibility and social-media spread while providing benchmarks for future prediction methods.

  • Takeaways & Limitations

    The repository’s credibility ground truth is inherited from publisher-level credibility labels, and its current coverage could be extended to other languages and local social media.

Abstract

from arXiv · show

First identified in Wuhan, China, in December 2019, the outbreak of COVID-19 has been declared as a global emergency in January, and a pandemic in March 2020 by the World Health Organization (WHO). Along with this pandemic, we are also experiencing an "infodemic" of information with low credibility such as fake news and conspiracies. In this work, we present ReCOVery, a repository designed and constructed to facilitate research on combating such information regarding COVID-19. We first broadly search and investigate ~2,000 news publishers, from which 60 are identified with extreme [high or low] levels of credibility. By inheriting the credibility of the media on which they were published, a total of 2,029 news articles on coronavirus, published from January to May 2020, are collected in the repository, along with 140,820 tweets that reveal how these news articles have spread on the Twitter social network. The repository provides multimodal information of news articles on coronavirus, including textual, visual, temporal, and network information. The way that news credibility is obtained allows a trade-off between dataset scalability and label accuracy. Extensive experiments are conducted to present data statistics and distributions, as well as to provide baseline performances for predicting news credibility so that future methods can be compared. Our repository is available at http://coronavirus-fakenews.com.

1 INTRODUCTION

ReCOVery addresses the COVID-19 infodemic by building a scalable multimodal repository for assessing news credibility and studying how low-credibility news spreads. It also supplies baseline prediction results for future comparison.

  • COVID-19 misinformation can prompt ineffective or dangerous protective actions, motivating research on news reliability.
  • ReCOVery investigates how low-credibility COVID-19 news is created and spread and how such news can be predicted.
  • The repository combines textual, visual, temporal, and network information about news content and its social-media spread.

2 RELATED WORK

Existing COVID-19 and fake-news datasets cover tracking, scholarly literature, claims, news, or social-media information, but they do not provide ground-truth credibility for coronavirus information. ReCOVery addresses this gap with multimodal news and social-engagement data while balancing scalability and label accuracy.

  • COVID-19 Datasets: COVID-19 datasets support epidemic tracking, scholarly discovery, and Twitter information analysis, but lack ground-truth credibility labels for coronavirus information.
  • “Fake” News and Rumor Datasets: Fake-news and rumor datasets variously contain full articles, claims, social-media posts, or combined content and social information.
  • ReCOVery: Compared with NELA-GT-2018, FEVER, and LIAR, ReCOVery provides multimodal information and social engagements of news articles.
  • ReCOVery: Compared with CREDBANK and FakeNewsNet, ReCOVery targets the coronavirus infodemic and uses an annotation approach trading off data scalability and label accuracy.

3 DATA COLLECTION

ReCOVery constructs a scalable COVID-19 news repository by selecting extreme-credibility publishers, collecting qualifying articles and multimodal content, and inheriting article labels from publishing sites.

  • Site selection: News credibility is assessed from the publishing media site to support scalable repository construction without labeling every article individually.The approach is intended to trade off dataset scalability and label accuracy.
  • Site selection: The researchers use NewsGuard and Media Bias/Fact Check to identify reliable and unreliable news sites.NewsGuard scores range from 0 to 100, while MBFC assigns six factual-accuracy levels.
  • Site selection: Reliable sites require NewsGuard scores above 90 and high or very high MBFC factual reporting, whereas unreliable sites require scores below 30 and MBFC reporting below mixed.These extreme thresholds are stricter than NewsGuard’s initial threshold of 60.
  • Site selection: 60 news sites are selected from approximately 2,000 investigated publishers, including 22 reliable and 38 unreliable sources.Several unavailable or defunct sites are excluded from the repository.
  • Content collection: COVID-19 articles are identified case-insensitively using the keywords SARS-CoV-2, COVID-19, or Coronavirus.The repository crawls article content and social information, including textual, visual, temporal, and network components.
  • Content collection: Each collected article includes identifiers, URL, publisher, publication date, authors, title, body text, and an image.These fields support verification, temporal analysis, credibility evaluation, and multimodal research.

4 DATA STATISTICS AND DISTRIBUTIONS

ReCOVery contains 2,029 coronavirus news articles with multimodal and social-spreading information, whose distributions reveal imbalanced classes and long-tailed activity across publishers, dates, authors, content, and Twitter spreaders.

  • Dataset Statistics: 2,029 news articles form an imbalanced dataset, with approximately twice as many reliable as unreliable articles; 2,017 include textual and visual information, and 1,747 were shared on social media.The dataset includes 78,659 users spreading reliable news and 17,323 spreading unreliable news, while 93,761 users appear overall, indicating some users spread both types.
  • News Publishers: Five unreliable publishers have no COVID-19 articles yet remain in the repository for future updates.The publisher distribution covers extremely reliable and extremely unreliable news sites.
  • Publication Dates: January-to-May publication counts increase significantly, described as exponentially, alongside the dataset’s 2020 publication-date distribution.The authors attribute this pattern to growing global cases and deaths, expanding worldwide discussion, article unavailability, and keyword-selection timing.
  • Authors and Collaborations: Most articles have five or fewer authors, while over 90% of 1,095 author-network nodes have at most two collaborators.Both author counts and collaboration degrees follow long-tail or power-law-like distributions; publisher names used as authors are recorded as zero authors.
  • News Content: News-content word counts follow a long-tail distribution, averaging approximately 800 words with a median near 600.The reported textual characteristics cover news titles and body text.
  • News Spreading: More than 80% of news articles were shared fewer than 100 times, while a small number received thousands of tweets; spreader counts differ because users can tweet repeatedly.The most popular spreader has over 40 million followers or 600,000 friends.

5 FORMING BASELINES: USING ReCOVery TO PREDICT COVID-19 NEWS CREDIBILITY

This section establishes baseline methods for predicting COVID-19 news credibility using ReCOVery data, comparing single-modal and multimodal approaches across traditional and neural learning frameworks. It reports benchmark performance and identifies directions for richer social-media-based methods.

  • 5.1 Methods: Baseline methods are organized by learning framework, using either traditional statistical learners or neural networks.Examples include SVM with LIWC features and neural methods such as Text-CNN and SAFE.
  • 5.1 Methods: Baseline methods are also grouped as single-modal or multimodal approaches.LIWC, RST, and Text-CNN are single-modal, whereas SAFE uses multimodal information.
  • 5.1 Methods: LIWC extracts 93 linguistic, psychological, and topical features from each news story for traditional statistical classification.The features count words assigned to one or more of 93 lexicon categories.
  • 5.1 Methods: RST represents each article through 45 counts of rhetorical relations obtained from a pretrained rhetorical-structure parser.The resulting features are classified within a traditional statistical learning framework.
  • 5.1 Methods: Text-CNN applies convolution and max pooling within a convolutional neural network for text classification.
  • 5.1 Methods: SAFE jointly learns news representations from textual and visual information and their relationship for fake-news detection.This multimodal design can recognize falseness in text, images, or their irrelevance.
  • 5.2 Implementation Details: The dataset is split randomly into training and testing sets at 0.8:0.2, with precision, recall, and F1 used because reliable articles outnumber unreliable ones by approximately 2:1.Traditional learners include LR, NB, k-NN, RF, DT, and SVM, with only the best-performing classifier presented.
  • 5.3 Experimental Results: F1 scores for reliable news range from 71% to 83%, while unreliable-news F1 scores range from 43% to 67%; multimodal features are generally more representative.The four baselines are content-based, motivating future methods that mine social-media information.

6 CONCLUSION

The conclusion presents ReCOVery as a multimodal repository and benchmark for COVID-19 news credibility research. It also identifies multilingual expansion and additional ground-truth labels as future extensions.

  • 6 CONCLUSION: ReCOVery provides textual, visual, temporal, and network information about COVID-19 news and its spread on social media.
  • 6 CONCLUSION: The repository balances data scalability and label accuracy while providing benchmark performances for predicting news credibility.
  • 6 CONCLUSION: Future work includes adding multilingual news and local social-media spread information, alongside labels for hate speech, clickbaits, and social bots.These extensions are intended to support study of virus-related bias, discrimination, and correlations among low-credibility information and accounts.
Loading 2006.05557v2…